I wrote last Monday’s FireStarter on Process and Peer Pressure because there were a few things bothering me that I needed to get out of my system, but I saved a lot for later. I didn’t really intend to write this followup so soon, but I saw that Cisco announced their own Software Development Lifecycle. I wanted to make some statements on SDL later this year when I begin publishing more concrete Secure Software Development Lifecycle (SSDL in Securosis parlance, SDL for most organizations) guidelines, but Cisco’s announcement changes things. I worry that sheer inertia will prompt the industry as a whole to rubber-stamp SDLs. Before you know it, HR reps will be including “SDL certification” requirements on every engineering job description, without a clue what they are demanding or why, so let’s stop this train before it runs too far off the tracks.
If you are thinking about incorporating Secure Development Lifecycle practices for software development, that’s great. If you have read about Microsoft’s SDL, witnessed Microsoft’s success, seen Cisco’s endorsement, and believe their model will work for you, just stop. It’s not going to work for you. It’s based on a lot of factors and assumptions that do not pertain to you. It’s not a template for your requirements.
Adopting MS-SDL wholesale is a little like a child putting on adult clothes because they want to be ‘big’. You cannot drop that particular process into your development organization and have it fit. More likely you will break everything. Your team will need to change their skills and priorities, and though it sounds cliche, people are resistant to change. Existing processes need to be adjusted to accommodate secure development processes and techniques. You will need new tools, or to augment existing ones. You will need a whole new class of metrics and tracking. And everything you pick the first time will need several iterations of alteration and adjustment before you get it right – this isn’t Microsoft’s first attempt either.
It’s not that the SDL is bad – it isn’t. Microsoft did an excellent job with their SDL. It’s very well thought out, incorporates most effective defect detection techniques, has clearly evolved through several revisions, and includes intelligent tradeoffs in places where there is no single ‘right’ answer. But it is their SDL, not yours. If you take the SDL Microsoft has described and try to implement it, you will fail. I am talking to the 99% of people out there who would think about implementing SDL and think “Hey, Microsoft published this new thingie for free; let’s use it and save ourselves the time and money!” Wrong.
Here’s why:
- Too Big: This process is geared for very large firms, with lots of resources, and a genuine desire to get better. You may not need all of it and frankly it would be overwhelming to start with. This is huge – I mean really huge. You are not going to swallow this elephant in a single gulp.
- Organic Evolution: Microsoft’s success is not just the introduction of process and techniques. It was not just hiring a handful of really good people and helping to educate the development staff. The MS-SDL reflects several years of focused evolution, and in software that is a lot. They spent a long time looking at the code and figuring out what was wrong. They developed their own tools to help discover problems. They developed software to help track their progress and provide metrics to demonstrate what worked and what did not. They evolved their own threat modeling. They tried, revised, fixed and re-implemented most of what they do several times over. Don’t think their publishing a guide can save you this pain – it cannot.
- Resources: People, Tools, and Time are the three classic resources you have when you build code. Resources are scare. Always. OK, if you have billions of dollars in the bank, or you are a bank, you might not be quite as pinched for resources. But developing quality code is expensive. Microsoft had the money to hire some of the best people, to buy or build the best tools, a willingness to take additional time for security before releasing software, and then hire some more of the best people. Your developers work nights and weekends to get the release out the door and collapse in a heap, dreaming about all the things they wanted to do before the code was released. Cisco? Yeah, they can do this. You? You don’t have the resources to do everything, so you need to pick and choose.
- Appropriateness of Techniques: Your program calls for white box testing. Great, but you don’t own critical code you rely on. You leverage open source where you can get code, but off-the-shelf software and even Microsoft tools do not provide source code. If you have four thousand web pages, and most of them don’t filter input values, do you really think you are going to fix this in the current release cycle, or are you going to deploy a WAF? If you are starting an application from scratch, your first step will be threat modeling. If you have a huge existing application, forget threat modeling for now – pen testing is probably much more effective and efficient. And it’s not just which techniques, but how you use them. Within all these techniques, there are many variations and supporting requirements that need tweaking so they can work for you. We discuss these tradeoffs in the Use Case portion of Understanding and Selecting a Web Application Security Program white paper, but the point is that the right choice for you is different than the right choice for Microsoft or Cisco, and you can’t discover what’s right for your environment by reading their SDLs.
- Do what Microsoft did, not what they do: Using the SDL as your program is a really bad thing to do. I really hope people don’t take this as a slam against Microsoft – my point is to follow Microsoft’s example, rather than their SDL. You need to do what Microsoft did, because you cannot simply jump ahead to what Microsoft is doing today. Microsoft’s journey to where they are now is far more interesting and useful than the specific tools and techniques they eventually settled on. You can learn by example, sure, but you need to answer the same questions – sometimes with different answers, as dictated by your own constraints – to evolve your own process from the beginning. That comes from hard work and analysis, with lots of trial and error mixed in.
There is no shortcut, no secret sauce, no package of Instant Code-Be-Secure. One size does not fit all. Admire what Microsoft has done, for their customers and for the community, learn, and then figure what is relevant to you. You don’t have to completely start from scratch, but you have got work to do in figuring out how you are going to build your own Secure Development Lifecycle.
Reader interactions
8 Replies to “FireStarter: Secure Development Lifecycle—You’re Doing It Wrong”
Good post!
Couple of points, though – we don’t put a lot of effort into security because we have lots of money to toss around. We put the effort in because it is the right thing to do, and we also know that it _costs_ a lot of money if we do not. It’s a key part of getting management buy-in – show them the ROI.
We do have an advantage of scale – since we have 3000 developers in Office, we can afford to dedicate a dozen or so to nothing but security. If you’ve got 30, then it averages out to a fraction of one person. Which means you (as we do) have to distribute the effort.
While pen testing is a lot of fun, and it will indeed find valuable problems, you can’t test quality into software. You have to make processes to deal with problems on a consistent basis.
By all means, take the bits of the SDL that work for you and use them. One of the most effective threat models I’ve ever done was on a large whiteboard with a bunch of people in the room.
Also, it shouldn’t be a WAF _or_ trying to secure your code. How about both? There’s no approach to securing things that’s perfect, so use them in layers, and where each layer makes sense.
Maybe I should set up an account here…
Adrian,
If management doesn’t accept security as a priority, it’s not going to be. It’s literally that simple in my mind. However, even if they accept it as a priority, that still doesn’t leave them with tools to actually treat it that way. But that’s not a unique problem to secure development.
Writing quality software is HARD, and it demands the need for engaged management at all levels and in various degrees. There has to be at a reasonable minimum some level of: vision, clear communication, market research, design & usability testing, accountability and verification. So, why then do people seem so surprised to find out that securing software has similar traits and expectations across all levels of the company? Likely, if I was to wager a guess, because they don’t write software well in the first place.
To your point, if companies literally have thousands of sites that are insecure but don’t have the resources to fix broken code… how would they do it if it was any other quality concern? The company has lots of reasonable options to fix it, including: retiring sites, consolidating sites, fixing critical issues in critical sites, hiring workers to implement changes, focusing a whole team on changes, etc… but they are most likely to choose the “cost effective” one. Which is fine, of course, if it actually worked.
I don’t dislike WAF’s for what they can do, I dislike them because by choosing them it almost GUARANTEES that the problems wont actually be fixed. Because the problems aren’t fixed, coupled with the fact that WAF’s aren’t nearly as strong as people sell them as, the company is at risk. Possibly less risk, but that’s ultimately specific to the company itself.
To your point, secure developers and WAF’s are at odds particularly because if you choose one you have essentially pooped on the other. PCI DSS makes it pretty clear, put in a WAF and you don’t need to verify/write code securely. To say that people could do both is irrelevant, they wont. If they DID the whole of this argument is effectively moot.
-A
@Andrew – I put the WAF reference in that bullet point because the half dozen other examples were too subtle to make the point. I considered writing a FireStarter called ‘Developers Hate WAF’. Maybe I should. People who believe in secure development practices are anti-WAF. They view WAF is an inefficient band-aid that breaks web applications. They spend time propping up WAF rules that could be spent fixing code. Fixing the code is the _right_ thing to do. But when it comes down to it, when the volume of crap—critical crap—you have to fix is measured in years, its your only choice. Well, that or get out of the business. I repeatedly argued this point in the past, and had my thinking adjusted by Jeremiah Grossman and the sheer weight of evidence he brought to the discussion. Nobody said necessity is pretty.
Your second point reinforces my argument: can I responsibly and reasonably do it from where I am at. You also interweave several points which were in my original draft but removed to focus on this issue.
1. Management buy in. It’s needed and necessary to be successful. But you want management buy in that security is a requirement, not how to achieve security. And the reason I wrote this post is comments by mid-level management looking to adopt SDL.
2. Several people I know in the research community feel Microsoft SDL is so far off from the appropriate starting point that I have heard more than one expert in secure code development wonder aloud if the only reason Microsoft published their SDL was for PR purposes. My feeling is Microsoft turned around a number dodge-y to downright broken products, and they were kind enough to share the tools that make them successful. Frankly had I spent that much time and money, I would push as much of this out there to show the world all the great things that were done. Yeah, it’s now a PR effort as well, but they earned the credibility to do it.
3. I under-played the metrics and measurements part of the argument. “Is the process capable of proving its value” is a serious issue because firms don’t collect and track many of the metrics they will need, and it will be several years to demonstrate value even after they do. “Security” is a nebulous concept to begin with, making this all the more difficult.
Adrian,
Overall I agree with your post. There are lots of nuances here that aren
@starbuck – thank you for the thoughtful comment. You have captured the motivation for the post better than I did.
@Marisa – I don’t have personal experience with Veracode products. Conceptually I like the idea of static binary as it helps uncover what is going on with third party products. I need to reach out to them for a briefing.
@Andre – Does this mean you don’t wear the “I ♥ WAF” shirt I sent you for Christmas?
-Adrian
“Before you know, it HR reps will be including “SDL certification” requirements on every engineering job description, without a clue what they are demanding or why, so let’s stop this train before it runs too far off the tracks.”
Damn right.
By the way, I didn’t really see the point of your article at first as it seems quite logic to me that adopting the methodology/process that one of the biggest software editors adopted would require very strong adaptation.
Then I remembered myself almost three years ago when I printed out the SDL process and came to the meeting room bragging about it “Yeah, that’s what we’ll do!!!” And I also remember the moment, one year after, when I realized that these models were just…models, that small ISVs like the one I was working for couldn’t afford both financially and technically.
Now, the most interesting (from my humble opinion) of your recommendations is the 5th one: Do what MS did, not what they do. That’s what happens, ironically: the SDL process describes Microsoft’s maturity model at its most mature stage but lacks guidance on how to reach it (the assessment kind of helps but…anyway). Every company has its own needs and resources and SDL does not provide any insights on how to identify the appropriate roadmap (aka: the cheapest and most risk-mitigating approach).
That’s a selling point of the OpenSAMM process, which proposes industry-oriented maturity roadmaps that should help the organization walk along the path towards a mature software development lifecycle. I am currently deploying security within an existing SDLC with a massive amount of developers, based on the OpenSAMM guidance. Within six months I hope I will be able to have some thoughts to share on the differences between working with SDL and working with OpenSAMM.
Let’s hope they will be more positive than my experience with SDL.
Thanks for your article!!
“Do what Microsoft did, not what they do”
I love that. Agreed on all fronts. To Microsoft’s credit though, they do have an arsenal of helpful documents and a fleet of consultants in their Pro Network to help you with the growing pains.
To point 4, how do you feel about the increasingly popular use of binary testing? (e.g. Veracode)
I agree with everything in this entire post. A very nice job!
However, since I’m here, I might as well leave the feedback that it would be nice if you just skip on the whole WAF conversation.
Nobody in their right mind is going to deploy a WAF because they affect performance very negatively and cause severe security problems, even when in “monitoring-only” mode. This is primarily because the technology sucks, but also because the companies that build and promote WAF are very immature and do not understand SDL-IT, SDL-Lite, or have their own customized SSDL like you explain here. If they don’t understand appsec for themselves, how are they going to protect your app?