In this post we will outline some of the key characteristics of DevOps. In fact, for those of you new to the concept, this is the most valuable post in this series. We believe that DevOps is one of the most disruptive trends to ever hit application development, and will be driving organizational changes for the next decade. But it’s equally disruptive for application security, and in a good way. It enables security testing, validation and monitoring to be interwoven with application development and deployment. To illustrate why we believe this is disruptive – both for application development and for application security, we are first going to delve into what Dev Ops is and talk about how it changes the entire development approach.

What is it?

We are not to dive too deep into the geeky theoretical aspects of DevOps as it stays outside our focus for this research paper. However, as you begin to practice DevOps you’ll need to delve into it’s foundational elements to guide your efforts, so we will reference several here. DevOps is born out of lean manufacturing, Kaizen and Deming’s principles around quality control techniques. The key idea is a continuous elimination of waste, which results in improved efficiency, quality and cost savings. There are numerous approaches to waste reduction, but key to software development are the concepts of reducing work-in-progress, finding errors quickly to reduce re-work costs, scheduling techniques and instrumentation of the process so progress can be measured. These ideas have been in proven in practice for decades, but typically applied to manufacturing of physical goods. DevOps applies these practices to software delivery, and when coupled with advances in automation and orchestration, become a reality.

So theory is great, but how does that help you understand DevOps in practice? In our introductory post we said:

DevOps is an operational framework that promotes software consistency and standardization through automation. Its focus is on using automation to do a lot of the heavy lifting of building, testing, and deployment. Scripts build organizational memory into automated processes to reduce human error and force consistency.

In essence development, quality assurance and IT operations teams automate as much of their daily work as they can, investing the time up front to make things easier and more consistent over the long haul. And the focus is not just the applications, or even an application stack, but the entire supporting eco-system. One of our commenters for the previous post termed it ‘infrastructure as code’, a handy way to think about the configuration, creation and management of the underlying servers and services that applications rely upon. From code check-in, through validation, to deployment and including run time monitoring; anything used to get applications into the hands of users is part of the assembly. Using scripts and programs to automate builds, functional testing, integration testing, security testing and even deployment, automation is a large part of the value. It means each subsequent release is a little faster, and a little more predictable, than the last. But automation is only half the story, and in terms of disruption, not the most important half.

The Organizational Impact

DevOps represents an cultural change as well, and it’s the change in the way the organization behaves that has the most profound impact. Today, development teams focus on code development, quality assurance on testing, and operations on keeping things running. In practice these three activities are not aligned, and in many firms, become competitive to the point of being detrimental. Under DevOps, development, QA and operations work together to deliver stable applications; efficient teamwork is the job. This subtle change in focus has a profound effect on the team dynamic. It removes much of the friction between groups as they no longer work on their pieces in isolation. It also minimizes many of the terrible behaviors that cause teams grief; incentives to push code before it’s ready, the fire drills to fix code and deployment issues at the release date, over-burdening key people, ad-hoc changes to production code and systems, and blaming ‘other’ groups for what amounts to systemic failures. Yes, automation plays a key role in tackling repetitive tasks, both reducing human error and allowing people to focus on tougher problems. But DevOps effect is almost as if someone opens a pressure relief value when teams, working together, identify and address the things that complicate the job of getting quality software produced. Performing simpler tasks, and doing them more often, releasing code becomes reflexive. Building, buying and integrating tools needed to achieve better quality, visibility and just make things easer help every future release. Success begets success.

Some of you reading this will say “That sounds like what Agile development promised”, and you would be right. But Agile development techniques focused on the development team, and suffers in organizations where project management, testing and IT are not agile. In our experience this is why we see companies fail in their transition to Agile. DevOps focuses on getting your house in order first, targeting the internal roadblocks that introduce errors and slow the process down. Agile and DevOps are actually complementary to one another, with Agile techniques like scrum meetings and sprints fitting perfectly within a DevOps program. And DevOps ideals on scheduling and use of Kanban board’s have morphed into Agile Scrumban tools for task scheduling. These things are not mutually exclusive, rather they fit very well together!

Problems it solves

DevOps solves several problems, many of them I’ve alluded to above. Here I will discuss the specifics in a little greater detail, and the bullets bullet items have some intentional overlap. When you are knee deep in organizational dysfunction, it is often hard to pinpoint the causes. In practice it’s usually multiple issues that both make thing more complicated and mask the true nature of the problem. As such I want to discuss what problems DevOps solve from multiple viewpoints.

• Reduced errors: Automation reduces errors that are common when performing basic – and repetitive – tasks. And more to the point, automation is intended to stop ad-hoc changes to systems; these commonly go un-recorded, meaning the same problem is forgotten over time, and needs to be fixed repeatedly. By including configuration and code updates within the automation process, settings and distributions are applied consistently – every time. If there is a incorrect setting, the problem is addressed in the automation scripts and then pushed into production, not by altering systems ad-hoc.
• Speed and efficiency: Here at Securosis we talk a lot about ‘reacting faster and better’, and ‘doing more with less’. DevOps, like Agile, is geared towards doing less, doing it better, and doing it faster. Releases are intended to occur on a more regular basis, with a smaller set of code changes. Less work means better focus, and more clarity of purpose with each release. Again, automation helps people get their jobs done with less hands-on work. But it also helps speed things up: Software builds can occur at programatic speeds. If orchestration scripts can spin up build or test environments on demand, there is no waiting around for IT to provision systems as it’s part of the automated process. If an automated build fails, scripts can pull the new code and alert the development team to the issue. If automated functional or regression tests fail, the information is in QA or developers hands before they finish lunch. Essentially you fail faster, with subsequent turnaround to identify and address issues being quicker as well.
• Bottlenecks: There are several bottlenecks in software development; developers waiting for specifications, select individuals who are overtasked, provisioning IT systems, testing and even process (i.e.: synchronous ones like waterfall) can cause delays. Both the way that DevOps tasks are scheduled, the reduction in work being performed at any one time, and in the way that expert knowledge is embedded within automation, once DevOps has established itself major bottlenecks common to most development teams are alleviated.
• Cooperation and Communication: If you’ve ever managed software releases, then you’ve witnessed the ping-pong match that occurs between development and QA. Code and insults fly back and forth between these two groups, that is when they are not complaining about how long it is taking IT to get things patched and new servers available for testing and deployment. The impact of having operations and development or QA work shoulder to shoulder is hard to articulate, but focusing the teams on smaller set of problems they address in conjunction with one another, friction around priorities and communication start to evaporate. You may consider this a ‘fuzzy’ benefit, until you’ve seen it first hand, then you realize how many problems are addressed through clear communication and joint creative efforts.
• Technical Debt: Most firms consider the job of development to produce new features for customers. Things that developers want – or need – to produce more stable code are not features. Every software development project I’ve ever participated in ended with a long list of things we needed to do to improve the work environment (i.e.: the ‘To Do’ list). This was separate and distinct from new features; new tools, integration, automation, updating core libraries, addressing code vulnerabilities or even bug fixes. As such, project managers ignored it, as it was not their priority, and developers fixed issues at their own peril. This list is the essence of technical debt, and it piles up fast. DevOps looks to reverse the priority set and target technical debt – or anything that slows down work or reduces quality – before adding new capabilities. The ‘fix-it-first’ approach produces higher quality, more reliable software.
• Metrics and Measurement: Are you better or worse than you were last week? How do you know? The answer is metrics. DevOps is not just about automation, but also about continuous and iterative improvements. The collection of metrics is critical to knowing where to focus your attention. Captured data – from platforms and applications – forms the basis for measuring everything from tangible things like latency and resource utilization, to more abstract concepts like code quality and testing coverage. Metrics are key to know what is working and what could use improvement.
• Security: Security testing, just like functional testing, regression testing, load testing or just about any other form of validation, can be embedded into the process. Security becomes not just the domain of security experts with specialized knowledge, but part and parcel to the development and delivery process. Security controls can be used to flag new features or gate releases within the same set of controls you would use to ensure custom code, application stack or server configurations are to specification. Security goes from being ‘Dr. No’ to just another set of tests to measure code quality.

And that’s a good place to end this post, as the remainder of this series will focus on blending security with DevOps. Specifically our next discussion will be on the role security should play within a DevOps environment.

In the next post I will dig into the role of security in DevOps, but I hope to get a lot of comments before I launch that post next week. I worked hard to capture the essence of DevOps from the research calls and personal experience in this post. And some of the advantages I mention are not all that clear unless used in cloud and virtual environments where the degree of automation changes what’s possible. That said I know that some of the ways I have phrased DevOps advantages will rub some people wrong, so please comment where you disagree or think things are mis-characterized.