Our last post reviewed key tools to conduct security tests in the development process, and before that we discussed big picture process adjustments to accommodate security testing, but didn’t fully how to integrate. Agile itself is in the middle of a major disruptive evolution, transforming into a new variant called DevOps, bringing significant long-term implications which are beneficial to security. The evolution of development security and Agile are closely tied together, so we can start by specifying how to integrate into the deployment pipeline, then discuss the implications of DevOps.

Understanding the Deployment Pipeline

The best way to integrate security testing into the development process is by integrating with the deployment pipeline. This is the series of tools an organization uses to take developed code from the brain of a developer into the hands of a customer. While products vary greatly, the toolchains themselves are relatively consistent, although not all organizations use all components.

• Integrated Development Environment (IDE): The IDE is where developers write code. It typically consists of a source code editor (a text editor), a compiler or an interpreter, a debugger, and other tools to help the programmer write code and build applications (such as a user interface editor, code snippet library, version control browser, etc.).
• Issue Tracker: A tracker is basically a project management tool designed to integrate directly into the development process. User stories are entered directly, broken down into features, and broken down again then specific developer tasks/assignments. Detected bugs also go into the issue tracker. This is the central tool for tracking the status of the development project – from earliest concepts, to updates, to production bugs.
• Version Control System/Source Code Management: Managing constantly changing code for even a small application is challenging. Source code is mostly a bunch of text files. And we mean a lot of files, which may be worked on by teams of tens, hundreds, or thousands of developers. The version control system/source code management tool keeps track of all changes and handles checkout, checkin, branching, forking, and otherwise keeping the code consistent and manageable. Whichever tool is used, this is typically referred to as the source code repository, or repo for short.
• Build Automation: Automation tools convert the text of source code into compiled applications. Most modern applications include many components which need to be compiled, integrated, and linked in the correct order. A build automation tool handles both simple and complex scenarios, according to scripts created by developers.
• Continuous Integration (CI) Server: A CI server is the next iteration of build automation. It connects to the source code repository and, based on rules, automatically integrates and compiles code as it is committed. Rather than manually running a build automation tool, the CI server grabs code, creates a build, and runs automated testing automatically when triggered – such as when a developer commits code from an IDE. CI servers can also automate the deployment process, pushing updated code onto production systems.

There are an unlimited range of possible deployment pipelines, and the pipeline is often actually a series of manual processes. But the broad steps are:

1. The product owner enters requirements for a feature into the issue tracker.
2. The product owner or someone else on the development team (such as the program manager) breaks the user story and features down into a set of developer assignments, which are then added to the backlog.
3. The program manager assigns specific tasks to developers.
4. A developer checks out the latest code, writes/edits in an IDE, tests and debugs locally, and then commits it to the source code repository using the version control system. The developer might for existing for independent development and testing, depending on the nature of the feature.
5. The build automation tool compiles the code into the main application and may perform automated testing. The compiled product is then sent to QA/testing and eventually to operations to push into production. If something breaks, that is marked as a bug in the issue tracker.
6. If the organization uses continuous integration the code will be automatically compiled, integrated, and tested using the CI server. It may be pushed into deployment or handed off for additional manual testing, such as user acceptance testing. Again, if something breaks that becomes a bug in the issue tracker, probably automatically.

Not every organization follows even this general process, but just about everyone running Agile uses some variation of it.

Integrating Security

If you map our security toolchain to the deployment pipeline there are clear opportunities for integration. The ones we most commonly see are:

• Security manages security issues and bugs in the issue tracker. Security features are often entered as user stories or feature requirements, in cooperation with the product owner or program manager. Security sensitive bugs are tagged as security issues. In some cases security teams monitor the issue tracker to help identify potential vulnerabilities that might have been entered as simple bug reports.
• Static analysis is integrated in the IDE, build automation tool, or CI server. Sometimes all of the above. For example when a developer commits code locally it can undergo static analysis, with issues highlighted back in the IDE for easy identification and remediation. Static analysis may also be triggered when code is committed to the source code repository.
• Dynamic analysis is also typically integrated at the build automation or CI server, using tests defined by security.
• Other security tests, such as unit, component, and regression testing, are also often best integrated at the build or CI server.
• Vulnerability analysis may be automated if the organization uses a CI server, but otherwise is often a manual or periodic process.
• Any problems discovered by the testing tools generate entries in the issue tracker, just like any other bugs. Ideally security needs to sign off on any unremediated security bugs before release.

Security and DevOps

There is no single definition of DevOps, but essentially it means deeper integration of development and operations in the software deployment process. A better way to phrase it is integration of more Operations in Development, and more Development in Operations. If you remember the corny Reese’s Peanut Butter Cup ads — “You got your peanut butter in my chocolate, no, you got your chocolate in my peanut butter.” — it’s kinda like that. Integrating the two make very good things possible.

Putting code into production is a very important step, one we’ve ignored in this series as it’s generally not part of the Agile development discussion. This step is incredibly error-prone because developers tend to work in their own vacuum, separated from production systems, and separated from the operations folks who deploy it. Heck, that’s often a security and compliance requirement. As a result their development and testing environments are very different from production, and the differences grow over time. This materially increases the chance things will break when they are deployed, because developers cannot account for all the differences.

The process is also very manual, usually with Operations teams running through deployment steps to get everything configured correctly on their systems and then updating the code. Even if an organization builds code using Agile, they often deploy infrequently to avoid breaking production: Agile development but Waterfall deployment!

DevOps addresses this through extensive automation. Thanks to virtualization and cloud computing, we have tools to consistently and programmatically define our environments and “rebuild production” as we update code. This is thanks to two key pieces we add to the process:

• Configuration Management: Newer configuration management tools completely manage and define a system using programmatic scripts. In DevOps we use these tools to define and push server configurations to enforce consistency and automatically update systems.
• Cloud and Virtualization Environment Configuration Tools: Some cloud providers support building out complete environments based on templates. There are also stand-alone tools to manage the same automation. At a basic level you can define networks, servers, configurations, and connections – all using scripts.

In both cases the tools are largely driven by text files from a version control repository, just like code. This means we can completely define the environment, the server configurations, and the application in text files; and build, test, and deploy consistently.

This supports continuous deployment, the heart of DevOps. A developer, working with Ops, defines server and environment requirements within the development process. Both Development and Operations use the same deployment pipeline, which now updates not only code, but also the environment in which it runs. This means, for example, that the CI server can create an entire test environment (on a local virtual platform or in the cloud) which is completely consistent with production. Operations testing is thus integrated with development to detect more problems earlier.

DevOps dramatically improves consistency and predictability, while (thanks to automation) also speeding up deployment if desired. It can be an extension to Agile, aligning with sprints, or become a truly continuous process where code is deployed into production as soon as it is complete and has passed automated tests. Some organizations using DevOps deploy new code into production dozens of times a day, without sacrificing security or resiliency. Errors still occur, but they are smaller, with a faster feedback cycle, and improved ability to roll back to known good states because everything is in version control.

DevOps is ideal for security integration because it enables us to integrate security requirements and testing into Development and Operations – often completely automated. This continuous integration is another key element of DevOps, ensuring all of the code and supporting services work together, but also an ideal place to automatically integrate security. For example we can add server hardening requirements into configuration management scripts. We also get improved audit trails because all production changes start at the beginning of the pipeline, rather than with administrators modifying servers in production.

This is only one aspect of DevOps, but it is the one most relevant to security teams and Agile development. DevOps itself is incredibly disruptive because it increases agility and resiliency while reducing operational overhead costs. At the same time improved consistency and ability to enforce security standards – without impeding the development process – bring tremendous security advantages.

Summary: Be Agile, for Agile

Security’s job is to manage risk for an organization. That is impossible if you don’t understand the risk, and few things are harder to understand than new untested applications. Development teams and security have historically battled due to their competing priorities: Development needs to hit deadlines and get code out the door, while Security needs to understand the code and ensure it’s safe.

If Security sits on one end of the pipeline and tries to evaluate and fix everything before it walks out the factory door, there is no way they can keep up. By the time they see problems – especially in a rapid Agile environment – it is late in the process, more expensive to fix, and sure to hold up business needs.

The answer is to integrate as directly as possible into the development process. Start early with threat modeling, define security requirements, use a shared issue tracker, and automate security testing into the development (or deployment) pipeline. Agile is all about being more responsive and better meeting customer requirements, and these days nobody can credibly argue that security isn’t high on the list. You just need to understand where developers are coming from and how to integrate into their process – trying to make development processes conform to security has a very poor track record, and Agile exacerbates the problem.