DevOps is an operational framework which promotes software consistency and standardization through automation. It helps address many nightmare development issues around integration, testing, patching, and deployment – both by breaking down barriers between different development teams, and also by prioritizing things which make software development faster and easier.

DevSecOps is the integration of security teams and security tools directly into the software development lifecycle, leveraging the automation and efficiencies of DevOps to ensure application security testing occurs in every build cycle. This promotes security and consistency, and helps to ensure that security is prioritized no lower that other quality metrics or features. Automated security testing, just like automated application build and deployment, must be assembled with the rest of the infrastructure.

And there lies the problem. Software developers have traditionally not embraced security. It’s not because they do not care about security, but they were incentivized to to focus on delivery of new features and functions. DevOps is raising the priority of automating build processes – making them faster, easier, and more consistent. But that does not mean developers are going out of their way to include security or security tooling. That’s often because security tools don’t easily integrate well with development tools and processes, tend to flood queues with unintelligible findings, and lack development-centric filters to help prioritize. Worse, security platforms – and the security professionals who recommend them – were difficult to work with, or even failed to offer API support for integration.

On the other side of equation are security teams, who fear automated software processes and commonly ask, “How can we get control over development?” This question misses the point of DevSecOps, and risks placing security in opposition to all other developer priorities: to improve velocity, efficiency, and consistency with each software release. The only way for security teams to cope with the changes within software development, and to scale their relatively small organizations, is to become just as agile as development teams by embracing automation.

Why Did We Write This Paper?

We discuss the motivation behind our research to help readers understand our goals and what we wish to convey. This is doubly relevant when we update a research paper, as it helps us spotlight recent changes in the industry which have made older papers inaccurate or inadequate to describe recent trends. DevOps has matured considerably in four years, so we have a lot to talk about.

This will be a major rewrite of our 2015 research on Building Security into DevOps, with significant additions around common questions security teams ask about DevSecOps and a thorough update on tooling and integration approaches. Much of this paper will reflect 400+ conversations since 2017 across 200+ security teams at Fortune 2000 firms. So we will include considerably more discussion derived from those conversations. But DevOps has now been around for years, so discussion of its nature and value is less necessary, and we can focus on the practicalities of how to put together a DevSecOps program.

Now let’s shake things up a bit.

Different Focus, Different Value

A plethora of new surveys and research papers are available, and some of them a very good. And there are more conferences and online resources popping up than I can count. For example Veracode recently released the latest iteration of its State of Software Security (SoSS) report and it’s a monster, with loads of data and observations. Their key takeaways are that the agility and automation employed by DevSecOps teams provide demonstrable security benefits, including faster patching cycles, shorter flaw persistence, faster reduction of technical debt, and ‘easier’ scanning – which leads to faster problem identification. Sonatype’s recently released 2019 State of the Software Supply Chain shows that “Exemplary Project Teams” who leverage DevOps principles drastically reduce code deployment failure rates, and remediate vulnerabilities in half the time of average groups. And we have events like All Day DevOps, where hundreds of DevOps practitioners share stories on cultural transformations, Continuous Integration / Continuous Deployment (CI/CD) techniques, site reliability engineering, and DevSecOps. All of which is great, and offers qualitative and quantitative data showing why DevOps works and how practitioners are evolving programs.

So that’s not what this paper is about. Those resources do not address the questions I am asked each and every week.

This paper is about putting together a comprehensive DevSecOps program. Overwhelmingly my questioners ask, “How do I put a DevSecOps program together?” and “How does security fit into DevOps?” They are not looking for justification or individual stories on nuances to address specific impediments. They want a security program in line with peer organizations, which embraces “security best practices”. These audiences are overwhelmingly comprised of security and IT practitioners, largely left behind by development teams who have at least embraced Agile concepts, if not DevOps outright. Their challenge is to understand what development is trying to accomplish, integrate with them in some fashion, and figure out how to leverage automated security testing to be at least as agile as development.

DevOps vs. DevSecOps

Which leads us to another controversial topic, and why this research is different: the name DevSecOps. We contend that calling out security – the ‘Sec’ in ‘DevSecOps’ – is needed in light of maturity and understanding of this topic.

Stated another way, practitioners of DevOps who have fully embraced the movement will say there is no reason to add ‘Sec’ into DevOps, as security is just another ingredient. The DevOps ideal is to break down silos between individual teams (e.g., architecture, development, IT, security, and QA) to better promote teamwork and better incentivize each team member toward the same goals. If security is just another set of skills blended into the overall effort of building and delivering software, there is no reason to call it out any more than quality assurance. Philosophically they’re right. But in practice we are not there yet. Developers may embrace the idea, but they generally suck at facilitating team integration. Sure, security is welcome to participate, but it’s up to them to learn where they can integrate, and all too often security is asked to contribute skills which they simply do not possess. It’s passive-aggressive team building!

Automated security testing, just like automated application build and deployment, takes time and skill to build out. In our typical engagements with clients, developers are absent from the call. A divide still exists, with little communication between security and usually dozens to hundreds of diverse development teams. When developers are present they explain that security team can create scripts to integrate security testing into the build server, codify security policies, and stitch together security analysis tools with trouble ticketing and metrics with a few lines of python code. After all, many IT practitioners are learning to script for configuration management and build templates to define infrastructure deployments, so why not security? But few security practitioners are capable of coding at this level. Worse, most firms we speak with have a ratio of around 100 developers to every security practitioner, so there is simply no way to scale their security resources across all development projects.

It does not help that many security professionals are early in understanding DevOps, while developers have been adopting various methods over the last decade to become more agile. Security is genuinely behind the curve, and it seems the bulk of the available research, as mentioned above, is not aimed at tackling security’s introduction and integration.

We have one more very important reason to use the DevSecOps name: security efforts for code security are very different than efforts to secure infrastructure and supporting components. The security checks to validate application code is secure (DevSec) are very different than the tooling and processes to verify the supporting infrastructure (SecOps) is secure. These are two different disciplines, with different tooling and approaches, and it is a mistake to discuss one as a subset of the other.

Common Questions

We went through our call notes from the last three years and tallied up the questions we were asked. Following are the most common questions in order of how often they were asked.

• We want to integrate security testing into the development pipeline, and are going to start with static analysis. How do we do this?
• How do we build an application security strategy in light of automation, CI/CD and DevOps?
• How can we start building out an application security strategy? What app security standards should I follow?
• Development is releasing code to production every day. How do we get control over development? Can we realistically modify developer behavior?
• What is the best way to introduce DevSecOps? Where do I start? What are the foundational pieces?
• How do we get different units to adopt the same program (different teams all do things differently)? Right now some use waterfall, some agile, and others DevOps?
• How should we (Security) work with Dev?
• We understand “shift left”, but are the tools effective? What tools do you recommend we start with?

These questions contain some common threads: they all come from firms with at least some DevOps teams already in place, where security has some understanding of the intent of DevOps, but they are all starting from scratch. Even security teams with tests already built into their development pipelines struggle with the value each tool provides, pushback from developers over false positives, how to work with developers, or how to scale consistently across many development teams. During calls and engagements, we find security doesn’t quite understand why developers embrace DevOps, often missing the thrust of their effort.

The following is a list of questions which security teams should ask but don’t.

• How do we fit – operationally and culturally – into DevSecOps?
• How do we get visibility into Development and their practices?
• How do we know changes are effective? What metrics should we collect and monitor?
• How do we support Development?
• Do we need to know how to code?

Over the course of this series we will address all these questions. Next we will provide a brief introduction to DevSecOps principles and the role of security in DevOps.