In our first paper on ‘Building Security Into DevOps’, given the ‘newness’ of DevOps for most of our readers, we included a discussion on the foundational principles and how DevOps is meant to help tackle numerous problems common to software delivery. Please refer to that paper is you want more detailed background information. For our purposes here we will discuss just a few principles that directly relate to the integration of security teams and testing with DevOps principles. These concepts lay the foundations for addressing the questions we raised in the first section, and readers will need to understand these as we discuss security tooling and approaches in a DevOps environment.

And before we dive in, let’s answer one of the most common questions from the previous section: “How do we get control over development?” The short answer is you do not. The longer answer is, in DevOps, you need to work along side your partner, not “control” them. Yes, a small percentage of organizations we spoke with gate all software releases by having security run a battery of tests prior to release, and certify every release from a security standpoint. It is rare that security gets to control software releases in this way, is anti-DevOps, but it can be very effective security control if not altogether efficient. That said, the remainder of this section should exemplify why your goal should not be to control development, and helpful approach to work with them.

DevOps and Security

Build Security In

It is a terrible truth, but wide use of application security techniques within the code development process is relatively new. Sure, the field of study is decades old, but application security was more often bolted on with network or application firewalls, not baked into the code itself. Security product vendors discovered that understanding application requests in order to detect and then block attacks is incredibly difficult to do outside the application. It is far more effective to fix vulnerable code and close off attack vectors when possible. Add-on tools are getting better – and some work inside the application context – but better to address the issues in the code when possible.

A central concept when building security in is ‘shift left’, or the idea that we integrate security testing earlier within the Software Development Lifecycle (SDLC) – the phases of which are typically listed left to right as design, development, testing, pre-production and production. Essentially we shift more resources away from production on the extreme right, and put more into design, testing and development phases. Born out of lean manufacturing, Kaizen and Deming’s principles, these ideas have been proven effective, but typically applied to the manufacture of physical goods. DevOps has promoted use in software development, demonstrating we can improve security at a lower cost by shifting security defect detection earlier in the process.

Automation

Automation is one of the keys to success for most firms we speak with, to the point that the engineering teams often equate DevOps and Automation as synonymous. The reality is the cultural and organizational changes that come with DevOps are equally important, it’s just that automation is sometimes the most quantifiable benefit.

Automation brings speed, consistency and efficiency to all parties involved. DevOps, like Agile, is geared towards doing less, better, and faster. Software releases occur more regularly, with less code change between them. Less work means better focus, more clarity of purpose with each release, resulting in fewer mistakes. It also means it’s easier to rollback in the event of mistakes. Automation helps people get their jobs done with less hands-on work, but as automation software does exactly the same things every time, consistency is the most conspicuous benefit.

The place where automation is first applied, where the benefits of automation are most pronounced, are the application build servers. Build servers (e.g.: Bamboo, Jenkins, ), commonly called Continuous Integration (CI) servers, automatically construct an application – and possibly the entire application stack – as code is changed. Once the application is built, these platforms may also launch QA and security tests, kicking back failed builds to the development team. Automation benefits other facets of software production, including reporting, metrics, quality assurance and release management, but security testing benefits are what we are focused on in this research.

On the outset this may not seem like much; calling security testing tools instead of manually running the tests. That perspective misses the fundamental benefits of automated security testing. Automation is how we ensure that each update to software includes security tests, ensuring consistency. Automation is how we help avoid mistakes and omissions common with repetitive and – let’s be totally transparent here – boring manual tasks. But most importantly, as security teams are typically outnumbered by developers at a ratio of 100 to one, automation is the key ingredient to scaling security coverage without having to scale security personnel headcount.

One Team

A key DevOps principle is to break down silos and have better cooperation between developers and supporting QA, IT, security and other teams. We have heard this idea so often that it sounds cliché, but the reality is few in software development actually made changes to implement this idea. Most DevOps centric firms are changing development team composition to include representatives from all disciplines; that means every team has someone who knows a little security and/or represents security interest, even on small teams. And for those that do, they realize the benefits not just of better communication, but true alignment of goals and incentives. Development in isolation is incentivized to write new features. Quality Assurance in isolation is incented to get code coverage for various tests. When everyone on a team is responsible for the successful release of new software, there is a change in priorities and changes to behavior.

This item remains a bit of a problem for many of the firms we have interviewed. The majority of firms we have spoken with are large in size, with hundreds of development teams located in different countries, some of which are third party (re: external) consultants. It is hard to get consistency across all of these teams, and even harder to get general participation. Managerial structure is set up so development managers manage developers, not IT personnel. The management tools for feature tracking, trouble-ticketing, resource allocation are geared towards a siloed structure. And many leading security tools are set to analyze and report defects to security professionals, not developers or IT personnel who resolve an issue. Progress is still measured in feature outputs and code coverage, and bonuses awarded accordingly.

The point here is this cultural change, and the great benefits derived, are not realized without some changes to the supporting systems and structures. This is a very hard adjustment, one where the various managers are all looking to implement policies as if they have full oversight, missing the point that they too need to adopt the ‘one team’ approach with their peers to effectively enact changes.

Security Practitioners and Application Security

Why security folks struggle with DevSecOps, and even application security in general, as they do not have backgrounds in software development. Most security practitioners come from a network security background, and many CISOs we speak with are more risk and compliance focused, so there is a general lack of understanding of software development. This lack of knowledge of development tooling and processes, along with common challenges developers are trying to overcome, means security teams seldom understand why automated build servers, central code repositories, containers, Agile and DevOps have caught fire and widely adopted in a very short time. Here we discuss some of the drivers for changes in development practices and the key areas security teams need to understand when trying to get a handle on application security.

• Knowledge of Process: We are not here to teach the nuances of development process, but we want to point out the reasons why processes change: Speed. Moving from Waterfall to various processes like Spiral, Prototype Evolutions, Extreme Programming, Agile and Agile with Scrum and other variations made over the last 20 years. Each was an attempt to satisfy the same goals: Simplify, hopefully avoiding bugs, and speed up delivery. The entirety of software development practice evolution in recent decades has been aimed at addressing these three goals, which helps to show that the process itself is not the important part. Daily scrums, bi-weekly software delivery (sprints), Kanban, Agile, test driven development and automated build servers are tools to advance the state of the art. So it is critical for security professionals to understand that security testing and policies should embrace these same ideals. And lastly, DevOps is process independent; you can embrace DevOps and still have a waterfall process, but certainly DevOps fits more naturally with Agile.
• Knowledge of Tools: Software development leverages many tools to manage code and process. The two most important to security are code repositories and build tools. Repositories like Git essentially manage application code, giving developers a shared location to store code, track versions and changes. Others, like Docker Registry, are specifically for containers. These tools are essential for developers to manage the code they are building, but also important to security as a place where code can be inspected. Build servers like Jenkins and Bamboo automate the building, testing, and delivery of code. But rather than at a component or module level, they are typically employed for full application stack testing. Developers and quality assurance teams use the build server to launch functional, regression, and unit testing; security teams should leverage these build servers to integrate security testing (such as SAST, DAST, Composition Analysis, and security unit tests) so it fits within the same build process and uses all the same management and communications tools. It is important for security teams to understand which tools the development team uses and who controls those resources, and arrange for integration of security testing.
• Everything Is Code: Applications are software. This is fairly well understood, but less appreciated is that in many environments – especially public cloud – servers, networks, messaging, IAM and every other bit of the infrastructure may be defined as configuration scripts, templates, or application code. IT teams now define entire data centers with templates comprised of a few hundred lines of script. The key for security practitioners are twofold: security policies can also be defined in scripts/code, and you can examine code repositories to ensure the templates, scripts and code are secure before they run. This is a fundamental change to security audits.
• Open Source: Open source software plays a huge part in application development, and is so universally embraced in the development community that it is almost impossible to find a new application development project which does not leverage it. This means a large portion of your code may not be tested the way you expect, or developers may intentionally use old, vulnerable versions, because they know the old version works with their code. If they change a library, it might break the application and require more work. Developers are incented to get code working and we have witnessed heroic efforts on their part to avoid new (patched) open source versions, for the sake of stability. So we want you to come away with two points: you need to test open source code before it hits production, and you need to ensure that developers do not surreptitiously swap out trusted debugged versions of open source libraries for older, probably vulnerable, versions.
• Tooling and Developer “Buy in”: The first step most security teams take when introducing security into application development is to run static analysis scans. The good part is that most security practitioners know what SAST is and does. The bad part is that security started with older SAST tools which were slow, produced output only intelligible by security folks, created “false positive” alerts, and lacked critical API needed to fully integrate with the build process. Overall, their efforts were developer hostile, and most development teams reacted by ignoring the scans or removing those tools from the build process. Two key points: select tools which fit the development model (faster, easier, better), and use tools which are actually effective. Left to their own decisions, developers always choose the easiest tool to integrate, not the most effective security scanner. It is important that the security team be part of the security tool selection process to ensure security scans provide adequate analysis.
• Security Friction & Cultural Dynamics: Most application security teams are playing catch-up. Development is (usually) already agile, and if some of your development organizations are embracing DevOps IT and QA may be as well. This means security folks are the non-agile anomaly– anything you do or ask adds time and complexity, the antithesis of software engineering goals. This topic is so important that I added the entire next section, “Scaling Security”, to address the cultural friction between security and development.
• SDLC and S-SDLC: Many application security teams approach application security by looking at the Software Development LifeCycle (SDLC), with the goal of applying some form of security analysis in each phase of the lifecycle. A Secure SDLC (S-SDLC) typically includes threat modeling during design, composition analysis in development, static analysis during build, and any number of tests pre-production. This is an excellent way to set up a process independent application security program. As many large organizations come to understand, each of your development teams employs a slightly different process, and it is entirely possible your company uses every known development process in existence. This is a huge headache but the S-SDLC can become your yardstick: Use it as your policy template and map security controls to the fit within the different processes.

Scaling Security

As mentioned in the introductory section, security teams are vastly outnumbered. As an example, I spoke with three midsized firms this week – their development personnel ranged from 800-2000 people, while their security teams ranged from 12 to 25. They typically had two or three security personnel with backgrounds in application security. They may be rare as unicorns, but that does not give them magical powers to cover all development operations, so they need ways to scale their experience across the enterprise. And they need to do it in a way which meshes with development objectives, getting software development teams to implement their security controls. Here are several methods that work.

• Automation: We have already discussed automation to some degree so I can keep it short here. Automation is how security analysis can be faster, more frequent, and without direct involvement from the security team. Security tools which perform automated analysis, either out of the box or your own custom checks, are critical to scale across development teams. And yes, this means you need to integrate security tools into every build pipeline in your company. But this means that not only are scans automated, but distribution of results is also integrated with other tools and processes. This is how teams scale, and necessary for the next two items.
• Failing the Build: Development and security teams commonly have friction with each other. Security typically provides development managers security scan results with thousands of defects. Development managers read them as saying “Dude, your code sucks, what’s wrong with you, fix it now!” One way to reduce friction between the two groups is to take the output from static or dynamic scans, discuss the scope of the problem, what critical defects mean, and come to agreement on what is reasonable to fix in the middle term. Once everyone agrees about what a critical issue is and what is a reasonable timeframe to fix it, you instruct the security tools to fail a build when critical bugs are discovered. This process takes some time to implement, and some pain to work through, but it changes the relationship between security and development. No longer is it security saying code is defective – instead it’s an unbiased tool reporting a defect to development. Security is no longer the bad guy standing in the way of progress– now development must meet a new quality standard, one focused on code quality in terms of security defects. It also shifts interactions because developers often need assistance understanding the defect, look for ways to tackle classes of defects instead of individual bugs, and seek help from security. Failing the build creates a sea change between groups. Any security tooling which produces false positives magnifies the difficulty in establishing this change, but this step is critical for DevOps teams.
• Metrics: Metrics are essential to understand the scope of application security issues, and security tools are how you collect most metrics. Even of you do not fail the build, and even if the results are not shared with anyone outside security, integrating security testing into build servers and code repositories is critical to gaining visibility and producing metrics. These metrics help you decide where to spend budget – whether on additional tooling, developer education, or runtime protection. And these metrics will be your guide to the effectiveness of the tools, education, and runtime protection you implement. Without metrics you’re just guessing.
• Security Champions: One of the most effective methods I have discovered to scale security is to deputize willing developers with an active interest in security to be “security champions” on their development teams. Most developers have some interest in security, and they know security education makes them more valuable, which often means raises. For questions on security the developer gains a liaison on a security team, and in turn can ask the champion their own questions. Typically security teams cultivate these relationships through education, a “center of excellence” where developers and security pros can interact (often a Slack channel these days), sending developers to security conferences, or simply sponsoring events like lunches where security topics are discussed. Regardless of how you do it, this is an excellent way to scale security without scaling headcount, and we recommend you set aside some budget and resources – it returns far more benefits than it costs.
• Education: If you want developers to understand security and threats to applications, educate them. Educational budgets for engineering leads and VPs are usually tightly restricted. To fill gaps it is not uncommon for security teams to shoulder the expense of educating select developers on relevant skills the organization lacks. Sometimes this is through purchase of security related CBT, sometimes through purchase of professional services from security vendors, and sometimes it is specific classes from SANS or other organizations. Understanding how to remediate application security issues, security reference architectures, how to perform threat modeling, and how to use security tools are all common topics.

This series will run a bit longer than most. Over the last few years we have acquired considerable research. We will attempt to be concise but there is a lot of material to cover to address the key questions.

Next I will discuss putting together a Secure SDLC and integration of security testing into the development process.