As we mentioned earlier, DevOps is not all about tools and technology – much of its success lies in how people work within the model. We have already gone into great detail about tools and process, and we approached much of this content from the perspective of security practitioners getting onboard with DevOps. This paper is geared toward helping security folks, so here we outline their role in a DevOps environment. We hope to help you work with other teams and reduce friction.

And while we deliberately called this paper “Enterprise DevSecOps”, please keep in mind that your development and IT counterparts likely think there is no such thing. To them security becomes part of the operational process of integrating and delivering code. We call security out as a separate thing because, even woven into the DevOps framework, it’s substantially more difficult for security personnel to fit in. You need to consider how you can improve delivery of secure code without waste and without introducing bottlenecks in a development process you may not be intimately familiar with. The good news is that security fits nicely within a DevOps model, but you need to tailor things to work within your organization’s automation and orchestration to be successful.

The Security Pro’s Responsibilities

• Learn the DevOps model: We have not even touched on theory or practice of DevOps in this series. There is a lot for you to learn on base concepts and practices. To work in a DevOps environment you need to understand what it is and how it works. You need to understand cultural and philosophical changes as well as how they effect process, tooling, and priorities. You need to understand your organization’s approach to optimally integrate security tooling and metrics. Once you understand the mechanics of the development team, you’ll have a better idea of how to work with them in their process.
• Learn how to be agile: Your participation in a DevOps team means you need to fit into DevOps – not the other way around. The goal of DevOps is fast, faster, fastest: small iterative changes which offer quick feedback, ultimately reducing Work In Progress (WIP). Small, iterative changes to improve security fit this model. Prioritize things which accelerate delivery of secure software over delivery of new features – keeping in mind that it is a huge undertaking to change culture away from “feature first”. You need to adjust requirements and recommendations so they fit into the process, often simplified into small steps, with enough information for tasks to be both automated and monitored. You can recommend manual code reviews or fuzz testing, so long as you understand where they fit within the process, and what can – and cannot – gate releases.
• Educate: Our experience shows that one of the best ways to bring a development team up to speed in security is training: in-house explanations or demonstrations, third-party experts to help with application security or threat modeling, eLearning, or various commercial courses. The historical downside has been cost, with many classes costing thousands of dollars. You’ll need to evaluate how best to use your resources – the answer typically includes some eLearning for all employees, and select people attending classes and then teaching peers. On-site experts can be expensive, but an entire group can participate in training.
• Grow your own support: There is simply no way for security teams to scale out their knowledge without help. This does not mean hundreds of new security employees – it means deputizing developers to help with product security. Security teams are typically small and outnumbered by developers 100:1. What’s more, security people are not present at most development meetings, so they lack visibility into day-to-day agile activities. To help extend the reach of the security team, see if you can get someone on each development team – a “security champion” – to act as a security advocate. This helps not only extend the security team’s reach, but also expand security awareness.
• Help DevOps team understand threats: Most developers don’t fully grasp how attackers approach systems, or what it means when a code or SQL injection attack is possible. The depth and breadth of security threats is outside their experience and most firms do not teach threat modeling. The OWASP Top Ten is a good guide to the types of code deficiencies which plague development teams, but you should map these threats back to real-world examples, show the damage that a SQL injection attack can cause, and explain how a Heartbleed type vulnerability can completely expose customer credentials. You can these real-world use cases as “shock and awe” to help developers “get it”.
• Advise on remediation practices: Your security program is inadequate if it simply says to “encrypt data” or “install WAF”. All too often, developers and IT have a singular idea of what constitutes security, centered on a single tool they want to set and forget. Help build out the elements of the security program, including both in-code enhancements and supporting tools. Teach how those each help to address specific threats, and offer help with deployment and policy setup. In the past, firms used to produce code style guides to teach younger developers what properly written code looked like. Typically these are not online. Consider a wiki page for security coding practices and other reference materials which are easily discovered and readable by people without a security background.
• Help evaluate security tools: It is unusual for people outside security to fully understand what security tools do or how they work. So you can help in two ways; first, help developers select tools. Misconceptions are rampant, and not just because vendors over-promise. Additionally it is uncommon for developers to evaluate code scanners, activity monitors, or even patch management system effectiveness. In your role as advisor it is your responsibility to help DevOps understand what the tools can provide and what fits within your shared testing framework. Sure, you might not be able to evaluate the quality of the API, but you can tell when a product fails to deliver meaningful results. Second, you should help position the expenditure, as it’s not always clear to the people holding the purse strings how specific tools address security and compliance requirements. You should specify functional and reporting requirements for tools which meet business needs.
• Help with priorities: During our research many security pros told us that all vulnerabilities started looking like high priorities, and it was incredibly difficult to differentiate a vulnerability with impact on the organization from one which did not. Exposure analysis is outside the developer skill set. You need to help fill this gap because not every vulnerability poses real risk. And security folks have a long history of sounding like the terrorism threat scale, with vague warnings about “severe risk” and “high threat levels”. None of these warnings are valuable without mapping a threat to possible exploitations, what the exploit might mean to the business, or what you can do to address – and reduce – risks. For example you might be able to remediate a critical application vulnerability in code, patch supporting systems, disable the feature if it’s not critical, block with IDS or firewalls, or even filter with WAF or RASP technologies. Sometimes code exploitation cannot actually harm the business, so “do nothing” is the right answer! Rather than the knee-jerk “OMG! Fix it now!” reaction we have historically seen, there are typically several options to address a vulnerability, so presenting tradeoffs to a DevOps team allows them to select the best fit.
• Write tests: DevOps has placed some operations and release management personnel in the uncomfortable position of having to learn to script, code, and expose their work for public review. It pushes people outside their comfort zones in the short term, but that is a key part of building a cohesive team in the medium term. It is perfectly acceptable for security folks to contribute tests to the team: scans that validate certificates, checks for known SQL injection attacks, open source tools for locating vulnerabilities, and so on. If you’re worried about it, help out and integrate unit and regression tests. Integrate and ingratiate! To participate in a DevOps team, where automation plays a key part, you will likely need to know how to write scripts or templates. The good news is that your policies are now embodied into the definition of the environment. Don’t be afraid that you don’t know source code control or the right format for the scripts – this is an area where developers are usually keen to assist. You need to relearn a bit on the scripting side before your tests can be integrated into build and deployment servers, but you’ll do more than preach security – you can contribute!
• Advocacy: One crucial area where you can help development is in understanding both corporate and external policy requirements. Just as a CVE entry tells you next to nothing about how to actually fix a security issue, internal security and privacy policy enforcement are often complete mysteries to the development team. Developers cannot Google those answers so offer yourself as an advisor. You should understand compliance, privacy, software licensing, and security policies at least as well as anyone on the development team, so they will probably appreciate the help.

That concludes our series on DevSecOps. Unlike our last paper, Putting Security Into DevOps, which was written to balance the needs of Development, IT and security, this paper is a reaction to market demand. The security folks are the ones who are tasked with getting security into DevOps, and struggling to integrate – culturally or mechanically – into DevOps. And they are the ones who have been asking questions and asking for help. They are the ones having trouble getting the attention of developers, much less developer participation. If you fall into this area, we hope you will find this material helpful. The pain of security testing, and the problem of security controls being outside the domain of developers and IT staff, can be mitigated with DevOps. Security can be largely automated, built slowly over time, and part of the entire release process. And security need no longer be the first casualty of the war for new features and functions — instead it becomes systemized in the delivery process. We have outlined the reasons we expect DevOps to be significant for software development teams in the future, and to advance security testing within application development teams far beyond where it’s stuck today.

We are under no illusion that the journey will be easy, as the path from traditional development to DevOps requires tons or hard work while jettisoning long held beliefs about how to build applications. Every organization hits roadblocks and failures along the way, but the good news is the entire approach is geared to assume failures will occur, and help you identify and move past them. The benefits of slow, consistent and demonstrable improvements benefit not just code code quality but security; not just developer relations with operations but with security professionals as well.

We hope you find this research helpful. If you have any questions on this topic, or want to discuss your situation specifically, feel free to send us a note at info@securosis.com or ask via the Securosis blog.