Research

April 7 Update: some research is emerging since I posted this that COVID related ARDS is not typical ARDS. Here’s the medical reference for providers but it’s very early evidence so far we should keep an eye on: COVID-19 Does Not Lead to a “Typical” ARDS. This was further validated by an article in MedScape that previews some emerging peer-reviewed research. Thus while my explanations of ARDS and ventilators is accurate, the ties to COVID-19 are not and new treatment protocols are emerging. Although this is a security blog, this post has absolutely nothing to do with security. No parallels from medicine, no mindset lessons, just some straight-up biology. As many readers know I am a licensed Paramedic. I first certified in the early 1990’s, dropped down to EMT for a while, and bumped back up to full medic two years ago. Recently I became interested in flight and critical care and completed an online critical care and flight medic course from the great team at FlightBridgeED. Paramedics don’t normally work with ventilators – it is an add-on skill specific for flight and critical care (ICU) transports. I’m a neophyte to ventilator management, with online and book training but no practice, but I understand the principles, and thanks to molecular biology back in college, have a decent understanding of cellular processes. COVID-19 dominates all our lives now, and rightfully so. Ventilators are now a national concern and one the technology community is racing to help with. Because of my background I’ve found myself answering a lot of questions on COVID-19, ARDS, and ventilators. While I’m a neophyte at running vents, I’m pretty decent at translating complex technical subjects for non-experts. Here’s my attempt to help everyone understand things a bit better. The TL;DR is that COVID-19 damages the lungs, which for some people triggers the body to overreact with too much inflammation. This extra fluid interferes with gas exchange in the lungs, and oxygen can’t as easily get into the bloodstream. You don’t actually stop breathing, so we use the ventilators to change pressure and oxygen levels, in an attempt to diffuse more oxygen through this barrier and into the lungs without, causing more damage by overinflating them. We start with respiration Before we get into COVID and ventilators we need to understand a little anatomy and physiology. Cells need oxygen to convert fuel into energy. Respiration is the process of getting oxygen into cells and removing waste products – predominantly CO2. We get oxygen from our environment and release CO2 through ventilation: air moving in and out of our lungs. Those gases are moved around in our blood, and the actual gas exchange occurs in super-small capillaries which basically wrap around our cells. The process of getting blood to tissues is called perfusion. Theis is all just some technical terminology to say: our lungs take in oxygen and release carbon dioxide, we move the gases around using our circulatory system, and we exchange gases in and out of cells in super-small capillaries. Pure oxygen is a toxin, and CO2 diffused in blood is an acid, so our bodies have all sorts of mechanisms to keep things running. Everything works thanks to diffusion and a few gas laws (Graham’s, Henry’s, and Dalton’s are the main ones). Our lungs have branches and end in leaves called alveoli. Alveoli are pretty wild – they have super-thin walls to allow gases to pass through, and are surrounded by capillaries to transfer gasses into and out of our blood. They look like clumps of bubbles, because they maximize surface area to facilitate the greatest amount of gas exchange in the smallest amount of space. Healthy alveoli are covered in a thin liquid called surfactant, which keeps them lubricated so they can open and close and slide around each other as we breathe. Want to know one reason smokers and vapers have bad lungs? All those extra chemicals muck up surfactant, thicken cell walls, and cause other damage. In smokers a bunch of the alveoli clump together, losing surface area, in a process called atelectasis (remember that word). Our bodies try to keep things in balance, and have a bunch of tools to nudge things in different directions. The important bit for our discussion today is that ventilation is managed through how much we breathe in for a given breath (tidal volume), and how many times a minute we breathe (respiratory rate). This combination is called our minute ventilation and is normally about 6-8 liters per minute. This is linked to our circulation (cardiac output), which is around 5 liters per minute at rest. The amount of oxygen delivered to our cells is a function of our cardiac output and the amount of oxygen in our blood. We need good gas exchange with our environment, good gas exchange into our bloodstream, and good gas exchange into our cells. COVID-19 screws up the gas exchange in our lungs, and everything falls apart from there. Acute Respiratory Distress Syndrome ARDS is basically your body’s immune system gone haywire. It starts with lung damage – which can be an infection, trauma, or even metabolic. One of the big issues with ventilators is that they can actually cause ARDS with the wrong settings. This triggers an inflammatory response. A key aspect of inflammation is various chemical mediators altering cell walls, especially those capillaries – and then they start leaking fluid. In the lungs this causes a nasty cascade: Fluid leaks from the capillaries and forms a barrier/buffer of liquid between the alveoli and the capillaries, and separates them. This reduces gas exchange. Fluid leaks into the alveoli themselves, further inhibiting gas exchange. The cells are damage by all this inflammation, triggering another stronger immune response. Your body is now in a negative reinforcement cycle and making things worse by trying to make them better. This liquid and a bunch of the inflammation chemicals dilute the surfactant and damage the alveolar walls, causing atelectasis. In later stages of ARDS your

This is the third post in our series, “Network Operations and Security Professionals’ Guide to Managing Public Cloud Journeys”, which we will release as a white paper after we complete the draft and have some time for public feedback. You might want to start with our first and second posts. Special thanks to Gigamon for licensing. As always, the content is being developed completely independently using our Totally Transparent Research methodology. Learning cloud adoption patterns doesn’t just help us identify key problems and risks – we can use them to guide operational decisions to address the issues they consistently raise. This research focuses on managing networks and network security, but the patterns include broad security and operational implications which cover all facets of your cloud journey. Governance issues aside, we find that networking is typically one of the first areas of focus for organizations, so it’s a good target for our first focused research. (For the curious, IAM and compliance are two other top areas organizations focus on, and struggle with, early in the process). Recommendations for a Safe and Smooth Journey Developer Led Mark sighed with relief and satisfaction as he validated the VPN certs were propagated and approved the ticket for firewall rule change. The security group was already in good shape and they managed to avoid having to add any kind of direct connect to the AWS account for the formerly-rogue project. He pulled up their new cloud assessment dashboard and all the critical issues were clear. It would still take the IAM team and the project’s developers a few months to scale down unneeded privileges but… not his problem. The federated identity portal was already hooked up and he would get real time alerts on any security group changes. “Now onto the next one,” he mumbled after he glanced at his queue and lost his short-lived satisfaction. “Hey, stop complaining!” remarked Sarah, “We should be clear after this backlog now that accounting is watching the credit cards for cloud charges; just run the assessment and see what we have before you start complaining.” Having your entire organization dragged into the cloud thanks to the efforts of a single team is disconcerting, but not unmanageable. The following steps will help you both wrangle the errant project under control, and build a base for moving forward. This was the first adoption pattern we started to encounter a decade ago as cloud starting growing, so there are plenty of lessons to pull from. Based on our experiences, a few principles really help manage the situation: Remember that to meet this pattern you should be new to either the cloud in general, or to this cloud platform specifically. These are not recommendations for unsanctioned projects covered by your existing experience and footprint. Don’t be antagonistic. Yes, the team probably knew better and shouldn’t have done it… but your goal now is corrective actions, not punitive. You goal is to reduce urgent risks while developing a plan to bring the errant project into the fold. Don’t simply apply your existing policies and tooling from other environments to this one. You need tooling and processes appropriate for this cloud provider. In our experience, despite the initial angst, these projects are excellent opportunities to learn your initial lessons on this platform, and to start building out for a larger supported program. If you keep one eye on immediate risks and the other on long-term benefits, everything should be fine. The following recommendations go a long way towards reducing risks and increasing your chances of success. But before the bullet points we have one overarching recommendation: As you gain control over the unapproved project, use it to learn the particulars of this cloud provider and build out your core cloud management capabilities. When you assess, set yourself up to support your next ten assessments. When you enable monitoring and visibility, do so in a way which supports your next projects. Wherever possible build a core service rather than a one-off. Step one is to figure out what you are dealing with: How many environments are involved? How many accounts, subscriptions, or projects? How are the environments structured? This involves mapping out the application, the PaaS services offered by the provider (they offer PaaS services such as load balancers and serverless capabilities), the IAM, the network(s), and the data storage. How are the services configured? How are the networks structured and connected? The Software Defined Networks (SDN) used by all major cloud platforms only look the same on the surface – under the hood they are quite a bit different. And, most importantly, Where does this project touch other enterprise resources and data?!? This is essential for understanding your exposure. Are there unknown VPN connections? Did someone peer through an existing dedicated network pipe? Is the project talking to an internal database over the Internet? We’ve seen all these and more. Then prioritize your biggest risks: Internet exposures are common and one of the first things to lock down. We commonly see resources such as administrative servers and jump boxes exposed to the Internet at large. In nearly every single assessment we find at least one instance or container with port 22 exposed to the world. The quick fix for these is to lock them down to your known IP address ranges. Cloud providers’ security groups are very effective because they just drop traffic which doesn’t meet the rules, so they are an extremely effective security control and a better first step than trying to push everything through an on-premise firewall or virtual appliance. Identity and Access Management is the next big piece to focus on. This research is focused more on networking, so we won’t spend much time on this here. But when developers build out environments they almost always over-privilege access to themselves and application components. They also tend to use static credentials, because unsanctioned projects are unlikely to integrate into your federated identity management. Sweep out static credentials, enable federation, and turn

This is the second post in our series, “Network Operations and Security Professionals’ Guide to Managing Public Cloud Journeys”, which we will release as a white paper after we complete the draft and have some time for public feedback. You might want to start with our first post. Special thanks to Gigamon for licensing. As always, the content is being developed completely independently using our Totally Transparent Research methodology. Understanding Cloud Adoption Patterns Cloud adoption patterns represent the most common ways organizations move from traditional operations into cloud computing. They contain the hard lessons learned by those who went before. While every journey is distinct, hands-on projects and research have shown us a broad range of consistent experiences, which organizations can use to better manage their own projects. The patterns won’t tell you exactly which architectures and controls to put in place, but they can serve as a great resource to point you in the right general direction and help guide your decisions. Another way to think of cloud adoption patterns is as embodying the aggregate experiences of hundreds of organizations. To go back to our analogy of hiking up a mountain, it never hurts to ask the people who have already finished the trip what to look out for. Characteristics of Cloud Adoption Patterns We will get into more descriptive detail as we walk through each pattern, but we find this grid useful to define the key characteristics. Characteristics Developer Led Data Center Transformation Snap Migration Native New Build Size Medium/Large Large Medium/Large All (project-only for mid-large) Vertical All (minus financial and government) All, including Financial and Government Variable All Speed Fast then slow Slow (2-3 years or more) 18-24 months Fast as DevOps Risk High Low(er) High Variable Security Late Early Trailing Mid to late Network Ops Late Early Early to mid Late (developers manage) Tooling New + old when forced Culturally influenced; old + new Panic (a lot of old) New, unless culturally forced to old Budget Owner Project based/no one IT, Ops, Security IT or poorly defined Project-based, some security for shared services Size: The most common organization sizes. For example developer-led projects are rarely seen in small startups, because they can skip directly to native new builds, but common in large companies. Vertical: We see these patterns across all verticals, but in highly-regulated ones like financial services and government, certain patterns are less common due to tighter internal controls and compliance requirements. Speed: The overall velocity of the project, which often varies during the project lifetime. We’ll jump into theis more, but an example is developer-led, where initial setup and deployment are very fast, but then wrangling in central security and operational control can take years. Risk: This is an aggregate of risk to the organization and of project failure. For example in a snap migration everything tends to move faster than security and operations can keep up, which creates a high chance of configuration error. Security: When security is engaged and starts influencing the project. Network Ops: When network operations becomes engaged and starts influencing the project. While the security folks are used to being late to the party, since developers can build their own networks with a few API calls, this is often a new and unpleasant experience for networking professionals. Tooling: The kind of tooling used to support the project. “New” means new, cloud-native tools. “Old” means the tools you already run in your data centers. Budget Owner: Someone has to pay at some point. This is important because it represents potential impact on your budget, but also indicates who tends to have the most control over a project. Characteristics of Cloud Adoption Patterns In this section we will describe what the patterns look like, and identify some key risks. In our next section we will offer some top-line recommendations to improve your chances of success. One last point before we jump into the patterns themselves: while they focus on the overall experiences of an organization, patterns also apply at the project level, and an organization may experience multiple patterns at the same time. For example it isn’t unusual for a company with a “new to cloud” policy to also migrate existing resources over time as a long-term project. This places them in both the data center transformation and native new build patterns. Developer Led Mark was eating a mediocre lunch at his desk when a new “priority” ticket dropped into his network ops queue. “Huh, we haven’t heard from that team in a while… weird.” He set the microwaved leftovers to the side and clicked open the request… Request for firewall rule change: Allow port 3306 from IP 52.11.33.xxx/32. Mission critical timeline. “What the ?!? That’s not one of our IPs?” Mark thought as he ran a lookup. “amazonaws.com? You have GOT to be kidding me? We shouldn’t have anything up there. Mark fired off emails to his manager and the person who sent the ticket, but he had a bad feeling he was about to get dragged into the kind of mess that would seriously ruin his plans for the next few months. Developer-led projects are when a developer or team builds something in a cloud on their own, and central IT is then forced to support it. We sometimes call this “developer tethering”, because these often unsanctioned and/or uncoordinated projects anchor an organization to a cloud provider, and drag the rest of the organization in after them. These projects aren’t always against policy – this pattern is also common in mergers and acquisitions. This also isn’t necessarily a first step into the cloud overall – it can also be a project which pulls an enterprise into a new cloud provider, rather than their existing preferrred cloud provider. This creates a series of tough issues. To meet the definition of this pattern we assume you can’t just shut the project down, but actually need to support it. The project has been developed and deployed without the input of security or

This is the first post in a new series, our “Network Operations and Security Professionals’ Guide to Managing Public Cloud Journeys”, which we will release as a white paper after we complete the draft and have some time for public feedback. Special thanks to Gigamon for licensing. As always, the content is being developed completely independently using our Totally Transparent Research methodology. Cloud computing is different, disruptive, and transformative. It has no patience for traditional practices or existing architectures. The cloud requires change, and there is a growing body of documentation on end states you should strive for, but a lack of guidance on how to get there. Cloud computing may be a journey, but it’s one with many paths to what is often an all-too-nebulous destination. Although every individual enterprise has different goals, needs, and capabilities for their cloud transition, our experience and research has identified a series of fairly consistent patterns. You can think of moving to cloud as a mountain with a single peak, with everyone starting from the same trailhead. But this simplistic view, which all too often underlies conference presentations and tech articles, fails to capture the unique opportunities and challenges you will face. At the other extreme, we can think of the journey as involving a mountain range with innumerable peaks, starting points, and paths… and a distinct lack of accurate maps. This is the view that tends to end with hands thrown up in the air, expressions of impossibility, and analysis paralysis. But our research and experience guide us between those extremes. Instead of a single constrained path which doesn’t reflect individual needs, or totally individualized paths which require you to build everything and learn every lesson from scratch, we see a smaller set of common options, with consistent characteristics and experiences. Think of it as starting from a few trailheads, landing on a few peaks, and only dealing with a handful of well-marked trails. These won’t cover every option, but can be a surprisingly useful way to help structure your journey, move up the hill more gracefully, and avoid falling off some REALLY sharp cliff edges. Introducing Cloud Adoption Patterns Cloud adoption patterns represent a consolidated set of cloud adoption journeys, compiled through discussions with hundreds of enterprises and dozens of hands-on projects. Less concrete than specific cloud controls, they are a more general way of predicting and understanding the problems facing organizations when moving to cloud, based on starting point and destination. These patterns have different implications across functional teams, and are especially useful for network operations and network security, because they tend to fairly accurately predict many architectural implications, which then map directly to management processes. For example there are huge differences between a brand-new startup or cloud project without any existing resources, a major data center migration, and a smaller migration of key applications. Even a straight-up lift and shift migration is extremely different if it’s a one-off vs. a smaller project vs. wrapped up in a massive data center move with a hard cutoff deadline (often thanks to an hosting contract which is not being renewed). Each case migrates an existing application stack into the cloud, but the different scope and time constraints dramatically affect the actual migration process. We’ll cover them in more detail in our next post, but the four patterns we have identified are: Developer led: A development team starts building something in the cloud outside normal processes, and then pulls the rest of the organization behind them. Data center transformation: An operations-led process defined by an organization planning a methodical migration out of existing data centers and into the cloud, sometimes over a decade or more. Snap migration: An enterprise is forced out of some or all their data centers on a short timeline, due to contract renewals or other business drivers. Native new build: The organization plans to build a new application or several completely in the cloud using native technologies. You likely noticed we didn’t mention some common terms like “refactor” and “new to cloud”. Those are important concepts but we consider them options on the journey, not to define the journey. Our four patterns are about the drivers for your cloud migration and your desired end state. Using Cloud Adoption Patterns The adoption patterns offer a framework for thinking about your upcoming (or in-process) journey, and help identify both strategies for success and potential failure points. These aren’t proscriptive like the Cloud Security Maturity Model or the Cloud Controls Matrix; they won’t tell you exactly which controls to implement, but are more helpful when choosing a path, defining priorities, mapping architectures, and adjusting processes. Going back to our mountain-climbing analogy, the cloud adoption patterns point you down the right path and help you decide which gear to take, but it’s still up to you to load your pack, know how to use the gear, plan your stops, and remember your sunscreen. These patterns represent a set of characteristics we consistently see based on how organizations move to cloud. Any individual organization might experience multiple patterns across different projects. For example a single project might behave more like a startup, even while you concurrently run a larger data center migration. Our next post will detail the patterns with defining characteristics. You can use it to determine your overall organizational journey, as well as to plan out individual projects with their own characteristics. To help you better internalize these patterns, we will offer fictional examples based on real experiences and projects. Once you know which path you are on, our final sections will include top-line recommendations for network operations and security, and tie back to our examples to show how they play out in real life. We will also highlight the most common pitfalls and their potential consequences. This research should help you better understand what approaches will work best for your project in your organization. We are focusing this first round on networking, but future work will build on this basis to

For Rich and me, it seems like forever that we’ve been doing this cloud thing. We previewed the first CCSK class back at RSAC 2011, so we’re closing in on 10 years of hands-on, in the weeds cloud stuff. It’s fundamentally changed Securosis, and we ended up as founders of DisruptOps as well. Yet as the cloud giveth, it also taketh away. Adrian’s unique perspective on application and cloud security made him a great candidate to join Bank of America, so he did. It’s a great opportunity, but we’ll certainly miss having him around during RSAC week. Especially since it means I’ll have to get the aspirin and Tums for you derelicts. But that’s not this year’s DRB theme. We picked (IM)MATURITY because it’s hard to keep in mind that we’re still in the very early stages of the cloud disruption on IT. The questions we get now are less _ “what is this cloud thing?”_ and more ” what does the journey look like?” We didn’t have a decent answer, so we set out to find one. That led us to partner with our friends at IANS to develop a Cloud Security Maturity Model that gives you a sense of the cloud journey and helps you understand how to increase your cloud security capability and maturity. At this year’s DRB, we’ll have the model on hand, as well as an online diagnostic where you can do a self-assessment against the model. We may even have a new strategic relationship to announce at the breakfast. (hush, hush – don’t tell anyone) Let’s celebrate both our maturity in the security space (yes, this is my 24th RSA Conference) while acknowledging the immaturity of securing the cloud by once again holding the most kick-ass breakfast of the year. There are breakfast impostors now, filling up the TableTop Tap House on the other mornings of the conference. But we are the true breakfast innovators of the security industry. There can be only ONE Disaster Recovery Breakfast. check out the full invite Kidding aside, the DRB happens only because of the support of our long-time supporters LaunchTech and DisruptOps, and our media partner Security Boulevard. We’re excited to welcome IANS, Cloud Security Alliance, Highwire PR, and AimPoint Group to the family as well. Please make sure to say hello and thank them for helping support your recovery. As always, the breakfast will be Thursday morning of RSA Week (February 27) from 8-11 at Tabletop Tap House in the Metreon (fka Jillian’s). It’s an open door – come and leave as you want. The breakfast spread will be awesome (it always is), and the bar will be open. I am still drinking Decaf, but I’ve traded in my Bailey’s for a little Amarula after sampling the nectar on my recent trip to South Africa. Please always always always remember what the DR Breakfast is all about. There will be no spin, no magicians, and we’re pretty sure Rich will keep his pants on -– it’s just a place to find a bit of respite amongst the maelstrom of RSAC. To help us estimate numbers, RSVP to rsvp (at) securosis (dot) com. Share:

I never thought I would say this, but I am leaving Securosis. By the time you read this I will have started a new position with Bank of America. I have been asked to help out with application and cloud security efforts. I have been giving a lot of thought to what I like to do, what makes me happy, and what I want to do with the rest of my career, and I came to the realization it is time for a change. There are aspects of the practice of security which I can never explore with Securosis or DisruptOps. The bank offers many challenges – and operates at a scale – which I have never experienced. That, and I will get to work with a highly talented team already in place. I could not really have written a better job description for myself, so I am jumping at this opportunity. 12 years ago I sat down with Rich to discuss, “What comes next?” when the demise of my former company was imminent. When he asked me to consider joining him, I asked, “What exactly is it you do?” I really didn’t know what analysts actually do, nor how the profession works. Mike will tell you I still don’t, and he is probably right. When I joined, friends and associates asked, “What the hell are you doing?” and said, “That is not what you are good at!”, and told me Securosis would never survive – there was no way we could compete against the likes of Gartner, Forrester, and IDC. But Securosis has been an amazing success. After a few years I think we proved that “No Bullshit” research, and our open Totally Transparent Research model, both work. We have been very lucky to have found such a workable niche, and had so much fun filling it. But more than anything I am very thankful for being able to work with Mike and Rich over the last decade. I simply could not ask for better business partners. Both are smart to the point of being prescient, tremendously supportive, and have an intuitive grasp of the security industry. And when we get it wrong – which have have done more than we like to admit – we learn from our mistakes and move on. I recently read a quote from Chuck Akre: “Good judgement comes from experience, and experience comes from bad judgement.” We always wondered how three guys with big egos and aspiration could coexist, but I think our ability to work as a team has been due in large part to learning from each other, learning from our mistakes, and constantly trying to improve the business. And we have constantly pushed this business to improve and move in new directions: from pure research, to training, to the research Nexus, to cloud security, security consulting, investment due diligence, and eventually launching DisruptOps. Each and every change was to address something in the industry we felt was not being served. Even the Disaster Recovery Breakfast stems from that ethos – another way we wanted to do things differently. And we have gained a lot of – ahem – experience along the way. It has been one hell of a ride! Thank you, Rich and Mike. &nbsp A hearty thank you to Chris Pepper for enduring my writing style and lack of grammar all these years. Early on one reader went so far as to compare my writing to Nazi atrocities against the English language, which was uncalled for, but perhaps not so far off the mark. Chris has helped me get a lot better, and for that I am very grateful. Finally, I wanted to thank all the readers of the posts, Friday Summaries, Incites, and research papers we have produced over the years. Some 15,000 blog posts, hundreds of research papers, and more webcasts than I can count. The support of the security community has made this work so rewarding. Thank you all for participating and helping us make Securosis a success. -Adrian Share:

Today we are launching our 2019 updated research paper from our recent series, Understanding and Selecting RASP (Runtime Application Self-Protection). RASP was part of the discussion on application security in just about every one of the hundreds of calls we have taken, and it’s clear that there is a lot of interest – and confusion – on the subject, so it was time to publish a new take on this category. And we would like to heartily thank you to Contrast Security for licensing this content. Without this type of support we could not bring this level of research to you, both free of charge and without requiring registration. We think this research paper will help developers and security professionals who are tackling application security from within understand what other security measures are at their disposal to protect application stacks from attack. And to be honest we were surprised by the volume of questions being asked. Each team was either considering RASP, or already engaged in a proof-of-concept with a RASP vendor. This was typically in response to difficulties with existing Web Application Firewalls (WAF) as those platforms have not fared well as development has gotten more agile. Since 2017 we have engaged in over 250 additional conversations on what has turned into a ‘DevSecOps’ phenomena, with both security and development groups asking about RASP, how it deploys and the realistic benefits it might provide. And make no mistake, it was not just IT security asking about WAF replacements, but security and development – facing a mountain of ‘technical debt’ with security defects – asking about monitoring/blocking malicious requests while in production. In this paper we cover what RASP is, how it works, use cases and how to differentiate one RASP product from another. And we address the perspectives and specific concerns of IT, application security and developer audiences. Again, thank you to Contrast Security for licensing this research. You can download from the attachment link below, or from the research library. And you can tune into our joint webcast on November 19 by registering here: Evaluating RASP Platforms. Understanding_RASP_2019_Final2.pdf Share:

As we mentioned earlier, DevOps is not all about tools and technology – much of its success lies in how people work within the model. We have already gone into great detail about tools and process, and we approached much of this content from the perspective of security practitioners getting onboard with DevOps. This paper is geared toward helping security folks, so here we outline their role in a DevOps environment. We hope to help you work with other teams and reduce friction. And while we deliberately called this paper “Enterprise DevSecOps”, please keep in mind that your development and IT counterparts likely think there is no such thing. To them security becomes part of the operational process of integrating and delivering code. We call security out as a separate thing because, even woven into the DevOps framework, it’s substantially more difficult for security personnel to fit in. You need to consider how you can improve delivery of secure code without waste and without introducing bottlenecks in a development process you may not be intimately familiar with. The good news is that security fits nicely within a DevOps model, but you need to tailor things to work within your organization’s automation and orchestration to be successful. The Security Pro’s Responsibilities Learn the DevOps model: We have not even touched on theory or practice of DevOps in this series. There is a lot for you to learn on base concepts and practices. To work in a DevOps environment you need to understand what it is and how it works. You need to understand cultural and philosophical changes as well as how they effect process, tooling, and priorities. You need to understand your organization’s approach to optimally integrate security tooling and metrics. Once you understand the mechanics of the development team, you’ll have a better idea of how to work with them in their process. Learn how to be agile: Your participation in a DevOps team means you need to fit into DevOps – not the other way around. The goal of DevOps is fast, faster, fastest: small iterative changes which offer quick feedback, ultimately reducing Work In Progress (WIP). Small, iterative changes to improve security fit this model. Prioritize things which accelerate delivery of secure software over delivery of new features – keeping in mind that it is a huge undertaking to change culture away from “feature first”. You need to adjust requirements and recommendations so they fit into the process, often simplified into small steps, with enough information for tasks to be both automated and monitored. You can recommend manual code reviews or fuzz testing, so long as you understand where they fit within the process, and what can – and cannot – gate releases. Educate: Our experience shows that one of the best ways to bring a development team up to speed in security is training: in-house explanations or demonstrations, third-party experts to help with application security or threat modeling, eLearning, or various commercial courses. The historical downside has been cost, with many classes costing thousands of dollars. You’ll need to evaluate how best to use your resources – the answer typically includes some eLearning for all employees, and select people attending classes and then teaching peers. On-site experts can be expensive, but an entire group can participate in training. Grow your own support: There is simply no way for security teams to scale out their knowledge without help. This does not mean hundreds of new security employees – it means deputizing developers to help with product security. Security teams are typically small and outnumbered by developers 100:1. What’s more, security people are not present at most development meetings, so they lack visibility into day-to-day agile activities. To help extend the reach of the security team, see if you can get someone on each development team – a “security champion” – to act as a security advocate. This helps not only extend the security team’s reach, but also expand security awareness. Help DevOps team understand threats: Most developers don’t fully grasp how attackers approach systems, or what it means when a code or SQL injection attack is possible. The depth and breadth of security threats is outside their experience and most firms do not teach threat modeling. The OWASP Top Ten is a good guide to the types of code deficiencies which plague development teams, but you should map these threats back to real-world examples, show the damage that a SQL injection attack can cause, and explain how a Heartbleed type vulnerability can completely expose customer credentials. You can these real-world use cases as “shock and awe” to help developers “get it”. Advise on remediation practices: Your security program is inadequate if it simply says to “encrypt data” or “install WAF”. All too often, developers and IT have a singular idea of what constitutes security, centered on a single tool they want to set and forget. Help build out the elements of the security program, including both in-code enhancements and supporting tools. Teach how those each help to address specific threats, and offer help with deployment and policy setup. In the past, firms used to produce code style guides to teach younger developers what properly written code looked like. Typically these are not online. Consider a wiki page for security coding practices and other reference materials which are easily discovered and readable by people without a security background. Help evaluate security tools: It is unusual for people outside security to fully understand what security tools do or how they work. So you can help in two ways; first, help developers select tools. Misconceptions are rampant, and not just because vendors over-promise. Additionally it is uncommon for developers to evaluate code scanners, activity monitors, or even patch management system effectiveness. In your role as advisor it is your responsibility to help DevOps understand what the tools can provide and what fits within your shared testing framework. Sure, you might not be able to evaluate the quality of the API, but

In this section we show you how to weave security into the fabric of your DevOps automation framework. We are going to address the questions “We want to integrate security testing into the development pipeline, and are going to start with static analysis. How do we do this?”, “We understand “shift left”, but are the tools effective?” and “What tools do you recommend we start with, and how do we integrate them?”. As DevOps encourages testing in all phases of development and deployment, we will discuss what a build pipeline looks like, and the tooling appropriate for stage. The security tests typically sit side by side with functional and regression tests your quality assurance teams has likely already deployed. And beyond those typical post-build testing points, you can include testing on a developer’s desktop prior to check-in, in the code repositories before and after builds, and in pre-deployment staging areas. Build Process During a few of the calls we had, several of the senior security executives did not know what constituted a build process. This is not a condemnation as many people in security have not participated in software production and delivery, so we want to outline the process and the terminology used by developers. If you’re already familiar with this, skip forward to ‘Building s Security Toolchain’. Most of you reading this will be familiar with “nightly builds”, where all code checked in the previous day is compiled overnight. And you’re just as familiar with the morning ritual of sipping coffee while you read through the logs to see if the build failed and why. Most development teams have been doing this for a decade or more. The automated build is the first of many steps that companies go through on their way toward full automation of the processes that support code development. Over the last several years we have mashed our ‘foot to the floor’, leveraging more and more automation to accelerate the pace of software delivery. The path to DevOps is typically taken in two phases: first with continuous integration, which manages the building and testing of code; then continuous deployment, which assembles the entire application stack into an executable environment. And at the same time, there are continuous improvements to all facets of the process, making it easier, faster and more reliable. It takes a lot of work to get here, and the scripts and templates used often take months just to build out the basics, and years to mature them into reliable software delivery infrastructure. Continuous Integration The essence of Continuous Integration (CI) is developers regularly check-in small iterative code advances. For most teams this involves many updates to a shared source code repository, and one or more builds each day. The key is smaller, simpler additions, where we can more easily and quickly find code defects. These are essentially Agile concepts, implemented in processes which drive code, rather than processes that drive people (such as scrums and sprints). The definition of CI has evolved over the last decade, but in a DevOps context CI implies that code is not only built and integrated with supporting libraries, but also automatically dispatched for testing. Additionally, DevOps CI implies that code modifications are not applied to branches, but directly into the main body of the code, reducing the complexity and integration nightmares that can plague development teams. This sounds simple, but in practice it requires considerable supporting infrastructure. Builds must be fully scripted, and the build process occurs as code changes are made. With each successful build the application stack is bundled and passed along for testing. Test code is built before unit, functional, regression, and security testing, and checked into repositories and part of the automated process. Tests commence automatically whenever a new application bundle is available, but it means the new tests are applied with each new build as well. It also means that before tests can be launched test systems must be automatically provisioned, configured, and seeded with the necessary data. Automation scripts must provide monitoring for each part of the process, and communication of success or failure back to Development and Operations teams as events occur. The creation of the scripts and tools to make all this possible requires Operations, Testing and Development teams to work closely together. The following graphic shows an automated build pipeline, including security test points, for containers. Again, this level of orchestration does not happen overnight, rather an evolutionary process that take months to establish the basics and years to mature. But that is the very essence of continuous improvement. Continuous Deployment Continuous Deployment looks very similar to CI, but focuses on the releasing software to end users rather than building it. It involves similar packaging, testing, and monitoring tasks, with some additional wrinkles. Upon a successful completion of a build cycle, the results feed the Continuous Deployment (CD) process. CD takes another giant step forward for automation and resiliency but automating the release management, provisioning and final configuration for the application stack, then launches the new application code. When we talk about CD, there are two ways people embrace this concept. Some teams simply launch the new version of their application into an existing production environment. The CD process automates the application layer, but not the server, data or network layers. We find this common with on-premise applications and in private cloud deployments, and some public cloud deployments still use this model as well. A large percentage of the security teams we spoke with are genuinely scared of Continuous Deployment. They state “How can you possibly allow code to go live without checks and oversight!”, missing the point that the code does not go live until all of the security tests have been passed. And some CI pipelines contain manual inspection points for some tests. In our experience, CD means more reliable and more secure releases. CD addresses dozens of issues that plague code deployment — particularly in the areas of error-prone manual changes, and discrepancies in

This post is intended to help security folks create an outline or structure for an application security program. We are going to answer such common questions as “How do we start building out an application security strategy?”, “How do I start incorporating DevSecOps?” and “What application security standards should I follow?”. I will discuss the Software Development Lifecycle (SDLC), introduce security items to consider as you put your plan in place, and reference some application security standards for use as guideposts for what to protect against. This post will help your strategy; the next one will cover tactical tool selection. Security Planning and your SDLC A Secure Software Development Lifecycle (S-SDLC) essentially describes how security fits into the different phases of a Software Development Lifecycle. We will look at each phase in an SDLC and discuss which security tools and techniques are appropriate. Note that an S-SDLC is typically drawn as a waterfall development process, with different phases in a linear progression, but that’s really just for clearer depiction – the actual SDLC which is being secured is as likely to be Agile, Extreme, or Spiral as Waterfall. There are good reasons to base an S-SDLC on a more modern SDLC; but the architecture, design, development, testing, and deployment phases all map well to development stages in any development process. They provide a good jumping-off point to adapt current models and processes into a DevOps framework. As in our previous post, we want you to think of the S-SDLC as a framework for building your security program, not a full step-by-step process. We recognize this is a departure from what is taught in classrooms and wikis, but it is better for planning security in each phase. Define and Architect Reference Security Architectures: Reference security architectures exist for different types of applications and services, including web applications, data processing applications, identity and access management services for applications, stream/event processing, messaging, and so on. The architectures are even more effective in public cloud environments, Kubernetes clusters, and service mesh environments – where we can tightly control via policy how each application operates and communicates. With cloud services we recommend you leverage service provider guidelines on deployment security, and while they may not call them ‘reference security architectures’ they do offer them. Educate yourself on the application platforms and ask software designers and architects which methods they employ. Do not be surprised if for legacy applications they give you a blank stare. But new applications should include plans for process isolation, segregation, and data security, with a full IAM model to promote segregation of duties and data access control. Operational Standards: Work with your development teams to define minimal security testing requirements, and critical and high priority issues. You will need to negotiate which security flaws will fail a build, and define the process in advance. You will probably need an agreement on timeframes for fixing issues, and some type of virtual patching to address hard-to-fix application security issues. You need to define these things up front and make sure your development and IT partners agree. Security Requirements: Just as with minimum functional tests which must run prior to code acceptance, you’ll have a set of security tests you run prior to deployment. These may be an agreed upon battery of unit tests for specific threats your which team writes. Or you may require all OWASP Top Ten vulnerabilities be mitigated in code or supporting products, mapping each threat to a specific security control for all web applications. Regardless of what you choose, your baseline requirements should account for new functionality as well as old. A growing body of tests requires more resources for validation and can slow your test and deployment cycle over time, so you have some decisions to make regarding which tests can block a release vs. what you scan for post-production. Monitoring and Metrics: If you will make small iterative improvements with each release, what needs fixing? Which code modules are problematic for security? What is working and how can you prove it? Metrics are key to answering all these questions. You need to think about what data you want to collect and build it into your CI:CD and production environments to measure how your scripts and tests perform. That means you need to engage developers and IT personnel in collecting data. You’ll continually evolve collection and use of metrics, but plan for basic collection and dissemination of data from the get-go. Design Security Design Principles: Some application security design and operational principles offer significant security improvement. Things like ephemeral instances to aid patching and reduce attacker persistence, immutable services to remove attack surface, configuration management to ensure servers and applications are properly set up, templating environments for consistent cloud deployment, automated patching, segregation of duties by locking development and QA personnel out of production resources, and so on. Just as important, these approaches are key to DevOps because they make delivery and management of software faster and easier. It sounds like a lot to tackle, but IT and development pitch in as it makes their lives easier too. Secure the Deployment Pipeline: With both development and production environments more locked down, development and test servers become more attractive targets. Traditionally these environments run with little or no security. But the need for secure source code management, build servers, and deployment pipelines is growing. And as CI/CD pipelines offer an automated pathway into production, you’ll need at minimum stricter access controls for these systems – particularly build servers and code repositories. And given scripts running continuously in the background with minimal human oversight, you’ll need additional monitoring to catch errors and misuse. Many of the tools offer good security, with digital fingerprinting, 2FA, logging, role-based access control, and other security features. When deployed in cloud environments, where the management plane allows control of your entire environment, great care must be taken with access controls and segregation of duties. Threat Modeling: Threat modeling remains one of the most productive exercises