Research

Something You Probably Should Include When Building Your Next Threat Models We are working on our threat modeling here at DisruptOps and I decided to refresh my knowledge of different approaches. One thing that quickly stood out is that nearly none of the threat modeling documentation or tools I’ve seen cover the CI/CD pipeline. Read the full post at DisruptOps Share:

Three of the Most Crucial Sections of the DevSecOps Roadmap As I mentioned in the (DevSec)Ops vs. Dev(SecOps) post, we’ve been traveling around to a couple of DevOpsDays conferences doing the Quick and Dirty DevSecOps talk. One of the things I tend to start with early in the talk is that like DevOps, DevSecOps is not a product. Or something you can deploy and forget. It’s a cultural change. It’s a process. It’s a journey. Read the full post at DisruptOps Share:

This is the second post in our series on Protecting What Matters: Introducing Data Guardrails and Behavioral Analytics. Our first post, Introducing Data Guardrails and Behavioral Analytics: Understand the Mission, introduced the concepts and outlined the major categories of insider risk. This post defines the concepts. Data security has long been the most challenging domain of information security, despite being the centerpiece of our entire practice. We only call it “data security” because “information security” was already taken. Data security must not impede use of the data itself. By contrast it’s easy to protect archival data (encrypt it and lock the keys up in a safe). But protecting unstructured data in active use by our organizations? Not so easy. That’s why we started this research by focusing on insider risks, including external attackers leveraging insider access. Recognizing someone performing an authorized action, but with malicious intent, is a nuance lost on most security tools. How Data Guardrails and Data Behavioral Analytics are Different Both data guardrails and data behavioral analytics strive to improve data security by combining content knowledge (classification) with context and usage. Data guardrails leverage this knowledge in deterministic models and processes to minimize the friction of security while still improving defenses. For example, if a user attempts to make a file in a sensitive repository public, a guardrail could require them to record a justification and then send a notification to Security to approve the request. Guardrails are rule sets that keep users “within the lines” of authorized activity, based on what they are doing. Data behavioral analytics extends the analysis to include current and historical activity, and uses tools such as artificial intelligence/machine learning and social graphs to identify unusual patterns which bypass other data security controls. Analytics reduces these gaps by looking not only at content and simple context (as DLP might), but also adding in history of how that data, and data like it, has been used within the current context. A simple example is a user accessing an unusual volume of data in a short period, which could indicate malicious intent or a compromised account. A more complicated situation would identify sensitive intellectual property on an accounting team device, even though they do not need to collaborate with the engineering team. This higher order decision making requires an understanding of data usage and connections within your environment. Central to these concepts is the reality of distributed data actively used widely by many employees. Security can’t effectively lock everything down with strict rules covering every use case without fundamentally breaking business processes. But with integrated views of data and its intersection with users, we can build data guardrails and informed data behavioral analytical models, to identify and reduce misuse without negatively impacting legitimate activity. Data guardrails enforce predictable rules aligned with authorized business processes, while data behavioral analytics look for edge cases and less predictable anomalies. How Data Guardrails and Data Behavioral Analytics Work The easiest way to understand the difference between data guardrails and data behavioral analytics is that guardrails rely on pre-built deterministic rules (which can be as simple as “if this then that”), while analytics rely on AI, machine learning, and other heuristic technologies which look at patterns and deviations. To be effective both rely on the following foundational capabilities: A centralized view of data. Both approaches assume a broad understanding of data and usage – without a central view you can’t build the rules or models. Access to data context. Context includes multiple characteristics including location, size, data type (if available), tags, who has access, who created the data, and all available metadata. Access to user context, including privileges (entitlements), groups, roles, business unit, etc. The ability to monitor activity and enforce rules. Guardrails, by nature, are preventative controls which require enforcement capabilities. Data behavioral analytics can be used only for detection, but are far more effective at preventing data loss if they can block actions. The two technologies then work differently while reinforcing each other: Data guardrails are sets of rules which look for specific deviations from policy, then take action to restore compliance. To expand our earlier example: A user shares a file located in cloud storage publicly. Let’s assume the user has the proper privileges to make files public. The file is in a cloud service so we also assume centralized monitoring/visibility, as well as the capability to enforce rules on that file. The file is located in an engineering team’s repository (directory) for new plans and projects. Even without tagging, this location alone indicates a potentially sensitive file. The system sees the request to make the file public, but because of the context (location or tag), it prompts the user to enter a justification to allow the action, which gets logged for the security team to review. Alternatively, the guardrail could require approval from a manager before allowing the action. Guardrails are not blockers because the user can still share the file. Prompting for user justification both prevents mistakes and loops in security review for accountability, allowing the business to move fast while minimizing risk. You could also look for large file movements based on pre-determined thresholds. A guardrail would only kick in if the policy thresholds are violated, and then use enforcement actions aligned with business processes (such as approvals and notifications) rather than simply blocking activity and calling in the security goons. Data behavioral analytics use historical information and activity (typically with training sets of known-good and known-bad activity), which produce artificial intelligence models to identify anomalies. We don’t want to be too narrow in our description, because there are a wide variety of approaches to building models. Historical activity, ongoing monitoring, and ongoing modeling are all essential – no matter the mathematical details. By definition we focus on the behavior of data as the core of these models, rather than user activity; this represents a subtle but critical distinction from User Behavioral Analytics (UBA). UBA tracks activity on a per-user basis. Data behavioral analytics (the acronym DBA is already taken, so we’ll

As we begin our series on Multi-cloud logging, we start with reasons some traditional logging approaches don’t work. I don’t like to start with a negative tone, but we need to point out some challenges and pitfalls which often beset firms on first migration to cloud. That, and it helps frame our other recommendations later in this series. Let’s take a look at some common issues by category. Tooling Scale & Performance: Most log management and SIEM platforms were designed and first sold before anyone had heard of clouds, Kafka, or containers. They were architected for ‘hub-and-spoke’ deployments on flat networks, when ‘Scalability’ meant running on a bigger server. This is important because the infrastructure we now monitor is agile – designed to auto-scale up when we need processing power, and back down to reduce costs. The ability to scale up, down, and out is essential to the cloud, but often missing from older logging products which require manual setup, lacking full API enablement and auto-scale capability. Data Sources: We mentioned in our introduction that some common network log sources are unavailable in the cloud. Contrawise, as automation and orchestration of cloud resources are via API calls, API logs become an important source. Data formats for these new log sources may change, as do the indicators used to group events or users within logs. For example servers in auto-scale groups may share a common IP address. But functions and other ‘serverless’ infrastructure are ephemeral, making it impossible to differentiate one instance from the next this way. So your tools need to ingest new types of logs, faster, and change their threat detection methods by source. Identity: Understanding who did what requires understandings identity. An identity may be a person, service, or device. Regardless, the need to map it, and perhaps correlate it across sources, becomes even more important in hybrid and multi-cloud environments Volume: When SIEM first began making the rounds, there were only so many security tools and they were pumping out only so many logs. Between new security niches and new regulations, the array of log sources sending unprecedented amounts of logs to collect and analyze grows every year. Moving from traditional AV to EPP, for example, brings with it a huge log volume increase. Add in EDR logs and you’re really into some serious volumes. On the server side, moving from network and server logs to add application layer and container logs brings a non-trivial increase in volume. There are only so many tools designed to handle modern event rates (X billio events per day) and volumes (Y terabytes per day) without buckling under the load, and more importantly, there are only so many people who know how to deploy and operate them in production. While storage is plentiful and cheap in the cloud, you still need to get those logs to the desired storage from various on-premise and cloud sources – perhaps across IaaS, PaaS, and SaaS. If you think that’s easy call your SaaS vendor and ask how to export all your logs from their cloud into your preferred log store (S3/ADLS/GCS/etc.). That old saw from Silicon Valley, “But does it scale?” is funny but really applies in some cases. Bandwidth: While we’re on the topic of ridiculous volumes, let’s discuss bandwidth. Network bandwidth and transport layer security between on-premise and cloud and inter-cloud is non-trivial. There are financial costs, as well as engineering and operational considerations. If you don’t believe me ask your AWS or Azure sales person how to move, say, 10 terabytes a day between those two. In some cases architecture only allows a certain amount of bandwidth for log movement and transport, so consider this when planning migrations and add-ons. Structure Multi-account Multi-cloud Architectures: Cloud security facilitates things like micro-segmentation, multi-account strategies, closing down all unnecessary network access, and even running different workloads in different cloud environments. This sort of segmentation makes it much more difficult for attackers to pivot if they gain a foothold. It also means you will need to consider which cloud native logs are available, what you need to supplement with other tooling, and how you will stitch all these sources together. Expecting to dump all your events into a syslog style service and let it percolate back on-premise is unrealistic. You need new architectures for log capture, filtering, and analysis. Storage is the easy part. Monitoring “up the Stack”: As cloud providers manage infrastructure, and possibly applications as well, your threat detection focus must shift from networks to applications. This is both because you lack visibility into network operations, but also because cloud network deployments are generally more secure, prompting attackers to shift focus. Even if you’re used to monitoring the app layer from a security perspective, for example with a big WAF in front of your on-premise servers, do you know whether you vendor has a viable cloud offering? If you’re lucky enough to have one that works in both places, and you can deploy in cloud as well, answer this (before you initiate the project): Where will those logs go, and how will you get them there? Storage vs. Ingestion: Data storage in cloud services, especially object storage, is so cheap it is practically free. And long-term data archival cloud services offer huge cost advantages over older on-premise solutions. In essence we are encouraged to store more. But while storage is cheap, it’s not always cheap to ingest more data into the cloud because some logging and analytics services charge based upon volume (gigabytes) and event rates (number of events) ingested into the tool/service/platform. Example are Splunk, Azure Eventhubs, AWS Kinesis, and Google Stackdriver. Many log sources for the cloud are verbose – both number of events and amount of data generated from each. So you will need to architect your solution to be economically efficient, as well as negotiate with your vendors over ingestion of noisy sources such as DNS and proxies, for example. A brief side note on ‘closed’ logging pipelines: Some vendors want to own your logging pipeline on top of your analytics toolset. This may

A Security Pro’s Cloud Automation Journey Catch me at a conference and the odds are you will overhear my saying “cloud security starts with architecture and ends with automation.” I quickly follow with how important it is to adopt a cloud native mindset, even when you’re bogged down with the realities of an ugly lift and shift before the data center contract ends and you turn the lights off. While that’s a nice quip, it doesn’t really capture anything about how I went from a meat and potatoes (firewall and patch management) kind of security pro to an architecture and automation and automation cloud native. Rather than preaching from the mount, I find it more useful to describe my personal journey and my technical realizations along the way. If you’re a security pro, or someone trying to up-skill a security pro for cloud, odds are you will end up on a very similar path. Read the full post at DisruptOps Share:

I have concluded that nobody is using Database Activity Monitoring (DAM) in public Infrastructure or Platform as a Service. I never see it in any of the cloud migrations we assist with. Clients don’t ask about how to deploy it or if they need to close this gap. I do not hear stories, good or bad, about its usage. Not that DAM cannot be used in the cloud, but it is not. There are certainly some reasons firms invest security time and resources elsewhere. What comes to mind are the following: PaaS and use of Relational: There are a couple trends which I think come into play. First, while user installed and managed relational databases do happen, there is a definite trend towards adopting RDBMS as a Service. If customers do install their own relational platform, it’s MySQL or MariaDB, for which (so far as I know) there are few monitoring options. Second, for most new software projects, a relational database is a much less likely choice to back applications – more often it’s a NoSQL platform like Mongo (self-managed) or something like Dynamo. This has reduced the total relational footprint. CI:CD: Automated build and security test pipelines – we see a lot more application and database security testing in development and quality assurance phases, prior to production deployment. Many potential code vulnerabilities and common SQL injection attacks are being spotted and addressed prior to applications being deployed. And there may not be a lot of reconfiguration in production if your installation is defined in software. Network Security: Between segmentation, firewalls/security groups, and port management you can really lock down the (virtual) network so only the application can talk to the database. Difficult for anyone to end-run around if properly set up. Database Ownership: Some people cling to the misconception that the database is owned and operated by the cloud provider, so they will take care of database security. Yes, the vendor handles lots of configuration security and patching for you. Certainly much of the value of a DAM platform, namely security assessment and detection of old database versions, is handled elsewhere. Permission misuse is harder. Most IaaS clouds offer dynamic policy-driven IAM. You can set very fine-grained access controls on database access, so you can block many types of ad hoc and potentially malicious queries. Maybe none of these reasons? Maybe all the above? I don’t really know. Regardless, DAM has not moved to the cloud. The lack of interest does not provide any real insights as to why, but it is very clear. I do still want some of DAM’s monitoring functions for cloud migrations, specifically looking for SQL injection attacks – which are still your issue to deal with – as well as looking for credential misuse, such as detecting too much data transfer or scraping. Cloud providers log API access to the database installation, and there are cloud-native ways to perform assessment. But on the monitoring side there are few other options for watching SQL queries. Share:

Disrupt:Ops: Consolidating Config Guardrails with Aggregators In Quick and Dirty: Building an S3 guardrail with Config we highlighted that one of the big problems with Config is you need to build it in all regions of all accounts separately. Now your best bet to make that manageable is to use infrastructure as code tools like CloudFormation to replicate your settings across environments. We have a lot more to say on scaling out baseline security and operations settings, but for this post I want to highlight how to aggregate Config into a unified dashboard. Read the full post at DisruptOps Share:

I had been planning to post on the recent announcement of the planned merger between Hortonworks and Cloudera, as there are a number of trends I’ve been witnessing with the adoption of Hadoop clusters, and this merger reflects them in a nutshell. But catching up on my reading I ran across Mathew Lodge’s recent article in VentureBeat titled Cloudera and Hortonworks merger means Hadoop’s influence is declining. It’s a really good post. I can confirm we see the same lack of interest in deployment of Hadoop to the cloud, the same use of S3 as a storage medium when Hadoop is used atop Infrasrtucture as a Service (IaaS), and the same developer-driven selection of whatever platform is easiest to use and deploy on. All in all it’s an article I wish I’d written, as he did a great job capturing most of the areas I wanted to cover. And there are some humorous bits like “Ironically, there has been no Cloud Era for Cloudera.” Check it out – it’s worth your time. But there are a couple other areas I still want to cover. It is rare to see someone install Hadoop into a public IaaS account. Customers (now) choose a cloud native variant and let the vendor handle all the patching and hide much of the infrastructure pieces from them. And they gain the option of spinning down the cluster when not in use, making it much more efficient. Couple that with all the work to set up Hadoop yourself, and it’s an easy decision. I was somewhat surprised to learn that things like AWS’s Elastic Map Reduce (EMR) are not always chosen as repository, but Dynamo is surprisingly popular – which makes sense, given its powerful query features, indexing, and ability to offer the best of relational and big data capabilities. Most public IaaS vendors offer so many database variants that it is easy to mix and match multiple variants to support applications, further reducing demand for classic Hadoop installations. One area continuing to drive Hadoop adoption is on-premise data collection and data lakes for logs. The most cited driver is the need to keep Splunk costs under control. It takes effort to divert some content to Hadoop instead of sending everything to the Splunk collectors – but data can be collected and held at drastically lower cost. And you need not sacrifice analytics. For organizations collecting every log entry, this is a win. We also see Hadoop adopted by Security Operations Centers, running side by side with other platforms. Part of the need is to fill gaps around what their SIEM keeps, part is to keep costs down, and part is to easily support deployment of custom security intelligence applications by non-developers. Another aspect not covered in any of the articles I have found so far is that Cloudera and Hortonworks both have deep catalogs of security capabilities. Together they are dominant. As firms use large “data lakes” to hold all sorts of sensitive data inside Hadoop, this will be a win for firms running Hadoop in-house. Identity management, encryption, monitoring, and a whole bunch of other great stuff. Big data is not the security issue it was 5 years ago. Hortonworks and Cloudera have a lot to do with that; their combined capabilities and enterprise deployment experience make them a powerful choice to help firms manage and maintain existing infrastructure. That is all my way of saving that some of their negative press is unwarranted, given the profitable avenues ahead. The idea that growth in the Hadoop segment appears to have been slowing is not new. AWS has been the largest seller of Hadoop-based data platforms, by revenue and by customer, for several years. The cloud is genuinely an existential threat to all the commercial Hadoop vendors – and comparable big data databases – if they continue to sell in the same way. The recent acceleration of cloud adoption simply makes it more apparent that Cloudera and Hortonworks are competing for a shrinking share of IT budgets. But it makes sense to band together and make the most of their expertise in enterprise Hadoop deployments, and should help with tooling and management software for cloud migrations. If Kubernetes is any indication, there are huge areas for improvement in tooling and services beyond what cloud vendors provide. Share:

Logging and monitoring for cloud infrastructure has become the top topic we are asked about lately. Even general conversations about moving applications to the cloud always seem to end with clients asking how to ‘do’ logging and monitoring of cloud infrastructure. Logs are key to security and compliance, and moving into cloud services – where you do not actually control the infrastructure – makes logs even more important for operations, risk, and security teams. But these questions make perfect sense – logging in and across cloud infrastructure is complicated, offering technical challenges and huge potential cost overruns if implemented poorly. The road to cloud is littered with the charred remains of many who have attempted to create multi-cloud logging for their respective employers. But cloud services are very different – structurally and operationally – than on-premise systems. The data is different; you do not necessarily have the same event sources, and the data is often different or incomplete, so existing reports and analytics may not work the same. Cloud services are ephemeral so you can’t count on a server “being there” when you go looking for it, and IP addresses are unreliable identifiers. Networks may appear to behave the same, but they are software defined, so you cannot tap into them the same way as on-premise, nor make sense of the packets even if you could. How you detect and respond to attacks differs, leveraging automation to be as agile as your infrastructure. Some logs capture every API call; while their granularity of information is great, the volume of information is substantial. And finally, the skills gap of people who understand cloud is absent at many companies, so they ‘lift and shift’ what they do today into their cloud service, and are then forced to refactor the deployment in the future. One aspect that surprised all of us here at Securosis is the adoption of multi-cloud; we do not simply mean some Software as a Service (SaaS) along with a single Infrastructure as a Service (IaaS) provider – instead firms are choosing multiple IaaS vendors and deploying different applications to each. Sometimes this is a “best of breed” approach, but far more often the selection of multiple vendors is driven by fear of getting locked in with a single vendor. This makes logging and monitoring even more difficult, as collection across IaaS providers and on-premise all vary in capabilities, events, and integration points. Further complicating the matter is the fact that existing Security Information and Event Management (SIEM) vendors, as well as some security analytics vendors, are behind the cloud adoption curve. Some because their cloud deployment models are no different than what they offer for on-premise, making integration with cloud services awkward. Some because their solutions rely on traditional network approaches which don’t work with software defined networks. Still others employ pricing models which, when hooked into highly verbose cloud log sources, cost customers small fortunes. We will demonstrate some of these pricing models later in this paper. Here are some common questions: What data or logs do I need? Server/network/container/app/API/storage/etc.? How do I get them turned on? How do I move them off the sources? How do I get data back to my SIEM? Can my existing SIEM handle these logs, in terms of both different schema and volume & rate? Should I use log aggregators and send everything back to my analytics platform? At what point during my transition to cloud does this change? How do I capture packets and where do I put them? These questions, and many others, are telling because they come from trying to fit cloud events into existing/on-premise tools and processes. It’s not that they are wrong, but they highlight an effort to map new data into old and familiar systems. Instead you need to rethink your logging and monitoring approach. The questions firms should be asking include: What should my logging architecture look like now and how should it change? How do I handle multiple accounts across multiple providers? What cloud native sources should I leverage? How do I keep my costs manageable? Storage can be incredibly cheap and plentiful in the cloud, but what is the pricing model for various services which ingest and analyze the data I’m sending them? What should I send to my existing data analytics tools? My SIEM? How do I adjust what I monitor for cloud security? Batch or real-time streams? Or both? How do I adjust analytics for cloud? You need to take a fresh look at logging and monitoring, and adapt both IT and security workflows to fit cloud services – especially if you’re transitioning to cloud from an on-premise environment and will be running a hybrid environment during the transition… which may be several years from initial project kick-off. Today we launch a new series on Building a Multi-cloud Logging Strategy. Over the next few weeks, Gal Shpantzer and I (Adrian Lane) will dig into the following topics to discuss what we see when helping firms migrate to cloud. And there is a lot to cover. Our tentative outline is as follows: Barriers to Success: This post will discuss some reasons traditional approaches do not work, and areas where you might lack visibility. Cloud Logging Architectures: We discuss anti-patterns and more productive approaches to logging. We will offer recommendations on reference architectures to help with multi-cloud, as well as centralized management. Native Logging Features: We’ll discuss what sorts of logs you can expect to receive from the various types of cloud services, what you may not receive in a shared responsibility service, the different data sources firms have come to expect, and how to get them. We will also provide practical notes on logging in GCP, Azure, and AWS. We will help you navigate their native offerings, as well as the capabilities of PaaS/SaaS vendors. BYO Logging: Where and how to fill gaps with third-party tools, or building them into applications and service you deploy in the cloud. Cloud or On-premise Management? We will discuss tradeoffs between moving log management into the cloud, keeping these activities on-premise, and using a

Disrupt:Ops: Quick and Dirty: Building an S3 Guardrail with Config In How S3 Buckets Become Public, and the Fastest Way to Find Yours we reviewed the myriad ways S3 buckets become public and where to look for them. Today I’ll show the easiest way to continuously monitor for public buckets using AWS Config. The good news is this is pretty easy to set up; the bad news is you need to configure it separately in every region in every account. Read the full post at DisruptOps Share: