Research

Let’s resume our discussion of endpoint attack prevention approaches with the options available once an attack actually begins to execute, or once it has already executed on a device. During Execution (Runtime) Once malicious code begin to execute, prevention of compromise requires recognizing bad behavior and blocking it before the attack can take control of the device. The first decision point is whether you want the protection to run in user mode (within the operating system and leveraging operating system protections) or kernel mode (at a lower level on the device, with access to everything – including interactions between the kernel and CPU). Many attacks exploit the operating system and applications which run within the OS, so it’s reasonable to protect in user mode. But you cannot preclude adversaries from attacking the kernel directly, so as so often, the best answer is often both. You need OS and application specific protections, but to comprehensively protect devices you need to monitor and protect the kernel as well. Otherwise you cannot defend against privileged processes and kernel-level rootkits. Exploit prevention: This is a large bucket of many techniques, designed to prevent exploits from compromising devices. Many advanced endpoint products use most (or even all) these techniques, and due to constant innovation by attackers they add new preventions on an ongoing basis. So understand this is a dynamic list.Exploit pathway blocking: This approach is driven by threat research, profiling behaviors observed when malware compromises devices and watching for those patterns in real time. It turns out there are a couple dozen ways to gain control of a machine (of course the actual number is up for debate), and if you make sure none of those patterns scenarios can be completed on a device you have a high level of protection. But be careful to monitor both false positives and resource consumption, because evaluating every function at the kernel level can have unintended consequences, starting with the predictable performance drain. This is a similar approach to HIPS (Host Intrusion Prevention), but detection is focused on device compromise at a much deeper device level. Memory protection: To detect the memory attacks described in our previous post (file-less malware), the memory usage of the operating system and applications need to be profiled; and memory must be monitored for abnormal memory activity which could indicate memory injection, encrypted memory, or hidden modules. Once again, this has driven an emphasis on endpoint threat research because profiling memory usage requires deep understanding of endpoint operating systems and how attackers manipulate devices. Macro protection: To protect against rogue macros, advanced endpoint prevention requires the ability to block unauthorized and potentially malicious macros. Similar to exploit pathway blocking and memory protection, threat research profiles legitimate macro behavior and malicious macros to develop a model for what macros can and should do. Anything that doesn’t fit into this model is blocked. Once again, this technique highlights the importance of threat research to ensure profiles are accurate and current. Script protection: The key to protecting against rogue scripts is to ensure that the logical chain of events makes sense. For instance a browser probably shouldn’t be launching a PowerShell script to execute command-line actions. If a device sees that behavior, block it. Likewise, a profile of legitimate scripting activity can be developed to detect and protect against malicious scripts. Registry protection: To maintain persistence adversaries increasingly store malware within the device registry. To prevent these attacks the registry needs to be profiled and monitored to prevent unauthorized changes, and if necessary to roll back undesired changes. Privilege escalation: At some point during an endpoint attack, the adversary will need to elevate privileges on the device to run the malware. The advanced endpoint agent can look for privilege escalation and new account creation as strong indicators of device compromise. Pros: You cannot really stop advanced exploits without protecting devices against these techniques, so it’s not really a question of whether to include these features or not. It’s about understanding how a vendor develops the models they use to distinguish legitimate behavior from illegitimate. Cons: These preventions require models of appropriate behavior, so false positives are always a concern, which comes down to opportunity cost. Whenever you need to spend time chasing down things that aren’t real issues, you aren’t doing something more useful. Ensuring that any agent provides granularity in terms of what gets blocked versus generating an alert is absolutely critical. Be aware of application impersonation, where a malicious application spoofs a legitimate one to access its privileges. Also consider differences between operating systems, in terms of ability to detect kernel activity or privilege escalation. Isolation: Another common technique is isolation within the operating system to shield critical system resources (such as memory, storage, and networking) from direct access by executables running on the system. This abstraction layer between applications and system services enables monitoring of system calls and blocking of abnormal behavior.Pros: Isolation is a time-honored approach to making sure a problem in one area of the environment doesn’t spread anywhere else. Abstracting operating system services and blocking malicious behavior before it can spread provides resilience to the device and prevents full compromise. Cons: Isolation of operating system functions is very complicated and resource-intensive on the device. This approach requires high-powered devices and considerable testing before rollout, to ensure it doesn’t break applications and impair employee productivity. Endpoint sandbox/emulation: A few years ago network-based malware sandboxes were all the rage. Files coming across ingress networks could be analyzed and unrecognized files would be executed inside the sandbox to see what it did. If a file showed malware characteristics it would be blocked at the perimeter. These devices worked great… until malware writers figured out how to evade them, at which point effectiveness took a hit, although there is still value in this approach and some prevention products detonate any unknown files in a sandbox on the endpoint to look for malicious characteristics. We’ll discuss this in more detail below, including integration with

We discussed specific attacks in our last post, so it’s time to examine approaches which can prevent them. But first let’s look at the general life cycle of an attack. Prevention Timeline As we dig into how to actually prevent the attacks described in the last post, the key principle is to avoid single points of failure, and then to ensure you have resilience so you can respond and restore normal operations as quickly as possible. You want multiple opportunities to block any attack. The most effective way to plan this out is to think about the attack on a timeline. We want an opportunity to prevent damage before execution, as early as possible during execution, and again in the worst case after execution. The earlier you can prevent an attack, the better, of course. In a perfect world you stop every attack before it gets anywhere. But, as we all discover seemingly every day, we don’t always get a chance to stop an attack before it starts. Even so, we still need to minimize damage, prevent data loss, and eliminate any attacker beachheads before they can move deeper into our systems. We focus on making sure you have numerous opportunities to determine whether code on a device is acting maliciously, and then block it. This timeline approach helps us provide failsafes and defense in depth, acknowledging that malware is very sophisticated and combines multiple attack types, which can change depending on the defenses in place on an endpoint. Let’s work through the techniques you can use to prevent attacks at every stage. We will describe each technique, and then go enumerate its pros and cons. Pre-execution The best time to prevent an attack is before it starts, and there are multiple ways to evaluate code about to run on a device to determine whether it’s malicious. Hygiene: This is a catch-all term to indicate proper strong configurations implemented on devices. Many organizations don’t consider these endpoint security controls, but the fact is that if you can block attacks by not leaving vulnerabilities on devices, that is pre-execution prevention.Patching: Keeping devices updated with the most recent patches prevents attackers from taking advantage of known vulnerabilities. Strong configurations: Each device should also be built from a strong configuration which disables unnecessary services and provides the device user with the minimum privilege to perform their job. Host firewall: Each device should have an operational firewall to prevent network attacks, blocking both non-standard protocols and traffic to known bad destinations. Host Intrusion Prevention: The firewall is to ensure unauthorized sites cannot communicate with the device (access control), and HIPS is about looking for attack patterns within the endpoint’s network stack. This is especially important for detecting reconnaissance and lateral movement. Device control: Finally, devices should be configured to disable capabilities such as USB storage to prevent introduction of malicious code via physical mechanisms. Pros: Hygiene is all about reducing device attack surface and removing low-hanging fruit to make things difficult for attackers. If by patching a system you can make their job harder, do that. If by shutting down USB ports you can prevent a social engineer from installing malware on a device via physical media, do that. Cons: Hygiene is a very low bar for protection. Even though you reduce attack surface, adversaries still have plenty of tactics available to compromise devices. Endpoint hygiene is necessary but not sufficient. File signatures: The most traditional endpoint defense involves a blacklist of known malicious file hashes and determining whether any file is on that list before allowing execution on a device. With billions of malicious files in circulation, it’s impractical to store all those file hashes on every device, or to search all those hashes every time a file executes, so integrating with a threat intelligence service to check file hashes which aren’t in the local cache is critical.Pros: Fool me once, shame on you. Fool me twice… File signatures are still used because it’s pathetic to be compromised by something you know is malicious and have seen before. The challenge is to leverage signatures efficiently, given the sheer number of items that need to be on any blacklist. Cons: It’s trivial to change the file hash of a malicious file. So the effectiveness of signature matching is abysmal, which is why every endpoint prevention offering uses additional techniques. Static analysis: Malicious files can have file attributes which indicate they are bad. These attributes include whether a file packer has been used (to change the hash), header details, embedded resources, inconsistent file metadata, etc. Static file analysis examines each file before execution, searching for these indicators. Endpoint prevention vendors typically use machine learning to analyze billions of malware files, searching for attributes which likely indicate malicious files. We will discuss machine learning later in this Buyer’s Guide.Pros: Static analysis is cheap and easy. Each endpoint prevention agent has a set of attributes to look for, and can quickly scan every file for those attributes before execution. Cons: As sophisticated as the machine learning models are which identify attributes likely to indicate a malicious file, this approach can have a high false positive rate. Static analysis is generally a coarse filter, used to determine whether a file warrants further analysis to determine whether it’s malicious. Whitelisting: The last pre-execution approach to mention is whitelisting. This entails assembling a list of all authorized files which can run on a device, and blocking anything not on the list. Malware is inherently unauthorized, so this is a good way to ensure only legitimate software runs.Pros: For devices without much variation in which applications run (such as customer support workstations and kiosks), whitelisting is a very powerful approach and can significantly reduce attack surface. Modern attacks involve downloading additional executables once the device is compromised, so even if a device is initially compromised an attacker should be unable to get additional malware files to run. Some solutions also enlist whitelisting as a supplementary technique to reduce the number of

About a year ago I first heard the dreaded acronym “MVC”. It was during a call about a potential project, and this contact kept namedropping it like Kanye or something – not that I knew what it meant at the time. I kept wondering how Model/View/Controller was so important to their deployment. Eventually I learned it stands for “Minimum Viable Cloud”. I want to take whichever consultant came up with that concept, dip them in chocolate, and toss them into a bear preserve. In the spring. Say around March or April. I’ve been hearing it more frequently since then, and here’s what it means, why I think it is a stupendously terrible idea, and a better alternative. Note that I don’t assume MVC is universally defined or understood – it seems to be more of a catchall term designed to assuage cloud fears while driving big consulting projects. The general consensus seems to be that you predefine and build your cloud environment, then shovel all your projects into it. Typically this is a single account (bye bye blasts radius management), with 1-3 virtual networks (dev/prod/???), and the full architecture built out like a single data center. All the security groups, subnets, and other major structures are predefined. These deployments are more likely to have a bunch of virtual appliance versions of the same tools used on-premise. There is a lot of complex work to set up and isolate subnets and such, some minimal cloud-level IAM and alerting, and a lot of baggage carried over from existing operations. It doesn’t work. Not for long. MVC fundamentally breaks agility and reinforces bad old habits. Even if you try to design a ‘friendlier’ MVC deployemnt, it doesn’t scale and doesn’t offer the security benefits of a cloud-native approach. With MVC everything you deploy has to fit an established pattern. Instead of fitting security to the project you are forced to fit the project to the security. Don’t interpret that statement as me saying security is a lower priority – it is an equal priority. The best security is when the parts are designed to cooperate and reinforce each other. You can’t do that with MVC. It is an anti-pattern. MVC also typically results in many assets of differing security contexts sharing the same virtual network and cloud account/subscription/project. It is often selected because, at the start it looks easier to manage, but in the long term it becomes harder, as you struggle to deal with all those conflicting contexts and isolate everything out in an environment not designed for that type of isolation. Instead follow the cloud-native pattern… which works for lift and shift as well as new builds. In this approach the application and security architecture teams work together and design in parallel (ideally – you could add in security later, just not too late). You fit the security to the application. At the start there is a lot of learning new things, but over time you learn and build a library of relatively standard design patterns. You deploy into a clean account/subscription/project each time if you can. This enables you to minimize the number of privileged users who need access to the cloud account and simplifies, overall, the configuration of the accounts. This approach helps you close in on immutable and indempotent deployments (for production – development environments are still more free-form). You now have an isolated environment working within very defined constraints/definitions. This reduces complexity and is a bit of a security dream. It does increase another kind of complexity: managing all these different environments. There are organizations managing thousands of cloud accounts today. Management shifts to automation, deployment pipelines, and maintaining security guardrails across accounts. The alternative is complexity within an account, often leading to conflicting and difficult-to-enforce security boundaries. And that’s the key. I don’t claim managing cloud-native deployments is necessarily easier, but it shifts management in a direction that improves inherent security. You gain stronger security boundaries and tighten control, but in exchange you need to adopt automation and new management techniques and tooling. MVC always fails over the long term. Always – you inevitably reach a point where too many things, across too many conflicting security contexts, are sharing a single implementation. It seems easier up front (and probably is, especially if you are new to the cloud), but sooner than you think you will need to make security compromises. It additionally inhibits your ability to properly design security for any individual project, because the applications are restricted to a pre-configured set of rules. MVC usage correlates highly with ‘monoclouds’: stuffing everything into a single account with a small number of virtual networks. We also see some MVC deployments where they create a standard template and then deploy it into multiple accounts. Those aren’t quite as bad, but you still cannot fit security to the application and deployment. This is a period of massive transition. Greater than corporate adoption of the Internet itself, because the cloud requires deeper reengineering of underlying architectures. This is an incredible opportunity to break out of constraints of the past which have inhibited security – especially backward-looking MVC and monoclouds. Focus on education, automation, and tooling. Instead of building an MVC take a cloud project (ideally a new one) and “right fit” its security. Then take those lessons and move onto the next project. You will trade off getting all your sh** into the cloud as quickly as possible, but gain security and be able to move even more quickly over the long term. Share:

As we previewed in the Introduction to our Endpoint Advanced Protection Buyer’s Guide, the first step to selecting an endpoint security product is figuring out what problem you are trying to solve. Then figure out which capabilities are most important to solve those problems. Only then can you start trying to find a vendor who meets those requirements. This is what we call establishing *selection criteria. In the Introduction we also explained how organizations need both prevention and detection/response to fully protect endpoints. But these two capabilities do not need to be bought or deployed together – the technologies can come from different vendors if their agents play nicely together, and not every endpoint needs extensive forensics capabilities. So these two main functions need to be treated differently. Though, to put a nice big caveat on that statement, there is value in leveraging prevention and detection/response from the same vendor. There is also value in having network security controls that work tightly with the endpoint security in place. Is that enough to drive you to a single vendor for everything? As usual it depends, and we’ll work through the decision points. Over the next 5 days, we will explain the main Prevention capabilities you need to understand to select and evaluate these solutions. We’ll start by explaining the latest categories of attacks because many demand new and innovative defenses. Then we’ll dig into the capabilities that can prevent these attacks. Finally we will dig into and explain how the foundational technologies underlying these new endpoint security platforms work. There are nuances to how each vendor implements these technologies, and they’ll be sure to tell you how and why their approach is better. But without a clear understanding of what they are talking about, you cannot really discern the differences between vendors. Attacks There are many types of attacks, which all have one thing in common: compromise of the endpoint device. To avoid exploding your cranium by trying to cram in infinite possibilities, we will categorize and describe the major attack techniques, which provide the basis for figuring out your best protection strategy. But before we get there, we will intentionally conflate the delivery of malware with device compromise. We do this because companies in this space describe their capabilities in terms of attacks – not necessarily by the means of defense. To illuminate a bit, consider that some malware may be delivered by a phishing message and then use a known vulnerability to compromise the device. Is that different than the same attack was delivered via a drive-by download in your browser? Of course not – stopping the attack on the vulnerability is all that matters, not the delivery method. But, alas, security industry marketing machinery prefers to describe these as two totally different attacks. File-based Attacks In the first attack bucket, an unsuspecting user executes a compromised file which executes malicious code to compromise the device. This is basically traditional malware, and protecting against these attacks is the basis of the endpoint protection business we know today. In these first two categories, files are allowed onto the machine by the device ‘owner’. This can happen via email or a legitimate web browsing session, or when a user allows a download onto their device (possibly through social engineering). In any case, the file shows up on the device and must be evaluated. Known files (classic AV): Someone has seen this file before, and we know it’s malicious. The file’s hash is in a database somewhere, and the endpoint security tool checks to see if each file is recognized as bad before it allows execution. The challenge with using a blacklist of malicious files is scale. There are billions of files known to be bad, and keeping a comprehensive list on each endpoint is not feasible. It’s also not efficient to check every file against the entire blacklist prior to execution. Unknown files Otherwise known as zero-day malware, these files have not yet been seen and hashed as malware, so any defenses based on matching file hashes will be unable to recognize the files or detect the attacks. The challenge in detecting this type of attack is that it’s very easy to change the complexion of a malware file (using a file packer or other technique to change its hash), which means the file won’t show up on blacklists. Additionally, adversaries have sophisticated labs to test their malware against common endpoint prevention offerings, further challenging today’s solutions. The next attacks are a bit more obfuscated and require different tactics for prevention and detection: Document/macro attacks: In this kind of attack malicious code is hidden within a known file type like PDF or Microsoft Office, typically as a macro. The content is the attack vector and requires interpretation by the user’s application, but the attack is not an executable binary program. When opening or performing some kind of activity with the file, its code will execute to compromise the device. These attacks also get around traditional signature-based defenses because the file is a legitimate document – it’s the (invisible) contents which are malicious. Legitimate software: Yet another way to deliver malicious code to a device is to hide it within legitimate software. This typically happens with common applications (like Adobe Reader), system files, and multimedia files. Unsuspecting users can click a link within a legitimate search engine and download what they think is a legitimate app, but it might not be. With this type of attack everything looks kosher. It’s a familiar app and looks like something the user wants. To protect against these attacks we need to focus more on what the file does instead of what it looks like. File-less Attacks Over the past decade savvy attackers realized the entire endpoint protection business was based on attacks leveraging files on the compromised device to store malicious code. But if they could deliver malware without storing it in files, their attacks would be much harder to detect. And they were right.

The team is back from the dead, and so are some really crappy cloud ideas. Watch or listen: Share:

As I was flying home from a meeting today I read two security stories that highlighted the differences between bad and less bad ways to report on security issues. Before I go into them, here is how I evaluate articles related to either stunt hacking or super-popular technology: Is there a real vulnerability? Is it exploitable, and to what degree? What are the actual, known, demonstrable consequences of exploitation? Would other controls or the real-world ecosystem limit either exploitation or impact? Who is writing the article or giving the presentation, who are their sources, and why are they talking about it? How did the vendor/target/whoever respond to the situation, and how is this reflected in the article? These are actually the same criteria I apply to original research reports and conference presentations. Now on to the articles: First, a contact at Apple pointed me to this article by Lily Hay Newman in Wired on “privacy risks” with CoreML. (Let’s be honest: I am known to have a real sore spot for this kind of article – the pointer wasn’t accidental. I’ll save you some time by summing it up: CoreML enables machine learning in apps. These apps can have access to your photos (with permission). Machine learning is hard, so bad actors can sneak in code to do things like find nudies or which products you have in the background of photos. This is against the App Store guidelines, but no one really knows whether Apple would detect it. There’s one small quote at the end from an actual security researcher admitting that such an app could just upload every photo to the cloud if it had this permission anyway. Here is how I’ve been summarizing these kinds of pieces, since basically the start of Securosis: There is a new technology getting some decent attention. Hypothetically someone might be able to do bad stuff with it. Let’s put “iPhone” or “critical infrastructure” in the headline so we get lots of clicks. (This list is growing, though – today I would add cars, airplanes, home automation, electronic toys, and robots/drones). Let’s barely mention that multiple other vendors or product categories have the same capability and often worse security controls. Because iPhones! I want to contrast Wired’s piece with a different piece at BleepingComputer on a backdoor in a satellite Internet system heavily used in shipping. The reason this article is a good contrast is because it starts with a similar premise – a researcher finding an issue and taking it to the press (in this case clearly to get some media coverage). I’m not convinced this basis for articles is usually a good thing because a lot of companies push their researchers for “big” findings like this to get attention. But some are legitimately important issues which do need coverage that vendors or whoever would otherwise try to cover up. In this case: Most ships use a popular satellite Internet system. There is a backdoor (literally named backdoor) in the system, plus another vulnerability. The system is at end-of-life, still in wide use, and will not be patched. The system is for Internet traffic only, not ship control, and the networks are separated. Exploiting this is hard but possible. Although you can’t get into control systems, it could be used for tracking or economic malfeasance. It is at least partially patched, and the vendor warned everyone. The key differences: This was a real exploitable vulnerability, not purely hypothetical. The article clearly defined the scope of potential exploitation. The piece was quickly updated with a statement from the product vendor that indicates the issue may not be even as bad as reported by the security vendor. Or an issue at all any more (but the update should be called out at the top, because it totally undermines the rest of the piece). Now, is this article great? No – the headline and section titles are more hyperbolic than the actual text – editors often do this after a writer submits an article. Also I think the refining statement should be at the top. According to Inmarsat’s statement (after release) the exploit requires physical access and remote exploitation is blocked on shoreside firewalls. The positives of the article are that it mostly balances the risk, highlights a really stupid mistake (the backdoor was insanely easy to exploit) and was based… on reality. Do you want to see a similar situation that involved a real exploit, real risks, a horrible vendor response, and resulting widespread action? Check out this article on a pacemaker recall due to exploitable vulnerabilities. It even highlights issues with how it was handled by both researchers and ÷vendors. Share:

We will close out this series with a look at several operational considerations for selecting a secrets management platform. There are quite a few secrets management tools, both commercial and otherwise, on the market, and each does things a bit differently. Rather than a giant survey of every product and how it works, we will focus on the facets of these products which enable them to handle the use cases discussed earlier. Central questions include how these platforms deploy, how they provide scalability and resiliency, and how they integrate with the services they supply secrets to? To better distinguish between products you need to understand why they were created, because core functions and deployment models are heavily influenced by a platform’s intended use. Classes of Products Secrets management platforms fall into two basic categories: general-purpose and single-purpose. General-purpose solutions provide secrets for multiple use cases, with many types of secrets. General-purpose systems can automatically provision secrets to just about any type of application – from sending user name and password to a web page, to issuing API keys, to dynamic cloud workloads. Single-purpose options – commonly called ‘embedded’ because they install into another platform – are typically focused on one use case. For example the embedded solutions focus on provisioning secrets to Docker containers, and nest into your orchestration manager (e.g.: Swarm, Kubernetes, DC/OS). This is a critical distinction, because a product embedded into a container manager may not work for non-container use cases. The good news is that many services are deployed this way so they are still useful in many environments, and because these tools leverage existing infrastructure they often integrate well and scale easily. These platforms typically leverage specific constructs of the orchestration manager or container environment to provide secrets. They also tend to make assumptions about how secrets are to be used; for example they may leverage Kubernetes’ ‘namespace’ to enforce policy or the UNIX ‘namespace’ to distribute secrets. Because containers are ephemeral, ephemeral or ‘dynamic’ secrets are often preferred for those secrets managers. The bad news is that some embedded tools assume your cluster is a secure environment, so they can safely pass and store secrets in clear text. Many embedded tools fully encrypt secrets, but they may not support diverse types of secrets or integrate with non-containerized applications. These specializations are neither good nor bad, depending on what you need for secrets management, but embedded systems may be limited. General-purpose products are typically more flexible and may take more time and to set up, but provide a breadth of functions not generally found in embedded tools. Deployment Models Solitary Servers Common among early tools focused on personal productivity, solitary servers are exactly what the name implies. They typically consist of a central secret storage database and a single server instance that manages it. Basically all functions – including user interfaces, storage management, key management, authentication, and policy management – are handled by a single service. These tools are commonly used via command-line interfaces or API, and work best for a small number of systems. Client-Server Architecture The label for this model varies from vendor to vendor. Primary/Secondary, Manager/Worker, Master/Slave, and Service/Agent are just some of the terms to describe the hierarchical relationship between the principal service which manages the repository of secrets, and the client which works with the calling application. This is by far the most common architecture. There is a repository where encrypted secrets are stored, usually a database which is shared or replicated across one or more manager nodes. And each manager can work with one or more agents to support the needs of their service or application. This architecture helps provide scalability and reliability by spawning new clients and servers as needed. These products often deploy each component as a container, leveraging the same infrastructure as the applications they serve. Many embedded products use this model to scale. Integration We already talked about how secrets are shared between a secrets management tool and a recipient, whether human or machine. And we covered integration with container management and orchestration systems, as many tools were designed to do. It’s time to mention the other common integration points and how each works. Build Servers: Tools like Jenkins and Bamboo are used by software development teams to automate the building and verification of new code. These tools commonly access one or more repositories to get updated code, grab automation scripts and libraries to set up new environments, connect to virtual or cloud services to run tests, and sign code before moving tested code into another repository or container registry. Each action requires specific credentials before it can take place. Secrets management integration is either performed as a plug-in component to the build server or as an external service it communicates with. IT Automation: Automated builds and the power of build managers have vastly improved development productivity, but orchestration tools are what move code at warp speed from developer desktops into production. Chef/Puppet/Ansible are the trio of popular orchestration tools automating IT and development tasks, the backbone of Continuous Integration and Continuous Deployment. Virtually any programable IT operation can be performed with these tools, including most VMware and all cloud services functions offered through API. As with build servers, secrets management typically installs as a component or add-on module of the orchestration tool, or runs as a service. Public Cloud Support: Public cloud is a special case. Conceptually, every use case outlined in this series is applicable to cloud services. And because every service in a public cloud is API enabled, it is the ideal playground for secrets management tools. What’s special about cloud services is how integration is managed; most secrets management tools which support the cloud directly integrate with either cloud native identity systems, cloud-native key management, or both. This offers advantages because secrets can then be provisioned in any region, to any supported service within that region, leveraging existing identities. The cloud service can fully define which user can access which secrets.

In this section we will discuss the core features of a secrets management platform. There are basic functions every secrets management platform needs to address the basic use cases. These include secure storage and disbursement of secrets, identity management, and API access, for starters. There are plenty of tools out there, many open source, and several bundled into other platforms. But when considering what you need from one of these platforms, the key thing to keep in mind is that most of them were originally developed to perform a single very specific task – such as injecting secrets into containers at runtime, or integrating tightly with a Jenkins build server, or supplementing a cloud identity service. Those do one thing well, but typically do not address multiple use cases. Now let’s take a closer look at the key features. Core Features Storing Secrets Secrets management platforms are software applications designed to support other applications in a very important task: securely storing and passing secrets to the correct parties. The most important characteristic of a secrets management platform is that it must never leave secret information sitting around in clear text. Secure storage is job #1. Almost every tool we reviewed provides one or more encrypted repositories – which several products call a ‘vault’ – to store secret information in. As you insert or update secrets in the repository, they are automatically encrypted prior to being written to storage. Shocking though it may be, at least one product you may come across does not actually encrypt secrets – instead storing them in locations its developers consider difficult to access. The good news is that most vaults use vetted implementations of well-known encryption algorithms to encrypt secrets. But it is worth vetting any implementation, with your regulatory and contractual requirements in mind, prior to selecting one for production use. With the exception of select platforms which provide ‘ephemeral secrets’ (more on these later), all secret data is stored within these repositores for future use. Nothing is stored in clear text. How each platform associates secrets with a given user identifier, credential, or role varies widely. Each platform has its own way of managing secrets internally, but mostly they use a unique identifier or key-value pair to identify each secret. Some store multiple versions of a secret so changes over time can be recalled if necessary for recovery or auditing, but the details are part of their secret sauce. The repository structure varies widely between offerings. Some store data in simple text or JSON files. Some use key-value pairs in a NoSQL style database. Others use a relational or Big Data database of your choice. A couple employ multiple repository types to increase isolation between secrets and/or use cases. Their repository architecture is seldom determined by strong security; more common drivers are low cost and ease of use for the product developers. And while a repository of any type can be secured, the choice of repository impact scalability, how replication is performed, and how quickly you can find and provision secrets. Another consideration is which data types a repository can handle. Most platforms we reviewed can handle any type of data you want to store: string values, text fields, N-tuple pairings, and binary data. Indexing is often performed automatically as you insert items, to speed lookup and retrieval later. Some of these platforms really only handle string, which simplifies programmatic API but limits their usability. Again, products tailored to a particular use case may be unsuitable for other uses or across teams. Identity and Access Management Most secrets management platforms concede IAM to external Active Directory or LDAP services, which makes sense because most firms already have IAM infrastructure in place. Users authenticate to the directory store to gain access, and the server leverages existing roles to determine which functions and secrets the user is authorized to access. Most platforms are also able to use a third-party Cloud Identity Service or Privileged Access Management service, or to directly integrate with cloud-native directory services. Note that a couple of the platforms we reviewed manage identity and secrets internally, rather than using an external identity store. This is not a bad thing because they then tend to include secrets management to supplement password or key management, and internally management of identity is part of their security architecture. Access and Usage Most platforms provide one or more programming interfaces. The most common, to serve secrets in automated environments, is an access API. A small and simple set of API calls are provided to authenticate a session, insert a record, locate a secret, and share a secret to a specific user or service. More advanced solutions also offer API access to advanced or administrative functions. Command-line access is also common, leveraging the same basic functions in a command-driven UNIX/Linux environment. A handful of others also offer a graphical user interface, either directly or indirectly, sometimes through another open source project. Sharing Secrets The most interesting aspect of a secrets management system is how it shares secrets with users, services, or applications. How do you securely provide a secret to its intended recipient? As in the repository, as discussed above, secrets in transit must be protected, which usually means encryption. And there are many different ways to pass secrets around. Let’s take a look at the common methods of secret passing. Encrypted Network Communications: Authenticated service or users are passed secrets, often in clear text, within an encrypted session. Some use Secure Sockets Layer (SSL), which is not ideal, for encrypted transport, but thankfully most use current versions of Transport Layer Encryption, which als authenticates the recipient to the secrets management server. PKI: Several secrets management platforms combine external identity management with a Public Key Infrastructure to validate recipients of secrets and transmit PKI encrypted payloads. The platform determines who will receive a secret, and encrypts the content with the recipient’s public key. This ensures that only the intended recipient can decrypt the secret, using their private key.

This post will discuss why secrets management is needed at all, along with the diverse use cases which teams need it to address. In every case there is some secret data which needs to be sent – hopefully not in plain text – to an application or service. And in every case we want the ability to provide secrets, both when an operator is present and automatically. The biggest single issue is that security around these secrets today is largely absent, and they are kept in cleartext within documents of various types. Let’s dive in. Use Cases API Gateways and Access Keys: Application Programming Interfaces are how software programs interact with other software and services. These API form the basic interface for joint operation. To use an API you must first authenticate yourself – or your code – to the gateway. This is typically done by providing an access key, token, or response to a cryptographic challenge. For ease of automation many developers hard-code access keys, leaving themselves vulnerable to simple file or code inspection. And all too often, even when kept in a private file on the developer’s desktop, keys are accidentally shared or posted, such as to public code repositories. The goal here is to keep access keys secret while still provisioning to valid applications as needed. Automated Services: Applications are seldom stand-alone entities. They are typically comprised of many different components, databases, and supporting services. And with current application architectures we launch many instances of an application to ensure scalability and resiliency. As we launch applications, whether in containers or as servers, we must provision them with configuration data, identity certificates, and tokens. How does a newly created virtual machine, container, or application discover its identity and access the resources it needs? How can we uniquely identify a container instance among a sea of clones? In the race to fully automate the environment, organizations have automated so fast that they got out over their skis, with little security and a decided imbalance towards build speed. Developers typically place credentials in configuration files which are conveniently available to applications and servers on startup. We find production credentials shared with quality assurance and developer systems, which are commonly far less secure and not always monitored. They are also frequently shared with other applications and services which should not have access. The goal is to segregate credentials without causing breakage or unacceptable barriers. Build Automation: Most build environments are insecure. Developers tend to feel security during development slows them down, so they often bypass security controls in development processes. Build environments are normally under developer control, on development-owned servers, so few outsiders know where they are or how they operate. Nightly build servers has been around for over a decade, with steadily increasing automation to improve agility. As things speed up we remove human oversight. Continuous Integration and Continuous Deployment use automation to speed software delivery. Build servers like Jenkins and Bamboo automatically regenerate application as code, formation templates, and scripts are checked into repositories. When builds are complete we automatically launch new environments to perform functional, regression, and even security tests. When these tests pass, some organizations launch the code in production! Build server security has become an important topic. We no longer have the luxury of leaving passwords, encryption keys, and access tokens sitting in unprotected files or scripts. But just as continuous integration and DevOps improve agility, we need to automate the provisioning of secrets into the process as well, and create an audit trail to prove we are delivering code and services securely. Encrypted Data: Providing encryption keys to unlock encrypted volumes and file stores is a common task, both on-premise and for cloud services. In fact, automated infrastructure makes the problem more difficult as the environment is less static, with thousands of services, containers and applications popping in and out of service. Traditionally we have used key management servers designed to handle secure distribution and management of keys, but a number of commercial key management tools (hardware and software) have not been augmented for Infrastructure and Platform as a Service. Additionally, developers demand better API integration for seamless use with applications. This capability is frequently lacking, so some teams use cloud native key management, while others opt for secrets management as a replacement. Sharing: Collaboration software has helped development, quality assurance, and product mamagement teams cooperate on projects; even though people in these groups are less and less likely to share office space. User are more likely to work from home, at least part time. In some contexts the issue is how to securely share information across a team of remote developers, but that use case overlaps with having IT share secret data across multiple data centers without exposing it in clear text, or exposed in random files. The databases that hold data for chat and collaboration services tend to not be very secure, and texting certificates to a co-worker is a non-starter. The solution is a central, robust repository, where a select group of users can store and retrieve secrets. Of course there are plenty more use cases. In interviews we discuss everything from simple passwords to bitcoin wallets. But for this research we need to focus on the issues developers and IT security folks asked about. Our next post will discuss the core features and functions of a secrets management system, as well as some advanced functions which differentiate commercial options from open source. We want to provide a sense of what is possible, and help guide readers to the subset of functions they need for their use cases. Share:

This week we are starting a new research series on Secrets Management. What is secrets management and why do you care? A good number of you in security will be asking these questions. Secrets Management platforms do exactly what the name implies; they store, manage and provide secrets. This technology addresses several problems most security folks don’t yet know they have. As development teams leverage automation and orchestration techniques, they are creating new security issues to be tackled. Let’s jump into some of the back story, and then outline what we will accomplish in this research effort. It sounds cliche, sure, but IT and application environments are genuinely undergoing radical change. New ways of deploying applications as microservices or into containers is improving our ability to cost-effectively scale services and large systems. Software defined IT stacks and granular control over services through API provide tremendous advantages in agility. Modern operational models such as Continuous Integration and DevOps amplify these advantages, bringing applications and infrastructure to market faster and more reliably. Perhaps the largest change currently affecting software development and IT is cloud computing: on-demand and elastic services offers huge advantages, but predicated on automated infrastructure defined as software. While cloud is not a necessary component to these other advancements, it’s makes them all the more powerful. Leveraging all these advancements together, a few lines of code can launch – or shut down – an entire (virtual) data center in minutes, with minimal human involvement or effort. Alongside their benefits, automation and orchestration raise new security concerns. The major issue today is secure sharing of secret information. Development teams need to share data, configurations, and access keys across teams to cooperate on application development and testing. Automated build servers need access to source code control, API gateways, and user roles to accomplish their tasks. Servers need access to encrypted disks, applications need to access databases, and containers must be provisioned with privileges as they start up. Automated services cannot wait around for users to type in passwords or provide credentials! So we need new agile and automated techniques to provision data, identity, and access rights. Obtaining these secrets is essential for automation scripts to function, but many organizations cling to the classic (simple) mode of operation: place secrets in files, or embed them into scripts, so tasks can complete without human intervention. Developers understand this is problematic, but it’s a technical problem most sweep under the rug. And they certainly do not go out of their way to tell security about how they provision secrets, so most CISOs and security architects are unaware of this emerging security issue. This problem is not new. No administrator wants to be called into work in the middle of the night to enter a password so an application can restart. So IT administrators routinely store encryption keys in files so an OS or application can access them when needed. Database administrators place encryption keys and passwords in files to facilitate automated reboots. Or they did until corporate networks came under even more attack; then we saw a shift to keys, certificates, and passwords. Since then we have relied upon everything from manual intervention, key management servers, and even hardware dongles to provide a root of trust to establish identity and provision systems. But those models not only break the automation we rely upon to reduce costs and speed up deployments, lack also the programmatic interfaces needed to integrate with cloud services. To address the changes described above, new utilities and platforms have been created to rapidly provide information across groups and applications. The term for this new class of product is “Secrets Management”; it is changing how we deliver identity, secrets, and tokens; as well as changing the way we validate systems for automated establishment of trust. In this research we will explore why this is an issue for many organizations, what sort of problems these new platforms tackle, and how they work in these newer environments. Specifically, we will cover: Use Cases: We will start by considering specific problems which make secret sharing so difficult: such as moving passwords in clear text, providing keys to encryption engines, secure disk storage, knowing which processes are trustworthy, and mutual (bidirectional) authentication. Then we will discus specific use cases driving secrets management. We will cover issues such as provisioning containers and servers, software build environments, database access, and encrypted disk & file stores; we will continue to examine sharing secrets across groups and controlling who can launch which resources in private and public cloud environments. Components and Features: This section will discuss the core features of a secrets management platform. We will discuss the vault/repository concept, the use of ephemeral non-vault systems, identity management for vault access, role-based authorization, network security, and replication for both resiliency and remote access. We will cover common interfaces such as CLI, API, and HTTP. We’ll contrast open source personal productivity tools with newer commercial products; we will also consider advanced features such as administration, logging, identity store integration, ability to provide secure connections, and policy enforcement. Deployment Considerations: Next we will discuss what is stored in a repository, and how secrets are shared or integrated with dependent services or applications. We will discuss both deployment models; as well as the secrets to be shared: passwords, encryption keys, access tokens, API keys, identity certificates, IoT key pairs, secure configuration data, and even text files. We will also offer some advice on product selection criteria and what to look for. As we leverage cloud services and rely more heavily on automation to provision applications and IT resources, we find more and more need to get secrets to applications and scripts securely. So our next post will start with use cases driving this market. Share: