Most people fail to consider the build environment when thinking about container security, but it is critical. The build environment is traditionally the domain of developers, who don’t share much detail with outsiders (meaning security teams). But with Continuous Integration (CI) or full Continuous Deployment (CD), we’re shooting new code into production… potentially several times a day. An easy way for an attacker to hack an application is get into its development or build environment – usually far less secure than production – and alter code or add new code to containers. The risk is aggravated by DevOps rapidly breaking down barriers between groups, and operations and security teams given access so they can contribute to the process. Collaboration demands a more complex and distributed working environment, with more stakeholders. Better controls are needed to restrict who can alter the build environment and update code, and an audit process to validate who did what.

It’s also prudent to keep in mind the reasons developers find containers so attractive, lest you try to adopt security controls which limit their usefulness. First, a container simplifies building and packaging application code – abstracting the app from its physical environment – so developers can worry about the application rather than its supporting systems. Second, the container model promotes lightweight services – breaking large applications down into small pieces, easing modification and scaling… especially in cloud and virtual environments. Finally, a very practical benefit is that container startup is nearly instant, allowing agile scaling up and down in response to demand. It is important to keep these feature in mind when considering security controls, because any control that reduces one of these core advantages is likely to be rejected or ignored.

Build pipeline security breaks down into two basic areas. The first is application security: essentially testing your code and its container to ensure it conforms to security and operational practices. This includes tools such as static analysis, dynamic analysis, composition analysis, scanners built into the IDE, and tools which monitor runtime behavior. We will cover these topics in the next section. The second area of concern is the tools used to build and deploy applications – including source code control, build tools, the build controller, container registries, container management facilities, and runtime access control. At Securosis we often call this the “management plane”, as these interfaces – whether API or GUI – are used to set access policies, automate behaviors, and audit activity. Let’s dive into build tool security.

Securing the Build

The problem is conceptually simple, but there are many tools used for building software, and most have several plug-ins which alter how data flows, so environments can get complicated. You can call this Secure Software Delivery, Software Supply Chain Management, or Build Server Security – take your pick, because these terms are equivalent for our purpose. Our goal is to shed light on the tools and processes developers use to build application, so you can better gauge the threats, as well as security measures to secure these systems.

Following is a list of recommendations for securing platforms in the build environment to ensure secure container construction. We include tools from Docker and others to automate and orchestrate source code, building, the Docker engine, and the repository. For each tool you select some combination of identity management, roles, platform segregation, secure storage of sensitive data, network encryption, and event logging.

1. Source Code Control: Stash, Git, GitHub, and several variants are common. Source code control has a wide audience because it is now common for Security, Operations, and Quality Assurance to all contribute code, tests, and configuration data. Distributed access means all traffic should run over SSL or VPN connections. User roles and access levels are essential for controlling who can do what, but we recommend requiring token-based or certificate-based authentication, or two-factor authentication at a minimum, for all administrative access. This is good housekeeping whether you are using containers or not, but containers’ lack of transparency, coupled with automated processes pushing them into production, amplifies the need to protect the build.
2. Build Tools and Controllers: The vast majority of development teams we speak with use build controllers like Bamboo and Jenkins, with these platforms becoming an essential part of their automated build processes. They provide many pre-, post-, and intra-build options, and can link to a myriad of other facilities. This is great for integration flexibility but can complicate security. We suggest full network segregation of the build controller system(s), and locking network connections to limit what can communicate with them. If you can deploy build servers as on-demand containers without administrative access to ensure standardization of the build environment and consistency of new containers. Limit access to the build controllers as tightly as possible, and leverage built-in features to restrict capabilities when developers need access. We also suggest locking down configuration and control data to prevent tampering with build controller behavior. Keep any sensitive data, including ssh keys, API access keys, database credentials, and the like in a secure database or data repository (such as a key manager, encrypted .dmg file, or vault) and pulling credentials on demand to ensure sensitive data never sits on-disk unprotected. Finally, enable the build controller’s built-in logging facilities or logging add-ons, and stream output to a secure location for auditing.
3. Container Platform Security: Whether you use Docker or another tool to compose and run containers, your container manager is a powerful tool which controls what applications run. As with build controllers like Jenkins, you’ll want to limit access to specific container administrator accounts. Limit network access to only build controller systems. Make sure Docker client access is segregated between development, test, and production, to limit who and which services can launch containers in production.
4. Container Registry Security: We need to discuss container registries, because developers and IT teams often make the same two mistakes. The first is to allow anyone to add containers to the registry, regardless of whether they have been vetted. In such an environment it’s all too easy to insert an insecure container into production. It’s common for developers to leverage open source tools and platforms to speed development, some of which are available as pre-built containers. But it’s also common for attackers to create ‘pre-pwned’ containers loaded with malware, which probe and attack their host and other accessible containers. You’ll want to ensure that your registry only accepts containers from trusted sources, and not just whatever some developer grabbed on a whim. Ensuring containers are vetted, and that only containers signed with a trusted key can be launched, helps protect from these problems. You’ll also want to ensure that your registries and clients require IAM credentials, to limit who can control the build or add images, and that images loaded into production musts come from your approved registry.

Next we will discuss how to secure containers at runtime.