We close out this research paper with two key areas: Monitoring and Auditing. We want to draw attention to them because they are essential to security programs, but have received only sporadic coverage in security blogs and the press. When we go beyond network segregation and network policies for what we allow, the ability to detect misuse is extremely valuable, which is where monitoring and logging come in. Additionally, most Development and Security teams are not aware of the variety of monitoring options available, and we have seen a variety of misconceptions and outright fear of the volume of audit logs to capture, so we need to address these issues.

Monitoring

Every security control discussed so far can be classed as preventative security. These efforts remove vulnerabilities or make them hard to exploit. We address known attack vectors with well-understood responses such as patching, secure configuration, and encryption. But vulnerability scans can only take you so far. What about issues you are not expecting? What if a new attack variant gets by your security controls, or a trusted employee makes a mistake? This is where monitoring comes in: it is how you discover unexpected problems. Monitoring is critical to any security program – it’s how you learn what works, track what’s really happening in your environment, and detect what’s broken.

Monitoring is just as important for container security, but container providers don’t offer it today.

Monitoring tools work by first collecting events, then comparing them to security policies. Events include requests for hardware resources, IP-based communication, API requests to other services, and sharing information with other containers. Policy types vary widely. Deterministic policies address areas such as which users and groups can terminate resources, which containers are disallowed from making external HTTP requests, and which services a container is allowed to run. Dynamic (also called ‘behavioral’) policies address issues such as containers connecting to undocumented ports, using more memory than normal, or exceeding runtime thresholds. Combining deterministic white and black lists with dynamic behavior detection offers the best of both worlds, enabling you to detect both simple policy violations and unexpected variations from the ordinary.

We strongly recommend you include monitoring container activity in your security program. A couple container security vendors offer monitoring tools. Popular evaluation criteria include:

• Deployment Model: How does the product collect events? What events and API calls can it collect for inspection? Typically these products use one of two models for deployment: either an agent embedded in the host OS, or a fully privileged container-based monitor running in the Docker environment. How difficult are collectors to deploy? Do host-based agents require a host reboot to deploy or update? You need to assess what types of events can be captured.
• Policy Management: You need to evaluate how easy it is to build new policies or modify existing ones. You will want a standard set of security policies from the vendor to speed deployment, but you will also stand up and manage your own policies, so ease of management is key to long-term happiness.
• Behavioral Analysis: What, if any, behavioral analysis capabilities are available? How flexible are they – what types of data are available for use in policy decisions? Behavioral analysis starts with system monitoring to determine ‘normal’ behavior. The pre-built criteria for detecting aberrations are often limited to a few sets of indicators, such as user ID or IP address, but more advanced tools offer a dozen or more choices. The more you have available – such as system calls, network ports, resource usage, image ID, and inbound and outbound connectivity – the more flexible your controls can be.
• Activity Blocking: Does the vendor offer blocking of requests or activity? Blocking policy violations helps ensure containers behave as intended. Care is required because such policies can disrupt new functionality, causing friction between Development and Security, but blocking is invaluable for maintaining Security’s control over what containers can do.
• Platform Support: You need to verify your monitoring tool supports your OS platforms (CentOS, CoreOS, SUSE, Red Hat, Windows, etc.) and orchestration tool (Swarm, Kubernetes, Mesos, or ECS).

Audit and Compliance

What happened with the last build? Did we remove sshd from that container? Did we add the new security tests to Jenkins? Is the latest build in the repository? You may not know the answers off the top of your head, but you know where to get them: log files. Git, Jenkins, JFrog, Docker, and just about every development tool creates log files, which we use to figure out what happened – and all too often, what went wrong. There are people outside Development – namely Security and Compliance – with similar security-related questions about what is going on in the container environment, and whether security controls are functioning. Logs are how you get answers for these teams.

Most of the earlier sections in this paper, covering areas such as build environments and runtime security, carry compliance requirements. These may be externally mandated like PCI-DSS or GLBA, or internal requirements from internal audit or security teams. Either way, auditors will want to see that security controls are in place and working. And no, they won’t just take your word for it – they will want audit reports for specific event types relevant to their audit. Similarly, if your company has a Security Operations Center, they will want all system and activity logs some time period to reconstruct events, and, investigate alerts, and/or determine whether a breach occurred. You really don’t want to get too deep into that stuff – just get them the data and let them worry about the details.

CIS offers benchmarks and security checklists for container security, orchestration manager security, and most compliance initiatives. These are a good starting point for conducting basic security and compliance assessments of your container environment. In addition ‘vendors’ – both open source teams and cloud service providers – offer security deployment and architecture recommendations to help produce dependable environments. Finally, we see configuration checkers arriving in the market – both open source and commercial tools – a quick search is likely to offer a couple options relevant to your environment, whatever it may be.

Container security programs are quite complex, due to the large number of areas which require attention. The good news is that most of what you need is already in place. During our investigation for this series we did not speak with any firms which did not already have Splunk, log storage, or SIEM on-premise, and in many cases all three were available. Additionally the vast majority of code repositories, build controllers, and container management systems – specifically the Docker runtime and Docker Trusted Registry – produce event logs in formats which can be consumed by various log management and SIEM systems without modification. As do most third-party image validation and monitoring security tools. You will need to determine how easy this is to leverage. Some simply dump syslog formatted information into a directory, at which point it’s up to you to drop this into Splunk, an S3 bucket, Loggly, or your SIEM tool. In other cases – most, actually – you can specify CEF, JSON, or some other format, and the tools can automatically link to your SIEM of choice, sending events as they occur.