In the first post of this series on Security and Privacy on the Encrypted Network, we argued that organizations need to encrypt more traffic. Unfortunately the inability to see and inspect encrypted traffic impairs the ability to enforce security controls/policies and meet compliance mandates. So let’s dig into how to strategically decrypt traffic in order to address a few key use cases – including enforcing security policies and monitoring for security and compliance. We also need to factor in the HR and privacy issues associated with decrypting traffic – you don’t want to end up on the wrong side of a worker council protesting your network security approach.

What to Decrypt

The first step in gaining visibility into the encrypted network is to set policies for when traffic will be decrypted and for how long. These decisions depend more on organizational culture than anything else, so you need to figure out what will work for your company. As security guys we favor more decryption than less, because that enables more comprehensive inspection… and therefore stronger monitoring and enforcement. But this is a company-specific choice.

Several factors influence decryption policies, most obviously the applications themselves. Let’s briefly cover the main applications you are most likely to decrypt:

1. Webmail: Employees think they are doing your organization a favor by working at all times of the day. But this always-on workforce requires use of personal devices, and may decide (however misguided) that it’s easiest to send work documents to personal machines via personal email accounts. What could go wrong? And of course there are more malicious uses for webmail in a corporate environment. There are endpoint DLP agents that should catch this behavior, but if you don’t have them deployed you should be inspecting outbound webmail traffic. The complication is that most webmail is now encrypted so you need to decrypt sessions to inspect the traffic.
2. Web browsing: Similarly social media sites and other web properties utilize user-generated content that may be protected or sensitive, so you need to ensure you can enforce policies on web application traffic as well. Many apps use SSL/TLS by default, so you will need to decrypt to enforce acceptable use policies and protect data.
3. SaaS Apps: Business functions are increasingly migrating to Software as a Service (SaaS) so it is important to inspect SaaS traffic. You may want to enforce tighter content policies on SaaS apps, but first you need to decrypt their traffic for inspection and enforcement.
4. Custom Apps: Similarly your custom web apps (or partner web apps) require scrutiny given the likelihood that they will use sensitive data. As with SaaS apps, you will want to enforce granular policies for these apps, which requires decryption.

To net it out, if an application has access to protected or critical data you should decrypt and inspect its traffic. Within each application defined above, secondary attributes may demand or preclude decryption. For example certain web apps/sites should be whitelisted because they handle private employee data, such as consumer healthcare and financial sites.

Another policy trigger will be individual employees and groups. Maybe you don’t want to decrypt traffic from the legal team, because it is likely protected and sensitive. And of course there are the folks who require exceptions. Like the CEO, who gets to do whatever he/she wants and may approve an exception for their own traffic. There will be other exceptions (we guarantee it), so make sure your policies include the ability to selectively decrypt and enforce policies. For example one app may need to always be inspected (regardless of user) based on the sensitivity of data it can access. Likewise perhaps one set of users won’t have their traffic inspected at all. You should have flexibility to decrypt traffic to enforce policies, based on applications and users/groups, to accurately map to business processes and requirements.

Regardless of the use case for decryption, you will want to be flexible about what gets decrypted, for whom, and when.

Where to Decrypt?

Now that you know what to decrypt you need to determine the best place to do it. This decision hinges on type of traffic (ingress vs. egress), which applications need to be inspected, and which devices you need to send data to for monitoring and/or enforcement.

1. Firewall: Firewalls frequently take on the decryption role because they is inline for both egress and ingress, and already enforcing policies – especially as they evolve toward application-aware Next Generation Firewalls (NGFW). Unfortunately decryption is computationally demanding, which creates scaling issues even for larger and more powerful firewalls.
2. IPS: IPS is an inspection technology, so an inability to inspect encrypted traffic is a serious limitation. To address this some organizations decrypt on their IPS devices. The IPS function is computationally demanding so these devices tend to have more horsepower, which helps when doing decryption. But as with firewalls, scalability can be an issue.
3. Web filter: Due to their role, web filters need to decrypt traffic. They tend to be a bit underpowered compared to other devices in the DMZ, so unless there is minimal encrypted traffic, they can run out of gas quickly.
4. Dedicated SSL decryption device: For organizations with a lot of encrypted traffic (which is becoming more common), a few dedicated decryption devices are available which specialize in decrypting traffic without disrupting employees, offering flexibility in how to route decrypted traffic for either active controls (FW, IPS, web filter, etc.) or monitoring, and then re-encrypting as it continues out to the Internet. We will get into specifics of selecting and deploying these devices in our next post.
5. Cloud-based offerings: As Security as a Service (SECaaS) offerings mature, organizations have the option to decrypt in the cloud, removing their responsibility for scalability. On the other hand this requires potentially sensitive data to be decrypted and inspected in the cloud, which may be a cultural or regulatory challenge.

These devices are typically deployed inside your network permiter, so you remain blind to attackers encrypting internal reconnaissance traffic, or traffic moving out of the data center to a staging server inside your internal network. To address these issues you might choose to put one of the devices above (most likely a firewall or IPS) in front of your data center to enforce security policies and inspect traffic. We increasingly see this deployment model for network security gear, although the internal network traffic characteristics are different than on the perimeter. Scale is important important in the data center environment, given multi-gigabit internal networks.

To ensure scalability you will want to test the performance impact of decryption. According to independent lab tests (from organizations such as NSS Labs), you may see a performance penalty of up to 80% when decrypting on firewalls and IPS devices. Obviously ensuring adequate throughput for typical traffic volumes is a hard architectural and deployment requirement

Enforcing Security Policies

Our first use case is active enforcement of security policies. This involves decrypting traffic and then enforcing policies using traditional devices, including firewalls, IPS, web filters, load balancers, etc. Harkening back to the first post in this series, it is very common for sophisticated attackers to encrypt traffic to and from compromised devices. So if you don’t decrypt both ingress and egress traffic you are blind to certain attacks. You can miss newly compromsied machines connecting to the mothership, along with the resulting malware downloads, because C&C traffic is encrypted. Another blind spot is exfiltration of sensitive data, which is consistently encrypted.

The key considerations to decrypt traffic and send it to an active security device are:

1. Throughput: Networks continue to get faster so you to inspect the traffic at (or very near) wire speed. Of course decryption and re-encryption are very resource intensive so you need to make sure your decryption capability can scale sufficiently.
2. Latency: Many applications are real-time or highly interactive, so you cannot afford to introduce major latency. So you need to not make sure that along with adequate throughput, scaling up doesn’t add unacceptable latency.
3. Full protocol support: Attackers can hide encrypted traffic in other protocols on different ports, so it is important for inspection engines to analyze the full traffic stream – not just port 443.
4. Policy granularity: Granular policies are necessary to dictate what gets decrypted by attributes such as protocol, user/group, application, web site category, etc.
5. Send to multiple devices: You may want both an IPS and a network-based malware sandbox to analyze traffic, so you will want the ability to send it to multiple devices without a lot of fuss.
6. Fail open or closed: If decryption fails do you just allow traffic to pass through, or will you block it? This can get sticky – ensure senior management and Legal sign off off. In many cases continuing business operations is prioritized over the possibility of an attack.

Given the performance impact of decryption you will need to manage expectations all along the way. You likely cannot buy enough decryption gear to keep pace with all your networks and security devices. So for traffic that needs to be decrypted and inspected, make sure everyone understands the potential performance impact. A little proactive communication with key stakeholders is essential for success.

Monitoring and Forensics

Improving incident response in the face of more sophisticated attacks requires that we monitor and capture more traffic for alerting and forensics. This use case differs signficantly from enforcing security policies because in this case you aren’t quickly re-encrypting or discarding data. The entire point is to either derive metadata from the packet stream, or actually capture packets for subsequent investigations.

When you decrypt to enforce a security policy the data may be unencrypted for a few seconds. But when you decrypt for monitoring and forensics, it may remain unencrypted indefinitely. You need to be sensitive to this change, and far more careful and stringent about how and how long you keep that unencrypted data.

Above we talked about kinds of applications and other attributes that might trigger or prevent decryption; use the same approach to define policies for monitoring and forensics. You also need a risk analysis on pretty much every policy, determining whether its traffic could contain sensitive information (either corporate or personal) and the risk of capturing it.

For example you might want to decrypt traffic from HR and send it to the IPS to check for attacks – HR tends to be a frequent phishing target. But you might avoid sending that decrypted stream to your packet capture device because sensitive personnel information could be captured, posing an unacceptable risk. There are no any simple right or wrong answers about what to decrypt or not – you need to ask the right questions when setting up policies.

This is a non-optimal situation for security; it doesn’t reconcile well with our general approach of capturing as much as we can and keeping it as long as we can afford. Unfortunately privacy issues, described in more detail below, force you to make tough choices about what you can decrypt for monitoring, and how long you can keep it. But even in places where packet capture is a no-go you will still want to decrypt and pull session metadata (source, destination, protocol, amount of data, application, etc.) to glean patterns for analysis by your SIEM (or other security analysis capability). The caveat is that in some geographies you cannot even do that basic traffic inspection, and adherence to local laws usually trumps corporate security policy.

You may be able to allay fears about capturing encrypted traffic by highlighting the security of the capture environment. If captured data is stored in a purpose-built device with proper access control and authorization protections, and the data is protected at rest somehow… that may comfort key influencers about decryption policies.

HR & Compliance Issues

As we have alluded, there are non-trivial privacy and compliance ramifications to decryption policy, and at some point (earlier rather than later) organizational lawyers should look at the policies. Here are a couple issues to consider:

1. Regional variations: If you do business is a geography that is very privacy-sensitive you are better off getting the local team engaged as soon as possible – especially in Europe. You will likely need to work with local authorities and/or worker councils to make sure your policies don’t violate local laws. There are locales that specifically restrict website impersonation: decryption of traffic using an intermediate certificate in a man-in-the-middle implementation. There is also a great deal of sensitivity about employee surveillance, so consider local attitudes to that as well.
2. Whitelisting: To address some of these issues you should consider whitelisting certain categories of applications and/or websites – typically healthcare and financial companies. This ensures you do not decrypt sites with a high likelihood of intercepting sensitive or protected information. Of course this is well known by attackers, who may use compromised servers within white-listed categories to evade decryption.
3. Culture: Some organizations walk a fine line in terms of pushing for more monitoring/security. If you are in such an environment you may be able to decrypt more. Others are very conservative, and will not monitor if there is even a slight chance employees could feel alienated or violated. That’s why working with senior management and legal counsel is so important to define policies. Avoid making the final call on what to decrypt and what to bypass unilaterally.
4. Documentation: Even if everyone signs off on the policies, they may have some issues later on. So make sure you can substantiate exactly what gets decrypted and when through solid documentation. Also make sure you can prove who has access to the data (especially when keeping data for monitoring and forensics), and the workflows for who access what during investigation. Do this by leveraging logs from decryption gear, and keep reports of the policies and response process workflows to prove what you are doing – and aren’t. Better to have too much documentation than ask worker councils to just trust you.

As usual, what should be a fairly straightforward technical discussion of what to decrypt and how devolves into a squishy reality of policies and privacy mandates. That is the game we have to play as security folks, and it won’t help to complain about it. Build your policies using an inclusive process to develop and update what gets deployed according to what is acceptable in your organization. Make sure to document everything because at some point you will be asked why that data or app or website was decrypted, and you will need answers and evidence.

Our next post will get back into our comfort zone, digging into technical details of how to select a decryption device and where to deploy it.