As we discussed previously, despite all the cool innovation happening to effectively prevent compromises on endpoints, the fact remains that you cannot stop all attacks. That means detecting the compromise quickly and effectively, and then figuring out how far the attack has spread within your organization, continues to be critical.

The fact is, until fairly recently endpoint detection and forensics was a black art. Commercial endpoint detection tools were basically black boxes, not really providing visibility to security professionals. And the complexity of purpose-built forensics tools put this capability beyond the reach of most security practitioners. But a new generation of endpoint detection and response (EDR) tools is now available, with much better visibility and more granular telemetry, along with a streamlined user experience to facilitate investigations – regardless of analyst capabilities.

Of course it is better to have a more-skilled analyst than a less-skilled one, but given the hard truth of the security skills gap, our industry needs to provide better tools to make those less-skilled analysts more productive, faster. Now let’s dig into some key aspects of EDR.

Telemetry/Data Capture

In order to perfrom any kind of detection, you need telemetry from endpoints. This begs the question of how much to collect from each device, and how long to keep it. This borders on religion, but we remain firmly in the camp that more data is better than less. Some tools can provide a literal playback of activity on the endpoint, like a DVR recording of everything that happened. Others focus on log events and other metadata to understand endpoint activity.

You need to decide whether to pull data from the kernel or from user space, or both. Again, we advocate for data, and there are definite advantages to pulling data from the kernel. Of course there are downsides as well, including potential device instability from kernel interference.

Again recommend the risk-centric view on protecting endpoints, as discussed in our prevention post. Some devices possess very sensitive information, and you should collect as much telemetry as possible. Other devices present less risk to the enterprise, and may only warrant log aggregation and periodic scans.

There are also competing ideas about where to store the telemetry captured from all these endpoint devices. Some technologies are based upon aggregating the data in an on-premise repository, others perform real-time searches using peer-to-peer technology, and a new model involves sending the data to a cloud-based repository for larger scale-analysis.

Again, we don’t get religious about any specific approach. Stay focused on the problem you are trying to solve. Depending on the organization’s sensitivity, storing endpoint data in the cloud may not be politically feasible. On the other hand it might be very expensive to centralize data in a highly distributed organization. So the choice of technology comes down to the adversary’s sophistication, along with the types and locations of devices to be protected.

Threat Intel

It’s not like threat intelligence is a new concept in the endpoint protection space. AV signatures are a form of threat intel – the industry just never calls it that. What’s different is that now threat intelligence goes far beyond just hashes of known bad files, additionally looking for behavioral patterns that indicate an exploit. Whether the patterns are called Indicators of Compromise (IoC), Indicators or Attack (IoA), or something else, endpoints can watch for them in real time to detect and identify attacks.

This new generation of threat intelligence is clearly more robust than yesterday’s signatures. But that understates the impact of threat intel on EDR. These new tools provide retrospection, which is searching the endpoint telemetry data store for newly emerging attack patterns. This allows you to see if a new attack has been seen in the recent past on your devices, before you even knew it was an attack.

The goal of detection/forensics is to shorten the window between compromise and detection. If you can search for indicators when you learn about them (regardless of when the attack happens), you may be able to find compromised devices before they start behaving badly, and presumably trigger other network-based detection tactics.

A key aspect of selecting any kind of advanced endpoint protection product is to ensure the vendor’s research team is well staffed and capable of keeping up with the pace of emerging attacks. The more effective the security research team is, the more emerging attacks you will be able to look for before an adversary can compromise your devices. This is the true power of threat intelligence.

Analytics

Once you have all of the data gathered and have enriched it with external threat intelligence, you are ready to look for patterns that may indicate compromised devices. Analytics is now a very shiny term in security circles, which we find very amusing. Early SIEM products offered analytics – you just needed to tell them what to look for. And it’s not like math is a novel concept for detecting security attacks. But security marketers are going to market, so notwithstanding the particular vernacular, more sophisticated analytics do enable more effective detection of sophisticated attacks today.

But what does that even mean? First we should define probably the term machine learning, because every company claims they do this to find zero-day attacks and all other badness with no false positives or latency. No, we don’t believe that hype. But the advance of analytical techniques, harnessed by math ninja known as data scientists, enables detailed analysis of every attack to find commonalities and patterns. These patterns can then be used to find malicious code or behavior in new suspicious files. Basically security research teams sets up their math machines to learn about these patterns. Ergo machine learning. Meh.

The upshot is that these patterns can be leveraged for both static analysis (what the file looks like) and dynamic analysis (what the software does), making detection faster and more accurate.

Response

Once you have detected a potentially compromised devices you need to engage your response process. We have written extensively about incident response (including Using TI in Incident Response and Incident Response in the Cloud Age), so we won’t go through the details of the IR process again here. Though as we have described, advanced endpoint protection tools now provide both more granular telemetry, and a way to investigate an attack within the management console.

Additionally, these tools increasingly integrate with other response tools in use within your environment. Advanced endpoint protection products bring several capabilities to response, including:

1. Attack Visualization: In many cases, being able to visualize the attack on a device is very instructive for understanding how the malware works and what it does to devices. The management consoles of some EAP products offer a visual map to follow the activity of malware on a device – including the process the attack impacted, kernel-level activity, and/or API calls. This timeline of sorts must also specify the files involved in the attack and track network connectivity.
2. Understanding Outbreaks: As discussed above, a key aspect of EAP products is their ability to aggregate telemetry and search after the fact to determine if other devices have been attacked by similar malware. This provides invaluable insight into how the attack has proliferated through your environment, and identifies specific devices in need of remediation or quarantine.
3. Forensics: You also need the endpoint agent to be able to gather raw telemetry from the device and provide tools to analyze the data. At times, especially when skilled forensicators are involved, they need full data to really dig into what the malware did. A key aspect of forensic analysis is the need to enforce chain of custody for collected data, especially if prosecution is an option.
4. Ease of Use: EAP tools have been built for more general security practitioners, rather than only forensics ninja, so user experience has been a focus for helping less experienced professionals be more productive. This requires a much easier workflow for drilling down into attacks, and pivoting to find the root cause.
5. Integration with Enterprise Tools: Another key criteria for EAP products is making sure they play nice with tools already in use. You’ll want to be able to send data directly to a SIEM for further correlation and analysis. You’ll also want to integrate with a case management system to track investigations. Finally, think about integrations with network security controls (including firewalls and web filters) to block C&C sites and other malicious addresses discovered on endpoints, preventing other devices from contacting known-bad Internet addresses.

Hunting

Finally we should acknowledge another very shiny concept in security circles: hunting. It seems every practitioner aspires to be a hunter nowadays. OK, maybe that’s a little exaggerated, but it’s a cool gig. Hunters go out and proactively look for adversary activity on networks and systems, as opposed to waiting for monitors to alert, and then investigating.

Psychologically, hunting is great for security teams because it puts the team more in control of their environment. Instead of waiting for a tool to tell you things are bad, you can go out and figure it out yourself.

But the reality is that hunting is primarily relevant to the most sophisticated and advanced security teams. It requires staff to look around, and unfortunately most organizations are not sufficiently staffed to achieve core operational goals, so there isn’t much chance they have folks sitting around, available to proactively look for bad stuff.

Keep in mind the tools used by hunters are largely the same ones useful to practitioners focused on validating attacks on endpoints. A hunter needs to be able to analyze granular telemetry from endpoints and other devices. They need to search through telemetry to find activity patterns that could be malicious. They need to forensically investigate a device when they find something suspicious. Hunters also need to retrospectively look for indicators of attack to understand which devices have been targeted. Pretty much what EDR tools do.

To be clear, we aren’t maligning hunting at all. If your organization can devote the resources to stand up a hunting function, that’s awesome. Our point is simply that the tools needed to hunt are pretty much the same tools used by responders to verify alerts.

That’s detection and response as part of an Endpoint Advanced Protection lifecycle. Our next post will wrap up with the sticky questions that need to be answered – including remediation once you find a compromised device, whether an EAP product can replace your existing AV, and how to integrate these tools with existing network and security management controls.