In the first post updating our research on Network-based Malware Detection, we talked about how attackers have evolved their tactics, even over the last 18 months, to defeat emerging controls like sandboxing and command & control (C&C) network analysis. As attackers get more sophisticated defenses need to as well. So we are focusing this series on tracking the evolution of malware detection capabilities and addressing issues with early NBMD offerings – including scaling, accuracy, and deployment. But first we need to revisit how the technology works. For more detail on the technology you can always refer back to the original Network-based Malware Detection paper.
Looking for Bad Behavior

Over the past few years malware detection has moved from file signature matching to isolating behavioral characteristics. Given the ineffectiveness of blacklist detection the ability to identify malware behaviors has become increasingly important. We can no longer judge malware by what it looks like – we need to actually analyze what a file does to determine whether it’s malicious. We discussed this behavioral analysis in Evolving Endpoint Malware Detection, focusing on how new approaches have added contextual determination to make the technology far more effective.

You can read our original paper for full descriptions of these kinds of tells that usually mean a device is compromised; but a simple list includes memory corruption/injection/buffer overflows; system file/configuration/registry changes; droppers, downloaders, and other unexpected programs installing code; turning off existing anti-malware protections; and identity and privilege manipulation. Of course this list isn’t comprehensive – it’s just a quick set of guidelines for kinds of information you can search devices for, when you are on the hunt for possible compromises. Other things you might look for include parent/child process inconsistencies, exploits disguised as patches, keyloggers, and screen grabbing. Of course these behaviors aren’t necessarily bad – that’s why you want to investigate as possible, before any outbreak has a chance to spread.

The innovation in the first generation of NBMD devices was running this analysis on a device in the perimeter. Early devices implemented a virtual farm of vulnerable devices in a 19-inch rack. This enabled them to explode malware within a sandbox, and then to monitor for the suspicious behaviors described. Depending on the deployment model (inline or out of band), the device either fired an alert or could actually block the file from reaching its target. It turns out the term sandbox is increasingly unpopular amongst security marketers for some unknown reason, but that’s what they use – a protected and monitored execution environment for risk determination. Later in this series we will discuss different options for ensuring the sandbox can to your needs.
Tracking the C&C Malware Factory

The other aspect of network-based malware detection is identifying egress network traffic that shows patterns typical of communication between compromised devices and controlling entities. Advanced attacks start by compromising and gaining control of a device. Then it establishes contact with its command and control infrastructure to fetch a download with specific attack code, and instructions on what to attack and when. In Network-based Threat Intelligence we dug deep into the kinds of indicators you can look for to identify malicious activity on the network, such as:

Destination: You can track the destinations of all network requests from your environment, and compare it against a list of known bad places. This requires an IP reputation capability – basically a list of known bad IP addresses. Of course IP reputation can be gamed, so combining it with DNS analysis to identify likely Domain Generation Algorithms (DGA) helps to eliminate false positives.
Strange times: If you have a significant volume of traffic which is out of character for that specific device or time – such as the marketing group suddenly performing SQL queries against engineering databases – it’s time to investigate.
File types, contents, and protocols: You can also learn a lot by monitoring all egress traffic, looking for large file transfers, non-standard protocols (encapsulated in HTTP or HTTPS), weird encryption of the files, or anything else that seems a bit off… These anomalies don’t necessarily mean compromise, but they warrant further investigation.
User profiling: Beyond the traffic analysis described above, it is helpful to profile users and identify which applications they use and when. This kind of application awareness can identify anomalous activity on devices and give you a place to start investigating.

Layers FTW

We focus on network-based malware detection in this series, but we cannot afford to forget endpoints. NBMD gateways miss stuff. Hopefully not a lot, but it would be naive to believe you can keep computing devices (endpoints or servers) clean. You still need some protection on your endpoints, but at least you should have controls that work together to ensure you have full protection, when the device is on the corporate network and when it is not.

This is where threat intelligence plays a role, making both network and endpoint malware detection capabilities smarter. You want bi-directional communication so malware indicators found by the network device or in the cloud are accessible to endpoint agents. Additionally, you want malware identified on devices to be sent to the network for further analysis, profiling, determination, and ultimately distribution of indicators to other protected devices.

This wisdom of crowds is key to fighting advanced malware. You may be one of the few, the lucky, and the targeted. No, it’s not a new soap opera – it just means you will see interesting malware attacks first. You’ll catch some and miss others – and by the time you clean up the mess you will probably know a lot about what the malware does, how, and how to detect it. Exercising good corporate karma, you will have the opportunity help other companies by sharing what you found, even if you remain anonymous. If you aren’t a high-profile target this information sharing model works even better, allowing you to benefit from the misfortune of the targeted.

The goal is to increase your chance of catching the malware before it wreaks havoc. Or at least shortly thereafter – which requires a coordinated effort on the network and on devices, leveraging a threat intelligence capability to make the industry as a whole smarter. Now that you understand how these devices detect malware on the network, it is time to push forward and address the scalability issues of first-generation of NBMD devices. We will do that next week.