Research

The first post in this series, Behind the 8 Ball, raised a number of key challenges practicing security in our current environment. These include continual advancement and innovation by attackers seeking new ways to compromise devices and exfiltrate data, increasing complexity of technology infrastructure, frequent changes to said infrastructure, and finally the systemic skills shortage which limits our resources available to handle all the challenges created by the other issues. Basically, practitioners are behind the 8-ball in getting their job done and protecting corporate data. As we discussed in that earlier post, thinking differently about security entails you changing things up to take a (dare we say it?) more enlightened approach, basically focusing the right resources on the right functions. We know it seems obvious that having expensive staff focused on rote and tedious functions is a suboptimal way to deploy resources. But most organizations do it anyway. We prefer to have our valuable, constrained, and usually highly skilled humans doing what humans are good at, such as: identifying triggers that might indicate malicious activity drilling into suspicious activity to understand the depth of attacks and assess potential damage figuring out workarounds to address attacks Humans in these roles generally know what to look for, but aren’t very good at looking at huge amounts of data to find those patterns. Many don’t like doing the same things over and over again – they get bored and less effective. They don’t like graveyard shifts, and they want work that teaches them new things and stretches their capabilities. Basically they want to work in an environment where they do cool stuff and can grow their skills. And (especially in security) they can choose where they work. If they don’t get the right opportunity in your organization, they will find another which better suits their capabilities and work style. On the other hand machines have no problem working 24/7 and don’t complain about boring tasks – at least not yet. They don’t threaten to find another place to work, nor do they agitate for broader job responsibilities or better refreshments in the break room. We’re being a bit facetious here, and certainly don’t advocate replacing your security team with robots. But in today’s asymmetric environment, where you can’t keep up with the task list, robots may be your only chance to regain balance and keep pace. So we will expand a bit on a couple concepts from our Intro to Threat Operations paper, because over time we expect our vision of threat operations to become a subset of SecOps. Enriching Alerts: The idea is to take an alert and add a bunch of common information you know an analyst will want to the alert, before to sending it to an analyst. This way the analyst doesn’t need to spend time gathering information from those various systems and information sources, and can get right to work validating the alert and determining potential impact. Incident Response: Once an alert has been validated, a standard set of activities are generally part of response. Some of these activities can be automated via integration with affected systems (networks, endpoint management, SaaS, etc.) and the time saved enables responders to focus on higher-level tasks such as determining proliferation and assessing data loss. Enriching Alerts Let’s dig into enriching alerts from your security monitoring systems, and how this can work without human intervention. We start with a couple different alerts, and some educated guesses as to what would be useful to an analyst. Alert: Connection to a known bad IP: Let’s say an alert fires for connectivity to a known bad IP address (thanks, threat intel!). With source and destination addresses, an analyst would typically start gathering basic information. 1. Identity: Who uses the device? With a source IP it’s usually straightforward to see who the address is allocated to, and then what devices that person tends to use. Target: Using a destination IP external site comes into focus. An analyst would probably perform geo-location to figure out where the IP is and a whois query to figure out who owns it. They could also figure out the hosting provider and search their threat intel service to see if the IP belongs to a known botnet, and dig up any associated tactics. Network traffic: The analyst may also check out network traffic from the device to look for strange patterns (possibly C&C or reconnaissance) or uncharacteristically large volumes to or from that device over the past few days. Device hygiene: The analyst also needs to know specifics about the device, such as when it was last patched and does it have a non-standard configuration? Recent changes: The analyst would probably be interested in software running on the device, and whether any programs have been installed or configurations changed recently. Alert: Strange registry activity: In this scenario an alert is triggered because a device has had its registry changed, but it cannot be traced back to authorized patches or software installs. The analyst could use similar information to the first example, but device hygiene and recent device changes would be of particular interest. The general flow of network traffic would also be of interest, given that the device may have been receiving instructions or configuration changes from external devices. In isolation registry changes may not be a concern, but in close proximity of a larger inbound data transfer the odds of trouble increase. Additionally, checking out web traffic logs from the device could provide clues to what they were doing that might have resulted in compromise. Alert: Large USB file transfer: We can also see the impact of enrichment in an insider threat scenario. Maybe an insider used their USB port for the first time recently, and transferred 1GB of data in a 3-hour window. That could generate a DLP alert. At that point it would be good to know which internal data sources the device has been communicating with, and any anomalous data volumes over the past few days, which

There are plenty of obvious questions you could ask an endpoint security vendor. But most won’t really help you understand the nuances of their approach, so we decided to distill the selection criteria down to a couple of key points. We’ll provide not just the questions, but the rationale behind them. Q1: Where do you draw the line between prevention and EDR? The clear trend is towards an integrated advanced endpoint protection capability addressing prevention, detection, response, and hunting. That said, it may not be the right answer for any specific organization, depending on the adversaries they face and the sophistication & capabilities of their internal team. As discussed under selection criteria for Prevention, simple EDR (EDR-lite) is already bundled into a few advanced prevention products, accelerating this integration and emphasizing the importance of deciding whether the organization needs separate tools for prevention and detection/response/hunting. Q2: How does your product track a campaign, as opposed to just looking for attacks on single endpoints? Modern attacks rarely focus on just one endpoint – they tend to compromise multiple devices as the adversary advances towards their objective. To detect and respond to such modern attacks, analysis needs to look not merely at what’s happening on a single endpoint, but also at how that endpoint is interacting with the rest of the environment – looking for broader indications of reconnaissance, lateral movement, and exfiltration. Q3: Is detection based on machine learning? Does your analysis leverage the cloud? How do your machine learning models handle false positives? Advanced analytics are not the only way to detect attacks, but they are certainly among the key techniques. This question addresses the vendor’s approach to machine learning, digs into where they perform analysis, and gets at the breadth of the data they use to train ML models. Finally, you want the vendor to pass a sniff test on false positives. If any vendor claims they don’t have false positives, run away fast. Q4: Does your endpoint agent work in user or kernel mode? What kind of a performance impact does your agent have on devices? The answer is typically ‘both’ because certain activities that cannot be monitored or prevented purely from user space or kernel mode. For monitoring and EDR, it’s possible to stay within user mode, but that limits automated remediation capability because some attacks need to be dealt with at the kernel level. Of course, with many agents already in use on typical endpoints, when considering adding another for EDR you will want to understand the performance characteristics of the new agent. Q5: Do we need “Full DVR”, or is collecting endpoint metadata sufficient? This question should reveal the vendor’s response religion – some believe comprehensive detection and/or response can work using only metadata from granular endpoint telemetry, while others insist that a full capture of all endpoint activity is necessary to effectively respond and to hunt for malicious activity. The truth is somewhere in the middle, depending on your key use case. Detection-centric environments can run well on metadata, but if response/hunting is your driving EDR function, access to full devie telemetry is more important because attackers tend to cover their tracks using self-deleting files and other techniques to obfuscate their activities. Keep in mind that the EDR architecture is a major factor here, as central analysis of metadata can provide excellent detection, with full telemetry stored temporarily on each device in case it is needed for response. Q6: How is threat intelligence integrated into your agent? This anser should be about more than getting patterns for the latest indicators of compromise and patterns for attacks involving multiple devices. Integrated threat intel provides the ability to search historical telemetry for attacks you didn’t recognize as attacks at the time (retrospective search). You should also be able to share intelligence with a community of similar organizations, and be able to integrate first-party intel from your vendor with third-party intel from threat intelligence vendors when appropriate. Additionally, the able to send unrecognized files to a network sandbox makes the system more effective and enables quicker recognition of emerging attacks. Q7: How does your product support searching endpoint telemetry for our SOC analysts? Can potentially compromised devices be polled in real time? What about searching through endpoint telemetry history? Search is king for EDR tools, so spend some time with the vendor to understand their search interface and how it can be used to drill down into specific devices or pivot to other devices, to understand which devices an attacker has impacted. You’ll also want to see their search responsiveness, especially with data from potentially hundreds of thousands of endpoints in the system. This is another opportunity to delve into retrospective search capabilities – key for finding malicious activity, especially when you don’t recognize it as bad when it occurs. Also consider the tradeoffs between retention of telemetry and the cost of storing it, because being able to search a longer history window makes both retrospective search and hunting more effective. Q8: Once I get an alert, does the product provide a structured response process? What kind of automation is possible with your product? What about case management? As we have discussed throughout this series, the security skills gap makes it critical to streamline the validation and response processes for less sophisticated analysts. The more structured a tool can make the user experience, the more it can help junior analysts be more productive, faster. That said, you also want to make sure the tool isn’t so structured that analysts have no flexibility to follow their instincts and investigate the attack a different way. Q9: My staff aren’t security ninjas, but I would like to proactively look for attackers. How does your product accelerate a hunt, especially for unsophisticated analysts? Given sufficiently capable search and visualization of endpoint activity, advanced threat hunters can leverage an EDR tool for hunting. Again, you’ll want to learn how the tool can make your less experienced folks more productive and enable them

Now let’s dig into some key EDR technologies which appear across all the use cases: detection, response, and hunting. Agent The agent is deployed to each monitored endpoint, so you be sensitive to its size and its performance hit on devices. A main complaint regarding older endpoint protection was performance impact on devices. The smaller the better, and the less performance impact the better (duh!), but just as important is agent deployability and maintainability. Full capture versus metadata: There are differing strong opinions on how much telemetry to capture and store from each device. Similar to the question of whether to do full network packet capture or to derive metadata from the packet stream, there is a level of granularity available with a full endpoint capture which isn’t available via metadata, for session reconstruction and more detail about what an adversary actually did. But full capture is very resource and storage intensive, so depending on the sophistication of your response team and process, metadata may be your best option. Also consider products that can gather more device telemetry when triggered, perhaps by an alert or connection to a suspicious network. Offline collection: Mobile endpoints are not always on the network, so agents much be able to continue collecting event data when disconnected. Once back on the network, cached endpoint telemetry should be uploaded to the central repository, which can then perform aggregate analysis. Multi-platform support: It’s a multi-platform world, and your endpoint security strategy needs to factor in not just Windows devices, but also Macs and Linux. Even if these platforms aren’t targeted they could be used in sophisticated operations as repositories, staging grounds, and file stores. Different operating systems offer different levels of telemetry access. Security vendors have less access to the kernel on both Mac and Linux systems than on Windows. Also dig into how vendors leverage built-in operating system services to provide sufficiently granular data for analysis. Finally, mobile devices access and store critical enterprise data, although their vulnerability is still subject to debate. We do not consider mobile devices as part of these selection criteria, although for many organizations an integrated capability is an advantage. Kernel vs. user space: There is a semi-religious battle over whether a detection agent needs to live at the kernel level (with all the potential device instability risks that entails), or accurate detection can take place exclusively at the kernel level. Any agent must be able to detect attacks at lower levels of the operating system – such as root kits – as well as any attempts at agent tampering (again, likely outside user space). Again, we don’t get religious, and we appreciate that user-space agents are less disruptive, but are not willing to compromise on detecting all attacks. Tamper proof: Speaking of tampering, to address another long standing issue with traditional EPP, you’ll want to dig into the product security of any agent you install on any endpoint in your environment. We can still remember the awesome Black Hat talks where EPP agent after EPP agent was shown to be more vulnerable than some enterprise applications. Let’s learn from those mistakes and dig into the security and resilience of the detection agents to make sure you aren’t just adding attack surface. Scalability: Finally, scale is a key consideration for any enterprise. You might have 1,000 or 100,000 devices, or even more; but regardless you need to ensure the tool will work for the number of endpoints you need to support, and the staff on your team – both in terms of numbers and sophistication. Of course you need to handle deployment and management of agents, but don’t forget the scalability and responsiveness of analysis and searching. Machine Learning Machine learning is a catch-all term which endpoint detection/response vendors use for sophisticated mathematical analysis across a large dataset to generate models, intended to detect malicious device activity. Many aspects of advanced mathematics are directly relevant to detection and response. Static file analysis: With upwards of a billion malicious file samples in circulation, mathematical malware analysis can pinpoint commonalities across malicious files. With a model of what malware looks like, detection offerings can then search for these attributes to identify ‘new’ malware. False positives are always a concern with static analysis, so part of diligence is ensuring the models are tested constantly, and static analysis should only be one part of malware detection. Behavioral profiles: Similarly, behaviors of malware can be analyzed and profiled using machine learning. Malware profiling produces a dynamic model which can be used to look for malicious behavior. Those are the main use cases for machine learning in malware detection, but there are a number of considerations when evaluating machine learning approaches, including: Targeted attacks: With an increasing amount of attacks specifically targeting individual organizations, it is again important to distinguish delivery from compromise. Targeted attacks use custom (and personalized) methods to deliver attacks – which may or may not involve custom malware – but once the attacker has access to a device they use similar tactics to a traditional malware attack, so machine learning models don’t necessarily need to do anything unusual to deal with targeted attacks. Cloud analytics: The analytics required to develop malware machine learning models are very computationally intensive. Cloud computing is the most flexible way to access that kind of compute power, so it makes sense that most vendors perform their number crunching and modeling in the cloud. Of course the models must be able to run on endpoints to detect malicious activity, so they are typically built in the cloud and executed locally on every endpoint. But don’t get distracted with where computation happens, so long as performance and accuracy are acceptable. Sample sizes: Some vendors claim that their intel is better than another company’s. That’s a hard claim to prove, but sample sizes matter. Looking at a billion malware samples is better than looking at 10,000. Is there a difference between looking at a hundred million samples and at a billion?

As we resume posting Endpoint Detection and Response (D/R) selection criteria, let’s start with a focus on the Detection use case. Before we get too far into capabilities, we should clear up some semantics about the word ‘detection’. Referring back to our timeline in Prevention Selection Criteria, detection takes place during execution. You could make the case that detection of malicious activity is what triggers blocking, and so a pre-requisite to attack prevention – without detection, how could you know what to prevent?. But that’s too confusing. For simplicity let’s just say prevention means blocking an attack before it compromises a device, and can happen both prior to and during execution. Detection happens during and after execution, and implies a device was compromised because the attack was not prevented. Data Collection Modern detection requires significant analysis across a wide variety of telemetry sources from endpoints. Once telemetry is captured, a baseline of normal endpoint activity is established and used to look for anomalous behavior. Given the data-centric nature of endpoint detection, an advanced endpoint detection offering should aggregate and analyze the following types of data: Endpoint logs: Endpoints can generate a huge number of log entries, and an obvious reaction is to restrict the amount of log data ingested, but we recommend collecting as much log data from endpoint as possible. The more granular the better, given the sophistication of attackers and their ability to target anything on a device. If you do not collect the data on the endpoint, there is no way to get it when you need it to investigate later. Optimally, endpoint agents collect operating system activity alongside all available application logs. This includes identity activity such as new account creation and privilege escalation, process launching, and file system activity (key for detection ransomware). There is some nuance to how long you retain collected data because it can be voluminous and compute-intensive to process and analyze – both on devices and centrally. Processes: One of the more reliable ways to detect malicious behavior is by which OS processes are started and where they are launched from. This is especially critical when detecting scripting attacks because attackers love using legitimate system processes to launch malicious child processes. Network traffic: A compromised endpoint will inevitably connect to a command and control network for instructions and to download additional attack code. These actions can be detected by monitoring the endpoint’s network stack. An agent can also watch for connections to known malicious sites and for reconnaisance activity on the local network. Memory: Modern file-less attacks don’t store any malicious code in the file system, so modern advanced detection requires monitoring and analyzing activity within endpoint memory. Registry: As with memory-based attacks, attackers frequently store malicious code within the device registry to evade file system detection. So advanced detection agents need to monitor and analyze registry activity for signs of misuse. Configuration changes: It’s hard for attackers to totally obfuscate what is happening on an endpoint, so on-device configuration changes can indicate an attack. File integrity: Another long-standing method attack detection is monitoring changes to system files, because changes to such files outside administrative patching usually indicates something malicious. An advanced endpoint agent should collect this data and monitor for modified system files. Analytics As mentioned above, traditional endpoint detection relied heavily on simple file hashes and behavioral indicators. With today’s more sophisticated attacks, a more robust and scientific approach is required to distinguish legitimate from malicious activity. This more scientific approach is centered around machine learning techniques (advanced mathematics) to recognize the activity of adversaries before and during attacks. Modern detection products use huge amounts of endpoint telemetry (terabytes) to train mathematical models to detect anomalous activity and find commonalities between how attackers behave. These models then generate an attack score to prioritize alerts. Profiling applications: Detecting application misuse is predicated on understanding legitimate usage of the application, so the mathematical models analyze both legitimate and malicious usage of frequently targeted applications (browsers, office productivity suites, email clients, etc.). This is a similar approach to attack prevention, discussed in our Prevention Selection Criteria guide. Anomaly detection: With profiles in hand and a consistent stream of endpoint telemetry to analyze, the mathematical models attempt to identify abnormal device activity. When suspicion is high they trigger an alert, the device is marked suspicious, and an analyst determines whether the alert is legitimate. Tuning: To avoid wasting too much time on false positives, the detection function needs to constantly learn what is really an attack and what isn’t, based on the results of detection in your environment. In terms of process, you’ll want to ensure your feedback is captured by your detection offering, and used to constantly improve your models to keep detection precise and current. Risk scoring: We aren’t big fans of arbitrary risk scoring because the underlying math can be suspect. That said, there is a role for risk scoring in endpoint attack detection: prioritization. With dozens of alerts hitting daily – perhaps significantly more – it is important to weigh which alerts warrant immediate investigation, and a risk score should be able to tell you. Be sure to investigate the underlying scoring methodology, track scoring accuracy, and tune scoring to your environment. Data management: Given the analytics-centric nature of EDR, being able to handle and analyze large amounts of endpoint telemetry collected from endpoints is critical. Inevitably you’ll run into the big questions: where to store all the data, how to scale analytics to tens or hundreds of thousands of endpoints, and how to economically analyze all your security data. But ultimately your technology decision comes down to a few factors: Cost: Whether or not the cost of storage and analytics is included in the service (some vendors store all telemetry in a cloud instance) or you need to provision a compute cluster in your data center to perform the analysis, there is a cost to crunching all the numbers. Make sure hardware, storage, and networking costs (including management)

As we resume posting Endpoint Detection and Response (D/R) selection criteria, let’s start with a focus on the Detection use case. Before we get too far into capabilities, we should clear up some semantics about the word ‘detection’. Referring back to our timeline in Prevention Selection Criteria, detection takes place during execution. You could make the case that detection of malicious activity is what triggers blocking, and so a pre-requisite to attack prevention – without detection, how could you know what to prevent?. But that’s too confusing. For simplicity let’s just say prevention means blocking an attack before it compromises a device, and can happen both prior to and during execution. Detection happens during and after execution, and implies a device was compromised because the attack was not prevented. Data Collection Modern detection requires significant analysis across a wide variety of telemetry sources from endpoints. Once telemetry is captured, a baseline of normal endpoint activity is established and used to look for anomalous behavior. Given the data-centric nature of endpoint detection, an advanced endpoint detection offering should aggregate and analyze the following types of data: Endpoint logs: Endpoints can generate a huge number of log entries, and an obvious reaction is to restrict the amount of log data ingested, but we recommend collecting as much log data from endpoint as possible. The more granular the better, given the sophistication of attackers and their ability to target anything on a device. If you do not collect the data on the endpoint, there is no way to get it when you need it to investigate later. Optimally, endpoint agents collect operating system activity alongside all available application logs. This includes identity activity such as new account creation and privilege escalation, process launching, and file system activity (key for detection ransomware). There is some nuance to how long you retain collected data because it can be voluminous and compute-intensive to process and analyze – both on devices and centrally. Processes: One of the more reliable ways to detect malicious behavior is by which OS processes are started and where they are launched from. This is especially critical when detecting scripting attacks because attackers love using legitimate system processes to launch malicious child processes. Network traffic: A compromised endpoint will inevitably connect to a command and control network for instructions and to download additional attack code. These actions can be detected by monitoring the endpoint’s network stack. An agent can also watch for connections to known malicious sites and for reconnaisance activity on the local network. Memory: Modern file-less attacks don’t store any malicious code in the file system, so modern advanced detection requires monitoring and analyzing activity within endpoint memory. Registry: As with memory-based attacks, attackers frequently store malicious code within the device registry to evade file system detection. So advanced detection agents need to monitor and analyze registry activity for signs of misuse. Configuration changes: It’s hard for attackers to totally obfuscate what is happening on an endpoint, so on-device configuration changes can indicate an attack. File integrity: Another long-standing method attack detection is monitoring changes to system files, because changes to such files outside administrative patching usually indicates something malicious. An advanced endpoint agent should collect this data and monitor for modified system files. Analytics As mentioned above, traditional endpoint detection relied heavily on simple file hashes and behavioral indicators. With today’s more sophisticated attacks, a more robust and scientific approach is required to distinguish legitimate from malicious activity. This more scientific approach is centered around machine learning techniques (advanced mathematics) to recognize the activity of adversaries before and during attacks. Modern detection products use huge amounts of endpoint telemetry (terabytes) to train mathematical models to detect anomalous activity and find commonalities between how attackers behave. These models then generate an attack score to prioritize alerts. Profiling applications: Detecting application misuse is predicated on understanding legitimate usage of the application, so the mathematical models analyze both legitimate and malicious usage of frequently targeted applications (browsers, office productivity suites, email clients, etc.). This is a similar approach to attack prevention, discussed in our Prevention Selection Criteria guide. Anomaly detection: With profiles in hand and a consistent stream of endpoint telemetry to analyze, the mathematical models attempt to identify abnormal device activity. When suspicion is high they trigger an alert, the device is marked suspicious, and an analyst determines whether the alert is legitimate. Tuning: To avoid wasting too much time on false positives, the detection function needs to constantly learn what is really an attack and what isn’t, based on the results of detection in your environment. In terms of process, you’ll want to ensure your feedback is captured by your detection offering, and used to constantly improve your models to keep detection precise and current. Risk scoring: We aren’t big fans of arbitrary risk scoring because the underlying math can be suspect. That said, there is a role for risk scoring in endpoint attack detection: prioritization. With dozens of alerts hitting daily – perhaps significantly more – it is important to weigh which alerts warrant immediate investigation, and a risk score should be able to tell you. Be sure to investigate the underlying scoring methodology, track scoring accuracy, and tune scoring to your environment. Data management: Given the analytics-centric nature of EDR, being able to handle and analyze large amounts of endpoint telemetry collected from endpoints is critical. Inevitably you’ll run into the big questions: where to store all the data, how to scale analytics to tens or hundreds of thousands of endpoints, and how to economically analyze all your security data. But ultimately your technology decision comes down to a few factors: Cost: Whether or not the cost of storage and analytics is included in the service (some vendors store all telemetry in a cloud instance) or you need to provision a compute cluster in your data center to perform the analysis, there is a cost to crunching all the numbers. Make sure hardware, storage, and networking costs (including management)

As we continue documenting what you need to know to understand Endpoint Advanced Protection offerings, it’s time to delve into Detection and Response. Remember that before you are ready to pick anything, you need to understand the problem you are trying to solve. Detecting all endpoint attacks within microseconds and without false positives isn’t really achievable. You need to determine the key use cases most important to you, and make an honest assessment of your team and adversaries. Why is this introspection necessary? Nobody ever says they don’t want to detect active attacks and hunt for adversaries. It’s cool and it’s necessary. Nobody wants to be perpetually reacting to attacks. That said, if you don’t have enough staff to work through half the high-priority alerts from your security monitoring systems, how can you find time to proactively hunt for stuff your monitoring systems don’t catch? As another example, your team may consist of a bunch of entry-level security analysts struggling to figure out which events are actual device compromise, and which are false positives. Tasking these less sophisticated folks with advanced memory forensics to identify file-less malware may not be a good use of time. To procure effective advanced Endpoint Detection and Response (EDR) technology, you must match what you buy to your organization’s ability to use it. Of course you should be able to grow into a more advanced program and capability. But don’t pay for an Escalade when a Kia Sportage is what you need today. Over the next 5 days we will explain what you need to know about Detection and Response (D/R) to be an educated buyer of these solutions. We’ll start by helping you understand the key use cases for D/R, and then delve into the important capabilities for each use case, the underlying technologies which make it all work, and finally some key questions to ask vendors to understand their approaches to your problems. Planning for Compromise Before we get into specific use cases, we need to level-set regarding your situation, which we highlighted in our introduction to the Endpoint Advanced Protection Buyer’s Guide. For years there was little innovation in endpoint protection. Even worse, far too many organizations didn’t upgrade to the latest versions of their vendor’s offerings – meaning they were trying to detect 2016 attacks with 2011 technology. Inevitably that didn’t work out well. Now there are better alternatives for prevention, so where does that leave endpoint detection and response? In the same situation it has always been: a necessity. Regardless of how good your endpoint prevention strategy is, it’s not good enough. You will have devices which get compromised. So you must to be in position to detect compromise and respond to it effectively and efficiently. The good news is that in the absence of perfect (and often even effective) prevention options, many organizations have gone down this path, investing in better detection and response. They have been growing network-based detection and centralized security monitoring infrastructure (which drove the wave of security analytics offerings hitting the market now), and these organizations also invested in technologies to gather telemetry from endpoints and make sense of it. To be clear, we have always been able to analyze what happened on an endpoint after an attack, assuming some reasonable logging and a forensic image of the device. There are decent open source tools for advanced forensics, which have always been leveraged by forensicators who charge hundreds of dollars an hour. What you don’t have is enough people to perform that kind of response and forensic analysis. You hardly have enough people to work through your alert queue, right? This is where advanced Endpoint Detection and Response (EDR) tools can add real value to your security program. Facing a significant and critical skills gap, the technology needs to help your less experience folks by structuring their activities and making their next step somewhat intuitive. But if a tool can’t make your people better and faster, then why bother? But all vendors say that, right? They claim their tools find unknown attacks, and don’t create a bunch of makework wasted identifying or confirming false positives. And help you prioritize activities. The magic tools even find attacks before you know they are attacks, bundled with a side of unicorn dust. Our objective with these selection criteria is to make sure you understand how to dig deeper into the true capabilities of these products, and know what is real and what is marketing puffery. Understand whether a vendor understands the entire threat landscape, or is focused on a small subset of high-profile attack vectors, and whether they will be an effective partner as adversaries and their tactics inevitably change. But as we mentioned above, you need to focus your selection process on the problem you need to solve, which comes down to defining your main use cases for EDR. Key Use Cases Let’s be clear about use cases. There are three main functions you need these tools to perform, and quite a bit of overlap with technologies underlying endpoint prevention. Detection: When you are attacked it’s a race against time. Attackers are burrowing deeper into your environment and working toward their goal. The sooner you detect that something is amiss on an endpoint, the better your chance to contain the damage. Today’s challenge is not just detecting an attack on a single endpoint, but instead figuring out the extent of a coordinated campaign against many endpoints and devices within your environment. Response: Once you know you have been attacked you need to respond quickly and efficiently. This use case focuses on providing an analyst the ability to drill down, validate an attack, and determine the extent of the attacker’s actions across all affected device(s), while assessing potential damage. You also need to able to figure out effective workarounds and remediations to instruct the operations team how to prevent further outbreaks of the same attack. Don’t forget the need to make sure evidence is gathered in a way

There are plenty of obvious questions you could ask an endpoint security vendor. But most won’t really help you understand the nuances of their approach, so we decided to distill the selection criteria down to a couple of key points. We’ll provide not just the questions, but the rationale behind them. Q1 If your prevention capabilities rely on machine learning, how and how often are your machine learning models retrained? An explanation here should provide some perspective on the vendor’s approach to math and the ‘half-life’ of their models, which indicates how quickly they believe malware attack behaviors change. Some espouse continuous retraining, while others maintain that very little changes daily, so it’s sufficient to retrain weekly or monthly. You’ll also want to understand whether model updates disrupt the end-user experience with significant downloads, restarts, or other intrusions on normal user activity. Q2 How do your machine learning models factor in false positives and minimize them? If a vendor claims they don’t have false positives, run away – quickly. Customer feedback and awareness of false positives are critical to keeping the products current, so get a sense of how they update their protection, models, and whitelists based on what doesn’t work in the field. Q3 Does your agent work in user or kernel mode? Do you protect the OS kernel? The answer is typically both, because some activities aren’t accessible from either user mode and others from kernel mode. For monitoring and EDR it’s possible to stay in user mode but if attacks need to be blocked there is a requirement for some kind of kernel access. This question enables you to get at how an EDR vendor has developed their offering to provide broader prevention. Q4 Can you block an attack before it loads into memory? If so, how? This digs into the vendor’s ability to block a file-less attack, a critical aspect of stopping advanced attacks. Q5 How do you prevent an attacker from gaining root access to the device? You are trying to understand the exploit prevention/blocking capability of the product; at some point to control the machine, an attacker will need root-level access. Q6 How is threat intelligence integrated into the prevention agent? This answer should be about more than getting patterns for the latest indicators of compromise. It should include the ability to block known bad IP addresses at the network layer, as well as cloud-based sandbox integration. Q7 How often and how large are agent updates? How do you age out old signatures to conserve space? How are updates distributed? Attacks change and machine learning models are imperfect, so agents need to be updated. Especially given that what’s old becomes new again, and care needs to be taken when specific signatures or behavioral models are no longer on endpoint agents. This question enables you to assess how the vendor distributes work between the agents and the cloud. It also gets at whether the vendor has a cloud-native management option, which wouldn’t require an on-premise aggregation point. Q8 How does the product integrate with other enterprise security solutions, including SIEM and/or EDR? If the vendor offers a full EDR capability use this question to figure out whether it’s a common agent between prevention and EDR, and the level of management integration. You can dig a bit into how the endpoint prevention product sends data to/from a SIEM, incident response, and network-based controls. Q9 Does the product support automation? If so, how? Given that you likely don’t have enough people to do what needs to be done, you’ll need to automate some functions as the only way to scale up your security operation. Some tools integrate with automation platforms (typically for incident response), while offering some auto-remediation for common problems (make sure you can control what gets done by the machine and what just sends an alert to a console). Q10 Does the product support application control to lock down some devices? How are exceptions to the white list handled? Can you lock down USB ports? For some devices it’s easier and safer to just lock them down and prevent unauthorized executables from running. This won’t work on all devices but having the option provides flexibility in designing endpoint controls. Likewise, locking down USB ports prevents a common mechanism of data leakage. We could ask another couple hundred questions, but these ten should provide a lot of the insight you need to differentiate between vendors. Next week we’ll post a similar set of criteria for Detection/Response. Enjoy the weekend! Share:

Every year, as I travel the security conference circuit, hallway conversations always turn to, “See anything interesting?”. To be honest, I can’t remember the last time I was excited about an honestly cool security technology (which I didn’t create myself, but let’s not go there today). I see plenty of cloud innovation, and plenty of security evolution, but not a lot of revolution. A week ago I picked up my iPhone X. Although I received a background brief on Face ID a couple weeks earlier, I hadn’t gotten my hands on it until then. And, really, didn’t get to play with it until the next day after spending 5 hours restoring my data (all 200 GB of it). Face ID is the most compelling security advance I have seen in a very long time. It’s game-changing not merely due to technology, but also thanks to design and implementation. Apple has created a new authentication modality. First things first: Face ID nails nearly every criteria I came up with to evaluate it. The false positive rate, within certain genetic constraints, is 1 in a million compared, to 1 in 50,000. The inherent security architecture doesn’t look quite as tied to hardware as Touch ID (because the phone needs the sensor package for other capabilities), but does appear to be either as strong (including the software implementation) or close enough in practical circumstances. Watch enough videos of journalists buying masks of their own faces, and it’s clear Face ID is more expensive to circumvent than Touch ID. We haven’t actually seen a public crack yet, but I always assume it will happen eventually. Because history. Apple sometimes has a weak spot underestimating adversaries in their threat models, but they did a good job on this one. In my pre-release article I wrote: Face ID doesn’t need to be the same as Touch ID – it just needs to work reasonably equivalently in real-world use. In my personal experience, and with every user I’ve talked with and in every article I’ve read, Face ID’s core usability is equal to or greater than Touch ID’s. For example, it doesn’t work as well at many angles you could touch your phone from, but it works better in the kitchen and after a shower/workout. I’ve tested it in all sorts of lighting conditions and haven’t found one that trips it up yet. The only downside is I can’t register my wife’s face, and we were become accustomed to using Touch ID on each other’s devices. I do believe it’s slower at actual recognition, but it’s nearly impossible to notice due to the implementation. Face ID is tightly bound to activity, which masks its latency. For example, the time to swipe your fingers is long enough to unlock, where with Touch ID recognition and unlocking were the same action, which made the latency more visible. But I think it’s time to justify that hyperbolic headline. Apple didn’t just throw a facial recognition sensor into the iPhone and replace a fingerprint sensor – they enabled a new security modality. I call this “continuous authentication”. When you use an iPhone you look at the iPhone (some calls and music listening excepted). Instead of unlocking your iPhone once and opening up everything, or requiring you to put your finger on the sensor when an app or feature wants to re-authenticate, the phone can quickly scan your face on demand. And the iPhone does this constantly. Here are the examples I’ve discovered so far: It’s already been widely reported that notification, by default, don’t show details on the lock screen until you look at the iPhone. This is my favorite new feature because it improves security with effectively zero usability impact. I always disabled Control Center on the lock screen for security reasons, but like notifications, just looking at my phone unlocks it. It’s just too bad my thumb can’t reach that upper right corner. Safari now (optionally) uses Face ID before filling in passwords on web sites. Previously, even with Touch ID, they filled in automatically when the phone was unlocked. Apple Pay and the App Store now authenticate with your face without separate authentication actions. Apps can authenticate as you open them. This is where I notice that Face ID is a likely bit slower, but because I don’t need to take another action it feels faster. The lock screen and Safari passwords are, to my knowledge, legitimately new modalities. The others are evolutions of previous use cases. Face ID allows your iPhone to authenticate you under nearly every circumstance you would use your phone and need to authenticate, but without requiring any user action. I think we are just scratching the surface of what’s possible here. Yes, we’ve used tools like Yubikeys plugged into devices to keep sessions open, but I think it’s clear how this is different. This is just the first generation of Face ID. Imagine the use cases once it evolves and can, for example, register multiple users. My Xbox Kinect (may it rest in peace) already does this pretty well, so we know it’s possible (Kinect’s implementation is as secure, and it’s a lot bigger). One of the biggest problems in healthcare security is quickly authenticating to shared workstations in clinical environments… I could see a future version of Face ID significantly addressing that problem. I previously said that Touch ID enables you to use a strong password with the convenience of no password at all. Face ID not only exceeds that mark, it may be the ultimate expression of it, by deeply integrating effortless authentication throughout the user experience without requiring new behaviors. That, my friends, is the power of security design, not just security engineering. Share:

After exploring prevention approaches, you should understand some common technologies which are foundational to endpoint advanced prevention offerings. Machine Learning Machine learning is a catch-all term to indicate that the endpoint protection vendor uses sophisticated mathematical analysis on a large set of data to generate models for detecting malicious files or activity on devices. There are a couple mathematical algorithms which can improve malware prevention. Static file analysis: With upwards of a billion malicious file samples in circulation, mathematical analysis of malware can pinpoint commonalities across malicious files. With a model of what malware looks like, advanced prevention products then Share:

As the velocity of technology infrastructure change continues to increase, it is putting serious stress on Security Operations (SecOps). This has forced security folks to face the fact that operations has never really been our forte. That’s a bit harsh, but denial never helps address serious problems. The case is fairly strong that most organizations are pretty bad at security operations. How many high-profile breaches could have been avoided if one of many alerts was acted upon? How many attacks were made possible by not having properly patched servers or infrastructure? How many successful compromises resulted from human error? If your answer to any of those questions was greater than zero, there is room for improvement. But there is no cavalry off in the distance to magically address operational issues. If anything, SecOps is going to get harder for five reasons: Adversary innovation: Our adversaries are innovating and finding ways to compromise devices using both old and new tactics. They follow the path of least resistance to achieve their mission with focus and persistence. Infrastructure complexity and velocity: With the advent of SaaS and the public cloud, technology infrastructure is getting more complicated and changes happen much faster than before. Data ends up in environments you don’t control and can’t really monitor, yet you still have to protect it. More devices, more places: It seems every employee nowadays has multiple devices which need to connect to sensitive stuff, and they want to access corporate systems from wherever they are. What could possibly go wrong with that? Compounding the issue are IoT and other embedded devices connecting to networks, dramatically increasing where you can be attacked. Maintaining visibility into and understanding of your attack surface and security posture continue to get harder. Hunters hunt: For a long time security folks could be blissfully unaware of the stuff they didn’t find. If the monitor missed it, what could they do besides clean up the mess afterwards? Now organizations proactively look for signs of active adversaries, and these hunters are good at what they do. So in additional to all those alerts, you need have to handle the stuff the hunters find. Skills gap: We’ve been talking about a serious security skills gap for a long time. But it’s not getting any better. There just aren’t enough security people to meet demand, and the problem gets more acute each day. Progress But the news isn’t all bad. By understanding the attacks which may be coming at you through more effective use of threat intelligence, you can benefit from the misfortune of others. You don’t need to wait until you experience an attack and then configure your monitoring environment to look for it. Additionally, enhanced security analytics makes it easier to wade through all the noise to find patterns of attacks, and to pinpoint anomalous behavior which may indicat malicious activity. Integration of threat intelligence and security analytics provides Security Decision Support. It is a key lever for scaling and improving the effectiveness of a security team. We will flesh out these ideas in detail in a blog series. But even with more actionable and prioritized alerts, someone still has to do something. You know: security operations. In many case, this is where everything falls apart. To illustrate, the security teams involved in two of the highest-profile breaches of the last few years (Target and Equifax) were alerted to adversary activity more than once before the breaches became apparent. They just didn’t execute on a strategy to stop either attack before it became a catastrophe. To be fair, it’s easy criticize organizations after they’ve suffered a massive breach. That’s not the point. We bring them up as reminders of a concept we have been talking about for more than a decade: Respond Faster and Better. That’s what it’s all about. As an industry we need to figure out how to more effectively operationalize world-class security practices, quickly and effectively. And yes, we do understand this is much easier to say than to do. But why is this so hard? Let’s examine what security operations tends to do with their time. Those of you with backgrounds in manufacturing probably remember time and motion studies performed to improve productivity of factory workers. Security is far from a factory floor, but the concept applies. Can SecOps be streamlined by figuring out and optimizing whatever takes up a lot of time? We believe the answer is a resounding yes. A lot of security operational tasks involve updates, policy changes, compliance reporting, and other tedious and rote tasks. Certainly there are periods of intense activity, such as triaging a new attack or trying to figure out an effective workaround to an attack. But there is plenty of time spent on distinctly unsexy things. This also causes unmet expectations for people entering the security field. Most entrants have dreams of being a l33t haXor or a threat hunter. Very few wake up excited to tackle change control for a list of firewall changes, or to reimage endpoints after the CEO clicked one of those links. Again. And even if you could find people who get excited about security operations, they would still be human. Which basically means they make errors. But when you need every update and every change to be done right, for fear of opening a hole in your environment large enough to drive a truck (or all your proprietary data – or all your customer data) through, perfection needs to be the goal – even though people are not perfect, no matter how hard they work. Behind the 8 Ball So SecOps is behind the 8 ball, by definition. The deck is stacked against us. The attack surface is growing, the adversaries are getting better, and all we have is ingenuity, a metric crap ton of alerts, and too few humans to get things done. Yep, it sounds like Mission: Impossible. So what? Do we give up? Just pack it in and take a job at a