To explain the importance of picking a platform, rather than a product, our last post compared Log Management to SIEM, like the difference between using kitchen appliances and running a machine shop. One is easy to use, but limited in applicability; the other requires more work on your part, but can accomplish much more. Our goal was to contrast use cases and levels of expectations between the two product classes; despite lower overall platform satisfaction and the greater amount of work required, SIEM is what many customers need to get their work done. Pushing the boundaries of what is possible involves some pain.

Customers grumble about the tremendous growth in event collection driven by all these new devices, but they need to collect nearly every event type, and often believe they need real-time response. The product had better be fast and provide detailed forensic audits. Customers depend on compliance reports for their non-technical audience, along with detailed operational reports for IT. SIEM customers have a daily yin/yang battle – between automation and generic results; between efficiency and speed; between easy and useful. Again, dissatisfaction is to be expected, but this exercise is to get real work done with a balky product.

SIEMulation

To illustrate why customers go through the re-evaluation process, here are some excerpts from customer conversations:

• “We had some data leakage a couple years ago; nothing serious, but it was a partner who discovered the issue. It took some time to determine why we did not see the activity with SIEM and other internal security systems. Needless to say our executive team was not happy, and wanted us to justify our security expenditures. Actually they said “Why did we not see this? What the hell are we paying for?” Our goal is to be able to get detection working the way we need it to work, and that means full packet capture and analysis. As you know, that means a lot more data, and we need longer retention periods as well.”
• “We upgraded from log management to SIEM two years ago in order to help with malware detection and to scale up general security awareness. The new platform is supposed to scale, but we don’t actually know if it does scale yet because we are still rolling it out. Talk to me again in a couple years – I ought to have it done by then.”
• “I want security analytics. I have systems to measure supply chain efficiency. I have business risk analysis systems. I want the same view into operational and security risk, but I can’t blend the analysis capabilities from these other platforms with the SIEM data. Our goal is to have the same type of analysis everywhere, and eventually a more unified system.”

When it comes to evaluating your current platform, you need to think about the issue from two perspectives. First formally evaluate how well your platform addresses your current and foreseeable requirements in order to quantify critical features you depend on and identify significant deficiencies. Second, look at evolving use cases and the impact of newer platforms on operations and deployment – both good and bad. Just because another vendor offers more features and performance does not mean you should replace your SIEM. The grass is not always greener on the other side. The former is critical for the decision process later in this series; the latter is essential for analyzing the ramifications of a replacement decision.

Sizing up the incumbent

The first step in the evaluation process uses the catalog of requirements you built already to critically assess how the current SIEM platform achieves your needs. This means spelling out each business function, how critical it is, and whether the current platform gets it done. You will need to discuss these questions with stakeholders from operations, security, compliance, and any other organizations that participate in the management of SIEM or take advantage of it. You cannot make this decision in a vacuum, and lining up support early in the process will pay dividends later on. Trust us on this.

Operations will be the best judge of whether the platform is easy to maintain and the complexity of implementing new policies. Security will have the best understanding of the product or service’s forensic auditing capabilities. Compliance teams can judge suitability of reports for audit preparation. And an increasingly common contributor is risk and/or data analysts who mine information and help prioritize allocation of resources. Each audience provides a unique perspective on the criticality of some function and the effectiveness of the current platform.

At this point you have already examined your requirements so you should understand what you have, what you want, and the difference between the two. In some cases you will find that the incumbent platform simply does not fill a hard requirement – which makes the analysis easy. In other cases the system works perfectly, but is a nightmare in terms of maintenance and care & feeding for any system or rule changes. Performance may be less than ideal, but it’s not necessarily clear what that really means, because any system could always be faster when investigating a possible breach. It may turn out the SIEM functions as designed but lacks the capacity to keep up with all the events you need to collect, or takes too long to generate actionable reports. Act like a detective, collecting these tidbits of information, no matter how small, to build the story of the existing SIEM platform in your environment. This information will come into play later when you weigh options, and we recommend using a format that makes it easy to compare and contrast issues.

Security, compliance, management, integration, reporting, analysis, performance, scalability, correlation, and forensic analysis are all areas you need to evaluate in terms of your revised requirements. Prioritization of existing and desired features helps streamline the analysis. We reiterate the importance of staying focused on critical items to avoid “shiny object syndrome” driving you to select the pretty new thing, perhaps ignoring a cheap dull old saw that gets the work done.

Next move to evaluation of other SIEM solutions. At this point in the process you have documented your requirements and rationally evaluated your current SIEM platform to determine what’s working and what’s not. This is critical because a thorough understanding of your existing platform’s strengths and weaknesses is the yardstick against which all other options will be measured. As you evaluate new platforms, you can objectively figure out whether it is time to move on and select another platform. At this point no decision has been made. You are doing your homework – no more, no less.

You face two major difficulties during this phase. First, you need to get close to some of the other SIEM solutions in order to dig in and determine what other SIEM providers legitimately deliver and what is marketing fluff. Second, you’re not exactly comparing apples to apples. Some new platforms offer advantages because they use different data models and deployment options, which demands careful analysis of how a new tool can and should fit into your IT environment and corporate culture. Some capabilities require you to push into new areas likely outside your comfort zone. Start by addressing common user complaints – and associated solutions – to highlight differences in function, architecture, and deployment.

The yardsticks

The most common complaints we hear include: the SIEM does not scale well enough, users need more and better data, the product needs to be easier to use while providing more value, and users need to react faster to investigate the types of attacks happening today.

• Scale: With the ever-growing number of events to monitor, it is simply not enough to buy bigger and/or more boxes to handle the exponential growth in event processing. Some SIEM vendors tried segregate reporting and alerting from collection and storage to offload processing requirements, which enables tuning each server to its particular role. This was followed by alternative deployment models, where log management collected the data (to meet scalability needs) and delivered a heavily filtered event stream to the SIEM to reduce its analysis load. But that is a band-aid rather than a solution. New platforms address many of the architectural scaling issues, with purpose-built data stores providing fully distributed processing. These platforms can flexibly divide event processing/correlation, reporting, and forensic analysis. For more information on SIEM scaling architectures, see our Understanding and Selecting a SIEM/Log Management paper.
• Data: More data. More data, and more types of data, from more types of applications and devices. Most platforms continue to collect data from an increasing number of devices, but many fail in two areas. First, they have failed to climb out of the network and server realms to start monitoring application assets in more depth. Second, they are having trouble with things that don’t fit neatly into an existing format like NetFlow or syslog. This, coupled with poorly executed correlation and enrichment, produces data of limited value for analysis and reporting – which defeats the purpose. For example, you might need to collect a binary file or pull data from social media. The format of this data is non-standard, and the important aspects of an application event or SQL query must be interpreted within the context of the application, requiring a deep understanding of what it does and why. You might feed these events through a typical data normalization routine and see nothing out of the ordinary. But if you examine the original transaction and dig into the actual query, you might find SQL injection. Better data means both broader data collection options and more effective processing of the collected data, within the context of the business process.
• Easier: This generic term encompasses several aspects: automation of common tasks, (real) centralized management, better visualization, and analytics. Rules that ship out of the box are traditionally immature (and mostly useless), as if written by tech companies with little understanding of your particular requirements – which they are. Automated reporting and alerting features got a black eye because they returned minimally useful information, requiring extensive human intervention to comb through thousands of false positives. The tool was supposed to help – not create even more work. Between better data collection, more advanced analytics engines, and easier policy customization, the automation capabilities of SIEM platforms have evolved quickly. Centralized management is not just a reporting dashboard across several unconnected products. We call that “integration on the screen”. To us, centralized management means analysis and reporting of events from across the enterprise, the ability to distribute rules from a central policy manager, and the capability to tune rules on an enterprise basis. Most products cannot do this, but in distributed environments where you want to push processing closer to the point of attack, you need it. Useful visualization – not just shiny pie charts, but real graphical representations of trends, meaningful to the business – can make decisions easier.
• Faster and Better: Collection, moving the data to a central location, aggregation, normalization, correlation, and then processing, is a somewhat antiquated SIEM model. Welcome to 2002. Newer SIEMs inspect events and perform some pre-processing prior to storage to enable near-real-time analysis – closer to the event source – as well as post-correlation analysis. And many forms of analysis require more advanced queries and pre-processing large amounts of data so they can provide statistical references and models of activity. These actions are computationally expensive, so recognize that these advances are predicated on an advanced product architecture and appropriate deployment model. As mentioned under Data, this requires SIEM deployment (analysis, correlation, etc.) to be pushed closer – or into – the collector nodes.