In the second half of Platform Evaluation for Security Management 2.0, we’ll cover evaluating other SIEM solutions. At this point in the process you have documented your requirements, and rationally evaluated your current SIEM platform to determine what’s working and what’s not. This step is critical because a thorough understanding of your existing platform’s strengths and weaknesses is the yardstick against which all other options will be measured. As you evaluate new platforms, you can objectively figure out if it’s time to move on and select another platform. Again, at this point no decision has been made. You are doing your homework – no more, no less.

We’ll walk you through the process of evaluating other security management platforms, in the context of your key requirements and your incumbent’s deficiencies. There are two major difficulties during this phase of the process. First, you need to get close to some of the other SIEM solutions in order to dig in and determine what the other SIEM providers legitimately deliver, and what is marketing fluff. Second, you’re not exactly comparing apples to apples. Some new platforms offer advantages because they use different data models and deployment options, which demands require careful analysis of how a new tool can and should fit into your IT environment and corporate culture. Accept that some capabilities require you to push into new areas that are likely outside your comfort zone. Let’s discuss the common user complaints – and associated solutions – which highlight differences in function, architecture, and deployment.

The most common complaints we hear include: the SIEM does not scale well enough, we need more and better data, the product needs to be easier to use while providing more value, and we need to react faster given the types of attacks happening today.

• Scale: With the ever growing number of events to monitor, it’s simply not enough to buy bigger and/or more boxes to handle the exponential growth in event processing. Some SIEM vendors tried segregating reporting and alerting from collection and storage to offload processing requirements, which enables tuning each server to its particular role. This was followed by deployment models, where log management collected the data (to meet scalability needs) and delivered a heavily filtered event stream to the SIEM to reduce the load it needed to handle. But this is a stopgap. New platforms address many of the architectural scaling issues, with purpose-built data stores provide fully distributed processing. These platforms can flexibly divide event processing/correlation, reporting, and forensic analysis. For more information on SIEM scaling architectures, consult our Understanding and Selecting a SIEM/Log Management report.
• Data: Most platforms continue to collect data from an increasing number of devices, but many fail in two areas. First, they have failed to climb out of the network and server realm, to start monitoring applications in more depth. Second, many platforms suffer from over-normalization – literally normalizing the value right out of collected data. For many platforms, normalization is critical to address scalability concerns. This, coupled with poorly executed correlation and enrichment, produces data of limited value for analysis and reporting – which defeats the purpose. For example, if you need detailed information for business analytics, you’ll need new agents on business systems – collecting application, file system, and database information that is not included in syslog! Oh, horrors, going beyond syslog is really no longer an option. The format of this data is non-standard, and the important aspects of an application event or SQL query are not easily extracted, and vary by application. At times, you might feed these events through a typical data normalization routine and see nothing out of the ordinary. But if you examine the original transaction and dig into the actual query, you might find SQL injection. Better data means both broader data collection options and more effective processing of the collected data.
• Easier: This encompasses several aspects: automation of common tasks, (real) centralized management, better visualization, and analytics. Rules that ship out of the box have traditionally been immature (and mostly useless) as they were written by tech companies with little understanding of your particular requirements. Automated reporting and alerting features got a black eye because they returned minimally useful information, requiring extensive human intervention to comb through thousands of false positives. The tool was supposed to help – not create even more work. Between better data collection, more advanced analytics engines, and easier policy customization, the automation capabilities of SIEM platforms have evolved quickly. Centralized management is not just a reporting dashboard across several products. We call that integration on the screen. To us, centralized management means both reporting events and the ability to distribute rules from a central policy manager and tune the rules on an enterprise basis. This is something most products cannot do, but is very important in distributed environments where you want to push processing closer to the point of attack. Useful visualization – not just shiny pie charts, but real graphical representations of trends, meaningful to the business – can help make decisions easier.
• Speed: Collection, moving the data to a central location, aggregation, normalization, correlation, and then processing is a somewhat antiquated SIEM model. Welcome to 2002. Newer SIEMs inspect events and perform some pre-processing prior to storage to ensure near-real-time analysis, as well as performing post-correlation analysis. These actions are computationally expensive, so recognize these advancements are predicated on an advanced product architecture and an appropriate deployment model. As mentioned in the data section, this requires SIEM deployment (analysis, correlation, etc.) to be pushed closer to the collector nodes – and in some cases even into the data collection agent.

Between your requirements and the SIEM advances you need to focus on, you are ready to evaluate other platforms. Here’s a roadmap:

1. Familiarize yourself with SIEM vendors: Compare and contrast their capabilities in the context of your requirements. There are a couple dozen vendors, but it’s fairly easy to eliminate many by reviewing their product marketing materials. You use a “magic vectored astrological table” to understand who all the competitors are, but it’s important to form your own opinions based on your requirements – not the generic average requirements of hundreds of other companies. We strongly suggest that you also contact peers in your industry and compare notes on what’s working, what’s not, and which vendors they have evaluated.
2. Create RFP/RFI and send to a select handful of vendors: Don’t go overboard with questions as that only encourages vendors to bend the truth to promise what you asked for. Base your questions on the requirements you identified. Consider discussing the requirements with your VAR and/or security service providers as well.
3. Select three for the short list and arrange hands-on demonstrations: But make sure you drive the demonstration – not the vendor. Be sure to ask for ad hoc demonstrations of the use cases you are interested in – not their optimized canned scenarios. If this is not possible via an on-site demonstration, ask for reference customers to speak with, and actually speak with them. Ask the vendor for performance metrics and ease of use information, and make sure to speak to a number of customers (not just the ones they suggest) to confirm their claims.
4. Evaluate based upon the weighted requirements you have established: This is about finding the right fit to compare to your existing platform, which means you must always stay true to your requirements. Shiny objects are great, but if a fancy feature doesn’t help meet your requirements you are wasting everyone’s time.
5. Conduct an in-house proof of concept (PoC): Here is where you figure out how the new platform really works. This involves setting up a test environment and evaluating your use cases on real traffic. Plan on customizing and administering the system yourself – do not let the vendor drive the PoC. As you know, this decision is one you’ll need to live with for years, so make sure you exercise the challengers.
6. Compare and contrast against your installed SIEM: After you’ve done your homework, it’s time to decide if you have found a better alternative.

At this point you should have the data to make a decision. In our next post we’ll walk through the decision process, providing tips on how to balance the cost and time of a new deployment against potential advancements in capabilities.