Security Management 2.5: Evaluating the Incumbent
To explain the importance of picking a platform, rather than a product, our last post compared Log Management to SIEM, like the difference between using kitchen appliances and running a machine shop. One is easy to use, but limited in applicability; the other requires more work on your part, but can accomplish much more. Our goal was to contrast use cases and levels of expectations between the two product classes; despite lower overall platform satisfaction and the greater amount of work required, SIEM is what many customers need to get their work done. Pushing the boundaries of what is possible involves some pain. Customers grumble about the tremendous growth in event collection driven by all these new devices, but they need to collect nearly every event type, and often believe they need real-time response. The product had better be fast and provide detailed forensic audits. Customers depend on compliance reports for their non-technical audience, along with detailed operational reports for IT. SIEM customers have a daily yin/yang battle – between automation and generic results; between efficiency and speed; between easy and useful. Again, dissatisfaction is to be expected, but this exercise is to get real work done with a balky product. SIEMulation To illustrate why customers go through the re-evaluation process, here are some excerpts from customer conversations: “We had some data leakage a couple years ago; nothing serious, but it was a partner who discovered the issue. It took some time to determine why we did not see the activity with SIEM and other internal security systems. Needless to say our executive team was not happy, and wanted us to justify our security expenditures. Actually they said “Why did we not see this? What the hell are we paying for?” Our goal is to be able to get detection working the way we need it to work, and that means full packet capture and analysis. As you know, that means a lot more data, and we need longer retention periods as well.” “We upgraded from log management to SIEM two years ago in order to help with malware detection and to scale up general security awareness. The new platform is supposed to scale, but we don’t actually know if it does scale yet because we are still rolling it out. Talk to me again in a couple years – I ought to have it done by then.” “I want security analytics. I have systems to measure supply chain efficiency. I have business risk analysis systems. I want the same view into operational and security risk, but I can’t blend the analysis capabilities from these other platforms with the SIEM data. Our goal is to have the same type of analysis everywhere, and eventually a more unified system.” When it comes to evaluating your current platform, you need to think about the issue from two perspectives. First formally evaluate how well your platform addresses your current and foreseeable requirements in order to quantify critical features you depend on and identify significant deficiencies. Second, look at evolving use cases and the impact of newer platforms on operations and deployment – both good and bad. Just because another vendor offers more features and performance does not mean you should replace your SIEM. The grass is not always greener on the other side. The former is critical for the decision process later in this series; the latter is essential for analyzing the ramifications of a replacement decision. Sizing up the incumbent The first step in the evaluation process uses the catalog of requirements you built already to critically assess how the current SIEM platform achieves your needs. This means spelling out each business function, how critical it is, and whether the current platform gets it done. You will need to discuss these questions with stakeholders from operations, security, compliance, and any other organizations that participate in the management of SIEM or take advantage of it. You cannot make this decision in a vacuum, and lining up support early in the process will pay dividends later on. Trust us on this. Operations will be the best judge of whether the platform is easy to maintain and the complexity of implementing new policies. Security will have the best understanding of the product or service’s forensic auditing capabilities. Compliance teams can judge suitability of reports for audit preparation. And an increasingly common contributor is risk and/or data analysts who mine information and help prioritize allocation of resources. Each audience provides a unique perspective on the criticality of some function and the effectiveness of the current platform. At this point you have already examined your requirements so you should understand what you have, what you want, and the difference between the two. In some cases you will find that the incumbent platform simply does not fill a hard requirement – which makes the analysis easy. In other cases the system works perfectly, but is a nightmare in terms of maintenance and care & feeding for any system or rule changes. Performance may be less than ideal, but it’s not necessarily clear what that really means, because any system could always be faster when investigating a possible breach. It may turn out the SIEM functions as designed but lacks the capacity to keep up with all the events you need to collect, or takes too long to generate actionable reports. Act like a detective, collecting these tidbits of information, no matter how small, to build the story of the existing SIEM platform in your environment. This information will come into play later when you weigh options, and we recommend using a format that makes it easy to compare and contrast issues. Security, compliance, management, integration, reporting, analysis, performance, scalability, correlation, and forensic analysis are all areas you need to evaluate in terms of your revised requirements. Prioritization of existing and desired features helps streamline the analysis. We reiterate the importance of staying focused on critical items to avoid “shiny object syndrome” driving you to select the pretty new