Security Management 2.5: The Decision Process
By this point you appreciate the difference large gap between what you need and what you have, so it’s time to dip your toes in the water to see what other platform vendors offer. But how? You need to figure out which vendors are worth investigating for their advantages, despite any disadvantages. Much of defining evaluation criteria and potential candidates involves wading objectively through vendor hyperbole to see what each offering actually does vs. drug-induced optimism in the vendor’s marketing department. As technology markets mature (and SIEM is pretty mature), the base capabilities of the platforms converge, making them all look alike. Complicating the issue, vendors adopt similar messaging regardless of actual features, making it increasingly difficult to differentiate between the platforms. But you still need to do it, because given your unhappiness with your current platform (or you wouldn’t be reading this, right?), you need to distill what a platform does and doesn’t do, as early in the process as you can. And make no mistake – there are significant differences! We divide the vendor evaluation process into two phases. First we will help you define a short list of potential replacements. Maybe you use a formal Request for Proposals or Information (RFP/RFI) to cull the 15 companies left in the space down to 3-5, or perhaps you don’t. You will see soon enough why you can’t run 10 vendors through even the first stage of the evaluation, so you need a way to narrow down the field to get started. At the conclusion of the short list exercise you will need to test one or two new platforms during a proof of concept (PoC), which we will detail. We don’t recommend skipping directly to the PoC, by the way. Each platform has strengths and weaknesses, and just landing in the upper-right quadrant of a magical chart doesn’t make a vendor the right choice for you. And the RFP process usually unearths items you had not considered, so the process is valuable for its own sake. It is time to do your homework. All of it. Even if you don’t feel like it. The Short List The goal at this point is to whittle the list down to 3-5 vendors who appear to meet your needs, based upon the results of the RFIs or RFPs you sent vendors. Their answers should quickly disqualify a few who lack critical capabilities. The next step, for the remaining vendors, is to get a little better sense of their products and firms. Your main tool at this stage is what we call the dog and pony show. The vendor brings in their sales folks and sales engineers (SEs) to tell you how their product is awesome and will solve every problem you have. Of course they won’t be ready (unless they read this paper as well) for the intensity of your KGB-style interrogation techniques. You know what is important to you, and you need confidence that any vendor passing through this gauntlet to the PoC can meet your requirements. Let’s talk a bit about tactics for getting the answers you need, based on deficiencies in your existing product (from your platform evaluation). You need detailed answers at these meetings to objectively evaluate any new platform. You don’t want a 30-slide PowerPoint walkthrough and a generic demo. Make sure the challenger understands those expectations ahead of the meeting so they have the right folks in the room. If they bring the wrong people cross them off. It’s as simple as that – it’s not like you have a lot of time to waste, right? We recommend defining a set of use cases/scenarios for the vendor to walk you through. Then their skilled folks with expertise using the tool can show you how they would solve the problem you mapped out. This forces them to think about your problems rather than their scripted demo and shows off the tool’s relevant capabilities, instead of the smoothness of the folks staging the demo. You don’t want to buy from the best presenter – you want to identify the product that best meets your needs, and that means making the vendor do what you need – not what shows off their product best. Here are a few scenarios to help guide you on how to set up these meetings. Prioritize this list based on your own needs, but this should get you 90% of the way through narrowing down the list. Security: The first scenario should focus on security. That’s what this ultimately boils down to, right? You want to understand how they would detect an attack based on their information sources, as well as how they configure rule sets and alerts. Make it detailed but not ridiculous. Basically, simplify your existing environment a bit and run them through an attack scenario you saw recently. This will be a good exercise for seeing how the data they collect solves a major use case, detecting an emerging attack quickly. Have the SE walk you through setting up and customizing a rule because you will often need to do both. Use your own scenario to reduce the likelihood of the SE having a pre-built rule. You want to really understand how the rules work because you will spend a lot of time configuring your rules, so it’s useful to see how easy it is for an expert create new policies. Compliance: Next you need to understand how much automation is available for compliance. Ask the SE to show you the process of preparing for an audit. And no, showing you a list of 2,000 reports, most called “PCI X.X”, is not sufficient. Nor is a drop-down list of 200 checkboxes with obscure names going to help you. Ask them to produce samples for a handful of critical reports you rely upon to see how closely they hit the mark – you can see the difference between reports developed by an engineer and those created by an auditor. You need to
