By this point you appreciate the difference large gap between what you need and what you have, so it’s time to dip your toes in the water to see what other platform vendors offer. But how? You need to figure out which vendors are worth investigating for their advantages, despite any disadvantages. Much of defining evaluation criteria and potential candidates involves wading objectively through vendor hyperbole to see what each offering actually does vs. drug-induced optimism in the vendor’s marketing department. As technology markets mature (and SIEM is pretty mature), the base capabilities of the platforms converge, making them all look alike. Complicating the issue, vendors adopt similar messaging regardless of actual features, making it increasingly difficult to differentiate between the platforms.

But you still need to do it, because given your unhappiness with your current platform (or you wouldn’t be reading this, right?), you need to distill what a platform does and doesn’t do, as early in the process as you can. And make no mistake – there are significant differences!

We divide the vendor evaluation process into two phases. First we will help you define a short list of potential replacements. Maybe you use a formal Request for Proposals or Information (RFP/RFI) to cull the 15 companies left in the space down to 3-5, or perhaps you don’t. You will see soon enough why you can’t run 10 vendors through even the first stage of the evaluation, so you need a way to narrow down the field to get started. At the conclusion of the short list exercise you will need to test one or two new platforms during a proof of concept (PoC), which we will detail. We don’t recommend skipping directly to the PoC, by the way. Each platform has strengths and weaknesses, and just landing in the upper-right quadrant of a magical chart doesn’t make a vendor the right choice for you. And the RFP process usually unearths items you had not considered, so the process is valuable for its own sake.

It is time to do your homework. All of it. Even if you don’t feel like it.

The Short List

The goal at this point is to whittle the list down to 3-5 vendors who appear to meet your needs, based upon the results of the RFIs or RFPs you sent vendors. Their answers should quickly disqualify a few who lack critical capabilities. The next step, for the remaining vendors, is to get a little better sense of their products and firms. Your main tool at this stage is what we call the dog and pony show. The vendor brings in their sales folks and sales engineers (SEs) to tell you how their product is awesome and will solve every problem you have. Of course they won’t be ready (unless they read this paper as well) for the intensity of your KGB-style interrogation techniques. You know what is important to you, and you need confidence that any vendor passing through this gauntlet to the PoC can meet your requirements.

Let’s talk a bit about tactics for getting the answers you need, based on deficiencies in your existing product (from your platform evaluation). You need detailed answers at these meetings to objectively evaluate any new platform. You don’t want a 30-slide PowerPoint walkthrough and a generic demo. Make sure the challenger understands those expectations ahead of the meeting so they have the right folks in the room. If they bring the wrong people cross them off. It’s as simple as that – it’s not like you have a lot of time to waste, right?

We recommend defining a set of use cases/scenarios for the vendor to walk you through. Then their skilled folks with expertise using the tool can show you how they would solve the problem you mapped out. This forces them to think about your problems rather than their scripted demo and shows off the tool’s relevant capabilities, instead of the smoothness of the folks staging the demo. You don’t want to buy from the best presenter – you want to identify the product that best meets your needs, and that means making the vendor do what you need – not what shows off their product best.

Here are a few scenarios to help guide you on how to set up these meetings. Prioritize this list based on your own needs, but this should get you 90% of the way through narrowing down the list.

• Security: The first scenario should focus on security. That’s what this ultimately boils down to, right? You want to understand how they would detect an attack based on their information sources, as well as how they configure rule sets and alerts. Make it detailed but not ridiculous. Basically, simplify your existing environment a bit and run them through an attack scenario you saw recently. This will be a good exercise for seeing how the data they collect solves a major use case, detecting an emerging attack quickly. Have the SE walk you through setting up and customizing a rule because you will often need to do both. Use your own scenario to reduce the likelihood of the SE having a pre-built rule. You want to really understand how the rules work because you will spend a lot of time configuring your rules, so it’s useful to see how easy it is for an expert create new policies.
• Compliance: Next you need to understand how much automation is available for compliance. Ask the SE to show you the process of preparing for an audit. And no, showing you a list of 2,000 reports, most called “PCI X.X”, is not sufficient. Nor is a drop-down list of 200 checkboxes with obscure names going to help you. Ask them to produce samples for a handful of critical reports you rely upon to see how closely they hit the mark – you can see the difference between reports developed by an engineer and those created by an auditor. You need to understand where the data comes from, and hopefully they have a demo data set to show you populated reports. The last thing you want is to learn that their reports don’t pull from the right data sources two days before an audit.
• Integration: In this part of the discussion, delve into how the product integrates with your existing IT stack. How does the platform pull data from your identity management system? CMDB? Learn about data collection. Are the connectors pre-built and maintained by the vendor? What about custom connectors? Is a SDK available, or does any customization require expensive professional services? Or both? What about collecting threat feeds and integrating with other products? Buyer beware – don’t leave any questions unanswered.
• Forensics: Vendors throw around the term root cause analysis frequently, while rarely substantiating how their tool works through an incident. Have the SE literally walk you through an investigation based on their sample data. Yes, you’ll test this yourself later, but get a feel for the built-in tools and how they can be used by the SE, who should really know how to use the system.
• Scalability: If your biggest issue is a requirement for more power you will want to know (at a very granular level) how each challenger solves the problem. Dive into their data model and their deployment architectures, and have them tell stories about their biggest implementations. If scalability is a problem for the incumbent you will know how big the system needs to get, and understand whether a proposed architecture passes the sniff test.
• Additional data types: What if you want to collect application data and build some rules? Have the SE walk you through the entire process: how they’d look at a data schema, set up the collector, and set up a few rules leveraging the new data. You aren’t looking for empirical correctness but you are evaluating whether unnatural acts are required to support additional data types. The less work required to get data in, the better off you are.

Depending on your requirements and your platform evaluation, you may need to go through other areas with each vendor. This list covers the major items – adjust and add scenarios to reflect your priorities.

This type of meeting could be considered cruel and unusual punishment. But you need this level of detail before you commit to actually testing a product or service. Remember, this evaluation happened because the incumbent failed to get it done. Shame on you if you don’t ask every question to avoid making the same mistakes again. Don’t worry about making the SE uncomfortable – providing technical answers is their job.

And don’t expect to get through a meeting like this in 30 minutes. You will likely need a half-day minimum to work through all these scenarios. That’s why you probably want to only bring 3-5 vendors in for these meetings. You will be spending days with each product during proof of concept, so try to disqualify products that won’t work before wasting much time on them. This initial meeting can be a painful investment of time – especially if you realize early in the meeting that a vendor won’t make the cut – but it is worth doing anyway. You can thank us later.

After you finish the ritual humiliation of vendor sales teams, you will have a very good idea of which products or services can fit. But that’s not good enough. You need to get hands on with the systems and run each through its paces for a couple days. The next step in the process, the Proof of Concept (PoC), is the most important. If you took part in the selection of the incumbent product, you will be better off this time. You didn’t know anything back then. Now you know what works – and more importantly, what doesn’t – and your evaluation will be better for it.

Security Ballet

With luck you have narrowed down your list three good candidates. If one or two more meet your initial requirements that is no bad thing, but it makes the PoC process longer and a bit bloodier. The Proof of Concept is where sales teams smell blood, and vendors bring their best and savviest to bear. They raise doubts about competitors. They highlight how their product was selected over those competitors at major accounts. They have phone numbers for customer references handy to speak with you. For now, forget all that. You are running this show, and the PoC needs to follow your script, not theirs.

Most SIEM vendors want to start by pushing you through a 3-5 day evaluation of their technology on their terms, with their guy driving. They like to airdrop tuned appliances or servers in, with their demo all prepared. You define a few key use cases, and then the vendor (the SE) stands the product up and runs through them – they say it’s easy because all their customers do the same thing. They like to bring in a defined set of activities for each day and end the test with a good idea of how their technology works, right? It looks like a well-rehearsed version of the Nutcracker, where each participant precisely executes their assigned task. Everything quick and painless – security is usually like that, right?

Wrong! Security is messy. Vendors design their PoC processes to highlight product strengths and hide weaknesses. We know this from first-hand experience – we have built them for vendors in past lives. You need to work through your situation, not theirs. Find the warts now – not when you are responding to an incident. It is bizarre that some vendors get scared by a real open PoC process, but their goal is to win the deal, so they put a lot of sweat into scripting a process so it goes smoothly. But smooth sailing is not the point! The vendor will always say “We can do that!” Your job is to find out how well – or how badly.

Before you start a PoC, we recommend you establish evaluation criteria based on your requirements and use cases from earlier in the process. Your criteria don’t need to be complicated. Your requirements should spell out the key capabilities you need, with a plan to further evaluate each challenger based on intangibles such as set-up/configuration, change management, customization, user experience/ease of use, etc. Before you start have your team assess your current platform against the criteria as a baseline.

Before you start the PoC, we recommend you invest in screen capture technology. It is hard to remember exactly what each tools did and how – especially after you have worked a few unfamiliar tools through the same procedures. So capture as much video as you can of the user experience – it will come in very handy as you reach the decision point.

Without further ado, let’s jump into the PoC.

Driving the PoC Bus

One advantage of testing security management products is that you can actually monitor production systems without worrying about blowing them up, taking them down, or adversely impacting anything. So do just that. Plan to pull data from your firewalls, your IDS/IPS systems, and your key servers. Not all devices, of course, but enough to get a feel for how you need to set up the collectors. The point is to run things according to your needs, with your data, alerting based on your policies. You drive the bus – don’t get taken for a ride. You will also want to configure a custom data source or two and integrate with your directory store to see how that works. Actually run through the full configuration and bootstrap the system in your environment.

If compliance is your key requirement, use PCI as an example. Start pulling data from your protected network segment. Pump that data through the PCI reporting process. Is the data correct and useful to everybody interested? Are the reports comprehensive? Will you need to customize the reports for any reason? How easy is that? You need to answer these kinds of questions during the PoC.

Also pay attention to visualization and user interfaces. Security systems are not only used by security professionals. A configurable UI makes it easier for a wider audience of users to contribute to, and benefit from, a SIEM platform. Configure some dashboards and see the results. Mess around with reports a bit. Tighten the thresholds of the alerts. Does the notification system work? Will alerts happen in a timely fashion at enterprise volumes? Is the information in the dashboards and reports useful? These are all things you need to do as part of kicking each challenger’s tires.

Run a Red Team

Run a simulated attack against yourself. We know actually attacking production systems would make you very unpopular with the ops folks, so set up a lab environment. Virtual environments are perfect for this, as you can use the same default images for each of the participating vendors. But you want as realistic a situation as possible. Have attackers breach test systems with attack tools. Have your defenders try to figure out what is going on as it’s happening. Does the system alert as it should? Will you need to heavily customize the rules? Can you identify the nature of the attack quickly? Does their super-duper forensic drill-down give you the view you need? The clock is ticking – how easy is it to use the system to search for clues?

Obviously this isn’t a real incident, so you can take some editorial liberties, and that’s fine. You want a feel for how the system performs in near-real-time. If an attacker is in your systems, will you find them? In time to stop or catch them? Once you know they are there, can you tell what they are doing? A Red Team exercise as part of the PoC will help you determine that.

The Postmortem

Hopefully nobody died in the process, but you do want to evaluate both successes and failures in terms of your priorities. When you finish a red team exercise, you should have a bunch of data that makes for a nice forensic investigation of what the attack team did – and perhaps what the defense team didn’t do as well as they could have. This is a learning experience for everyone, and we find that real attack scenarios weigh heavily in the success of these platforms. Knowing a tool will hold up in the heat of battle goes a long way to giving security and operations teams confidence when they go live. Now ask yourself, “How does it compare to our existing product for comparable functions?” This should go a long way for determining suitability of the contender to actually win your business.

You cannot fully test scalability during a PoC, so focus on the stuff you can see, feel, and touch. That’s the user experience, and there is no better way to distill out the effectiveness of each challenger than to stage an attack. Have your team grade each challenger while memory is fresh and perceptions are raw. After spending a week or two with another product, they won’t remember what they liked and what they didn’t about earlier ones – another reason screen grabs come in handy.

Lather, Rinse, Repeat

You will probably test more than one product or service, so you get to do it all again. Given the resource-intensive nature of the testing process, you probably cannot put more than a couple products through a comprehensive PoC, but use the same scenarios for each product. Consistency helps make the challenge fair, and comparisons more meaningful.

Now you have all the information you need to make a decision, so it is time to figure out what to do, and to gather data to substantiate your choice for the internal sales process. You can use the grades and videos you collected for each competitor – especially to make the case for a new platform, if that’s what you decide. See? There is method to our madness.