As we discussed in the last post, when considering new security management platforms, it’s critical to cull your short list based on your requirements, and to then move into the next step of the evaluation process – the Proof of Concept (PoC). Our PoC process is somewhat controversial – mostly because vendors hate it. Why? Because it’s about you and your needs, not them and their product. But you are the buyer, right? Always remember that.

Most SIEM vendors want to push you through a 3-5 day eval of their technology on their terms, with their guy driving. You already have a product in place so you know the drill. You defined a few use cases important to you, and then the vendor (and their SE) stood the product up and ran through those use cases. They brought in a defined set of activities for each day, and you ended the test with a good idea of how their technology works, right?

Actually, wrong. The vendor PoC process is built to highlight their product strengths and hide their weaknesses. We know this from first hand experience – we have built them for vendors in our past roles. Your objective must be to work through your paces, not theirs. To find the warts now – not when you are responding to an incident. It’s wacky that some vendors get scared by a more open PoC process, but their goal is to win the deal, and they put a lot of sweat into scripting their process so it goes smoothly for everyone involved. We hate to say it, but smooth sailing is not the point! The vendor will always say “We can do that!” – it’s your job to find out how well – or how awkwardly.

So set up evaluation criteria based on your requirements and use cases. Your criteria don’t need to be complicated. Your requirements should spell out the key capabilities you need, and then plan to further evaluate each challenger based on intangibles such as set-up/configuration, change management, customization, user experience/ease of use, etc. Before you start, have your team assess your current platform as a basis for comparison.

As you start the PoC, we recommend you invest in screen capture technology. It’s hard to remember what these tools did and how they did it later – especially after you’ve seen a few of them work through the same procedures. So capture as much video as you can of the user experience – it will come in very handy when you need to make a decision. We’ll discuss that in the next post.

Without further ado, let’s jump into the PoC.

Stand it up, for reals

One of the advantages of testing security management products is that you can actually monitor production systems without worrying about blowing them up, taking them down, or adversely impacting anything. So we recommend you do just that. Plan to pull data from your firewalls, your IDS/IPS systems, and your key servers. Not all devices, of course, but enough to get a feel for how you need to set up the collectors. You will also want to configure a custom data source or two and integrate with your directory store to see how that works. Actually do a configuration and bootstrap the system in your environment.

Keep in mind that the PoC is a great time to get some professional services help – gratis. This is part of the sales process for the vendors, so if you want to model out a targeted attack and then enumerate the rules in the system, have the SE teach you how to do it yourself. Then model out another attack and build the rules yourself, without help. The key is to learn how to run the system and to get comfortable – if you do switch you will be living with your choice for a long time.

Focus on visualization, your view into the system. Configure some dashboards and see the results. Mess around with the reports a bit. Tighten the thresholds of the alerts. Does the notification system work? Will the alerts be survivable at production levels for years? Is the information useful? These are all things you need to do as part of kicking each challenger’s tires.

If compliance is your key requirement use PCI as an example. Start pulling data from your protected network segment. Pump that data through the PCI reporting process. Is the data correct and useful for everybody with an interest? Are the reports comprehensive? Will you need to customize the report for any reason? You need to answer this kind of questions during the PoC.

Run a Red Team

Run a simulated attack against yourself. We know actually attacking production systems would make you very unpopular with the ops folks, so set up a lab environment. But otherwise, you want this situation to be as realistic as possible. Have attackers breach test systems with attack tools. Have your defenders try to figure out what is going on, as it’s happening. Does the system alert as it should? Will you need to heavily customize the rule set? Can you identify the nature of the attack quickly? Does their super-duper forensic drill-down give you the view you need? The clock is ticking, so how easy is it to use the system to search for clues?

Obviously this isn’t a real incident situation, so you’ll take some editorial liberties, and that’s fine. You want a feel for how the system performs in near-real-time. If an attacker is in your systems, will you find them? In time to stop or catch them? Once you know they are there, can you tell what they are doing? A Red Team PoC will help you determine that.

Do a Post-Mortem

Once you are done with the Red Team exercise, you should have a bunch of data that will make for a nice forensic investigation of what the attack team did, and perhaps what the defense team didn’t do as well as they could have. Remember this is a learning experience for everyone. Will the tool hold up in the heat of battle? How does it compare to your existing product for comparable functions?

As important is the experience of running a simulated attack on your team. You cannot possibly prevent all attacks from succeeding, so you need practice on your incident investigation and response processes. This type of simulation forces you to exercise facets of the product you might otherwise miss.

You can’t fully test scalability during the PoC so focus on the stuff you can see, feel, and touch. That’s the user experience, and there is no better way to distill out the effectiveness of each challenger than to stage an attack. Remember to have your team grade the challenger while their memory is fresh and their perception is raw. After spending 1-2 weeks with another product, they won’t be able to remember what they liked and what they didn’t – which is where the screen grabs come in handy.

Lather, Rinse, Repeat

You will probably test more than one product or service, so you get to do this all again. Given the resource-intensive nature of this testing process, you probably cannot put more than 2 products through a comprehensive PoC, but do use the same scenarios in each PoC. That consistency helps make the challenge fair and makes your comparison more meaningful.

Now you have all the information you need to make a decision, so that’s what the next post will focus on – figuring out what’s right and gathering data to substantiate your choice. This is where you get to use the grades and videos you collected for each challenger – especially in making the case for a new platform if that’s what you decide on. See? There is some method to our madness.