With vendor evaluations in hand, you are ready to make your decision, right? The answer is both yes and no. We know the importance of this decision – you are here because your first attempt at this project wasn’t as successful as it needed to be. After the vendor evaluation process you are in a position to distinguish innovative technologies from pigs with fresh lipstick. But now you need to see which of the vendors is actually the best fit for you! Successful decision-making on SIEM replacement goes beyond vendor evaluation – it entails evaluating yourself too. It is important to differentiate between the two because you cannot make a decision without taking a long hard look at yourself, your team, and your company. This is an area where many projects fail, so let’s break the decision down to ensure you can make a good recommendation and feel comfortable with it – from both internal and external perspectives.

But remember the selection of the ‘right’ vendor may come down to more than matching needs against capabilities. The output of our Security Management 2.5 process is not really a decision – it’s more of a recommendation. The final decision will likely be made in the executive suite. That’s why we focused so much on gathering data (quantitative where possible) – you will need to defend your recommendation until the purchase order is signed. And probably afterwards.

Defensible Position

We won’t mince words. This decision generally isn’t about objective or technical facts – especially since most of you reading this have an incumbent in play, typically part of a big company with important relationships with heavies inside your shop. This could get political, or the decision might be entirely financial, so you need your ducks in a row and a compelling argument for any change. And even then you might not be able to push through a full replacement. In that case the answer might be to supplement. In this scenario you still aggregate information with the existing platform, but then you feed it to the new platform for analysis, reporting, forensics, etc. across the enterprise. Given the economic cost of running both, this is unacceptable for some organizations, but if your hands are tied on replacement, this kind of creative approach is worth considering.

But that is still only the external part of the decision process. In many cases the (perceived) failure of your existing SIEM may be self-inflicted. So we also need to evaluate and explain the causes of the failure, with assurance that you can avoid those issues this time. If not your successor will be in the same boat in another 2-3 years. So before you put your neck on the chopping block and advocate for change (if that is what you decide), do some deep internal analysis as well.

Looking in the mirror

First, let’s make sure you really re-examined the existing platform in terms of the original goals. Did your original goals adequately map your needs at the time, or was there stuff you did not anticipate? How have your goals changed over time? Be honest! Do not let ego get in the way of doing what’s right, and take a hard and fresh look at the decision to ensure you don’t repeat previous mistakes. Did you kick off this process because you were pissed at the original vendor? Or because they got bought and seemed to forget about the platform? Do you know what it will take to get the incumbent where it needs to be – and whether that is even possible? Is it about throwing professional services at the issues? Is there a fundamental technology problem?

Did you assess the issues critically the first time around? If it was a skills issue, have you successfully addressed it? Can your folks build and maintain the platform moving forward? Are you looking at a managed service to take that concern off the table? If it was a resource problem, do you now have enough staff for proper care and feeding? Yes, the new generation of platforms requires less expertise to keep operational, but don’t be naive – no matter what any sales rep says, you cannot simply set and forget them. Whatever you pick will require expertise to deploy, manage, tune, and analyze reports. These platforms are not self-aware – by a long shot.

Remember, there are no right or wrong answers here, but the truth (and your commitment) will become clear when you need to sell something to management. Some of you may worry that management will see the need for replacement as “your fault” for choosing the incumbent, so make sure you have answers to these questions and that you aren’t falling into a self-delusional trap. You need your story straight and your motivations clear. Have a straightforward and honest assessment of what is going right and wrong, so you are not caught off guard when asked to justify changes and new expenses.

Setting Expectations

Revisiting requirements provides insight into what you need the security management platform to do. Remember, not everything is Priority #1, so pick your top three must-have items and prioritize the requirements. You can prioritize specific use cases (compliance, security, forensics, operations), and have a pretty good feeling about whether the new platform or incumbent will meet your expectations.

If you love some new features of the challenger(s), will your organization leverage them? Firing off alerts faster won’t help if your team takes a week to investigate each issue, or cannot keep up with the increased demand. The new platform’s ability to look at application and database traffic doesn’t matter if your developers won’t help you understand normal behavior to build the rule set. Fancy network flow analysis can be a productivity sink if your DNS and directory infrastructure is a mess and you can’t reliably map an IP to user ID.

Does your existing product have too many features? Yes, some organizations simply cannot take advantage of (or even handle) complex multi-variate correlation across the enterprise. Do you need to aggregate logs because organizational politics, or your team’s resources or skill set, prevent you from getting the job done otherwise? This might be a good reason to outsource or use a managed service. There isn’t a right or a wrong answer here. And not being honest will land you in the hot seat again.

Is is realistic to think you can deploy a common rule set across your enterprise? All these extra capabilities are great, and we recommend central management and reporting to ensure consistency, but all too often companies delegate IT maintenance down to each division, and it takes an act of Congress to get them in the same room – much less to agree on policies. Don’t expect people or politics to change just because the organization gains the ability to monitor across the enterprise. Having a technical capability doesn’t mean you will have agreement on whether or how to use it. Don’t forget the realities of your organization, either. Will people regard this whole project as snooping or unwelcome interference? That might impact project success, so you may need to actively manage expectations.

If you kickstarted this effort because the existing SIEM missed something and that resulted in a breach, can you honestly say the new thing would (not might) detect that attack? Was it really because you were just missing some data that would have enabled detection, or was the failure more systemic? We have certainly seen high profile breaches result in tossing the old and bringing in the new (someone has to pay, after all), but make sure you address the root cause of the real problem. And make sure you evaluate the right tool. If you were compromised by a persistent attacker you might be looking at the wrong technology. Maybe you really need full packet capture. Maybe not, but at least ask the question.

Economics

Even if introspection and expectations line up, there is still economic reality. IT groups need to do more with less, and that’s not changing. So you need to weigh the cost of getting your existing product functional against replacing it. Even if you want to stack the deck against the incumbent, don’t forget the cost (and time) to train your folks on the new tool – or the professional services to migrate your existing rules, data sources, and data. If you managed to get some of that done during the proof of concept that’s great, but there is certain to be more as you move toward full deployment.

Be sure you compare apples to apples. Sure, the new platform may do a lot more, but at what cost? If essential reports do not get produced, or you find yourself running on what feels like a deprecated platform, you still need to account for purchase, maintenance, deployment, customization, and training on the new thing. Make sure you consider total and not merely procurement cost. Even OpEx is real money, at least last time we checked.

Documentation

The end goal is a recommendation, so you need to document what you think and then present it to the folks with the money. You may not always be in the room when decisions are made, so your documentation must clearly articulate the reasons for change. Given all we know, here is how we would normally structure that artifact of your decision process:

• Requirements: Tell them what you need, and who said you need it. Compliance and security requirements come from different groups so make sure these dependencies are referenced.
• Current product evaluation: What works and doesn’t with your existing solution within the context of your requirements, both now and for the next set of use cases.
• Challenger assessment: Summarize the work you did to find the next platform. Which vendors did you disqualify and why – you never know if your CFO’s brother-in-law works for a vendor. What did you learn in the proof of concept? Which competitor came out on top?
• Cost estimate: What would it cost to move to the new platform? Which is capital expense and which is operational? What kind of investment in professional services would be required? How does that differ from making incremental improvements to the incumbent, and what functionality are you sacrificing in the move?
• Migration plan: If you ultimately decide to replace the SIEM what will the migration look like? How long will it take? Will there be a disruption of service? Will there be any exposures and for how long? You need all these answers before you pitch to the powers that be. Not a Gantt chart – that comes at the end – but enough to answer the tough questions.
• Recommendation: Your entire document should be building to this point, where you put the best path down on paper. If it is a surprise to your audience you did something wrong. This is about telling them what they already know, and making sure they have an opportunity to ask any more questions.

But even once the recommendation is in, the decision is far from done. Now you have to negotiate with the new vendor (or the incumbent) and this is when your process is most vulnerable. An incumbent losing the deal will act desperately to save it. Challengers will do likewise if they think you are staying put or choosing a competitor.