Blog

Understanding and Selecting SIEM/Log Management: Introduction

By Mike Rothman

Over the past decade business processes have been changing rapidly. We focus on collaboration, both inside and outside our own organizations. We have to support more devices in different form factors, many of which IT doesn’t directly control. We add new applications on a monthly basis, and are currently witnessing the decomposition of monolithic applications into dozens of smaller loosely connected application stacks. We add virtualization technologies and SaaS for increased efficiency. Now we are expected to provide anywhere access while maintaining accountability, but we have less control. A lot less control.

If that wasn’t enough, bad things are happening much faster. Not only are our businesses always on, the attackers don’t take breaks either. New exploits are discovered, ‘weaponized’, and distributed to the world within hours. So we have to be constantly vigilant and we don’t have a lot of time to figure out what’s under attack and how to protect ourselves before the damage is done.

Compound the 24/7 mindset with the addition of new devices implemented to deal with new threats. Every device, service, and application streams zillions of log files, events, and alerts. Our regulators now mandate we analyze this data every day. But that’s not the issue.

The real issue is pretty straightforward: of all the things flashing at us every minute, we don’t know what is really important. We have too much data, but not enough information.

This lack of information compounds the process of preparing for the inevitable audit(s), which takes way too long for folks who would rather be dealing with security issues. Sure, most folks just bludgeon their auditors with reams of data, none of which provides context or substantiation for the control sets in place relative to the regulations in play. But that’s a bad answer for both sides. Audits take too long and security teams never look as good as they should, given they can’t prove what they are doing.

Ask any security practitioner about their holy grail and the answer is twofold: They want one alert telling exactly what is broken, on just the relevant events, with the ability to learn the extent of the damage. They need to pare down the billions of events into actionable information.

And they want to make the auditor go away as quickly and painlessly as possible, which requires them to streamline both the preparation and presentation aspects of the audit process.

Security Information and Event Management (SIEM) and Log Management tools have emerged to address those needs and continue to generate a tremendous amount of interest in the market, given the compelling use cases for the technology.

Defining SIEM and Log Management

Security Information and Event Management (SIEM) tools emerged about 10 years ago as the great hope of security folks constantly trying to reduce the chatter from their firewalls and IPS devices. Historically, SIEM consisted of two distinct offerings: SEM (security event management), which collected and aggregated for security events; and SIM (security information management), which correlated and normalized the collected security event data.

These days, integrated SIEM platforms provide pseudo-real-time monitoring of network and security devices, with the idea of identifying the root causes of security incidents and collecting useful data for compliance reporting. The standard perception is that the technology is at best a hassle, and at worst an abject failure. SIEM is believed to be too complex, and too slow to implement, without providing enough customer value to justify the investment.

While SIM & SEM products focused on aggregation and analysis of security information, Log Management platforms were designed within a broader context of the collection and management of any log files. Log Management solutions don’t have the negative perception of SIEM because they do what they say they do – basically aggregate, parse, and index logs.

Log Management has helped get logs under control, but underdelivered on the opportunity to pluck value from the archives. Collection, aggregation, and reporting is enough to check the compliance box; but not enough to impact security operations – which is what organizations are really looking for. End users want simple solutions that improve security operations, while checking the compliance box.

Given that backdrop, it’s clear the user requirements that were served by separate SIEM and Log Management solutions have fused. As such, these historically disparate product categories have fused as well. If not from an integrated architecture standpoint; certainly from the standpoint of user experience, management console, and value proposition. There really aren’t independent SIEM and Log Management markets any more.

The key features we see in most SIEM/Log Management solutions include:

  • Log Aggregation: Collection and aggregation of log records from the network, security, servers, databases, identity systems, and applications.
  • Correlation: Attack identification by analyzing multiple data sets from multiple devices to identify patterns not obvious when looking at only one data source.
  • Alerting: Defining rules and thresholds to display console alerts based on customer-defined prioritization of risk and/or asset value.
  • Dashboards: Presentation of key security indicators within an interface to identify problem areas and facilitate investigation.
  • Forensics: Providing the ability to investigate incidents by indexing and searching relevant events.
  • Reporting: Documentation of control sets and other relevant security operations or compliance activities.

Prior to this series we have written a lot about SIEM and Log Management, but mostly on current events and trends within this market. Given the rapid evolution of the SIEM and Log Management markets, and unprecedented interest from our readers, we are now embarking on a thorough analysis of the space, in order to help end user organizations select products more quickly and successfully, by becoming more educated buyers.

It is time to spotlight both the grim realities and real benefits of SIEM. The vendors are certainly not going to tell you about the bad stuff in their products, but instead shout out the same fantastic advantages the last vendor did. Trust us when we say there are a lot of pissed-off SIEM users, but there are a lot of happy ones as well. We want to reset expectations so you can avoid joining the former category. Since Adrian and I have worked in and around the SIEM market, we’ll share our practical experiences in development, deployment, and integration of these products.

Understanding and Selecting

As with our previous Understanding and Selecting research, we follow a fairly standard methodology. First off, we start with the use cases driving the need for SIEM and Log Management solutions. These include improving security (reacting faster to emerging threats), increasing security efficiency (doing more with less), and of course compliance automation. Yes, there are more, but these are the use cases driving the bulk of the customer projects out there.

Then we will work through the business justification: why you need these tools and how to sell the project to your management. Next, we’ll talk about the key features of today’s SIEM/Log Management platforms, including log collection/aggregation, correlation, alerting, reporting, and forensics. We’ll also dive deep into the technical architectures, and how different architectures work for the different use cases.

Then we’ll dig into some of the advanced features from some of the leading-edge vendors, as well as how to distinguish one solution from the other – since all the vendor marketing pitches sound the same.

We will also spend some time speculating about what the future holds for the category and which capabilities will become absolutely critical over the next couple years. Finally, we’ll finish up with hard deployment advice, helping to guide your selection process.

So fasten your seat belts. It’s time to jump aboard the Understand and Selecting SIEM and Log Management Express.

No Related Posts
Comments

@Steve

I believe that Prism has a strong - and VERY useful! - log management offering, but SIEM is a bit different. Or maybe I am simply mistaken and need to take another look at your most recent solution…

By Anton Chuvakin


Great article.

If we are being honest, who really has time to look through logs 24/7/365?  Between putting out fires and other day to day priorities, it is almost impossible for someone to actually have the time to look through logs daily.  If an IT Manager is being truly honest, they don’t look at logs daily, which actually puts the organization in a compromising position allowing them to be vulnerable to a possible attack.  Everyone would agree that if the logs aren’t being viewed daily, then the the organization is at risk from an Information Security perspective. 

If we are to truly try to put together a way of looking at logs daily, then an organization must have a crew of at least four full-time professionals working 8 hour shifts, with one full-timer on standby for staff when they are off or on vacation.  This costs the organization on average a minimum of at least $50,000 per person with salary and benefits which will total at least $200,000 per year.

Organizations that try to save money by working their IT person to death with little or no other assistance are begging for problems.  I’ve never heard a CFO or hiring manager say that we are hiring you to watch and view logs all day long, that’s ridiculous!  Even if an organization has the smartest person in the world working for them who understood how to work a SIEM which includes reading logs, continued education on log management since technology changes daily, and having the expertise to fix whatever attacks that those logs discover, there just isn’t enough time in a day to perform their day to day duties.  We are in a recession right now, so what happens if that IT Manager quits for a higher paying job elswhere?  What if their SIEM tool alerts them at 2AM via email or pager and that person is asleep?  What happens at 3AM on Christmas morning when there is an alert and everyone is on vacation or your lone IT guy is sick and sleep from the robitussin?

In my opinion, it is best to go with a SaaS that offers Log Management with 24/7/365 eyes from certified intrusion analysts from the Sans Institue that speak clear English and can fix the problem if the organization can’t be reached via email, pager, or phone call. 

The reputation of the company is certainly worth the cost, and I’m sure it would be cheaper than the $200,000 minimum that I initially mentioned to have a crew to monitor the logs 24/7 if the organization is truly serious about Information Security.  If they are simply looking for a checkbox from an auditor, then Information Security pertaining to Log Management really doesn’t matter, until they get breached.

Maceo D. Wattley, M.A.
Information Security Expert

By Maceo D. Wattley, M.A.


To Anton’s point of “everyone with logs needs log management, but not everyone is mature enough for a SIEM”. I would agree but to a slightly different degree—the reality is there are actually a very small minority of people mature enough for a SIEM simply because the grand vision oversells what SIEM does, which leads to disillusionment, or worse, a false sense of security on the part of the Users.

Adrian says of the Holy Grail for security practitioners: “They want one alert that tells what the problem is…

By Steve Lafferty


Q1 can be said to have SIEM but not log management, based on my experience.

As for confusion, I completely agree, SIM, SIEM, SEM….its like cloud computing. It means everything, and nothing.

By Brian Jones


@Adrian

>Do you know of a SIEM vendor that does not offer Log Management today?

No, there isn’t any. They all learned the lessons and build/bought LM (all except vendor N, I think :-)). Everything else you say is 100% true, IMHO.

However, the opposite is just not true. A lot of smaller log mgt tools vendors have truly nothing to do with a grand vision of SIEM. Think Prism, GFI, even Sawmill, and many others.

So, there is no credible SIEM without LM, but there is plenty of LM without SIEM. As I said in the recent paper, “everybody who has logs needs LM”, but not everybody is mature enough to use a SIEM

Even splunk is very useful for LM and is clearly not a SIEM.

By Anton Chuvakin


@Anton - I am surprised by the comment. Do you know of a SIEM vendor that does not offer Log Management today? Seems they were all forced to adopt the LM technology because customers viewed log collection, log management and log analysis as complementary pieces that solve specific business problems. For the same reason there was a SIM & a SEM, I believe it was the _vendor_ community, attempting to differentiate technology differences/advantages that prompted the analyst community to split hairs. And it’s buyer requirements pushing vendors to offer unified solutions. Same goes for the prominent log management vendors needed to adopt some SEM capabilities to remain competitive. There is a risk of confusing people because these have been separate markets with separate names, but the vendors are well into the process of integrating SIEM & LM because of customer demand.

Just not sure where the confusion would be. We are open to the suggestion, but we would need to validate this claim, and I am not sure I totally understand your reasoning.

This is a really important point so I am wondering if you would elaborate?

-Adrian

By Adrian Lane


While completely merging SIEM and LM is fine for high-level guys like you guys, I think it has a risk of confusing people. So, if SIEM=LM or SIEM+LM=<some new joint thing>, does it make a $50 logging product “a SIEM?”

The use cases for free and low cost log mgt tools have little to do with SIEM.

By Anton Chuvakin


In this series i am looking forward to the “dig into some of the advanced features from some of the leading-edge vendors”

By Ramki B Ramakrishnan


If you like to leave comments, and aren’t a spammer, register for the site and email us at info@securosis.com and we’ll turn off moderation for your account.