I am happy to announce the release of a research paper a long time in the making: Security Analytics with Big Data. This topic generates tons of questions from end users, and we get them from large and mid-sized enterprises alike. The goals of this research project were threefold:

The research outline

  • Describes what security analytics with big data is and what it looks like
  • Discusses how it is different than past tools and platforms
  • Discusses the main use cases

These topics mirror our early discussions around security analytics. Big data is a very new and very disruptive trend, so how we might use big data to help with security problems was interesting to the community as a whole. Answering questions about how to leverage virtually free NoSQL analytics tools to do a better job of detecting security events is important – both for what is possible and to provide a picture of where the industry is heading.

The story behind the research

But a funny thing happened during the research – during interviews people invariably wanted to know how it works within their environment. Many people did not want to just start evaluating security analytics options – they were keen to leverage existing investments and build on infrastructure they already own. The backstory is relevant because this ended up becoming three contiguous research projects, and then we massaged the content into this final paper to address the full breadth of questions.

When I begun this work a year ago I wanted to fully describe the skunkworks projects I was seeing at some small and mid-sized firms. Both security companies and motivated individuals were using multiple NoSQL variants to detect security problems, often either with a new approach or at a unprecedented cost we had not seen before. Those trends are reflected in this research. Along the way I spoke with 20 large enterprises, and I kept getting the same request: “We are interested in security analytics, but we want to blend both the data and analysis with existing investments”. Most of the time these firms were referring to SIEM, but occasionally they had data warehouses with other information they wished to reference as well. That is also reflected in the paper. But when I got to this point, things got a bit odd.

Once our research papers are completed we see if companies are interested in licensing our research to educate employees, customers, or the larger IT community. The responses I got were, “This is not in line with our position”, “This research does not reflect what we see”, “This research does not differentiate our solution” and “Our SIEM was big data before there was big data”. The broader scope of this research generated a degree of negative feedback which got me thinking I had totally missed the mark, asked the wrong questions, or simply talked to too few of customers. I spent another 6 months going through new interviews with a broader set of questions, and speaking to more data architects, vendors, and would-be customers. Retracing my steps reaffirmed that the research was on target, and I feel this paper captures the market today. Customer interest and inquiries outpace what the vendor community is prepared to offer, and customers are asking for capabilities outside the vendor storylines. So this paper tells a decidedly different story than what you are likely to hear elsewhere.


First and foremost, this is a research paper to educate end users on what security analytics with big data is, the value it provides, and how to distinguish big data solutions from pretenders. That is its core value.

If you are going to “roll your own” big data security analytics cluster, this research provides a sample of what other firms are doing, architectures they use, and the underlying components they leverage to support their work. It will help you understand what types of data you probably already have at your disposal, and what observations you can derive from it.

If you are looking to acquire a big data analytics solution this research will help you understand potential risks in realizing your investment and help with rollout and integration.

You can download a copy on its landing page: Security Analytics with Big Data.

We hope you find this information helpful, and as always please ask questions or provide feedback on the blog.