It seems like BigData is all the rage. With things like NoSQL and Hadoop getting all the database wonks hot under the collar, smart forward-thinking folks like Amrit and Hoff increasingly point out the applicability of these techniques to security, and they’re right. I certainly agree that many of these new technologies will have a huge impact on our ability to figure out what’s happening in our environments. And not a moment too soon. Hoff wrote a couple recent posts discussing the coming renaissance of Big Data and Security (InfoSec Fail: The Problem with BigData is Little Data and More on Security and BigData…Where Data Analytics and Security Collide), and Amrit followed up with BigData, Hadoop, and the Impending Informationpocalypse, making great points about the fragility of any (relatively) new technology, as well as the need to really know what we are looking for. That’s the biggest fly in this BigData/security ointment. We need proper context to draw useful conclusions about anything. More data does not provide more context. If anything, it provides less because these analysis tools are only as good as the rules they use to alert us to stuff. It’s non-trivial to get this right. Even with the best infrastructure, monitoring everything all the time, you still need to know what to look for. And it won’t get any easier. Knowing what to look for will get much more complicated. The volume of data promises to mushroom over the next few years, as full packet capture starts to hit the mainstream and more folks start seriously monitoring databases and applications. This will ripple through the entire monitoring ecosystem. Now any company claiming the ability to do security management/analysis will need to not only have some security ninja on staff (to know what to look for), but also some legitimate BigData qualifications. This isn’t a new direction for the SIEM players. More than one vendor calls what they do security intelligence, modeled after the business intelligence market. That entails a BigData approach to business analysis. To get there, the SIEM vendors have built their own BigData platforms. This means they each have a purpose-built data store that can provide the kind of analysis and correlation required to find the proverbial needle in a stack of haystacks. They invested not because they wanted to build their own data stores, but because no commercial or open source technology could satisfy their requirements. Do Hadoop and these other technologies change that? Maybe. As Amrit points out, new technologies can be brittle, so it will be a while before tools (or services) based on these latest technologies are ready for prime time. But the writing is on the wall. Security is a BigData problem, and it’s not a stretch to think that some enterprising souls will apply BigData technologies to the security intelligence problem. Which is a great thing – we certainly have not solved the problem. OMG, maybe we will see some innovation in security soon. But I’m not holding my breath. Share:
