Research

This is part 2 of our series on skepticism in security. You can read part 1 here. Being a bit of a science geek, over the past year or so I’ve become addicted to The Skeptics’ Guide to the Universe podcast, which is now the only one I never miss. It’s the Skeptics’ Guide that first really exposed me to the scientific skeptical movement, which is well aligned with what we do in security. We turn back to Wikipedia for a definition of scientific skepticism: Scientific skepticism or rational skepticism (also spelled scepticism), sometimes referred to as skeptical inquiry, is a scientific or practical, epistemological position in which one questions the veracity of claims lacking empirical evidence. … Scientific skepticism utilizes critical thinking and inductive reasoning while attempting to oppose claims made which lack suitable evidential basis. … Characteristics: Like a scientist, a scientific skeptic attempts to evaluate claims based on verifiability and falsifiability rather than accepting claims on faith, anecdotes, or relying on unfalsifiable categories. Skeptics often focus their criticism on claims they consider to be implausible, dubious or clearly contradictory to generally accepted science. This distinguishes the scientific skeptic from the professional scientist, who often concentrates their inquiry on verifying or falsifying hypotheses created by those within their particular field of science. The skeptical movement has expanded well beyond merely debunking fraudsters (such as that Airborne garbage or cell phone radiation absorbers) into the general promotion of science education, science advocacy, and the use of the scientific method in the exploration of knowledge. Skeptics battle the misuse of scientific theories and statistics, and it’s this aspect I consider essential to the practice of security. In the security industry we never lack for theories or statistics, but very few of them are based on sound scientific principles, and often they cannot withstand scientific scrutiny. For example, the historic claim that 70% of security attacks were from the “insider threat” never had any rigorous backing. That claim was a munged up “fact” based on the free headline from a severely flawed survey (the CSI/FBI report), and an informal statement from one of my former coworkers made years earlier. It seems every day I see some new numbers about how many systems are infected with malware, how many dollars are lost due to the latest cybercrime (or people browsing ESPN during lunch), and so on. I believe that the appropriate application of skepticism is essential in the practice of security, but we are also in the position of often having to make critical decisions without the amount of data we’d like. Rather than saying we should only make decisions based on sound science, I’m calling for more application of scientific principles in security, and increased recognition of doubt when evaluating information. Let’s recognize the difference between guesses, educated guesses, facts, and outright garbage. For example – the disclosure debate. I’m not claiming I have the answers, and I’m not saying we should put everything on hold until we get the answers, but all sides do need to recognize we have no effective evidentiary basis for defining general disclosure policies. We have personal experience and anecdote, but no sound way to measure the potential impact of full disclosure vs. responsible disclosure vs. no disclosure. Another example is the Annualized Loss Expectancy (ALE) model. The ALE model takes losses from a single event and multiplies that times the annual rate of occurrence, to give ‘the probable annual loss’. Works great for defined assets with predictable loss rates, such as lost laptops and physical theft (e.g., retail shrinkage). Nearly worthless in information security. Why? Because we rarely know the value of an asset, or the annual rate of occurrence. Thus we multiply a guess by a guess to produce a wild-assed guess. In scientific terms neither input value has precision or accuracy, and thus any result is essentially meaningless. Skepticism is an important element of how we think about security because it helps us make decisions on what we know, while providing the intellectual freedom to change those decisions as what we know evolves. We don’t get as hung up on sticking with past decisions merely to continue to validate our belief system. In short, let’s apply more science and formal skepticism to security. Let’s recognize that just because we have to make decisions from uncertain evidence, we aren’t magically turning guesses and beliefs into theories or facts. And when we’re presented with theories, facts, and numbers, let’s apply scientific principles and see which ones hold up. Share:

Note: This is the first part of a two part series on skepticism in security; click here for part 2. Securosis: A mental disorder characterized by paranoia, cynicism, and the strange compulsion to defend random objects. For years I’ve been joking about how important cynicism is to be an effective security professional (and analyst). I’ve always considered it a core principle of the security mindset, but recently I’ve been thinking a lot more about skepticism than cynicism. My dictionary defines a cynic as: a person who believes that people are motivated purely by self-interest rather than acting for honorable or unselfish reasons : some cynics thought that the controversy was all a publicity stunt. * a person who questions whether something will happen or whether it is worthwhile : the cynics were silenced when the factory opened. 1. (Cynic) a member of a school of ancient Greek philosophers founded by Antisthenes, marked by an ostentatious contempt for ease and pleasure. The movement flourished in the 3rd century BC and revived in the 1st century AD. Cynicism is all about distrust and disillusionment; and let’s face it, those are pretty important in the security industry. As cynics we always focus on an individual’s (or organization’s) motivation. We can’t afford a trusting nature, since that’s the fastest route to failure in our business. Back in physical security days I learned the hard way that while I’d love to trust more people, the odds are they would abuse that trust for self-interest, at my expense. Cynicism is the ‘default deny’ of social interaction. Skepticism, although closely related to cynicism, is less focused on individuals, and more focused on knowledge. My dictionary defines a skeptic as: a person inclined to question or doubt all accepted opinions. * a person who doubts the truth of Christianity and other religions; an atheist or agnostic. 1. Philosophy an ancient or modern philosopher who denies the possibility of knowledge, or even rational belief, in some sphere. But to really define skepticism in modern society, we need to move past the dictionary into current usage. Wikipedia does a nice job with its expanded definition: an attitude of doubt or a disposition to incredulity either in general or toward a particular object; the doctrine that true knowledge or knowledge in a particular area is uncertain; or the method of suspended judgment, systematic doubt, or criticism that is characteristic of skeptics (Merriam-Webster). Which brings us to the philosophical application of skepticism: In philosophy, skepticism refers more specifically to any one of several propositions. These include propositions about: an inquiry, a method of obtaining knowledge through systematic doubt and continual testing, the arbitrariness, relativity, or subjectivity of moral values, the limitations of knowledge, a method of intellectual caution and suspended judgment. In other words, cynicism is about how we approach people, while skepticism is about how we approach knowledge. For a security professional, both are important, but I’m realizing it’s becoming ever more essential to challenge our internal beliefs and dogmas, rather than focusing on distrust of individuals. I consider skepticism harder than cynicism, because we are often forced to challenge our own internal beliefs on a regular basis. In part 2 of this series I’ll talk about the role of skepticism in security. Share:

Last week, Mike Rothman of eIQ wrote a thoughtful piece on the struggles of the SIEM industry. He starts the post by saying the Security Information and Event Management space has struggled over the last decade because the platforms were too expensive, too hard to implement, and (paraphrasing) did not scale well without investing a pound of flesh. All accurate points, but I think these items are secondary to the real issues that plagued the SIEM market. The issue with SIEM’s struggles in my mind was twofold: fragmented offerings and disconnection with customer issues. It is clear that the data SIM, SEM, and log management vendors collected could be used to provide insights into many different security issues, compliance issues, data collection functions, or management functions – but each vendor covered a subset. The fragmentation of this market, with some vendors doing one thing well but sucking at other important aspects, while claiming only their niche merited attention, was the primary reason the segment has struggled. They created a great deal of confusion through attempts to differentiate and get a leg up. Some did a good job at real-time analysis, some provide forensic analysis and compliance, and others excel at log collection and management. They targeted security, they targeted compliance, they targeted criminal forensics, and they targeted systems management – but the customer need was always ‘all of the above’. Mike is dead on that the segment has struggled and it’s their own fault due to piecemeal offerings that solved only a portion of the problems that needed solving. More attention was being paid to competitive positioning than actually solving customer problems. For example, the entire concept of aggregation (boiling all events into a single lowest common denominator format) was ‘innovation’ for the benefit of the vendor platform and was a detriment for solving customer problems. Sure, it reduced storage requirements and sped up reporting, but those were the vendor’s problems more than customer problems. The SIEM marketplace has gotten beyond this point, and it is no longer a segment struggling for an identity. The offerings have matured considerably in the last 3-4 years, and gone is the distinction between SIM, SEM and log management. Now you have all three or you don’t compete. While you still see some vendors pushing to differentiate one core value proposition over another, most vendors recognize the convergence as a requirement, as evidenced by this excellent article from Dominique Levin at Loglogic on the Convergence of SIEM and log management, as well as this IANS interview with Chris Peterson of LogRhythm. The convergence is necessary if you are going to meet the requirements and customer expectations. While I was more interested in some of the problems SIEM has faced over the years, I have to acknowledge the point Mike was making in his post: the SIEM market is being hurt as platforms are oversold. Are vendors over-promising, per Dark Reading? You bet they are, but when have you met a successful software salesperson who didn’t oversell to some degree? A common example I used to see was some of the sales teams claiming they offered DLP equivalent value. While some of the vendors pay lip service to the ability to provide ‘deep content inspection’ and business analytics, we need to be clear that regular expression checks are not deep content analysis, and capturing network packets is a long way from providing transactional analysis for fraud detection or policy compliance. What gets oversold in any given week will vary, but any technology where the customer has limited understanding of the real day-to-day issues is a ripe target. Conversely, I find customers I speak with being equally guilty as they promote the ‘overselling’ behavior. SIEM platforms are at the point where they can collect just about every meaningful piece of event data within the enterprise, and they will continue to evolve what is possible in analysis and applicability. Customers are not stupid – they see what is possible with the platforms, and push vendor as hard as they can to get what they want for less. Think about it this way: If you are a customer looking for tools to assist with PCI-DSS, and the platform cannot a) provide the near-real time analysis, b) provide forensic analysis, and c) safely protect its transaction archives, you move onto the next vendor who can. The first vendor who can (or successfully lies about it) wins. Salesmen are incentivized to win, and telling the customer what they want to hear is a proven strategy. So while they are not stupid, customers do make mistakes, and they need to perform their due diligence and challenge vendor claims, or hire someone who can do it for them, to avoid this problem. I am very interested to see how each vendor invests in technology advancement, and what they think the next important step in meeting business requirements will be. What I have seen so far indicates most will “cover more and do more”, meaning more platform coverage and more analysis, which is a safe choice. Similarly, most continue to offer more policies, reports, and configurations that speed up deployment and reduce set-up costs. Some have the vision to ‘move up the stack’, and look at business processing; some will continue to push the potential of correlation; while others will provide meaningful content inspection of the data they already have. Given that there are a handful of leading vendors in this space on a pretty even footing, which advancement they choose, and how they spin that value, can very quickly alter who leads and who follows. The value proposition provided by SIEM today is clearer than at any time in the segment’s history, and perhaps more than anything else, SIEM platforms are being leveraged for multiple business requirements across multiple business units. And that is why we are seeing SIEM expand despite economic recession. Because many of the vendors are meeting revenue goals, we will both see new investments in the technology, and

I first met Mike Andrews about 3 years ago at a big Black Hat party. Turns out we both worked in the concert business at the same time. Despite being located nowhere near each other, we each worked some of the same tours and had a bit of fun swapping stories. Mike managed to convince his employer to put up a well-designed series of webcasts on the basics of web and web application security. Since Mike wrote one of the books, he’s a great resource. Here’s Mike’s blog post, and a direct link to the WebSec 101 series hosted by his employer (he also gives out the slides if you don’t want to listen to the webcast). This is 101-level stuff, which means even an analyst can understand it. Share: