Research

“Hi. I’m Mike. And I’m an addict.” I start every chapter of the Pragmatic CSO with those very words. There there are many things you can be addicted to. Thrills. Sex. Sugar. Booze. Drugs. Twitter. Pr0n. Caffeine. Food. Some are worse than others, though none of them really good for you. But now I have to face up to another addiction. The need for gadgets. I’m jonesing for a new MacBook Air. Big time. Like waking up in the middle of the night wanting some SSD goodness in a petite 2lb package. Jonesing, I say, and it’s not pretty. Now there are folks with much worse gadget addiction than me. They are the ones standing in line at Best Buy for the latest Zune. Those folks have a problem. To be clear, so do I. I have a perfectly workable 15” MacBook Pro. It’s been a workhorse for two and a half years. For what I need, it’s fine. Why can’t I be happy with it? Why do I long for something new? The problem is my gear isn’t shiny anymore. I need a new trophy. Need. It. Now. I feel inadequate with a late 2008 MBP. In the bagel shop where I was writing this morning, there was a guy with an MB Air. I felt envy. Not enough to poach his machine when he tok a leak (by the way, it’s two frickin’ pounds – you can take it to the loo), but definitely envy. But then I looked over my other shoulder and saw a guy with an old school Apple laptop. And I mean old school. Like before they had a MagSafe connector, meaning a PowerBook G4. Oh, the horrors. I don’t know how that guy gets out of bed in the morning. And it’s worse when we have a Securosis meeting. Rich gets all the new toys. He’s got an MB Air 11”. I know he scoffs at my MBP. My laptop is older than his kids. Really. But Adrian is a different animal. He’s into high end audio equipment and dogs. My addiction is cheaper. At least I have that going for me. Over two years with the same laptop is a lifetime for me. Some guys trade in their wives every couple years. I trade in my laptop. The Boss likes that approach much better. Normally it’s not an issue, since I tend to hold down a job for 15 months, so I get a new toy every time I get a new job. I get my fix and have no issue, right? Not so much anymore – I’m not changing jobs any time soon. At least that’s what Rich and Adrian keep telling me. But I am getting smarter. Knowing this little issue I have, I made proper provisions this year by doing a side project over the winter and expressly earmarking those fees to breathe the (MB) Air. I’ve got motive. I’ve got opportunity. I’ve even got the funds. I know, you are wondering why I don’t just hop on the Apple web site and order it? This is why. They expect a new Air in the summer. That’s only what, 2 months away? It’ll be worth the wait. That’s what I keep telling myself. It will be smokin’ fast. And shinier. The next 2 months will be a struggle. I want it now. But I’m repressing my urges because I know how bad I’d feel when someone else got the shiny fast one, 4 days after I took delivery of my slow, dull one. I need to do some NLP to associate those bad feelings with the late 2010 MB Air. I will awaken the giant within, just you watch. That will keep me off the gadget juice. I’ll hold out because I have a plan. Every day, I’ll do my affirmations to convince myself that I’m still a good person, even though I use a late 2008 MBP. It will work. I know it will. The power of positive thinking in action. I’ll send a DM to my sponsor every day because I’m not addicted to Twitter. Not yet anyway. That will keep me on the straight and narrow. And doggone it, people like me, right? But we all know what happens when you repress an urge for too long. Gosh, that iPad 2 looks awfully shiny… -Mike Photo credits: “Apple addiction” originally uploaded by new-york-city Since I don’t do enough writing here on the Securosis blog, I figured I’d inflict some pithy verbiage on the victims, I mean readers, of Dark Reading. I’ll be posting on their Hacked Off blog monthly, and started with a doozy on why the RSA breach disclosure was pretty good. Surprisingly enough, I took a contrarian view to all those folks who think they should know everything, even if they aren’t RSA customers. It’s not about you, folks – sorry to bruise your egos. Incite 4 U Mea culpa roll with a side of SQLi: Do you ever wonder what a Barracuda roll tastes like? You can ask the folks in Hong Kong who used an automated SQLi attack to feast on Barracuda’s customer list over the weekend. The good news is that not much data was lost. Some customer and partner names and emails. The bad news is the breach happened because of an operational FAIL to put WAF back into blocking mode. As usual, people are the weakest link. But this disclosure is a great example of how to own it, explain it, and help everyone learn from it. A side of SQLi is not quite as tasty as miso soup, but news of the attack goes down a lot easier with a large serving of mea culpa. – MR Trust No One: I keep stealing a slide Gunnar did a while back (from Chris Hoff, who showed it to me first). It’s a table showing all the big advances in the web and web applications, and then the security tools we use to secure them. In every case, it’s firewalls and SSL. But between the Comodo breach and the

So your key security metrics are collected and shared safely. What comes next? Now we need to start deriving value from the data. Remember, metrics and numbers aren’t worth the storage to keep them, unless you use them as management tools. You need to start comparing the data, drawing conclusions, and adjusting your security program based on the data. OMG, actually making changes based on data rather than shiny objects, breaches, airline magazine articles, and compliance mandate changes. How novel. Remember the goal of this entire endeavor: to show relative progress. Now we get to figure out relative means, which involves defining peer group(s) for comparison. The first group you’ll compare your data to is actually yourself. Yes, this is trend analysis on your own metrics. It will provide some perspectives on whether you are improving – but improving against yourself does not provide perspective on whether you are ‘good’, spending too much money, or focusing on the right stuff. This is where you need to think about benchmarking, or going beyond security metrics. Peer Groups There are ways to define your peer group: Industry: This is your vertical market. Initially (until you have access to loads of data), you will focus on big industry buckets – like defense, healthcare, financial, hospitality, etc. Obviously there are differences between investment banks and insurance companies within the financial vertical, but businesses in the same category will have many consistent business processes which involve collecting very similar types of data. These organizations also tend to have similar geographic profiles – as for example a typical retailer will have a headquarters, regional distribution centers, and tons of stores. Additionally these companies exist under similar compliance/regulatory regimes. They also tend to be relatively consistent in terms of to technology adoption/maturity, which is critical for making relevant comparisons. Company size: Similar to the consistencies we find among companies in the same vertical/industry, we also find many similarities between companies of roughly the same size. For instance large enterprises (10,000+ employees) are generally global by definition – it is very difficult to get that big while focusing on a single geographic region. So organizational models and scale tend to be fairly consistent within a company-size segment. These companies also tend to spend similarly on security. Of course there are always outliers and some industries show less consistency, but we aren’t looking for perfection here. Region: Regional comparisons support many interesting comparisons. Culture and attitudes toward security can be enhanced or hindered by government funding and compliance regimes. We also see relatively consistent technology maturity/adoption within regions – largely based on local drivers such as compliance with laws and other rules, infrastructure, and available talent. Of course, not all metrics apply to any peer group. So when you define your benchmark peer groups, factor this in. The best way is to figure out how the specific metrics correlate for each peer group. We know, it’s math, but you’ll figure out pretty quickly whether there are any useful patterns or consistency within any particular metric. Focus on the metrics with the best correlation across a peer group. Sample Size Now that we’re talking about math, we have to address sample size. That’s basically how much data you need before the benchmark is useful. And as usual it depends, but push for statistical significance over the long term. Why? Because by definition statistical significance means a result is unlikely to occur by chance. You don’t want to be making decisions based on chance and randomness, so that’s our benchmark. More to the point, you want to stop making decisions based on chance. But it’s likely to take some time to get to a statistically significant dataset, so what can you do in the meantime? Look at the distribution, remove the outliers (which screw up your trend lines), and start comparing yourself against the trends you can spot. You can get a decent trend with only a handful of data points for metrics that correlate strongly. Always remember to keep the goal clearly in focus, and that is to identify gaps and highlight success, neither of which requires a huge amount of data. But to be clear, you are looking over time for statistical significance. Reverting to the Mean Another issue is whether you want to “revert to the mean,” meaning you look like everyone else in the peer group. Once again, it depends. Let’s take a look at a couple of likely metrics categories: For spending, it’s unlikely that you are getting a reasonable return from security spending 3 standard deviations above the mean. Not unless you can differentiate your product/offering on security, which is rare. For incidents, you want to be better than the mean. Most likely significantly so. Why? Because all your years of hard work can be unwound with one high profile breach. So the more effectively and quickly you respond and contain the damage, the better. Here you definitely don’t want to be in the bottom quartile, which indicates a failure of incident response and should be unacceptable to senior management. For efficiency, effectiveness, and coverage metrics (most of the easily quantifiable and operational metrics), you want to be better than the mean. That shows operational competence. In terms of importance, your spending is usually the most visible (to the folks who pay the bills, at least), so be in the ballpark there. Incidents come next, as they have a direct impact on issues like availability and brand damage. Then comes the operational stuff – it’s certainly important to how you run the security program, but rarely interesting to the muckety-mucks. Now it’s time to tell those muckety-mucks what you found, which means focusing on the commmunication strategy underlying your benchmarking program, so that’s where we’ll focus in the next post. Share: