Research

As we wrap up our series on Security Benchmarking, we find it instructive to actually walk through a scenario and apply the process. Yes, the scenario is a bit contrived, but we’ll use it to hit the high points of the process, deciding where to start, collecting the data, establishing the peer group and communicate the findings. Keep in mind that we focus on getting quick wins, showing immediate value, building momentum and leveraging that momentum for programatic success. Scenario For our case study, let’s use a mid-tier financial company as our example. I’d say large enterprise, but in reality there are a lot of nuances and moving pieces within a large enterprise that need more detailed discussion. So let’s keep it relatively simple. Likewise, we picked the financial vertical because of 1) need and 2) availability of data. The reality of the financial industries regulatory oversight has created a general perspective of security first and data-centricity (yes, these are the folks that try to do risk management for a living) means these businesses are move likely to embrace a benchmarking mentality. In our (contrived) scenario, the Board drove the hiring a new CISO to “fix security.” As easy as it is to think this was just catering to a board directive, the senior team seems to have a commitment to fix things and do it the right way. So the CISO has a clear honeymoon period and some leeway in thinking somewhat unconventionally about how to build the security program. The new CISO still spends some time figuring out what’s installed and what’s not working, but he knows the organization has AV deployed, they use an external scanning service, and do a pretty good job of patching on internal systems. Yet, like many smaller financial institutions they use hosted applications for most of their business processing. So a lot of their data is not within their direct control. Over the past few years, the organization has had a handful of incidents, but none really resulted in major data loss. Thus the CISO was pleasantly surprised when he got the mandate to fix the security program, when it wasn’t outwardly broken. The senior team came to the conclusion they are living on borrowed time and want to act decisively to make sure they are ready when the brown stuff hits the fan (which it inevitably will). See? We told the you the scenario was contrived, but without a senior-level mandate to make changes in implement a security program, getting any kind of security metrics/benchmarking initiative going will be difficult. Where Do You Start? Now the CISO has to figure out where to start. He’s decided that he wants to figure out where his most apparent gaps are. You know, the ones you can drive a Mack Truck through. So he starts with a comprehensive risk assessment to build a baseline, but he also wants to compare his environment to other like-sized companies (both in and out of his industry) to figure out how he compares to those organizations. Keep in mind, boiling the ocean and trying to do everything at this point is a bad idea. He’d get buried in the nuances of the data and not get anything done, which could endanger his entire security program. So he needs to ask the following questions: What do you need to achieve? Where are the key operational problems? This is where you always have to start. In our case study, the CISO is looking to identify his most critical gaps, and given the luck they’ve had in not having a huge data loss even with a few breaches, he wants to start with incident response. What data do you have? Next you have to figure out if you have the data or can get it easily. With incident data, the reality is the findings from the forensics investigations exist, but haven’t been put in any kind of format for comparison. But the data exists, so it makes sense to keep pressing down this path. If the data doesn’t exist or can’t be gathered quickly, then it’s time to look at Plan B. You don’t want to hold up the effort because it’s all about getting the quick win. Where will be most impactful to show comparative data? Selecting to focus initially on incident response represents a pretty shrewd move for the new CISO. He knows the board and senior management is sensitive to not getting nailed, as well as having a set of reasonable consensus metrics available (from CIS), and having the data. This increases the chances of success. Peer Groups and Service Providers Next, our CISO has to define the peer group for analysis. This isn’t brain surgery. He’ll need to compare to other financials (duh!), but also companies in other regulated industries (like healthcare and utilities) of a similar size. The good news is there are a ton of mid-sized hospital groups, as well as many community utilities, with similarly sensitive data. But how do they get their hands on that kind of data for comparison purposes? Now we go back and revisit the selection criteria for any kind of provider you’d think about for benchmarking services. Remember, these folks have to 1) have access to the data you’d need and 2) be able to protect the data you share with them. To be clear, you may not be able to get everything done with just one provider. In our case study here, the CISO will actually pick two. The first is his regional bank ISAC, who has been gathering data from its members for a while. The second is a commercial benchmarking offering, since they have more data about other industries that aren’t the focus of the ISAC. In reality, the CISO would like to just have one provider, but until a critical mass of data for many verticals is captured, he’ll need to piecemeal the solution to solve the problem. Analyze Equipped with data regarding his first

As is (now) our custom, we post a set of links to each blog series as it wraps up. This both gives us an easy way to find all our posts, and acknowledges that not everyone wants our complete feed and may want to read posts once they’re all written. As always, we love feedback on our work in progress. Yes, it’s time consuming to take time out for comment on specific posts. But remember that pretty much all our research is available free of charge, so it’s not too much to ask for a little constructive criticism on our work, is it? Please take a look and sink your teeth in. Introduction Security Metrics (from 40,000 feet) Collecting Data Systematically Sharing Data Safely Defining Peer Groups and Analyzing Data Communications Strategies Continuous Improvement You Can’t Benchmark Everything Benchmarking in Action As always, we thank you for reading, commenting, and making our research better. Share:

We have spent much of this series on why benchmarking is important. But we also need to point out some situations where benchmarking may not be appropriate. There are clearly situations where you can’t benchmark, particularly is on granular operational data, which I call Ninja Metrics. Dependency: Peer Group Data Most organizations have ‘nascent’ metrics programs, which may actually be too kind. But not all. Some have embraced detailed programs that gathers all sorts of data, mostly focused on operations. This represents the next step of a metrics program, and can be represented by some of the ideas put forth through our Quant research projects. We have created highly granular process maps (with associated metrics) for Patch Management, Network Security Operations, and Database Security. Each report specifies 50+ distinct metrics you can measure for that discipline. Yes, they are comprehensive. But there is a clear issue regarding benchmarking at this level. You will have a hard time finding similarly granular data from other companies for comparison. So the key dependency in implementing a benchmarking effort is the availability of peer group data for comparison. Compare to Yourself What do world class athletes do when they reach the top of the heap? You know, folks like Michael Phelps, who has basically shattered every record there is to shatter. They start comparing themselves to their past performance. Improvement is measured internally rather than externally. Even if no one else has ever done better, you know you can. And this is what you will likely need to do the most granular operational functions. When you take a step back this makes a lot of sense. The reality is that you aren’t necessarily trying to ‘win’ relative to operational excellence. You want to improve. That said, it is important to have an idea of where you stand in comparison to everybody else, at least on the high-level operational metrics. But for the most granular metrics, not so much. We hope that over time enough companies will start tracking granular operational metrics, and become comfortable enough with benchmarking, to share their data. But that’s not going to happen tomorrow or even the day after. In the meantime you can (and should) continue to push your metrics program forward – just understand your comparisons may need to be internal. As we wrap up the Benchmarking series, we’ll look at how to get some Quick Wins and see the process in action. Share:

The last two nights, we have celebrated Passover. Basically, we have a big dinner commemorating the escape of our forefathers from bondage and slavery in Egypt. At least that’s how the story goes, although I wasn’t there, so I maintain a healthy skepticism regarding burning bushes, parting seas, and plagues. But the point remains whether or not the stories are true. It’s really an excuse to party with friends and family, and enjoy some time together outside the craziness of day-to-day existence. I’m not unique in having a pretty hectic existence. For instance, the twins play baseball/softball, which means we were at the field both Saturday and Sunday, for a total of 5 games. Combined with the oldest one preparing for a dance show in a few weeks, we hardly had time to hit the head all weekend. But a close friend had a birthday party to celebrate her 40th Friday, so we had to take a break and celebrate. I did not squander the opportunity, and got rather festive with the help of some vanilla rum. OK, it might have been a lot of vanilla rum. If you find my liver, feel free to mail it back to Securosis Central. Adrian the mailman likes that kind of care package. Many of us don’t intentionally party enough. So I actually appreciate the religious holidays interspersed throughout the year. For me it’s not about the dogma, or whether what we are celebrating actually happened or not. And most of the time we don’t spontaneously start throwing food at each other. It’s about turning off the distractions and focusing on family and friends, if only for a night or two. We actually talk, as opposed to planning the next day’s activities. We eat (too much) and until you’ve experienced it, you can’t appreciate a Manischewitz Concord Grape hangover. A lot of our personal history is tied to these holiday celebrations, providing stories we tell for a lifetime. Like when – despite Mom’s stern warning not to get dirty – I fell into a stream behind my babysitter’s house, fancy corduroy pants and all. It was great fun but Mom was not amused. I think she’s still fuming. And it didn’t even involve Concord Grape. We can even make the wacky traditions fun. For instance, on Passover the kids hunt for a piece of Matzoh hidden in the house (it’s called the Afikomen), and if they find it they get a couple bucks. Which is huge progress, because I was lucky to get a piece of chocolate from my grandfather back in the day. Given this year’s bounty ($2 for each kid), and my oldest daughter’s big spending plans, she was very concerned that I wouldn’t make good on my financial obligations. I’m afraid I didn’t help the situation when I mentioned my new policy of charging $2 per month for rent. Imagine that – I can be difficult sometimes. Obviously I made good on the gift, but not before I had her unknowingly play back one of my favorite movie scenes. I asked her to say “I want my $2” about 10 times, and she didn’t understand why I was rolling on the floor. Too bad it was a school night, or I would’ve made her get on her bike and chase me around the neighborhood screaming “I want my $2.” Really, that’s not bad parenting, is it? Some folks figure they are Better Off Dead than suffering through yet another family holiday. But not me – I can make almost any occasion a big party. And I do. -Mike Photo credits: “La Tomatina / Spain, Bunol” originally uploaded by flydime I would be negligent if I didn’t call attention to a major milestone that one of us hits today. That’s right, the baby of the bunch, the rich mogul turns 40. Today. I’d say that’s old, but I still have 2+ years on him, and a lot more gray hair. Rich is taking a vacation day (as he should) and my hope is that he’ll take a step back to appreciates all he has and has done over the past 4 decades. He has a great wife and kids, he’s building a great business, and he’s one of the top dogs in this little game we play. So when you have your nightcap, after a typically hard day in the trenches of security, raise your glass to Rich and know that the next 40 will be better than the last. Incite 4 U Understand the real threat: Given all the (justified) bluster around the Verizon Data Breach Report, we can’t forget the need to understand what’s really at risk and how it is most likely to be compromised. Ax0n does a great job of reminding us by talking about the real insider threat, reminiscing about the hoops he’s had to jump through in order to remotely manage a server (legitimately, apparently). Then he contrasts that against the fact that other folks take the company’s most sensitive data outside on laptops and USB keys, posing a much more serious risk than a conscientious admin trying to fix things from home. Especially when the internal controls make life hard for people who don’t care about security. His point is that we need to match the controls (and security rhetoric) to the threat, and make sure it’s not onerous to drive creative folks to find a way around security. Remember, most folks believe security is not their job – it’s yours. You can make the case that it’s everyone’s job and you wouldn’t be wrong. But sales guys have to meet their quota each quarter, and that’s more important than meeting your rules. – MR DBIR poop commences: It took about a nanosecond, but as Rich predicted, the Verizon Data Breach Investigations Report is already being misquoted and misinterpreted. More breaches being investigated does not necessarily mean there were more breaches, but that’s the poop already hitting the wire. I understand the rush to get an article live, but they should at least read some of the report before editorializing. The general public

In a world full of TLAs (three letter acronyms), none resonates for security people as strongly as FUD. Or Fear, Uncertainty, and Doubt for you n00bs. Many of us rail at the offensive use of FUD in security sales. But let’s take a step back and acknowledge that security is like insurance. With very rare exceptions, security doesn’t help anyone sell more stuff. It doesn’t really help companies operate more efficiently. It’s basically about controlling downside risk. So it’s like insurance. You don’t buy health insurance because you want to. It doesn’t add anything to your life. It prevents you from going belly up if you have some catastrophic issue. It can maybe cut your medical bills if you are chronically ill or injury prone. But clearly you buy insurance because you feel you need to, not because you want to. Insurance brokers (at least all those I’ve dealt with) also leverage FUD in their sales cycle. They paint the picture of downside risk, which always involves preying on some fear of getting hurt, sick, etc. If I had a crystal ball and knew I (or anyone in my family) wouldn’t get sick, I’d drop health insurance like a hot potato. And your senior management is in exactly the same boat. If they thought there was no risk of losing protected data or intellectual property, you’d be out on your ass. So there will always be some level of FUD in our activities as security folks. I talked about using FUD as an end user a few years back, as well as more recently. So there can be legitimate uses of FUD to create urgency and provide a catalyst for funding. But let’s stay focused on security vendors using FUD to get you to buy their stuff. I realize it’s part of the game and I have accepted that. I don’t like it, but I accept it. But that doesn’t mean all FUD is created equal. So let’s attempt to break FUD down into a couple categories and (with your help) understand the impact of each type of FUD on the sales cycle. In this post, I’ll break down the categories of FUD we see most frequently. I started this discussion last week on Twitter, and got some great feedback. Hopefully we’ll get some more feedback on the blog (You! Yes, you! Get over to the blog and add some comments!) and come to some consensus about which kinds of FUD are common in practice. Then we’ll put together a survey to see if we can get some level of understanding about what is acceptable FUD vs. unacceptable. Dare I say it – maybe even useful FUD. In a perfect world, all our friends in the vendor community would take this feedback to heart and stop slinging bad FUD. Oy, such optimism. So here goes (in no particular order): Attack du jour press release: You know what I’m talking about here because these press releases show up in your inbox just about every day. This is the “you can stop StuxNet with our box” type release, where the vendor is trying to capitalize on some external event to get you to answer the phone. Similar to getting a call for travel insurance just after an airliner goes down. Threat reports: Almost every vendor has some kind of research capability now, so these reports basically list out which attacks and/or vulnerabilities they are seeing. Maybe they throw in some trend analysis as well. The idea is to keep your attention on common attacks, which are then addressed by the vendor’s widget or service. Breach reports: These reports are different from threat reports in that the objective is to actually study breaches – in an attempt to pinpoint both the breach’s impact and root causes. With this analysis a vendor/service provider hopes to educate potential customers on what causes breaches and how to address the risks (hopefully with their own products/services). Of course, the Verizon Data Breach report is the granddaddy of this kind of analysis. Check out Rich’s analysis of the 2010 report. Vendor surveys/peer group FUD: If you are a CISO, you get probably a dozen calls/emails a week to fill out one survey or another. Do you do this? Have you suffered from that? The vendors and researchers (like Ponemon) then assemble the data to build a case about what the masses are doing, or more likely aren’t doing. James McGovern accurately called this peer group FUD because it tries to trigger action by pointing out that either buddies friends are (or aren’t) doing something specific, and therefore you should. This also applies to the Security Benchmarking research I’m doing right now. Making security/compliance easy: One of my personal favorites: you still see vendors market events and position products with promises that using their gear will make either security or compliance (or both!) easy. And if you aren’t using their gear, your life is unnecessarily hard. Sponsored lab tests: You tend to see this kind of FUD during the sales cycle, when a vendor tries to convince you they are great and the competitor is crap, because the vendor paid some guy in a lab to run a test to which demonstrated something attractive about the vendor’s product or service. Some publications also run lab tests which straddle the line. It’s rare for money to directly change hands, but there can be backroom ad-buying hijinx. Our legal budget is rather limited so I won’t name names – these folks tend to be rather litigious – but you know who I’m talking about. Competitor sniping: Don’t you love it when vendors come in, and spend more time talking about why their competitors suck than about why they are good and how they can help you? Yeah, I hate that too. That’s competitor sniping in all its seedy glory. Cost of breach/attack analysis: We also see folks (like Larry Ponemon) who have built great businesses doing more targeted surveys, trying to understand what these security/compliance/breach issues actually cost companies. Clearly the idea is to derive an objective number that you (the practitioner) can use internally to talk about how bad it would be if something unfortunate were to happen. Yes, this

If you don’t already have attackers in your environment you will soon enough, so we have been spending a lot of time with clients figuring out how to respond in this age of APT (Advanced Persistent Threat) attackers and other attacks you have no shot at stopping. You need to detect and respond more effectively. We call this philosophy “React Faster and Better”, and have finally documented and collected our thoughts on the topic. Here are a couple excerpts from the paper to give you a feel for the issue and how we deal with it: Incident response is near and dear to our philosophy of security – it’s impossible to prevent everything (we see examples of this in the press every week), so you must be prepared to respond. The sad fact is that you will be breached. Maybe not today or tomorrow, but it will happen. We have made this point many times before (and it has even happened to us, indirectly). So response is more important than any specific control. But it’s horrifying how unsophisticated most organizations are about response. In this paper we’ll focus on pushing the concepts of incident response past the basics and addressing gaps in how you respond relative to today’s attacks. Dealing with advanced threats requires advanced tools. React Faster and Better is about taking a much broader and more effective approach on dealing with attacks – from what data you collect, to how you trigger higher-quality alerts, to the mechanics of response/escalation, and ultimately to remediation and cleaning activities. This is not your grandpappy’s incident response. To be clear, a lot of these activities are advanced. That’s why we recommend you start with our Incident Response Fundamentals from last year to get your IR team and function in decent shape. Please be advised that we have streamlined the paper a bit from the original blog series, cutting some of the more detailed information on setting up response tiers. We do plan to post the more complete paper at some point over the next couple months, but in the meantime you can refer back to the RFAB index of posts for the full unabridged version. A special thanks to NetWitness for sponsoring the research. Download: React Faster and Better: New Approaches for Advanced Incident Response (PDF) Share:

Just in case you had nothing to do over the weekend, I came up with some homework to catch you up on our Security Benchmarking series. We’re clicking right along and think the content is kickass. So check it out, comment, and let us know if we are smoking crack. Introduction Security Metrics (from 40,000 feet) Collecting Data Systematically Sharing Data Safely Defining Peer Groups and Analyzing Data Communications Strategies Continuous Improvement We’ll be wrapping the series up next week with 3 more posts. So please contribute while you have the chance. Share:

The simple fact is that most folks senior security folks came from the technical side of the house. They started as competent (if not studly) sysadmins or security administrators, drew the short straw, and ended up with management responsibility. But very few of these folks ever studied management, gone through management training, or done anything but learned on the job. This creates a situation where senior security folks spend a lot of time doing stuff, but not enough time talking about it. The huge disconnect is inadequate communication of both success and failure up and down the management stack to key security stakeholders. In fact, the Pragmatic CSO methodology originated largely to help technical folks figure out how to deal with their management responsibilities. The inability to communicate to key stakeholders will absolutely kill a benchmarking program because benchmarking entails ongoing incremental effort to gather metrics, as well as to compare against benchmarks and perform analysis. The benchmark must provide additional value, which must be communicated in order to make the effort worthwhile. As we all know, nothing really happens by itself. You need to build a systematic communications/outreach effort to leverage the benchmark data, specifically targeting a number of constituencies important to the success of any security practitioner. Let’s dig into how that’s done, because it’s a critical success factor for any benchmarking initiative. Understanding your audience The first rule of communications is to do it consistently and repetitively by telling them what you are going to say, saying it, and then telling them what you just said. It sounds silly, but given today’s over-saturated environment where the typical C-level exec has the attention span of a 2-year-old, you don’t have a choice. Effective communications requires more than just talking a lot – you need to tailor your message to the audience. This is something security folks have always stunk at. If you’ve ever uttered the words “AV coverage” or “firewall rules” in a management meeting you know what I mean. Senior management If there is one thing you should appreciate about senior management, it’s that they are fairly predicable. Their interests involve things that directly impact revenues/expenses. Period. They don’t want to know the details of how you do something unless it’s off the rails. They want to know the bottom line and whether/how it will impact their ability to get paid their full bonus at the end of the year. So we focus on incident data and budget efficiency. They want to know whether incidents have impacted availability and thus cost them money. They need to know about disclosures, with an eye towards brand damage. And they need to know how you do relative to peers – if only make themselves feel better that their competitors probably won’t be getting those bonuses this year either. Getting time with senior folks is challenging. So you’ll be doing well if you can get quarterly face time to go through the metrics/results/benchmarks. At a minimum you need to make your case annually ahead of budgeting, but that is not really frequent enough to get sufficient attention to successfully execute on your program. Finally, how can benchmark data help you with these folks? You can use the fact that in terms of overhead functions most senior managers are lemmings – if everybody else is doing it (whatever it is), they will be likely to follow suit. It’s an ugly job, but someone has to do it. CIO Odds are you report in through the technology stack, which means you’ll spend some time with the CIO. This is a good thing, but keep in mind that the CIO’s primary goal is to look good to senior management. We all know that security issues can make him/her look very bad. So we can focus on what interests senior management: incidents and budget efficiency. But with the CIO you should add high-level operational trending data, which highlights issues and/or shows progress on efficiency. Given the spend on security, the CIO needs to pay attention to and increase efficiency. How often should you be communicating with the CIO? Hopefully monthly, if not more often. We know it’s hard to book time around golf outings with the big systems, storage, and networking vendors. But you still need access and face time to make sure there is a clear understanding of where the security program is and what needs to be addressed. Benchmark data helps substantiate the need for specific projects/investments, driven either by peer group adoption or efficiency/effectiveness gaps. Again, your opinion about what’s important and needed is interesting, but not necessarily relevant. Having data to substantiate your arguments makes the discussion much easier. IT Ops teams Brown stuff tends to flow downhill, so your pals in IT ops tend to focus on looking good to the CIO. You need their support to execute on any kind of security program, because ops can make it protection difficult, and that would be a problem for you and the CIO. But ops isn’t interested in the same things as senior managers. You need to focus those discussions on areas where changes or activities depend on operational resources. As with all things operational, it’s about increasing efficiency and reducing error, so we want data which highlighting issues, gaps, and/or areas to improve. Ops folks may not appreciate being told they may need to do things differently. This is another place where benchmark data can be your ace in the hole. By showing relative performance and ability to execute on operational processes, the data substantiates your arguments and helps avoid you having to go back to the CIO to complain “Ops sucks and makes our life hard!” and hoping the CIO will make them play nice. Security team As valuable as benchmark data is for telling a better story to stakeholders and key influencers of the security program, the benchmark data is also a key management tool for your own security team. We all want our groups to work better and improve continuously – as we

“Hi. I’m Mike. And I’m an addict.” I start every chapter of the Pragmatic CSO with those very words. There there are many things you can be addicted to. Thrills. Sex. Sugar. Booze. Drugs. Twitter. Pr0n. Caffeine. Food. Some are worse than others, though none of them really good for you. But now I have to face up to another addiction. The need for gadgets. I’m jonesing for a new MacBook Air. Big time. Like waking up in the middle of the night wanting some SSD goodness in a petite 2lb package. Jonesing, I say, and it’s not pretty. Now there are folks with much worse gadget addiction than me. They are the ones standing in line at Best Buy for the latest Zune. Those folks have a problem. To be clear, so do I. I have a perfectly workable 15” MacBook Pro. It’s been a workhorse for two and a half years. For what I need, it’s fine. Why can’t I be happy with it? Why do I long for something new? The problem is my gear isn’t shiny anymore. I need a new trophy. Need. It. Now. I feel inadequate with a late 2008 MBP. In the bagel shop where I was writing this morning, there was a guy with an MB Air. I felt envy. Not enough to poach his machine when he tok a leak (by the way, it’s two frickin’ pounds – you can take it to the loo), but definitely envy. But then I looked over my other shoulder and saw a guy with an old school Apple laptop. And I mean old school. Like before they had a MagSafe connector, meaning a PowerBook G4. Oh, the horrors. I don’t know how that guy gets out of bed in the morning. And it’s worse when we have a Securosis meeting. Rich gets all the new toys. He’s got an MB Air 11”. I know he scoffs at my MBP. My laptop is older than his kids. Really. But Adrian is a different animal. He’s into high end audio equipment and dogs. My addiction is cheaper. At least I have that going for me. Over two years with the same laptop is a lifetime for me. Some guys trade in their wives every couple years. I trade in my laptop. The Boss likes that approach much better. Normally it’s not an issue, since I tend to hold down a job for 15 months, so I get a new toy every time I get a new job. I get my fix and have no issue, right? Not so much anymore – I’m not changing jobs any time soon. At least that’s what Rich and Adrian keep telling me. But I am getting smarter. Knowing this little issue I have, I made proper provisions this year by doing a side project over the winter and expressly earmarking those fees to breathe the (MB) Air. I’ve got motive. I’ve got opportunity. I’ve even got the funds. I know, you are wondering why I don’t just hop on the Apple web site and order it? This is why. They expect a new Air in the summer. That’s only what, 2 months away? It’ll be worth the wait. That’s what I keep telling myself. It will be smokin’ fast. And shinier. The next 2 months will be a struggle. I want it now. But I’m repressing my urges because I know how bad I’d feel when someone else got the shiny fast one, 4 days after I took delivery of my slow, dull one. I need to do some NLP to associate those bad feelings with the late 2010 MB Air. I will awaken the giant within, just you watch. That will keep me off the gadget juice. I’ll hold out because I have a plan. Every day, I’ll do my affirmations to convince myself that I’m still a good person, even though I use a late 2008 MBP. It will work. I know it will. The power of positive thinking in action. I’ll send a DM to my sponsor every day because I’m not addicted to Twitter. Not yet anyway. That will keep me on the straight and narrow. And doggone it, people like me, right? But we all know what happens when you repress an urge for too long. Gosh, that iPad 2 looks awfully shiny… -Mike Photo credits: “Apple addiction” originally uploaded by new-york-city Since I don’t do enough writing here on the Securosis blog, I figured I’d inflict some pithy verbiage on the victims, I mean readers, of Dark Reading. I’ll be posting on their Hacked Off blog monthly, and started with a doozy on why the RSA breach disclosure was pretty good. Surprisingly enough, I took a contrarian view to all those folks who think they should know everything, even if they aren’t RSA customers. It’s not about you, folks – sorry to bruise your egos. Incite 4 U Mea culpa roll with a side of SQLi: Do you ever wonder what a Barracuda roll tastes like? You can ask the folks in Hong Kong who used an automated SQLi attack to feast on Barracuda’s customer list over the weekend. The good news is that not much data was lost. Some customer and partner names and emails. The bad news is the breach happened because of an operational FAIL to put WAF back into blocking mode. As usual, people are the weakest link. But this disclosure is a great example of how to own it, explain it, and help everyone learn from it. A side of SQLi is not quite as tasty as miso soup, but news of the attack goes down a lot easier with a large serving of mea culpa. – MR Trust No One: I keep stealing a slide Gunnar did a while back (from Chris Hoff, who showed it to me first). It’s a table showing all the big advances in the web and web applications, and then the security tools we use to secure them. In every case, it’s firewalls and SSL. But between the Comodo breach and the

So your key security metrics are collected and shared safely. What comes next? Now we need to start deriving value from the data. Remember, metrics and numbers aren’t worth the storage to keep them, unless you use them as management tools. You need to start comparing the data, drawing conclusions, and adjusting your security program based on the data. OMG, actually making changes based on data rather than shiny objects, breaches, airline magazine articles, and compliance mandate changes. How novel. Remember the goal of this entire endeavor: to show relative progress. Now we get to figure out relative means, which involves defining peer group(s) for comparison. The first group you’ll compare your data to is actually yourself. Yes, this is trend analysis on your own metrics. It will provide some perspectives on whether you are improving – but improving against yourself does not provide perspective on whether you are ‘good’, spending too much money, or focusing on the right stuff. This is where you need to think about benchmarking, or going beyond security metrics. Peer Groups There are ways to define your peer group: Industry: This is your vertical market. Initially (until you have access to loads of data), you will focus on big industry buckets – like defense, healthcare, financial, hospitality, etc. Obviously there are differences between investment banks and insurance companies within the financial vertical, but businesses in the same category will have many consistent business processes which involve collecting very similar types of data. These organizations also tend to have similar geographic profiles – as for example a typical retailer will have a headquarters, regional distribution centers, and tons of stores. Additionally these companies exist under similar compliance/regulatory regimes. They also tend to be relatively consistent in terms of to technology adoption/maturity, which is critical for making relevant comparisons. Company size: Similar to the consistencies we find among companies in the same vertical/industry, we also find many similarities between companies of roughly the same size. For instance large enterprises (10,000+ employees) are generally global by definition – it is very difficult to get that big while focusing on a single geographic region. So organizational models and scale tend to be fairly consistent within a company-size segment. These companies also tend to spend similarly on security. Of course there are always outliers and some industries show less consistency, but we aren’t looking for perfection here. Region: Regional comparisons support many interesting comparisons. Culture and attitudes toward security can be enhanced or hindered by government funding and compliance regimes. We also see relatively consistent technology maturity/adoption within regions – largely based on local drivers such as compliance with laws and other rules, infrastructure, and available talent. Of course, not all metrics apply to any peer group. So when you define your benchmark peer groups, factor this in. The best way is to figure out how the specific metrics correlate for each peer group. We know, it’s math, but you’ll figure out pretty quickly whether there are any useful patterns or consistency within any particular metric. Focus on the metrics with the best correlation across a peer group. Sample Size Now that we’re talking about math, we have to address sample size. That’s basically how much data you need before the benchmark is useful. And as usual it depends, but push for statistical significance over the long term. Why? Because by definition statistical significance means a result is unlikely to occur by chance. You don’t want to be making decisions based on chance and randomness, so that’s our benchmark. More to the point, you want to stop making decisions based on chance. But it’s likely to take some time to get to a statistically significant dataset, so what can you do in the meantime? Look at the distribution, remove the outliers (which screw up your trend lines), and start comparing yourself against the trends you can spot. You can get a decent trend with only a handful of data points for metrics that correlate strongly. Always remember to keep the goal clearly in focus, and that is to identify gaps and highlight success, neither of which requires a huge amount of data. But to be clear, you are looking over time for statistical significance. Reverting to the Mean Another issue is whether you want to “revert to the mean,” meaning you look like everyone else in the peer group. Once again, it depends. Let’s take a look at a couple of likely metrics categories: For spending, it’s unlikely that you are getting a reasonable return from security spending 3 standard deviations above the mean. Not unless you can differentiate your product/offering on security, which is rare. For incidents, you want to be better than the mean. Most likely significantly so. Why? Because all your years of hard work can be unwound with one high profile breach. So the more effectively and quickly you respond and contain the damage, the better. Here you definitely don’t want to be in the bottom quartile, which indicates a failure of incident response and should be unacceptable to senior management. For efficiency, effectiveness, and coverage metrics (most of the easily quantifiable and operational metrics), you want to be better than the mean. That shows operational competence. In terms of importance, your spending is usually the most visible (to the folks who pay the bills, at least), so be in the ballpark there. Incidents come next, as they have a direct impact on issues like availability and brand damage. Then comes the operational stuff – it’s certainly important to how you run the security program, but rarely interesting to the muckety-mucks. Now it’s time to tell those muckety-mucks what you found, which means focusing on the commmunication strategy underlying your benchmarking program, so that’s where we’ll focus in the next post. Share: