In the last post, we talked about outcomes important to the business, and what types of security metrics can help make decisions to achieve those outcomes. Most organizations do pretty well with the initial gathering of this data. You know, when the reports are new and the pie charts are shiny. Then the reality – of the amount of work and commitment required to implement a consistent measurement and metrics process – sets in. Which is when most organizations lose interest and the metrics program falls by the wayside. Of course, if the there is a clear and tangible connection between gathering data and doing your job better, you make the commitment and stick with it. So it’s critical, especially within the early phases of a fact-based network security process, to get a quick win and capitalize on that momentum to cement the organization’s commitment to this model. We’ll discuss that aspect later in the series. But consistency is only one part of implementing this fact-based network security process. In order to get a quick win and warrant ongoing commitment, you need to make sense of the data. This issue has plagued technologies such as SIEM and Log Management for years – having data does not mean you have useful and valuable information. We want to base decisions on facts, not faith. In order to do that, you need to make gathering security metrics an ongoing and repeatable process, and ensure you can interpret the data efficiently. The keys to these are automation and visualization. Automating Data Collection Now that you know what kind of data you are looking for, can you collect it? In most cases the answer is yes. From that epiphany, the focus turns to systematically collecting the types of data we discussed in the last post. Data sources like device configuration, vulnerability, change information, and network traffic can be collected systematically in a leveraged fashion. There is usually a question of how deeply to collect data, whether you need to climb the proverbial stack in order to gather application and database events/logs/transactions, etc. In general, we Securosis folk advocate collecting more rather than less data. Not all of it may be useful now (or ever). But once you miss the opportunity to capture data you don’t get it back. It’s gone. And of course which data sources to leverage depends on the problems you are trying to solve. Remember, data does not equal information, and as much as we’d like to push you to capture everything, we know it’s not feasible. So balance data breadth and fidelity against cost and storage realities. Only you can decide how much data is enough to answer the questions of prioritizing activities. We tend to see most organizations focus on network, security, and server logs/events – at least initially. Mostly because that information is plentiful and largely useful in pinpointing attacks and substantiating controls. It’s beyond the scope of this paper to discuss the specifics of different platforms for collecting and analyzing this data, but you should already know the answer is not Excel. There is just too much data to collect and parse. So at minimum you need to look for some kind of platform to automate this process. Visualizaton Next we come come up against that seemingly intractable issue of making sense of the data you’ve collected. In this case, we see (almost every day) that a picture really is worth thousands of words (or a stream of thousands of log events). In practice, pinpointing anomalies and other suspicious areas which demand attention, is much easier visually – so focusing on dashboards, charts, and reports become a key part of operationalizing metrics. Right, those cool graphics available in most security management tools are more than eye candy. Who knew? So which dashboards do you need? How many? What should they look like? Of course it depends on which questions you are trying to answer. At the end of this series we will walk through a scenario to describe (at a high level, of course) the types of visualizations that become critical to detecting an issue, isolating its root cause, and figuring out how to remediate it. But regardless of how you choose to visualize the data you collect, you need a process of constant iteration and improvement. It’s that commitment thing again. In a dynamic world, things constantly change. That means your alerting thresholds, dashboards, and other decision-making tools must evolve accordingly. Don’t say we didn’t warn you. Making Decisions As we continue through our fact-based network security process, you now have a visual mechanism for pinpointing potential issues. But if your environment is like others we have seen, you’ll have all sorts of options for what you can do. We come full circle, back to defining what is important to your organization. Some tools have the ability to track asset value, and show visuals based on the values. Understand that value in this context is basically a totally subjective guess as to what something is worth. Someone could arbitrarily decide that a print server is as important as your general ledger system. Maybe it is, but this gets back to the concept of “relative value” earlier in the series. This relative understanding of an asset/business system’s value yields a key answer for how you should prioritize your activities. If the visualization shows something of significant value at risk, then fix it. Really. We know that sounds just too simple, and may even be so obvious it’s insulting. We mean no offense, but most organizations have no idea what is important to them. They collect very little data and thus have little understanding of what is really exposed or potentially under attack. So they have no choice but to fly blind and address whatever issue is next on the list, over and over again. As we have discussed, that doesn’t work out very well, so we need a commitment to collecting and then visualizing data, in order to
