Question everything, including the dataBy Mike Rothman
The good news about being in security is that you don’t have to look too far for criticism of your work. Most of the time it’s constructive criticism, so overall interaction with the security community makes your work markedly better. Which is why we live by the Totally Transparent Research process. It makes our work better.
But when our pals at Verizon clogged up my Twitter timeline this morning with their annual DBIR masterpiece (you can also check out our guidance on the DBIR), they dragged my attention back to a post by Jericho from Attrition: “Threat Intelligence”, not always that intelligent, prompted by Symantec’s most recent security trends report.
Jericho summed up the value of security trend reports as only he can, and explained why folks tend not to challenge them often.
The reason? Security companies, professionals, and journalists are complacent. They are happy to get the numbers that help them. For some, it sells copy. For others, it gets security budget. Since it helps them, their motivation to question or challenge the data goes away. They never realize that their “threat intelligence” source is stale and serving up bad data.
It’s not in the machine’s best interest to question the data. That’s why most folks (besides, me I guess) don’t poke at the vendor-sponsored survey data or other similar nonsense put forth as gospel in the security business. Anything that helps sell security is good, right?
Well, no. Decisions based on faulty data tend to be faulty decisions. So Jericho presents a number of inconsistencies between Symantec’s vulnerability data and the OSVDB dataset he contributes to. It’s pretty compelling stuff.
But we shouldn’t minimize either the effort involved in building these reports or the value they do provide. There is a lot of value in these threat and data breach reports, if the data is reasonably accurate. We’re security people. We question everything, so it’s reasonable to question the data you use to make the case for your existence.
Photo credit: “Question” originally uploaded by ACU Library