Friday Summary: Biased Analysis Edition
Glenn Fleishman (@GlennF) tweeted “Next month’s Wired: ‘We painstakingly reconstructed Steve Jobs’ wardrobe so you can wear it, too.’” A catty response to Wired Magazine’s recent reconstruction of Steve Jobs’ stereo system. Unlike Mr. Fleishman I was highly interested in this article, and found it relevant to current events. For people who love music and quality home music reproduction, iTunes’ disgustingly low-resolution MP3 files seem at odds with Jobs’ personal interest in HiFi. The equipment surrounding Jobs in the article’s lead picture was not just good stereo equipment, and not ‘name brand’ equipment either – but instead esoteric brands aimed at aficionados (indicating Jobs was very serious about music reproduction and listening). The irony is that someone who was heavily invested in HiFi would become the principal purveyor of what audiophiles deem unholy evil. Sure, MP3s are a great convenience – just not so great for music quality. This picture has made HiFi trade magazines over the years, and while Jobs was alive the vanishingly small population of audiophiles held out hope that we would someday get high-resolution music from iTunes. The rumor – of which confirmation would be a great surprise – is that we may finally get HiRes files from iTunes, which I suspect is why this picture was the subject of such scrutiny. The market for high-quality headphones has jumped 10-fold in the last 7 years, and vinyl record sales have gone up 6-fold in the same period, showing public interest in higher quality audio while CD sales plummet. Even piracy-paranoid anti-consumer vendors like Sony have begun to sell HiRes DSD files, so Apple has likely noticed these trends and we can hope they will follow suit. Garbage in, garbage out is a basic axiom I learned when I first started programming database applications, and it remains true for any database, including NoSQL variants. Write any query you want – if the data is bad, the results are meaningless. But even if the data is completely accurate, depending on how you write your queries, you may produce results that don’t mean what you think they do. The learning curve with NoSQL is even weirder – many data scientists are still learning how to use these platforms. Consider that for many NoSQL users, the starting point is often just looking for stuff – we don’t necessarily know what we are looking for, but we often discover interesting patterns in the data. And when we do, we try to make sense of them. This itself is a form of bias. In this process we may write and rewrite data queries many times over, trying to refine a hypothesis. But the quality and completeness of the data, as well as your ability to mine it effectively with queries, can lead to profound revelations – or perhaps to poop. More likely it’s somewhere in-between, but both extremes are a possibility. One of Gunnar’s key themes from a post earlier this year is to understand the balance between objective and subjective aspects of metrics. He said, “I am very tired of quant debates where … the supposed quant approach beats the subjective approach.” It is not a question of whether you are subjective or not – it is there in your biases when you make the model… “To me the formula for infosec is objective measures through logging and monitoring, subjective decisions on where to place them, and what depth, a mix of subjective and objective review of the logs and data feedback from the system’s performance over time.” I raise these points because while we examine our navels for effective uses of analytics for business, operations, and security metrics, practiced FUD-ites work their magic to make analysis irrelevant. An exaggerated example to make a point is this post on discrimination potential in big data use, where we see political opponents claiming big data is biased before it has been put to use. A transparent attempt to kill funding based on data analysis, without analysis to back it up! It is easier for a politician to generate fear by labeling this mysterious thing called “big data” as discriminatory in order to get their way than to discredit an actual analysis. They are feeding off audience bias (popular opinion). Many people naively believe “It’s big data so it’s evil” in response to NSA spying and corporations performing what feels like consumer espionage. It does not even matter if the data or tools will be used used effectively – bias and fear are used to kill metrics-based decisions. Ironic, right? As a security example: in each of the last three years – always a few months after the release of the Verizon DBIR – a handful of vendors has told me how the DBIR says the number one threat is from insiders! When I point out that the report says the exact opposite, they always argue that an outsider becomes an insider once they have breached your systems. And post-Snowden many enterprises are mostly worried about being Snowdened – regardless of any breach statistics. I don’t have any lesson here, or a specific safety tip to offer, but if you have metrics and data for decision support perform your own review. It will help remove some bias from the analysis. People who are financially invested in a specific worldview deliberately misinterpret, discredit, and fund biased studies, to support their position – their biased arguments drive you to conclusions that benefit them. On to the Summary: Webcasts, Podcasts, Outside Writing, and Conferences Rich quoted on SecDevOps. Favorite Securosis Posts David Mortman: NoSQL Security: Understanding NoSQL Platforms. Adrian Lane: XP Users Twisting in the Wind. For the picture, if nothing else. Mike Rothman: NoSQL Security: Understanding NoSQL Platforms. I have long said Adrian has forgotten more about databases than most of us know. He has proven it once again with this primer on NoSQL databases… Other Securosis Posts Incite 4/30/2014: Sunscreen. Firestarter: The Verizon DBIR. Defending Against Network-based Distributed Denial of Service Attacks [New Paper]. Summary: Time and Tourists. Pass the Hemlock. Favorite Outside Posts Mike Rothman: UltraDNS Dealing with DDoS Attack. The cyber equivalent of going up to someone and hitting them with
