Incite 11/7/2012: And the winner is… Math
Yesterday was Election Day in the US. That means hundreds of millions of citizens braved the elements, long lines, voter suppression attempts, flaky voting machines, and other challenges to exercise our Constitutional right to choose our leaders. After waiting for about 3 hours in 2008, I got smart and voted early this year. It took me about 45 minutes and it was done. Luckily I don’t live in a swing state, so I think I saw maybe 1 or 2 political ads throughout the cycle when I was traveling. I know folks that have been pummeled by non-stop robocalls, TV ads, radio blitzes, and annoying canvassers knocking on their doors will appreciate the relative silence they’ll hear tomorrow. But that’s all part of the process. US presidential candidates have the most sophisticated targeting and marketing machines in existence. Think about it. Each candidate probably spent $1B on the campaign, funded largely by big donors, and spent largely over the past 3-4 months. That’s a similar spend to what a Fortune 500 consumer products company spends on marketing, if not more. And all that marketing is to influence the “story” told by the mass media. Trying to manipulate press coverage to portray momentum, define story lines about candidates, and ultimately rile up the base and depress the competition. Amazingly enough, it’s very effective. Talking heads (many on the payrolls of political parties or specific candidates) appear daily to talk about how everything is rosy in their world, how their candidate has the momentum and will win in a landslide. There really is no unbiased view of a campaign. Then there are the polls. Hundreds of polls. Every day. With different results, all seemingly within the margin of error. And the polling numbers spun however they want. Let’s be clear about polls. They are biased because they take a statistical sample and apply certain voter turnout estimates to derive their numbers. That’s why some polls are consistently skewed towards one party or the other. But what happens if you average all the polls, build a big-ass model, and apply defensible algorithms to eliminate perceived poll bias for a decent estimate of the current state of the race? You get a predictive model of a likely outcome of the election. Which is exactly what Nate Silver has built. He was a former baseball analyst who built sophisticated models to estimate baseball player performance, and then applied his sabermetric kung fu to politics. His website was acquired by the NY Times a few years ago, and his accuracy has been uncanny. He called 49 out of 50 states in the 2008 presidential election and did well in 2010 as well. Could it be luck? Maybe, but probably not. Not if you believe in math, as opposed to punditry and hope. Since early in the Spring he’s shown the incumbent President as a solid favorite to be re-elected. Turns out he was right. Absolutely, totally right. Of course, throughout the campaign he became a target of folks on the other side of the aisle. Similar to the Salem witch hunts, folks who understand math have had to convince luddites that he isn’t a witch. What these folks don’t understand is that Nate Silver may have a specific ideological bent, but that’s not what his model is about. The data says what it says, and he reports a likelihood of victory. Not a projection. Not a guarantee. A likelihood. Models don’t lend themselves to exact precision. Nate would be the first to say there is a likelihood that his model was wrong and the election could have gone to the other candidate. That would have given his detractors the ability to put him and his models in a box. But it didn’t happen. Math won because math works. Models get better over time. They are never exact – not on complex systems anyway. Silver’s a numbers guy, which means he will continue to refine the model in every subsequent election. But it’s pretty close now, and that’s very impressive. The baseball pundits hated it when the math guys showed up and proved there is something to quantitative analysis. Now all the other sports are embracing the concepts. And yes, the politicians will pay more attention to quantitative methods over time as well. Anecdote is fine. Qualitative research has a place. But over time math wins. Which scares a lot of people because then pundits and other qualitative windbags have a lot less to talk about. When math wins, we all are winners… Especially guys like Rob Graham, who understand the models and how to game them for fun and profit. –Mike Photo credits: Math Doesn’t Suck originally uploaded by John Baichtal Heavy Research We’re back at work on a variety of blog series, so here is a list of the research currently underway. Remember you can get our Heavy Feed via RSS, where you can get all our content in its unabridged glory. And you can get all our research papers too. Building an Early Warning System The Early Warning Process Introduction Implementing and Managing Patch and Configuration Management Defining Policies Integrate and Deploy Technologies Understanding and Selecting a Key Manager Introduction Understanding and Selecting Identity Management for Cloud Services Introduction Newly Published Papers Defending Against Denial of Service Attacks Securing Big Data: Security Recommendations for Hadoop and NoSQL Environments Pragmatic WAF Management: Giving Web Apps a Fighting Chance Incite 4 U Taking the path of least resistance: If I was a bad guy (and yes, I’m a bad guy, but I’m not a bad guy), I’d go after small business. Maybe that’s because I know too much. I know how much effort and money is spent by enterprises to protect themselves. They still stink, but they try. PCI guarantees that. But small business tends to spend far less and take security far less seriously. That means they are sitting ducks. And as Krebs shows time and time again, those ducks get slaughtered. This latest story