Research

Continuing our discussion of core SIEM and Log Management technology, we now move into event correlation. This capability was the holy grail that drove most investment in early SIEM products, and probably the security technology creating the most consistent disappointment amongst its users. But ultimately the ability to make sense of the wide variety of data streams, and use them to figure out what is under attack or compromised, is essential to any security practice. This means that despite the disappointments, there will continue to be plenty of interest in correlation moving forward. Correlation Defining correlation is akin to kicking a hornet’s nest. It immediately triggers agitated debates because there are several published definitions and every expert has their favorite. As usual, we need to revisit the definitions and level-set, not to create controversy (though that tends to happen), but to make things easy to understand. As we search for a pragmatic definition, we need to simplify concepts to make subjects understandable to a wider audience at the expense of precision. We understand our community is not a bunch of shrinking violets, so we welcome your comments and suggestions to make our research more relevant. Let’s get back to the end-user problem driving SIEM and log management. Ultimately the goal of this technology is to interpret security-related data to improve security, increase efficiency, and/or document security controls. If a single file contained all the information required for security analysis, we would not bother with the collection and association of events from multiple sources. The truth is that each log or event contains a piece of information, which forms part of the puzzle, but lacks context necessary to analyze the big picture. In order to make meaningful decisions about what is going on with our applications and within our network, we need to combine events from different sources. Which events we want, and what pieces of data from those events we need, vary based on the problem we are trying to solve. So what is correlation? Correlation is the act of linking multiple events together to detect strange behavior. It is the association of different but related events to provide broader context than a single event can provide. Keep in mind that we are using a broad definition of ‘event’ because as the breadth of analysis increases, data may expand beyond traditional events. Seems pretty simple, eh? Let’s look at an example of how correlation can help achieve one of our key use cases: increasing the efficiency of the security team. In this case an analyst gets events from multiple locations and device types (and/or applications), and is expected to figure out whether there is an issue. The attacker might first scan the perimeter and then target an externally facing web server with a series of known exploits. Upon successfully compromising the web server, the attacker sets up a new user account and start scanning internally to find more exploitable hosts. The data is available to catch this attack, but not in a single place. The firewalls see the initial scans. The IDS/IPS sees the series of exploits. And the user directory sees the new account on the compromised server. The objective of correlation is to see all these events come through and recognize that the server has been compromised and needs immediate attention. Easy in concept, very hard in practice. Historically, the ability to do near real time analysis and event correlation was one of the ways SIEM differed from log management, although the lines continue to blur. Most of the steps we have discussed so far (collecting data, then aggregating and normalizing it) help isolate the attributes that link events together to make correlation possible. Once data is in manageable form we apply rules to detect attacks and misuse. These rules are comprised of the granular criteria (e.g., specific router, user account, time, etc.), and determine if a series of events reaches a threshold requiring corrective action. But the devil is in the details. The technology implements correlation as a linear series of events. Each comparison may be a simple case of “if X=Y, then” do something else, but we may need to string several of these comparisons together. Second, note that correlation is built on rules for known attack patterns. This means we need some idea of what we are looking for to create the correlation rules. We have to understand attack patterns or elements of a compliance requirement in order to determine which device and event types should be linked. Third, we have to factor in the time factor, because events do not happen simultaneously, so there is a window of time within which events are likely to be related. Finally the effectiveness of correlation also depends on the quality of data collection, normalization, and tagging or indexing of information to feed the correlation rules. Development of rules takes time and understanding, as well as ongoing maintenance and tuning. Sure, your vendor will provide out-of-the-box policies to help get you started, but expect to invest significant time into tweaking existing rules for your environment, and writing new policies for security and compliance to keep pace with the very dynamic security environment. Further complicating matters: more rules and more potentially-linked events to consider increase computational load exponentially. There is a careful balancing act to be performed between the number of policies to implement, the accuracy of the results, and the throughput of the system. These topics may not immediately seem orthogonal, but generic rules detect more threats at a cost of more false positives. The more specific the rule, and the more precisely tailored to find specific threats, the less it will find new problems. This is the difficulty in getting correlation working effectively in most environments. As described in the Network Security Fundamentals series, it’s important to define clear goals for any correlation effort and stay focused on them. Trying to boil the ocean always yields disappointing results. Alerting Once events are correlated, analysis performed, and weirdness

A holy grail of technology marketing is to define a product category. Back in the olden days of 1998, it was all about establishing a new category with interesting technology and going public, usually on nothing more than a crapload of VC money and a few million eyeballs. Then everything changed. The bubble popped, money dried up, and all those companies selling new products in new categories went bust. IT shops became very risk averse – only spending money on established technologies. But that created a problem, in that analysts had to sell more tetragon reports, which requires new product categories. My annoyance with these product categories hit a fever pitch last week when LogLogic announced a price decrease on their SEM (security event management) technology. Huh? Seems they dusted off the SEM acronym after years on the shelf. I thought Gartner had decreed that it was SIEM (security information and event management) when it got too confusing between the folks who did SEM and SIM (security information management) – all really selling the same stuff. Furthermore, log management is now part of that deal. Do they dare argue with the great all-knowing oracles in Stamford? Not that this expanded category definition is controversial. We’ve even posted that log management or SIEM isn’t a stand-alone market – rather it’s the underlying storage platform for a number of applications for security and ops professionals. The lesson most of us forget is that end users don’t care what you call the technology, as long as you solve their problems. Maybe the project is compliance automation or incident investigation. SIEM/Log Management can be used for both. IT-GRC solutions can fit into the first bucket, while forensic toolkits fit into the latter. Which of course confuses the hell out of most end users. What do they buy? And don’t all the vendors say they do everything anyway? The security industry – along with the rest of technology – focuses on products, not solutions. It’s about the latest flashing light in the new version of the magic box. Sure, most of the enterprise companies send their folks to solution selling school. Most tech company websites have a “solution” area, but in reality it’s always an afterthought. Let’s consider the NAC (network access control) market as another example. Lots of folks think Cisco killed the NAC market by making big promises and not delivering. But ultimately, end users didn’t care about NAC – they cared about endpoint assessment and controlling guest access, and they solved those problems through other means. Again, end users need to solve problems. They want answers and solutions, but they get a steady diet of features and spiels on why one box is better than the competitors. They get answers to questions they don’t ask. No wonder most end users turn off their phones and don’t respond to email. Vendors spin their wheels talking about product category leadership. Who cares? Actually, Rich reminded me that the procurement people seem to care. We all know how hard it is to get a vendor in the wrong quadrant (or heaven forbid no quadrant at all) through the procurement gauntlet. Although the users are also to blame for accepting this behavior, and the dumb and lazy ones even like it. They wait for a vendor to come in and tell them what’s important, as opposed to figuring out what problem needs to be solved. From where I sit, the buying dynamic is borked, although it’s probably just as screwy in other sectors. So what to do? That’s a good question, and I’d love your opinion. Should vendors run the risk of not knowing where they fit by not identifying with a set of product categories – and instead focus on solutions and customer problems? Should users stop sending out RFPs for SIEM/Log Management, when what they are really buying is compliance automation? Can vendors stop reacting to competitive speeds and feeds? Can users actually think more strategically, rather than whether to embrace the latest shiny upgrade from the default vendor? I guess what I’m asking is whether it’s possible to change the buying dynamic. Or should I just quiet down, accept the way the game is played, and try to like it? Share:

Recently Michael Zalewski posted a rant about the state of security engineering in Security engineering: broken promises. I posted my initial response to this on Twitter: “Great explanation of the issue, zero thoughts on solutions. Bored now.” I still stand behind that response. As a manager, problems without potential solutions are useless to me. The solutions don’t need to be deep technical solutions – sometimes the solution is to monitor or audit. Sometimes the solution is to do nothing, accept the risk, and make a note of it in case it comes up in conversation or an audit. But as I’ve mulled over this post over the last two weeks, there is more going on here. There seems to be a prevalent attitude among security practitioners in general, and researchers in particular, that if they can break something it’s completely useless. There’s an old Yiddish saying that loosely translates to: “To a thief there is no lock.” We’re never going to have perfect security, so picking on something for being imperfect is just disingenuous and grandstanding. We need to be asking ourselves a pragmatic question: Does this technology or process make things better? Just about any researcher will tell you that Microsoft’s SDL has made their lives much harder, and they have to work a lot more to break stuff. Is it perfect? No, of course not! But is it a lot better then it used to be for all involved (except the researchers Microsoft created the SDL to impede)? You betcha. Are CWE and CVSS perfect? No! Were they intended to be? No! But again, they’re a lot better than what we had before. Can we improve them? Yes, CVSS continues to go through revisions and will get better. As will the Risk Management frameworks. So really, while bitching is fun and all, if you’re not offering improvements, you’re just making things worse. Share:

We get a lot of requests to sponsor this blog. We got several this week. Not just the spammy “Please link with us,” or “Host our content and make BIG $$$” stuff. And not the PR junk that says “We are absolutely positive your readers would just love to hear what XYZ product manager thinks about data breaches,” or “We just released 7.2.2.4 version of our product, where we changed the order of the tabs in our web interface!” Yeah, we get fascinating stuff like that too. Daily. But that’s not what I am talking about. I am talking about really nice, personalized notes from vendors and others interested in supporting the Securosis site. They like what we do, they like that we are trying to shake things up a bit, and they like the fact that we are honest in our opinions. So they write really nice notes, and they ask if they can give us money to support what we do. To which we rather brusquely say, “No”. We don’t actually enjoy doing that. In fact, that would be easy money, and we like as much easy money as we can get. More easy money is always better than less. But we do not accept either advertising on the site or sponsorship because, frankly, we can’t. We just cannot have the freedom to do what we do, or promote security in the way we think best, if we accept payments from vendors for the blog. It’s like the classic trade-off in running your own business: sacrifice of security for the freedom to do things your own way. We don’t say “No,” to satisfy some sadistic desire on our part to be harsh. We do it because we want the independence to write what we want, the way we want. Security is such a freakin’ red-headed stepchild that we have to push pretty hard to get companies, vendors, and end users to do the right thing. We are sometimes quite emphatic to knock someone off the rhythm of that PowerPoint presentation they have delivered a hundred times, somehow without ever critically examining its content or message. If we don’t they will keep yakking on and on about how they address “Advanced Persistant Threats.” Sometimes we spotlight the lack of critical reasoning on a customer’s part to expose the fact that they are driven by politics without a real plan for securing their environment. We do accept sponsorship of events and white papers, but only after the content has gone through community review and everyone has had a chance to contribute. Many vendors and a handful of end-users who talk with us on the phone know we can be pretty harsh at times, and they still ask if they economically support our research. And we still say, “No”. But we appreciate the interest, and we thank you all for for participating in our work. On to the Summary: Webcasts, Podcasts, Outside Writing, and Conferences Adrian’s Dark Reading article on What Oracle Gets In The Secerno Buy. Rich quoted in a Dark Reading article on database passwords. Did we mention Rich was on NPR Science Friday? The full transcript is up. Unfortunately – since it has all the “you knows” and “ums” in it. Adrian’s DAM Deployment Issues to Avoid launched this week. Rich on the Network Security Podcast. Adrian quoted in CRN Tech on database security. Mike quoted in SC Magazine. Favorite Securosis Posts Rich: Code Re-engineering. This applies to so much more than code. I’ve been on everything from mountain rescues to woodworking projects where the hardest decision is to stop patching and nuke it from orbit. We are not mentally comfortable throwing away hours, days, or years of work; and the ability to step back, analyze, and start over is rare in any society. Mike Rothman: Code Re-engineering. Adrian shows his development kung fu. He should get pissed off more often. David Mortman: Gaming the Tetragon. Adrian Lane: The Secerno Technology. Just because you need to understand what this is now that Oracle has their hands on it. Other Securosis Posts Understanding and Selecting SIEM/LM: Aggregation, Normalization, and Enrichment. Quick Wins with DLP Presentation. Incite 5/26/2010: Funeral for a Friend. Understanding and Selecting SIEM/LM: Data Collection. A Phish Called Tabby. Thoughts on Diversity and False Diversity. FireStarter: The Only Value/Loss Metric That Matters. The Laziest Phisher in the World. Favorite Outside Posts Rich: Data Loss Prevention and Enterprise Rights Management; Complimentary or alternative? For 6 months or so I’ve been getting a lot of “which is better, DRM or DLP?” questions. The problem is that they are not alternative technologies, but complementary. The trick is to figure out which one might be more appropriate to implement first, not which can replace the other. Besides, I think they are on the path to complete convergence in the long term, and we already have early samples of combined solutions. Adrian: Bejtlich’s Forget Pre-Incident Cost, How Much Did Your Last Incident Cost? Almost picked Rich’s post The Only Value/Loss Metric That Matters for my internal fave of the week, but this is like a two-fer. Mike Rothman: Google Secure Search and Security Overkill. Boaz makes the point that not all security is worth it. Playing at a security theater near you…. David Mortman: Privacy Theater. Project Quant Posts DB Quant: Discovery And Assessment Metrics (Part 2) Identify Apps. DB Quant: Discovery And Assessment Metrics (Part 1) Enumerate Databases. DB Quant: Planning Metrics (Part 4). Research Reports and Presentations Understanding and Selecting a Database Encryption or Tokenization Solution. Low Hanging Fruit: Quick Wins with Data Loss Prevention. Report: Database Assessment. Top News and Posts TabNabbing was the big news this week. Three indicted on $100M Rogue Software Scam. Mozilla Plugin Check via Brian Krebs. Supposed Vuln in iPhone Encryption. Oopsie. Why does the IRS never have a problem like this? Your Privacy in Their Hands via LiquidMatrix. Can you have a PCI Compliant Virtual Site? Good question. New School blog announces The

When I was abroad on vacation recently, the conversation got to the relative cost of petrol (yes, gasoline) in the States versus pretty much everywhere else. For those of you who haven’t travelled much, fuel tends to be 70-80% more expensive elsewhere. Why is that? It comes down to the fact that the US Government bears many of real costs of providing a sufficient stream of petroleum. Those look like military, diplomatic, and other types of spending in the Middle East to keep the oil flowing. I’m not going to descend into either politics or energy dynamics here, but suffice it to say we’d be investing a crapload more money in alternative energy if US consumers had to directly bear the full brunt of what it costs to pull oil out of the Middle East. With that thought in the back of my mind, I checked out one of Bejtlich’s posts last weekend which talked about the R&D costs of the bad guys. Basically these folks run businesses like anyone else. They have to invest in their ‘product’, which is finding new vulnerabilities and exploiting them. They also have to invest in “customer service,” which is basically staying invisible once they are inside to avoid detection. And these costs are significant, but compared to the magnitude of the ‘revenue’ side of their equation, I’m sure they are happy to make the investment. Cyber-fraud is big business. But what about other hidden costs of providing security? We had a great discussion on Monday with the FireStarter talking about value/loss metrics, but do these risk models take into account some of the costs we don’t necessarily see as part of security? Like our network traffic. How much bandwidth is wasted on reconnaissance traffic looking for holes in our perimeters? What about the amount of your inbound pipe congested with spam, which you need to analyze and then drop. One of the key reasons anti-spam services took off is because the bandwidth demand of spam was transferred to the service provider. What would we do differently if we had to allocate those hidden costs to the security team? I know, at the end of the day it’s all just overhead, but what if? Would it change our behavior or our security architectures? I suspect we’d focus much more on providing clean pipes and having more of our security done in the cloud, removing some of these hidden costs from our IT stack. That makes economic sense, and we all know most of what we do ultimately is driven by economics. How about the costs of cleaning up an incident? Yes, there are some security costs in there from the standpoint of investigation and forensics, but depending on the nature of the attack there will be legal and HR resources required, which usually don’t make it into the incident post-mortem. Or what about the opportunity cost of 1,000 folks losing their authentication tokens and being locked out of the network? Or the time it takes a knowledge worker to jump through hoops to get around aggressive web filtering rules? Or the cost of false positives on the IPS that block legitimate business traffic and break critical applications? We know how big the security budget is, but we don’t have a firm grasp of what security really costs our businesses. If we did, what would we do differently? I don’t necessarily have an answer, but it’s an interesting question. As we head into Memorial Day weekend here in the US, we need to remember obviously, all the soldiers who give all. But we also need to remember the ripple effect of every action and reaction to the bad guys. Every time I go through a TSA checkpoint in an airport, I’m painfully aware of the billions spent each month around the world to protect air travel, regardless of whether terrorists will ever attack air travel again. I guess the same analogy can be used with security. Regardless of whether you’re actually being attacked, the costs of being secure add up. Score another one for the bad guys. Share:

In the last post on Data Collection we introduced the complicated process of gathering data. Now we need to understand how to put it into a manageable form for analysis, reporting, and long-term storage for forensics. Aggregation SIEM platforms collect data from thousands of different sources because these events provide the data we need to analyze the health and security of our environment. In order to get a broad end-to-end view, we need to consolidate what we collect onto a single platform. Aggregation is the process of moving data and log files from disparate sources into a common repository. Collected data is placed into a homogenous data store – typically purpose-built flat file repositories or relational databases – where analysis, reporting, and forensics occur; and archival policies are applied. The process of aggregation – compiling these dissimilar event feeds into a common repository – is fundamental to Log Management and most SIEM platforms. Data aggregation can be performed by sending data directly into the SIEM/LM platform (which may be deployed in multiple tiers), or an intermediary host can collect log data from the source and periodically move it into the SIEM system. Aggregation is critical because we need to manage data in a consistent fashion: security, retention, and archive policies must be systematically applied. Perhaps most importantly, having all the data on a common platform allows for event correlation and data analysis, which are key to addressing the use cases we have described. There are some downsides to aggregating data onto a common platform. The first is scale: analysis becomes exponentially harder as the data set grows. Centralized collection means huge data stores, greatly increasing the computational burden on the SIEM/LM platform. Technical architectures can help scale, but ultimately these systems require significant horsepower to handle an enterprise’s data. Systems that utilize central filtering and retention policies require all data to be moved and stored – typically multiple times – increasing the burden on the network. Some systems scale using distributed processing, where filtering and analysis occur outside the central repository, typically at the distributed data collection point. This reduces the compute burden on the central server and allows processing to occur on smaller, more manageable data sets. It does require that policies, along with the code to process them, be distributed and kept current throughout the network. Distributed agent processes are a handy way to “divide and conquer”, but increase IT administration requirements. This strategy also adds a computational burden o the data collection points, degrading their performance and potentially slowing enough to drop incoming data. Data Normalization If the process of aggregation is to merge dissimilar event feeds into one common platform, normalization takes it one step further by reducing the records to just common event attributes. As we mentioned in the data collection post, most data sources collect exactly the same base event attributes: time, user, operation, network address, and so on. Facilities like syslog not only group the common attributes, but provide means to collect supplementary information that does not fit the basic template. Normalization is where known data attributes are fed into a generic template, and anything that doesn’t fit is simply omitted from the normalized event log. After all, to analyze we want to compare apple to apples, so we throw away an oranges for the sake of simplicity. Depending upon the SIEM or Log Management vendor, the original non-normalized records may be kept in a separate repository for forensics purposes prior to later archival or deletion, or they may simply be discarded. In practice, discarding original data is a bad idea, since the full records are required for any kind of legal enforcement. Thus, most products keep the raw event logs for a user-specified period prior to archival. In some cases, the SIEM platform keeps a link to the original event in the normalized event log which provides ‘drill-down’ capability to easily reference extra information collected from the device. Normalization allows for predicable and consistent storage for all records, and indexes these records for fast searching and sorting, which is key when battling the clock in investigating an incident. Additionally, normalization allows for basic and consistent reporting and analysis to be performed on every event regardless of the data source. When the attributes are consistent, event correlation and analysis – which we will discuss in our next post – are far easier. Technically normalization is no longer a requirement on current platforms. Normalization was a necessity in the early days of SIEM, when storage and compute power were expensive commodities, and SIEM platforms used relational database management systems for back-end data management. Advances in indexing and searching unstructured data repositories now make it feasible to store full source data, retaining original data, and eliminating normalization overhead. Enriching the Future In reality, we are seeing a number of platforms doing data enrichment, adding supplemental information (like geo-location, transaction numbers, application data, etc.) to logs and events to enhance analysis and reporting. Enabled by cheap storage and Moore’s Law, and driven by ever-increasing demand to collect more information to support security and compliance efforts, we expect more platforms to increase enrichment. Data enrichment requires a highly scalable technical architecture, purpose-built for multi-factor analysis and scale, making tomorrow’s SIEM/LM platforms look very similar to current business intelligence platforms. But that just scratches the surface in terms of enrichment, because data from the analysis can also be added to the records. Examples include identity matching across multiple services or devices, behavioral detection, transaction IDs, and even rudimentary content analysis. It is somewhat like having the system take notes and extrapolate additional meaning from the raw data, making the original record more complete and useful. This is a new concept for SIEM, so the enrichment will ultimately encompass is anyone’s guess. But as the core functions of SIEM have standardized, we expect vendors to introduce new ways to derive additional value from the sea of data they collect. Other Posts in Understanding and Selecting SIEM/LM Introduction. Use Cases,

I don’t like to think of myself as a sentimental guy. I have very few possessions that I really care about, and I don’t really fall into the nostalgia trap. But I was shaken this week by the demise of a close friend. We were estranged for a while, but about a year ago we got back in touch and now that’s gone. I know it’s surprising, but I’m talking about my baseball glove, a Wilson A28XX, vintage mid-1980’s. You see, I got this glove from my Dad when I entered little league, some 30+ years ago. It was as big as most of my torso when I got it. The fat left-handed kid always played first base, so I had a kick-ass first baseman’s glove and it served me well. I stopped playing in middle school (something about being too slow as the bases extended to 90 feet), played a bit of intramural in college, and was on a few teams at work through the years. A few of my buddies here in ATL are pretty serious softball players. They play in a couple leagues and seem to like it. So last year I started playing for my temple’s team in the Sunday morning league with lots of other old Jews. I dug my glove out of the trunk, and amazingly enough it was still very workable. It was broken in perfectly and fit my hand like a glove (pun intended). It was like a magnet – if the ball was within reach, that glove swallowed it and didn’t give it up. But the glove was showing signs of age. I had replaced the laces in the webbing a few times over the years, and the edges of the leather were starting to fray. Over this weekend the glove had a “leather stroke”, when the webbing fell apart. I could have patched it up a bit and probably made it through the summer season, but I knew the glove was living on borrowed time. So I made the tough call to put it down. Well, not exactly down, since the leather is already dead, but I went out and got a new glove. Like with a trophy wife, my new glove is very pretty. A black leather Mizuno. No scratches. No imperfections. It even has a sort-of new-car smell. I’ll be breaking it in all week and hopefully it’ll be ready for practice this weekend. For an anti-nostalgia guy, this was actually hard, and it will be weird taking the field with a new rig. I’m sure I’ll adjust, but I won’t forget. – Mike Photo credits: “Leather and Lace” originally uploaded by gfpeck Incite 4 U I want to personally thank Rich and the rest of the security bloggers for really kicking it into gear over the past week. Where my feed reader had been barren of substantial conversations and debate for (what seemed like) months, this week I saw way too much to highlight in the Incite. Let’s keep the momentum going. – Mike. Focus on the problem, not the category – Stepping back from my marketing role has given me the ability to see how ridiculous most of security marketing is. And how we expect the vendors to lead us practitioners out of the woods, and blame then when they find another shiny object to chase. I’m referring to NAC (network access control), and was a bit chagrined by Joel Snyder’s and Shimmy’s attempts to point the finger at Cisco for single-handedly killing the NAC business. It’s a load of crap. To be clear, NAC struggled because it didn’t provide must-have capabilities for customers. Pure and simple. Now clearly Cisco did drive the hype curve for NAC, but amazingly enough end users don’t buy hype. They spend money to solve problems. It’s a cop-out to say that smaller vendors and VCs lost because Cisco didn’t deliver on the promise of NAC. If the technology solved a big enough problem, customers would have found these smaller vendors and Cisco would have had to respond with updated technology. – MR I can haz your ERP crypto – Christopher Kois noted on his blog that he had ‘broken’ the encryption on the Microsoft Dynamics GP, the accounting package in the Dynamics suite from the Great Plains acquisition. Encrypting data fields in the database, he noticed odd behavioral changes when altering encrypted data. What he witnessed was that if he changed a single character, only two bytes of encrypted data changed. With most block ciphers, if you change a single character in the plaintext, you get radically different output. Through trial and error he figured out the encryption used was a simple substitution cipher – and without too much trouble Kois was able to map the substitution keys. While Microsoft Dynamics does run on MS SQL Server, there are some components that still rely upon Pervasive SQL. Christopher’s discovery does not mean that MS SQL Server is secretly using the ancient Caesar Cipher, but rather that some remaining portion Great Plains does. It does raise some interesting questions: how do you verify sensitive data has been removed from Pervasive? If the data remains in Pervasive, even under a weak cipher, will your data discovery tools find it? Does your discovery tool even recognize Pervasive SQL? – AL Blame the addicts – When I was working at Gartner, nothing annoyed me more than those client calls where all they wanted me to do was read them the Magic Quadrant and confirm that yes, that vendor really is in the upper right corner. I could literally hear them checking their “talked to the analyst” box. An essential part of the due diligence process was making sure their vendor was a Leader, even if it was far from the best option for them. I guess no one gets fired for picking the upper right. Rocky DeStefano nails how people see the Magic Quadrant in his Tetragon of Prestidigitation post. Don’t blame the analyst

I just ran across a really interesting blog post by Joel Spolsky from last April: Things You Should Never Do, Part 1. Actually. the post pissed me off. This is one of those hot-button topics that I have had to deal with several times in my career, and have had to manage in the face of entrenched beliefs. His statement is t hat you should never rewrite a code base from scratch. The reasoning is “No major firm has ever successfully survived a product rewrite. Just look at Netscape … ” Whatever. I am a fixer. I was the guy who was able to make code reliable. I was the guy who found and fixed the obscure bugs. As I progressed in my career and started to manage teams of developers, more often than not I was handed the really crummy re-engineering projects because I could fix the problems and make customers happy. Sometimes success is its own penalty. I have inherited code so bad that bug fixes cost 4x in time and usually created new bugs in the process. I have inherited huge bodies of Java code written entirely as if Java were a 3G procedural language – ignoring the object-oriented paradigm completely. I have been tasked with fixing code that – for a simple true/false comparison – made 12 comparisons, 8 database, insertions and 7 deletions – causing an 180x performance penalty. I have inherited code so bad it broke the compiler. I have inherited code so bad that you could not change a back-end database query without breaking the GUI! It takes a real gift for bad programming to do these things. There are times when the existing code – all or part – simply needs to be thrown away. There are times that code is so tightly intertwined that you cannot simply fix one piece at a time. And in some cases there are really good business reasons, like your major customers say your code is crap and needs to be thrown away. Bad code can bleed a company to death with lost sales, brand impairment, demoralization, and employee turnover. That said, I agree with Joel’s basic premise that re-writing your product can kill your company. And I even agree about a lot of the social behaviors he describes that create failure. There is absolutely no reason to believe that the people who developed bad code the first time will not do the same thing the next time. But I don’t agree that you should never rewrite. I don’t agree that it has never been done successfully. I know because I have done it successfully. Twice. Out of three attempts, but hey, I got the important projects right. We tend not to hear about successful rewrites because the companies that carried it off really don’t want everyone knowing that previous versions were terrible. They would rather focus on happy customers and competitive products. It’s very likely that companies who need to rewrite code will screw up a second time. Honestly, there are a lot more historic rewrite flameouts than success stories. Companies know what they want to fix in the code, but they don’t understand what they need to fix in the company. I contend this is because there are company behaviors that promote failure, and if they did it once, they are likely to do it again. And again. Until, mercifully, the company goes down in flames. There are a lot of reasons why re-architecture and re-implementations projects fail. In no particular order … Big eyes: You are the chief developer and you hate your current product. You have catalogued everything that is wrong with it and how you would fix it. You have extensive lists of features you would like to implement. You have a grand vision of how this product should function, how it should be architected, and how it will be implemented. This causes your re-engineering effort to fail because you think that you are going to build perfect software, tackle every problem, and build every feature, in the first revision. And you commit to do so, just to get the project green-lighted. Resources: You current product sucks. It really sucks. It has atrocious quality and low performance, and is miserable to manage. It’s so freaking bad that customers ask for their money back, and sales falter. This causes your re-engineering effort to fail because there is simply not enough time, and not enough revenue to pay for your rebuild. Not with customers breathing down management’s neck, and investors looking for the quick “liquidity event”. So marketing keeps on marketing, sales keeps on selling, and you keep on supporting the old mess you have. Bad blood: When you car gets old and dies, you don’t expect someone to give you a new one for free. When your crappy old code no longer supports your customers, in essence you need to pay for new code. Yes, it is unfortunate that you bought a lemon last time, but you need to make additional investments in time and development resources, and fix the problems that led you down the wrong path. Your project fails because management is so bitter about the failure that they muck around with development practices, apply more pressure and try to get more involved with day-to-day development, when the opposite is needed. Expectations: Not only is the development team excited at not having to work on the atrocious code you have now, but they are really looking forward to working on a product that has semi-modern design. The whole department is buzzing, and so is management! This causes your re-engineering effort to fail because the Chickens think that no only are you going to deliver perfect software, but you are going to deliver every feature and function of the old crappy product, as well as a handful of new and extraordinary features as well. And it’s unlikely that management will let you adjust the ship date to accommodate the new demands.

Rich highlighted a great post from Rocky DiStefano of Visible Risk in today’s Incite: Blame the addicts – When I was working at Gartner, nothing annoyed me more than those client calls where all they wanted me to do was read them the Magic Quadrant and confirm that yes, that vendor really is in the upper right corner. I could literally hear them checking their “talked to the analyst” box. An essential part of the due diligence process was making sure their vendor was a Leader, even if it was far from the best option for them. I guess no one gets fired for picking the upper right. Rocky DeStefano nails how people see the Magic Quadrant in his Tetragon of Prestidigitation post. Don’t blame the analyst for giving you what you demand – they are just giving you your fix, or you would go someplace else. – RM Rocky is dead on – there are a number of constituencies that leverage information like the Magic Quadrant, and they all have different perspectives on the report. I don’t need to repeat what Rocky said, but I want to add a little more depth about each of the constituencies and provide some anecdotes from my travels. To be clear, Gartner (and Forrester, for that matter) place all sorts of caveats on their vendor rankings. They say not to use them to develop a short list, and they want clients to call to discuss their specific issues. But here’s the rub: They know far too many organizations use the MQ as a crutch to support either their own laziness and stupidity, or to play the game and support decisions they’ve already made. Institutionally they don’t care. As Rich pointed out, (most of) the analysts hate it. But the vendor rankings represent enough revenue that they don’t want to mess with them. Yes, that’s a cynical view, but at the end of the day both of the big IT research shops are public companies and they have to cater to shareholders. And shareholders love licensing 10-page documents for $20K each to 10 vendors. Rocky uses 3 cases to illuminate his point, first a veteran information security professional, and those folks (if they have a clue) know that they’ve got to focus their short list on vendors close to the Leader Quadrant. If not, they’ll spend more time justifying another lesser-ranked vendor than implementing the technology. It’s just not worth the fight. So they don’t. They pick the best vendor from the leader quadrant and move on. This leads us to the second case, the executive, who basically doesn’t care about the technology, but has a lot of stuff on his/her plate and figures if a vendor is a leader, they must have lots of customers calling Gartner and their stuff can’t be total crap. Most of the time, they’d be right. And the third case is vendors. Rocky makes some categorizations about the different quadrants, which are mostly accurate. Vendors in the “niche” space (bottom left) don’t play into the large enterprise market, or shouldn’t be. Those in the “challenger” quadrant (top left) are usually big companies with products they bundle into broad suites, so the competitiveness of a specific offering is less important. Those in the “visionary” sector (bottom right) delude themselves into thinking they’ve got a chance. They are small, but Gartner thinks they understand the market. In reality it doesn’t matter because the vast majority of the market – dumb and/or lazy information security professionals – see the MQ like this: In most enterprise accounts the only vendors with a chance are the ones in the leader quadrant, so placement in this quadrant is critical. I’ve literally had CEOs and Sales VPs take out a ruler and ask why our arch-nemesis was 2mm to the right of our dot. 2 frackin millimeters. You may think I’m kidding, but I’m not. So many of the high-flying vendors make it their objective to spend whatever resources it takes to get into the leader quadrant. They have customers call into Gartner with inquiries about their selection process (even though the selection is already made) to provide data points about the vendor. Yes, they do that, and the vendors provide talking points to their clients. They show up at the conferences and take full advantage of their 1on1 meeting slots. They buy strategy days. To be clear, you cannot buy a better placement on the MQ. But you can buy access, which gives a vendor a better opportunity to tell their story, which in many cases results in better placement. Sad but true. Vendors can game the system to a degree. Which is why Rich, Adrian, and I made a solemn blood oath that we at Securosis would never do a vendor ranking. We’d rather focus our efforts on the folks who want advice on how to do their job better. Not those trying to maximize their Tetris time. Share:

Yesterday I gave this presentation as a webcast for McAfee, but somehow my last 8 slides got dropped from the deck. So, as promised, here is a PDF of the slides. McAfee is hosting the full webcast deck over at their blog. Since we don’t host vendor materials here at Securosis, here is the subset of my slides. (You might still want to check out their full deck, since it also includes content from an end user). Presentation: Quick Wins with DLP Share: