The CIS Consensus Metrics and Project Quant

Just before release, the Center for Internet Security sent us a preview copy of the CIS Consensus Metrics. I’m a longtime fan of the Center, and, once I heard they were starting on this project, was looking forward to the results. Overall I think they did a solid job on a difficult problem. Is it perfect? Is it complete? No, but it’s a heck of a good start. There are a couple things that stand out: They do a great job of interconnecting different metrics, and showing you how you can leverage a single collected data attribute across multiple higher-level metrics. For example, a single “technology” (product/version) is used in multiple places, for multiple metrics. It’s clear they’ve designed this to support a high degree of automation across multiple workflows, supporting technologies, and operational teams. I like how they break out data attributes from security metrics. Attributes are the feeder data sets we use to create the security metrics. I’ve seen other systems that intermix the data with the metrics, creating confusion. Their selected metrics are a reasonable starting point for characterizing a security program. They don’t cover everything, but that makes it more likely you can collect them in the first place. They make it clear this is a start, with more metrics coming down the road. The metrics are broken out by business function – this version covering incident management, vulnerability management, patch management, application security, configuration management, and financial. The metric descriptions are clear and concise, and show the logic behind them. This makes it easy to build your own moving forward. There are a few things that could also be improved: The data attributes are exhaustive. Without automated tool support, they will be very difficult to collect. The document suggests prioritization, but doesn’t provide any guidance. A companion paper would be nice. This isn’t a mind-bending document, and we’ve seen many of these metrics before, but not usually organized together, freely available, well documented, or from a respected third party. I highly recommend you go get a copy. Now on to the CIS Consensus Metrics and Project Quant… I’ve had some people asking me if Quant is dead thanks to the CIS metrics. While there’s the tiniest bit of overlap, the two projects have different goals, and are totally complementary. The CIS metrics are focused on providing an overview for an entire security program, while Quant is focused on building a detailed operational metrics model for patch management. In terms of value, this should provide: Detailed costs associated with each step of a patch management process, and a model to predict costs associated with operational changes. Measurements of operational efficiency at each step of patch management to identify bottlenecks/inefficiencies and improve the process. Overall efficiency metrics for the entire patch management process. CIS and Quant overlap for the last goal, but not for the first two. If anything, Quant will be able to feed the CIS metrics. The CIS metrics for patch management include: Patch Policy Compliance Patch Management Coverage Mean Time to Patch I highly suspect all of these will appear in Quant, but we plan on digging into much greater depth to help the operational folks directly measure and optimize their processes. Share:

Read Post

Acquisitions and Strategy

There have been a couple of acquisitions in the last two weeks that I wanted to comment on; one by Oracle and one by McAfee. But between a minor case of food poisoning followed shortly by a major case of influenza, pretty much everything I wanted to do in the last 12 days, blogging notwithstanding, was halted. I am feeling better and trying to catch up on the stuff I wanted to talk about. At face value, neither of the acquisitions I want to mention are all that interesting. In the big picture, the investments do spotlight product strategy, so I want to comment on that. But before I do, I wanted to make some comments about how I go about assessing the value of an acquisition. I always try to understand the basic value proposition to the acquiring company, as well as other contributing factors. There are always a set of reasons why company A acquires company B, but understanding these reasons is much harder than you might expect. The goals of the buyers and the seller are not always clear. The market strategy and self-perception of each firm come into play when considering what they buy, why they bought it, and how much they were willing to pay. The most common motivators are as follows: Strategic: You want to get into a new market and it is either cheaper or faster to acquire a company that is already in that segment rather than organically develop and sell your own product. Basically this is paving the road for a strategic vision. Buying the major pieces to get into a new market or new growth opportunities in existing markets. No surprises here. Tactical: Filling in competitive gaps. A tactical effort to fill in a piece of the puzzle that your existing customers really need, or complete a product portfolio to address competitive deficiencies within your product. For example, having network DLP was fine up until a point, and then endpoint became a de facto requirement. We saw this with email security vendors who had killer email security platforms, but were still getting hammered in the market for not having complete web security offerings as well. Neither is surprising, but there are many more than these basic two reasons. And this is where things can get weird. Other motivating factors that make the deal go forward may not always be entirely clear. A couple that come to mind: Accretive Acquisition: Buying a solid company to foster your revenue growth curve. Clear value from the buyer’s perspective, but not so clear why profitable companies are willing to sell themselves for 2-4 times revenue when investor hopes, dreams, and aspirations are often much more than that. You have to view this from the seller’s side to make sense of it. There are many small, profitable companies out there in the $15-35M range, with no hope of going public because their market is too small and their revenue growth curve is too shallow. But the investors are pushing for an IPO that will take years, or possibly never happen. So what is your exit strategy? Which firms decide they want the early exit vs. betting their fortunes on a brighter future? You would think that in difficult economic times it is often based upon the stability of their revenue in the next couple of quarters. More often it comes down to which crazy CEOs still swear their firm is at the cusp of greatness for a multi-billion-dollar-a-year market and can convince their boards, vs. pragmatists who are ready to move on. I am already aware of a number of mid-sized companies and investment firms trying to tell “the wheat from the chaff” and target viable candidates, and a handful of pragmatic CEOs willing to look for their next challenge. Look for a lot more of these acquisitions in the next 12 months. Leveraged/Platform Enabler: Not quite strategic, not quite tactical, but a product or feature that multiple products can leverage. For example a web application server, a policy management engine, or a reporting engine may not be a core product offering, but could provide a depth of service that makes all your other products perform better. And better still, where a small firm could not achieve profitability, a large company might realize value across their larger customer base/product suite far in excess of the acquisition price. Good Tech, Bad Company: These firms are pretty easy to spot in this economy. The technology is good and the market is viable, but the company that produces the technology sucks. Wrong sales model, bad positioning, bad leadership decisions, or whatever – they simply cannot execute. I also call this “bargain bin”’ shopping because this is one of the ways mid-sized and larger firms can get cutting edge technology at firesale prices, and cash shortfalls force vendors to sell quickly! Still, it’s not always easy to distinguish the “over-sold bad tech” or “overfunded and poorly managed bad technology” firms from the “good tech, bad management” gems you are after. We have seen a few of these in the last 12 months, and we will see more in the coming 12 months as investors balk and lose confidence. The Hedge: This is where you want into a billion dollar market, but you cannot afford to buy one of the leaders, or your competitors have already bought all of them. What do you do? You practice the art of fighting without fighting: You buy any other player that is a long way from being the front-runner and market that solution like crazy! Sure, you’re not the leader in the category, but it’s good enough not to lose sales, and you paid a fraction of the price. It may even give you time to build a suitable product if you want to, but more often than not, you ride the positive perception train till it runs off the rails. Sellers know this game as well, and you will often see firms

Read Post

Totally Transparent Research is the embodiment of how we work at Securosis. It’s our core operating philosophy, our research policy, and a specific process. We initially developed it to help maintain objectivity while producing licensed research, but its benefits extend to all aspects of our business.

Going beyond Open Source Research, and a far cry from the traditional syndicated research model, we think it’s the best way to produce independent, objective, quality research.

Here’s how it works:

  • Content is developed ‘live’ on the blog. Primary research is generally released in pieces, as a series of posts, so we can digest and integrate feedback, making the end results much stronger than traditional “ivory tower” research.
  • Comments are enabled for posts. All comments are kept except for spam, personal insults of a clearly inflammatory nature, and completely off-topic content that distracts from the discussion. We welcome comments critical of the work, even if somewhat insulting to the authors. Really.
  • Anyone can comment, and no registration is required. Vendors or consultants with a relevant product or offering must properly identify themselves. While their comments won’t be deleted, the writer/moderator will “call out”, identify, and possibly ridicule vendors who fail to do so.
  • Vendors considering licensing the content are welcome to provide feedback, but it must be posted in the comments - just like everyone else. There is no back channel influence on the research findings or posts.
    Analysts must reply to comments and defend the research position, or agree to modify the content.
  • At the end of the post series, the analyst compiles the posts into a paper, presentation, or other delivery vehicle. Public comments/input factors into the research, where appropriate.
  • If the research is distributed as a paper, significant commenters/contributors are acknowledged in the opening of the report. If they did not post their real names, handles used for comments are listed. Commenters do not retain any rights to the report, but their contributions will be recognized.
  • All primary research will be released under a Creative Commons license. The current license is Non-Commercial, Attribution. The analyst, at their discretion, may add a Derivative Works or Share Alike condition.
  • Securosis primary research does not discuss specific vendors or specific products/offerings, unless used to provide context, contrast or to make a point (which is very very rare).
    Although quotes from published primary research (and published primary research only) may be used in press releases, said quotes may never mention a specific vendor, even if the vendor is mentioned in the source report. Securosis must approve any quote to appear in any vendor marketing collateral.
  • Final primary research will be posted on the blog with open comments.
  • Research will be updated periodically to reflect market realities, based on the discretion of the primary analyst. Updated research will be dated and given a version number.
    For research that cannot be developed using this model, such as complex principles or models that are unsuited for a series of blog posts, the content will be chunked up and posted at or before release of the paper to solicit public feedback, and provide an open venue for comments and criticisms.
  • In rare cases Securosis may write papers outside of the primary research agenda, but only if the end result can be non-biased and valuable to the user community to supplement industry-wide efforts or advances. A “Radically Transparent Research” process will be followed in developing these papers, where absolutely all materials are public at all stages of development, including communications (email, call notes).
    Only the free primary research released on our site can be licensed. We will not accept licensing fees on research we charge users to access.
  • All licensed research will be clearly labeled with the licensees. No licensed research will be released without indicating the sources of licensing fees. Again, there will be no back channel influence. We’re open and transparent about our revenue sources.

In essence, we develop all of our research out in the open, and not only seek public comments, but keep those comments indefinitely as a record of the research creation process. If you believe we are biased or not doing our homework, you can call us out on it and it will be there in the record. Our philosophy involves cracking open the research process, and using our readers to eliminate bias and enhance the quality of the work.

On the back end, here’s how we handle this approach with licensees:

  • Licensees may propose paper topics. The topic may be accepted if it is consistent with the Securosis research agenda and goals, but only if it can be covered without bias and will be valuable to the end user community.
  • Analysts produce research according to their own research agendas, and may offer licensing under the same objectivity requirements.
  • The potential licensee will be provided an outline of our research positions and the potential research product so they can determine if it is likely to meet their objectives.
  • Once the licensee agrees, development of the primary research content begins, following the Totally Transparent Research process as outlined above. At this point, there is no money exchanged.
  • Upon completion of the paper, the licensee will receive a release candidate to determine whether the final result still meets their needs.
  • If the content does not meet their needs, the licensee is not required to pay, and the research will be released without licensing or with alternate licensees.
  • Licensees may host and reuse the content for the length of the license (typically one year). This includes placing the content behind a registration process, posting on white paper networks, or translation into other languages. The research will always be hosted at Securosis for free without registration.

Here is the language we currently place in our research project agreements:

Content will be created independently of LICENSEE with no obligations for payment. Once content is complete, LICENSEE will have a 3 day review period to determine if the content meets corporate objectives. If the content is unsuitable, LICENSEE will not be obligated for any payment and Securosis is free to distribute the whitepaper without branding or with alternate licensees, and will not complete any associated webcasts for the declining LICENSEE. Content licensing, webcasts and payment are contingent on the content being acceptable to LICENSEE. This maintains objectivity while limiting the risk to LICENSEE. Securosis maintains all rights to the content and to include Securosis branding in addition to any licensee branding.

Even this process itself is open to criticism. If you have questions or comments, you can email us or comment on the blog.