Securosis

Research

FireStarter: There is No Market for Security Innovation

I often hear that there is no innovation left in security. That’s complete bullshit. There is plenty of innovation in security – but more often than not there’s no market for that innovation. For anything innovative to survive (at least in terms of physical goods and software) it needs to have a market. Sometimes, as with the motion controllers of the Nintendo Wii, it disrupts an existing market by creating new value. In other cases, the innovation taps into unknown needs or desires and succeeds by creating a new market. Security is a bit of a tougher nut. As I’ve discussed before, both on this blog and in the Disruptive Innovation talk I give with Chris Hoff, security is reactive by nature. We are constantly responding to changes in the underlying processes/organizations we protect, as well as to threats evolving to find new pathways through our defenses. With very few exceptions, we rarely invest in security to reduce risks we aren’t currently observing. If it isn’t a clear, present, and noisy danger, it usually finds itself on the back burner. Innovations like firewalls and antivirus really only succeeded when the environment created conditions that showed off value in these tools. Typically that value is in stopping pain, and not every injury causes pain. Even when we are proactive, there’s only a market for the reactive. The pain must pass a threshold to justify investment, and an innovator can only survive for so long without customer investment. Innovation is by definition almost always ahead of the market, and must create its own market to some degree. This is tough enough for cool things like iPads and TiVos, but nearly impossible for something less sexy like security. I love my TiVo, but I only appreciate my firewall. As an example, let’s take DLP. By bringing content analysis into the game, DLP became one of the most innovative, if not the most innovative, data security technologies we’ve seen. Yet 5+ years in, after multiple acquisitions by major vendors, we’re still only talking about a $150M market. Why? DLP didn’t keep your website up, didn’t keep the CEO browsing ESPN during March Madness, and didn’t keep email spam-free. It addresses a problem most people couldn’t see without DLP a DLP tool! Only when it started assisting with compliance (not that it was required) did the market start growing. Another example? How many of you encrypted laptops before you had to start reporting lost laptops as a data breach? On the vendor side, real innovation is a pain in the ass. It’s your pot of gold, but only after years of slogging it out (usually). Sometimes you get the timing right and experience a quick exit, but more often than not you either have to glom onto an existing market (where you’re fighting for your life against competitors that really shouldn’t be your competitors), or you find patient investors who will give you the years you need to build a new market. Everyone else dies. Some examples? PureWire wasn’t the first to market (ScanSafe was) and didn’t get the biggest buyout (ScanSafe again), but they timed it right and were in and out before they had to slog. Fidelis is forced to compete in the DLP market, although the bulk of their value is in managing a different (but related) threat. 7+ years in and they are just now starting to break out of that bubble. Core Security has spent 7 years building a market- something only possible with patient investors. Rumor is Palo Alto has some serious firewall and IPS capabilities, but rather than battling Cisco/Checkpoint, they are creating an ancillary market (application control) and then working on the cross-sell. Most of you don’t buy innovative security products. After paying off your maintenance and licens renewals, and picking up a few widgets to help with compliance, there isn’t a lot of budget left. You tend to only look for innovation when your existing tools are failing so badly that you can’t keep the business running. That’s why it looks like there’s no security innovation – it’s simply ahead of market demand, and without a market it’s hard to survive. Unless we put together a charity fund or those academics get off their asses and work on something practical, we lack the necessary incubators to keep innovation alive until you’re ready to buy it. So the question is… how can we inspire and sustain innovation when there’s no market for it? Or should we? When does innovation make sense? What innovation are we willing to spend on when there’s no market? When and how should we become early adopters? Share:

Share:
Read Post

Some DLP Metrics

One of our readers, Jon Damratoski, is putting together a DLP program and asked me for some ideas on metrics to track the effectiveness of his deployment. By ‘ask’, I mean he sent me a great list of starting metrics that I completely failed to improve on. Jon is looking for some feedback and suggestions, and agreed to let me post these. Here’s his list: Number of people/business groups contacted about incidents – tie in somehow with user awareness training. Remediation metrics to show trend results in reducing incidents – at start of DLP we had X events, after talking to people for 30 days about incidents we now have Y events. Trend analysis over 3, 6, & 9 month periods to show how the number of events has reduced as remediation efforts kick in. Reduction in the average severity of an event per user, business group, etc. Trend: number of broken business policies. Trend: number of incidents related to automated business practices (automated emails). Trend: number of incidents that generated automatic email. Trend: number of incidents that were generated from service accounts – (emails, batch files, etc.) I thought this was a great start, and I’ve seen similar metrics on the dashboards of many of the DLP products. The only one I have to add to Jon’s list is: Average number of incidents per user. Anyone have other suggestions? Share:

Share:
Read Post

Announcing NetSec Ops Quant: Network Security Metrics Suck. Let’s Fix Them.

The lack of credible and relevant network security metrics has been a thorn in my side for years. We don’t know how to define success. We don’t know how to communicate value. And ultimately, we don’t even know what we should be tracking operationally to show improvement (or failure) in our network security activities. But we in the echo chamber seem to be happier bitching about this, or flaming each other on mailing lists, than focusing on finding a solution. Some folks have tried to drive towards a set of metrics that make sense, but I can say most of the attempts are way too academic and also cost too much to collect to be usable in everyday practice. Not to mention that most of our daily activities aren’t even included in the models. Not to pick on them too much, but I think these issues are highlighted in the way the Center for Internet Security has scoped out network security metrics. Basically, they didn’t. They have metrics on Incident Management, Vulnerability Management, Patch Management, Configuration Change Management, Application Security, and Financial Metrics. So the guy managing the network security devices doesn’t count? Again, I know CIS is working towards a lot of other stuff, but the reality is the majority of security spending is targeted at the network and endpoint domains, and there are no good metrics for those. So let’s fix it. Today, we are kicking off the next in our series of Quant projects. This one is called Network Security Operations Quant, and we aim to build a process map and underlying cost model for how organizations manage their network security devices. The project’s formal objective and scope are: The objective of Network Security Operations Quant is to develop a cost model for monitoring and managing network security devices that accurately reflects the associated financial and resource costs. Secondarily, we also want to: Build the model in a manner that supports use as an operational efficiency model to help organizations optimize their network security monitoring and management processes, and compare costs of different options. Heavily engage the community and produce an open model with wide support and credibility, using the Totally Transparent Research process. Advance the state of IT metrics, particularly operational security metrics. We are grateful to our friends at SecureWorks, who are funding this primary research effort. As with all our quant processes, our methodology is: Establish the high level process map via our own research. Use a broad survey to validate and identify gaps in the process map. Define a set of subprocesses for each high-level process. Build metrics for each subprocess. Assemble the metrics into a model which can be used to track operational improvement. From a scoping standpoint, we are going to deal with 5 different network security processes: Monitoring firewalls Monitoring IDS/IPS Monitoring server devices Managing firewalls Managing IDS/IPS Yes, we know network security is bigger than just these 5 functions, but we can’t boil the ocean. There is a lot of other stuff we’ll model out using the Quant process over the next year, but this should be a good start. Put up or shut up We can’t do this alone. So we are asking for your help. First off, we are going to put together a “panel” of organizations to serve as the basis for our initial primary research. That means we’ll be either doing site visits or detailed phone interviews to understand how you undertake network security processes. We’ll also need the folks on the panel to shoot holes in our process maps before they are posted for public feedback. We are looking for about a dozen organizations from a number of different verticals and company sizes (large enterprise to mid-market). As with all our research, there will be no direct attribution to your organization. We are happy to sign NDAs and the like. If you are interested in participating, please send me an email directly at mrothman (at) securosis . com. Once the initial process maps are posted, we will post a survey to find out whether you actually do the steps we identify. We’ll also want your feedback on the process via posts that describe each step in the process. Everyone has an opportunity to participate and we hope you will take us up on it. This is possibly the coolest research project I’ve personally been involved with and I’m really excited to get moving on it. We look forward to your participation, so we finally can get on the same page, and figure out how to measure how we “network security plumbers” do our business. Share:

Share:
Read Post

Totally Transparent Research is the embodiment of how we work at Securosis. It’s our core operating philosophy, our research policy, and a specific process. We initially developed it to help maintain objectivity while producing licensed research, but its benefits extend to all aspects of our business.

Going beyond Open Source Research, and a far cry from the traditional syndicated research model, we think it’s the best way to produce independent, objective, quality research.

Here’s how it works:

  • Content is developed ‘live’ on the blog. Primary research is generally released in pieces, as a series of posts, so we can digest and integrate feedback, making the end results much stronger than traditional “ivory tower” research.
  • Comments are enabled for posts. All comments are kept except for spam, personal insults of a clearly inflammatory nature, and completely off-topic content that distracts from the discussion. We welcome comments critical of the work, even if somewhat insulting to the authors. Really.
  • Anyone can comment, and no registration is required. Vendors or consultants with a relevant product or offering must properly identify themselves. While their comments won’t be deleted, the writer/moderator will “call out”, identify, and possibly ridicule vendors who fail to do so.
  • Vendors considering licensing the content are welcome to provide feedback, but it must be posted in the comments - just like everyone else. There is no back channel influence on the research findings or posts.
    Analysts must reply to comments and defend the research position, or agree to modify the content.
  • At the end of the post series, the analyst compiles the posts into a paper, presentation, or other delivery vehicle. Public comments/input factors into the research, where appropriate.
  • If the research is distributed as a paper, significant commenters/contributors are acknowledged in the opening of the report. If they did not post their real names, handles used for comments are listed. Commenters do not retain any rights to the report, but their contributions will be recognized.
  • All primary research will be released under a Creative Commons license. The current license is Non-Commercial, Attribution. The analyst, at their discretion, may add a Derivative Works or Share Alike condition.
  • Securosis primary research does not discuss specific vendors or specific products/offerings, unless used to provide context, contrast or to make a point (which is very very rare).
    Although quotes from published primary research (and published primary research only) may be used in press releases, said quotes may never mention a specific vendor, even if the vendor is mentioned in the source report. Securosis must approve any quote to appear in any vendor marketing collateral.
  • Final primary research will be posted on the blog with open comments.
  • Research will be updated periodically to reflect market realities, based on the discretion of the primary analyst. Updated research will be dated and given a version number.
    For research that cannot be developed using this model, such as complex principles or models that are unsuited for a series of blog posts, the content will be chunked up and posted at or before release of the paper to solicit public feedback, and provide an open venue for comments and criticisms.
  • In rare cases Securosis may write papers outside of the primary research agenda, but only if the end result can be non-biased and valuable to the user community to supplement industry-wide efforts or advances. A “Radically Transparent Research” process will be followed in developing these papers, where absolutely all materials are public at all stages of development, including communications (email, call notes).
    Only the free primary research released on our site can be licensed. We will not accept licensing fees on research we charge users to access.
  • All licensed research will be clearly labeled with the licensees. No licensed research will be released without indicating the sources of licensing fees. Again, there will be no back channel influence. We’re open and transparent about our revenue sources.

In essence, we develop all of our research out in the open, and not only seek public comments, but keep those comments indefinitely as a record of the research creation process. If you believe we are biased or not doing our homework, you can call us out on it and it will be there in the record. Our philosophy involves cracking open the research process, and using our readers to eliminate bias and enhance the quality of the work.

On the back end, here’s how we handle this approach with licensees:

  • Licensees may propose paper topics. The topic may be accepted if it is consistent with the Securosis research agenda and goals, but only if it can be covered without bias and will be valuable to the end user community.
  • Analysts produce research according to their own research agendas, and may offer licensing under the same objectivity requirements.
  • The potential licensee will be provided an outline of our research positions and the potential research product so they can determine if it is likely to meet their objectives.
  • Once the licensee agrees, development of the primary research content begins, following the Totally Transparent Research process as outlined above. At this point, there is no money exchanged.
  • Upon completion of the paper, the licensee will receive a release candidate to determine whether the final result still meets their needs.
  • If the content does not meet their needs, the licensee is not required to pay, and the research will be released without licensing or with alternate licensees.
  • Licensees may host and reuse the content for the length of the license (typically one year). This includes placing the content behind a registration process, posting on white paper networks, or translation into other languages. The research will always be hosted at Securosis for free without registration.

Here is the language we currently place in our research project agreements:

Content will be created independently of LICENSEE with no obligations for payment. Once content is complete, LICENSEE will have a 3 day review period to determine if the content meets corporate objectives. If the content is unsuitable, LICENSEE will not be obligated for any payment and Securosis is free to distribute the whitepaper without branding or with alternate licensees, and will not complete any associated webcasts for the declining LICENSEE. Content licensing, webcasts and payment are contingent on the content being acceptable to LICENSEE. This maintains objectivity while limiting the risk to LICENSEE. Securosis maintains all rights to the content and to include Securosis branding in addition to any licensee branding.

Even this process itself is open to criticism. If you have questions or comments, you can email us or comment on the blog.