Security Benchmarking, Beyond Metrics: You Can’t Benchmark everything

We have spent much of this series on why benchmarking is important. But we also need to point out some situations where benchmarking may not be appropriate. There are clearly situations where you can’t benchmark, particularly is on granular operational data, which I call Ninja Metrics. Dependency: Peer Group Data Most organizations have ‘nascent’ metrics programs, which may actually be too kind. But not all. Some have embraced detailed programs that gathers all sorts of data, mostly focused on operations. This represents the next step of a metrics program, and can be represented by some of the ideas put forth through our Quant research projects. We have created highly granular process maps (with associated metrics) for Patch Management, Network Security Operations, and Database Security. Each report specifies 50+ distinct metrics you can measure for that discipline. Yes, they are comprehensive. But there is a clear issue regarding benchmarking at this level. You will have a hard time finding similarly granular data from other companies for comparison. So the key dependency in implementing a benchmarking effort is the availability of peer group data for comparison. Compare to Yourself What do world class athletes do when they reach the top of the heap? You know, folks like Michael Phelps, who has basically shattered every record there is to shatter. They start comparing themselves to their past performance. Improvement is measured internally rather than externally. Even if no one else has ever done better, you know you can. And this is what you will likely need to do the most granular operational functions. When you take a step back this makes a lot of sense. The reality is that you aren’t necessarily trying to ‘win’ relative to operational excellence. You want to improve. That said, it is important to have an idea of where you stand in comparison to everybody else, at least on the high-level operational metrics. But for the most granular metrics, not so much. We hope that over time enough companies will start tracking granular operational metrics, and become comfortable enough with benchmarking, to share their data. But that’s not going to happen tomorrow or even the day after. In the meantime you can (and should) continue to push your metrics program forward – just understand your comparisons may need to be internal. As we wrap up the Benchmarking series, we’ll look at how to get some Quick Wins and see the process in action. Share:

Read Post

Data Security: Dropbox Should Mimic CrashPlan

I love it when people froth at the mouth once they finally realize the blazingly obvious! For today’s example let’s look at the big Dropbox data privacy controversy. There are a few serious problems with Dropbox, such as not requiring a password after a host is added, making it super easy for someone to pretend to be you (if they get your host ID) and access your data. That’s not great, but there are far worse things out there I worry about. But the big controversy is that… ghasp… Dropbox employees could access your data! But if you know anything about security you know that if you get a nice, pretty web interface; then somewhere, somehow, the odds are an admin at the service provider can access your data. There are techniques around this using creative programming, but one look at the Dropbox code in your browser makes it clear they aren’t using anything like that. This is because the Dropbox web servers need to see your data to show you the web interface. Ergo, the servers can decrypt your data. Ergo, someone at Dropbox can see it. Now this doesn’t need to be true – they could have restricted the web UI to metadata and still encrypted file contents, then used a browser plugin (or maybe even JavaScript) to decrypt the files. But both options entail usability and security tradeoffs. A great example of how to manage issues like these is the CrashPlan backup service. CrashPlan offers a cascade of security options, each with usability tradeoffs, and all available to users. (All these options protect your symmetric encryption key, not the data itself): Protect with your account password. CrashPlan can access and see your data if needed. Protect with a separate data password stored locally. CrashPlan admins can’t access your data (even to restore it). You need to keep and secure an extra password. Set your own encryption key. Can be on a per-machine basis. Very secure, requiring more management. There is, of course, much more to their encryption scheme – this is just the user-controllable portion. Dropbox could do something similar: Standard (perhaps the only option on their free plan): Basic account username/password as they have now. Enhanced Security: Set a personal password, with metadata in the clear. You can manipulate your files, but they can only be downloaded by the local agent (not via a browser) and you need to remember the password (no password restore capability). You can still share public files, which are stored in a separate directory using your account password as on the old system. High Security: Metadata and file data encrypted using your personal passphrase, separate from your account passphrase. Web UI can only manage public files – everything else is accessible only through their client. These would require serious development effort, and I don’t want to gloss over the complexity or importance of implementing this type of security correctly and safely. This stuff is hard. But it would be manageable if they made it a priority. But seriously, people – if you want something free/cheap with a pretty web interface to manage your data, odds are you are trading off security. I use Dropbox extensively and just encrypt the things I consider too private to expose. Share:

Read Post

Oracle CVSS: ‘Partial+’ is ‘Useful-’

Oracle announced the April 2011 CPU this week, with just a few moderate security issues for the database. Most DBAs monitor Oracle’s Critical Patch Updates (CPU) and are already familiar with the Common Vulnerability Scoring System (CVSS). For those of you who are not, it’s a method of calculating the relative risk of software and hardware vulnerabilities, resulting in a score that describes the potential severity of the vulnerability if an attacker were to exploit the problem. The scores are provided to help IT and operations teams decide what to patch and when. Vendors are cagey about providing vulnerability information – under the belief that any information helps attackers create exploits – so CVSS is a compromise to help customers without overly helping adversaries. Oracle uses CVSS scoring to categorize vulnerabilities, and publishes the scores with the quarterly release of their CPUs. When Oracle database vulnerabilities are found, they provide the raw data fed into the scoring system to generate the score included with the patch announcement. Most of the DBA community is not happy with the CVSS system, as it provides too little information to make informed decisions. The scoring methodology of assembling ‘base metrics’ with time and environmental variables is regarded as fuzzy logic, intended to obfuscate the truth more than to help DBAs understand risk. The general consensus is that risk scores have low value, but anything with a high score warrants further investigation. Google and 3rd party researchers become catalysts for patching decisions. Still, it’s better than nothing, and most DBAs are simply too busy to make much fuss about it, so there is little more that quiet grumbling in the community. Things seem a bit different with the April 2011 CPU. One of the bugs (CVE-2010-0903) was very similar in nature and exploit method to the last Oracle patch release (CVE-2011-0806), but had a dramatically lower risk score. The modification to the CVSS security score was based on Oracle’s modification to the CVSS scoring system to include a ‘Partial+’ impact metric. I have not spoken to anyone at Oracle about this, so maybe they have a threat model that demonstrates an attacker cannot get out of the compromised database, but I doubt it. It looks like an attempt to “game the system” by producing lower risk scores. Why do I say that? Because a ‘Partial’ reference makes sense if the scope of a vulnerability is localized to a very small part of the database. If it’s the entire database – which is what ‘Partial+’ indicates – pwnage is complete. Lowering of CVSS scores by saying the compromise is ‘Partial+’, instead of ‘Complete’ deliberately(?) misunderstands the way attackers work. Once they get a foot in the door they will automatically start looking for what to attack next. To reduce the risk score you would need to understand what else would be exposed by exploiting this vulnerability. Most people in IT – if they do a threat analysis at all – do it from the perspective of before the exploit. Few fully consider the scope of potential damage if the database were compromised and used against you. I can’t see how ‘Partial+’ makes things better or provides more accurate reporting, but it’s certainly possible the Oracle team has some rationale for the change I have not thought of. To me, though Partial+ means a database is an attacker platform for launching new attacks. And if you have been following any of the breach reports lately, you know most involve a chain of vulnerabilities and weaknesses strung together. Does this change make sense to you? Share:

Read Post

Totally Transparent Research is the embodiment of how we work at Securosis. It’s our core operating philosophy, our research policy, and a specific process. We initially developed it to help maintain objectivity while producing licensed research, but its benefits extend to all aspects of our business.

Going beyond Open Source Research, and a far cry from the traditional syndicated research model, we think it’s the best way to produce independent, objective, quality research.

Here’s how it works:

  • Content is developed ‘live’ on the blog. Primary research is generally released in pieces, as a series of posts, so we can digest and integrate feedback, making the end results much stronger than traditional “ivory tower” research.
  • Comments are enabled for posts. All comments are kept except for spam, personal insults of a clearly inflammatory nature, and completely off-topic content that distracts from the discussion. We welcome comments critical of the work, even if somewhat insulting to the authors. Really.
  • Anyone can comment, and no registration is required. Vendors or consultants with a relevant product or offering must properly identify themselves. While their comments won’t be deleted, the writer/moderator will “call out”, identify, and possibly ridicule vendors who fail to do so.
  • Vendors considering licensing the content are welcome to provide feedback, but it must be posted in the comments - just like everyone else. There is no back channel influence on the research findings or posts.
    Analysts must reply to comments and defend the research position, or agree to modify the content.
  • At the end of the post series, the analyst compiles the posts into a paper, presentation, or other delivery vehicle. Public comments/input factors into the research, where appropriate.
  • If the research is distributed as a paper, significant commenters/contributors are acknowledged in the opening of the report. If they did not post their real names, handles used for comments are listed. Commenters do not retain any rights to the report, but their contributions will be recognized.
  • All primary research will be released under a Creative Commons license. The current license is Non-Commercial, Attribution. The analyst, at their discretion, may add a Derivative Works or Share Alike condition.
  • Securosis primary research does not discuss specific vendors or specific products/offerings, unless used to provide context, contrast or to make a point (which is very very rare).
    Although quotes from published primary research (and published primary research only) may be used in press releases, said quotes may never mention a specific vendor, even if the vendor is mentioned in the source report. Securosis must approve any quote to appear in any vendor marketing collateral.
  • Final primary research will be posted on the blog with open comments.
  • Research will be updated periodically to reflect market realities, based on the discretion of the primary analyst. Updated research will be dated and given a version number.
    For research that cannot be developed using this model, such as complex principles or models that are unsuited for a series of blog posts, the content will be chunked up and posted at or before release of the paper to solicit public feedback, and provide an open venue for comments and criticisms.
  • In rare cases Securosis may write papers outside of the primary research agenda, but only if the end result can be non-biased and valuable to the user community to supplement industry-wide efforts or advances. A “Radically Transparent Research” process will be followed in developing these papers, where absolutely all materials are public at all stages of development, including communications (email, call notes).
    Only the free primary research released on our site can be licensed. We will not accept licensing fees on research we charge users to access.
  • All licensed research will be clearly labeled with the licensees. No licensed research will be released without indicating the sources of licensing fees. Again, there will be no back channel influence. We’re open and transparent about our revenue sources.

In essence, we develop all of our research out in the open, and not only seek public comments, but keep those comments indefinitely as a record of the research creation process. If you believe we are biased or not doing our homework, you can call us out on it and it will be there in the record. Our philosophy involves cracking open the research process, and using our readers to eliminate bias and enhance the quality of the work.

On the back end, here’s how we handle this approach with licensees:

  • Licensees may propose paper topics. The topic may be accepted if it is consistent with the Securosis research agenda and goals, but only if it can be covered without bias and will be valuable to the end user community.
  • Analysts produce research according to their own research agendas, and may offer licensing under the same objectivity requirements.
  • The potential licensee will be provided an outline of our research positions and the potential research product so they can determine if it is likely to meet their objectives.
  • Once the licensee agrees, development of the primary research content begins, following the Totally Transparent Research process as outlined above. At this point, there is no money exchanged.
  • Upon completion of the paper, the licensee will receive a release candidate to determine whether the final result still meets their needs.
  • If the content does not meet their needs, the licensee is not required to pay, and the research will be released without licensing or with alternate licensees.
  • Licensees may host and reuse the content for the length of the license (typically one year). This includes placing the content behind a registration process, posting on white paper networks, or translation into other languages. The research will always be hosted at Securosis for free without registration.

Here is the language we currently place in our research project agreements:

Content will be created independently of LICENSEE with no obligations for payment. Once content is complete, LICENSEE will have a 3 day review period to determine if the content meets corporate objectives. If the content is unsuitable, LICENSEE will not be obligated for any payment and Securosis is free to distribute the whitepaper without branding or with alternate licensees, and will not complete any associated webcasts for the declining LICENSEE. Content licensing, webcasts and payment are contingent on the content being acceptable to LICENSEE. This maintains objectivity while limiting the risk to LICENSEE. Securosis maintains all rights to the content and to include Securosis branding in addition to any licensee branding.

Even this process itself is open to criticism. If you have questions or comments, you can email us or comment on the blog.