Why We Didn’t Pick the Cloud (Mostly), and That’s Okay

It’s no secret that we are currently working on a new software platform to deliver actionable security research to a broader market, engage folks, and… umm… feed our families. As you might expect, like any software project, it’s running about 30% late and 70% over budget. I just can’t seem to stop making our developers find exactly the right imagery and user experience to best represent the Securosis brand. Mike has coined a new term, ‘analness’, to describe the gyrations we’ve gone through, but I’m okay with that because we have spent years building our reputation and aren’t about to roll out a huge steaming pile of crap just to hit a delivery date. As we close in on the finish line, we faced a huge decision on how to host this. Our current provider is pretty good, but we ran into some issues earlier this year that prompted us to look at alternatives. And we are co-hosted, which won’t work once we start loading sensitive content into a paid service. So we began the long evaluation process of picking the right architecture and host. Well, that and satisfying our paranoia regarding site security. Despite being heavy cloud folks, we eventually decided on a dedicated server model offered by a specialized hosting company. Yes, we understand that’s probably counterintuitive, so here’s why we didn’t go that way. Co-hosting and VPS For the most part our current site is totally fine with our current load, and our hosting provider is a lot more security-conscious than most. I launched as a blog over at Bluehost, on a WordPress co-host. It worked totally fine, but as we started expanding it was clear that platform couldn’t meet our growing needs. We decided to switch to a better content management system (ExpressionEngine), and while we could technically run it there, we decided to go with a more specialized provider ( We have been mostly happy with the change, even though EH is considerably more expensive, because we get a lot more for what we pay. They also have excellent growth options to expand to a Virtual Private Server or even dedicated boxes if needed. But it’s still a co-host model. The one problem we hit earlier this year appeared after a major platform upgrade. Our back end became nearly unusable due to performance problems, and when I submitted a support request they kept blaming our configuration or plugins. We are big boys, and willing to accept when we screw up. We turned our system upside down and couldn’t find anything that would kill the performance of the admin console. As it turned out we were right. Another client in our cluster over-used resources – as I had initially suggested. We were bothered by their lack of investigation, and by the (realized) potential for another customer to impact us. That convinced us we need to get off co-hosting, and into VPS or cloud. We also had to factor in all the security reasons to drop a co-hosted model once we have content we want to protect. VPS vs. Cloud We quickly ruled out VPS. As our knowledge and experience working with various cloud services grew, we saw no reason to pick VPS over a pure cloud model. To be honest, while I see co-hosting surviving for a while, I definitely see the allure of VPS cratering in the next few years, as customers keep comparing VPS offerings against the rapidly evolving public cloud offerings. I decided we would go completely cloud. Aside from the lack of advantages to VPS, we were conscious of the importance of eating our own dogfood, now that we are working so deeply with the Cloud Security Alliance and advising people on cloud projects. Our criteria for a cloud provider including a security conscious shop, judged on both what they publish and checks with various industry connections. We wanted some IPS/firewall and patch management support options to improve our baseline security and reduce our management overhead. As our IT guy, I simply don’t have the time to manage all our patches/fixes myself. If I were caught on an international flight when we needed to block and fix a critical 0day, we could be screwed. That was unacceptable. Other factors included our plan to use a cloud-based WAF. Not that it could block everything, but the combination of blocking basic scans and providing better analytics was attractive. We also factored in performance, as we know our potential audience is self-limiting, and what we are delivering isn’t very CPU intensive. We need a little beef, and more importantly the capability to grow, but we couldn’t forsee a need for anything too crazy. It’s not like we are Netflix or anything (yet). So there we were – I thought we were all set, until… From Cloud to Dedicated I wasn’t fully satisfied with the options I found (all of which cost a heck of a lot more than a basic AWS deployment), but I felt confident that we could get what we need at a reasonable price. Then we mentioned what we were doing to a trusted friends in the industry. For now I won’t mention who we are working with, but someone we highly respect offers dedicated hosting in a special section of a major data center they lease (their own cage). I am not sure they expected us to take them up on the offer. It’s not like they were soliciting our business – this came up over beer. These folks are as paranoid as we are (maybe more), and aside from hosting the site they will implement some stringent and unusual security controls we couldn’t possibly get anywhere else for any reasonable price. Normally they don’t use this model even with their existing clients, and we are going to be their first test case beyond internal infrastructure. As a bonus, their data center guarantees 100% infrastructure uptime. In writing. (Note: this doesn’t mean our boxes, just their network and power). Trusted

Read Post

Security Benchmarking, Beyond Metrics: Benchmarking in Action

As we wrap up our series on Security Benchmarking, we find it instructive to actually walk through a scenario and apply the process. Yes, the scenario is a bit contrived, but we’ll use it to hit the high points of the process, deciding where to start, collecting the data, establishing the peer group and communicate the findings. Keep in mind that we focus on getting quick wins, showing immediate value, building momentum and leveraging that momentum for programatic success. Scenario For our case study, let’s use a mid-tier financial company as our example. I’d say large enterprise, but in reality there are a lot of nuances and moving pieces within a large enterprise that need more detailed discussion. So let’s keep it relatively simple. Likewise, we picked the financial vertical because of 1) need and 2) availability of data. The reality of the financial industries regulatory oversight has created a general perspective of security first and data-centricity (yes, these are the folks that try to do risk management for a living) means these businesses are move likely to embrace a benchmarking mentality. In our (contrived) scenario, the Board drove the hiring a new CISO to “fix security.” As easy as it is to think this was just catering to a board directive, the senior team seems to have a commitment to fix things and do it the right way. So the CISO has a clear honeymoon period and some leeway in thinking somewhat unconventionally about how to build the security program. The new CISO still spends some time figuring out what’s installed and what’s not working, but he knows the organization has AV deployed, they use an external scanning service, and do a pretty good job of patching on internal systems. Yet, like many smaller financial institutions they use hosted applications for most of their business processing. So a lot of their data is not within their direct control. Over the past few years, the organization has had a handful of incidents, but none really resulted in major data loss. Thus the CISO was pleasantly surprised when he got the mandate to fix the security program, when it wasn’t outwardly broken. The senior team came to the conclusion they are living on borrowed time and want to act decisively to make sure they are ready when the brown stuff hits the fan (which it inevitably will). See? We told the you the scenario was contrived, but without a senior-level mandate to make changes in implement a security program, getting any kind of security metrics/benchmarking initiative going will be difficult. Where Do You Start? Now the CISO has to figure out where to start. He’s decided that he wants to figure out where his most apparent gaps are. You know, the ones you can drive a Mack Truck through. So he starts with a comprehensive risk assessment to build a baseline, but he also wants to compare his environment to other like-sized companies (both in and out of his industry) to figure out how he compares to those organizations. Keep in mind, boiling the ocean and trying to do everything at this point is a bad idea. He’d get buried in the nuances of the data and not get anything done, which could endanger his entire security program. So he needs to ask the following questions: What do you need to achieve? Where are the key operational problems? This is where you always have to start. In our case study, the CISO is looking to identify his most critical gaps, and given the luck they’ve had in not having a huge data loss even with a few breaches, he wants to start with incident response. What data do you have? Next you have to figure out if you have the data or can get it easily. With incident data, the reality is the findings from the forensics investigations exist, but haven’t been put in any kind of format for comparison. But the data exists, so it makes sense to keep pressing down this path. If the data doesn’t exist or can’t be gathered quickly, then it’s time to look at Plan B. You don’t want to hold up the effort because it’s all about getting the quick win. Where will be most impactful to show comparative data? Selecting to focus initially on incident response represents a pretty shrewd move for the new CISO. He knows the board and senior management is sensitive to not getting nailed, as well as having a set of reasonable consensus metrics available (from CIS), and having the data. This increases the chances of success. Peer Groups and Service Providers Next, our CISO has to define the peer group for analysis. This isn’t brain surgery. He’ll need to compare to other financials (duh!), but also companies in other regulated industries (like healthcare and utilities) of a similar size. The good news is there are a ton of mid-sized hospital groups, as well as many community utilities, with similarly sensitive data. But how do they get their hands on that kind of data for comparison purposes? Now we go back and revisit the selection criteria for any kind of provider you’d think about for benchmarking services. Remember, these folks have to 1) have access to the data you’d need and 2) be able to protect the data you share with them. To be clear, you may not be able to get everything done with just one provider. In our case study here, the CISO will actually pick two. The first is his regional bank ISAC, who has been gathering data from its members for a while. The second is a commercial benchmarking offering, since they have more data about other industries that aren’t the focus of the ISAC. In reality, the CISO would like to just have one provider, but until a critical mass of data for many verticals is captured, he’ll need to piecemeal the solution to solve the problem. Analyze Equipped with data regarding his first

Read Post

Security Benchmarking, Beyond Metrics: Index

As is (now) our custom, we post a set of links to each blog series as it wraps up. This both gives us an easy way to find all our posts, and acknowledges that not everyone wants our complete feed and may want to read posts once they’re all written. As always, we love feedback on our work in progress. Yes, it’s time consuming to take time out for comment on specific posts. But remember that pretty much all our research is available free of charge, so it’s not too much to ask for a little constructive criticism on our work, is it? Please take a look and sink your teeth in. Introduction Security Metrics (from 40,000 feet) Collecting Data Systematically Sharing Data Safely Defining Peer Groups and Analyzing Data Communications Strategies Continuous Improvement You Can’t Benchmark Everything Benchmarking in Action As always, we thank you for reading, commenting, and making our research better. Share:

Read Post

Totally Transparent Research is the embodiment of how we work at Securosis. It’s our core operating philosophy, our research policy, and a specific process. We initially developed it to help maintain objectivity while producing licensed research, but its benefits extend to all aspects of our business.

Going beyond Open Source Research, and a far cry from the traditional syndicated research model, we think it’s the best way to produce independent, objective, quality research.

Here’s how it works:

  • Content is developed ‘live’ on the blog. Primary research is generally released in pieces, as a series of posts, so we can digest and integrate feedback, making the end results much stronger than traditional “ivory tower” research.
  • Comments are enabled for posts. All comments are kept except for spam, personal insults of a clearly inflammatory nature, and completely off-topic content that distracts from the discussion. We welcome comments critical of the work, even if somewhat insulting to the authors. Really.
  • Anyone can comment, and no registration is required. Vendors or consultants with a relevant product or offering must properly identify themselves. While their comments won’t be deleted, the writer/moderator will “call out”, identify, and possibly ridicule vendors who fail to do so.
  • Vendors considering licensing the content are welcome to provide feedback, but it must be posted in the comments - just like everyone else. There is no back channel influence on the research findings or posts.
    Analysts must reply to comments and defend the research position, or agree to modify the content.
  • At the end of the post series, the analyst compiles the posts into a paper, presentation, or other delivery vehicle. Public comments/input factors into the research, where appropriate.
  • If the research is distributed as a paper, significant commenters/contributors are acknowledged in the opening of the report. If they did not post their real names, handles used for comments are listed. Commenters do not retain any rights to the report, but their contributions will be recognized.
  • All primary research will be released under a Creative Commons license. The current license is Non-Commercial, Attribution. The analyst, at their discretion, may add a Derivative Works or Share Alike condition.
  • Securosis primary research does not discuss specific vendors or specific products/offerings, unless used to provide context, contrast or to make a point (which is very very rare).
    Although quotes from published primary research (and published primary research only) may be used in press releases, said quotes may never mention a specific vendor, even if the vendor is mentioned in the source report. Securosis must approve any quote to appear in any vendor marketing collateral.
  • Final primary research will be posted on the blog with open comments.
  • Research will be updated periodically to reflect market realities, based on the discretion of the primary analyst. Updated research will be dated and given a version number.
    For research that cannot be developed using this model, such as complex principles or models that are unsuited for a series of blog posts, the content will be chunked up and posted at or before release of the paper to solicit public feedback, and provide an open venue for comments and criticisms.
  • In rare cases Securosis may write papers outside of the primary research agenda, but only if the end result can be non-biased and valuable to the user community to supplement industry-wide efforts or advances. A “Radically Transparent Research” process will be followed in developing these papers, where absolutely all materials are public at all stages of development, including communications (email, call notes).
    Only the free primary research released on our site can be licensed. We will not accept licensing fees on research we charge users to access.
  • All licensed research will be clearly labeled with the licensees. No licensed research will be released without indicating the sources of licensing fees. Again, there will be no back channel influence. We’re open and transparent about our revenue sources.

In essence, we develop all of our research out in the open, and not only seek public comments, but keep those comments indefinitely as a record of the research creation process. If you believe we are biased or not doing our homework, you can call us out on it and it will be there in the record. Our philosophy involves cracking open the research process, and using our readers to eliminate bias and enhance the quality of the work.

On the back end, here’s how we handle this approach with licensees:

  • Licensees may propose paper topics. The topic may be accepted if it is consistent with the Securosis research agenda and goals, but only if it can be covered without bias and will be valuable to the end user community.
  • Analysts produce research according to their own research agendas, and may offer licensing under the same objectivity requirements.
  • The potential licensee will be provided an outline of our research positions and the potential research product so they can determine if it is likely to meet their objectives.
  • Once the licensee agrees, development of the primary research content begins, following the Totally Transparent Research process as outlined above. At this point, there is no money exchanged.
  • Upon completion of the paper, the licensee will receive a release candidate to determine whether the final result still meets their needs.
  • If the content does not meet their needs, the licensee is not required to pay, and the research will be released without licensing or with alternate licensees.
  • Licensees may host and reuse the content for the length of the license (typically one year). This includes placing the content behind a registration process, posting on white paper networks, or translation into other languages. The research will always be hosted at Securosis for free without registration.

Here is the language we currently place in our research project agreements:

Content will be created independently of LICENSEE with no obligations for payment. Once content is complete, LICENSEE will have a 3 day review period to determine if the content meets corporate objectives. If the content is unsuitable, LICENSEE will not be obligated for any payment and Securosis is free to distribute the whitepaper without branding or with alternate licensees, and will not complete any associated webcasts for the declining LICENSEE. Content licensing, webcasts and payment are contingent on the content being acceptable to LICENSEE. This maintains objectivity while limiting the risk to LICENSEE. Securosis maintains all rights to the content and to include Securosis branding in addition to any licensee branding.

Even this process itself is open to criticism. If you have questions or comments, you can email us or comment on the blog.