Securing Hadoop: Security Recommendations for NoSQL platforms [New Series]

It’s been three and a half years since we published our research paper on Securing Big Data. That research paper has been one of the more popular papers we’ve ever written. And it’s no wonder as NoSQL adoption was faster than we expected; we see hundreds of new projects popping up, leveraging the scale, analytics and low cost of these platforms. It’s not hyperbole to claim it has revolutionized the database market over the last 5 years, and community support behind these platforms – and especially Hadoop – is staggering. At the time we wrote the last paper security, Hadoop – much less the other platforms – was something of a barren wasteland. They did not include basic controls for data protection, most third party tools could not scale along with NoSQL and thus were of little use to developers, and leaders of NoSQL firms directed resources to improving performance and scalability, not security. Heck, in 2012 the version of Hadoop I evaluated did not even require and administrative password! But when it comes to NoSQL security, and Hadoop specifically, things have changed dramatically. As we advise clients on how to implement security controls, there are many new options to consider. And while there remains some gaps in monitoring and assessment capabilities, Hadoop has (mostly) reached security parity with the relational platforms of old. We can’t call it a barren wasteland any longer, so to accurately advise people on approaches and tools to leverage, we can no longer refer them back to that original paper. So we are kicking off a new research series to refresh this paper. Most of the content will be new. And this time we will do this a little bit differently that the last time. First, we are going to provide less background on what makes NoSQL different than relational databases, as most people in IT are now comfortable with the architectural and functional distinctions between the two. Second, most of our recommendations will still apply to NoSQL platforms in general, but this research will be more focused on Hadoop as we get a majority of questions on Hadoop security despite dozens of alternatives. Finally, as there are lots more aspects to talk about, we’ll weave preventative and detective controls into a more operational (i.e.: day to day management) model for both data and database infrastructure. Here is how we are laying out the series: Hadoop Architecture and Assembly — The goal with this post is to succinctly outline what Hadoop and similar styles of NoSQL clusters look like, how they are assembled and how they are used. In this light we get a better idea of the security challenges and what sort of protections need to be leveraged. As developers and data scientists stretch systems from a performance and scalability standpoint, and custom assemblage of open source and commercial products, there really is no such thing as a standard Hadoop deployment. So with these considerations in mind we will map out threats to the cluster. Use Cases & Security Architectures — This post will discuss the strategic considerations for deploying security for big data. Depending upon which model you choose, you change where certain types of threats are addressed, and consequentially what tools you will rely upon to provide security. Or stated another way, the security model you choose will dictate what security technologies you need to prevent and detect threats. There are several approaches that organizations take to secure Hadoop and other NoSQL clusters. These range from securing the network around the cluster, Identity Management, to maintaining security controls on each node within the cluster, or even taking a data centric approach to security. We’ll go over the major trends we see today, and discuss the advantages and pitfalls of each approach. Building Security Into the Cluster — Here is where we discuss how all of the pieces fit together. There are many security controls available, and each address a specific threat vector an attacker may employ. We’ll focus on security controls you want to build into your cluster from the start: identity, authorization, transport layer security, application security and data encryption. This will focus on the base security controls that allow you to define how the cluster should be used from a security standpoint. Operational Security — Here we will focus on the day to day security controls for monitoring ongoing security and discovering user behavior and ongoing security operations. Aspects like configuration management, patching, logging, monitoring, and node validation. We’ll even discuss integrating a DevOps approach to cluster administration to improve speed and consistency. Commercial Hadoop and NoSQL variants — Hadoop is the dominant flavor of ‘big data’ in use today. In this section we will discuss what the commercial Hadoop platform vendors are doing to promote security for their customers with a blend of open source, home grown and 3rd party security product support. There is no reason to roll you’re own security out of necessity as commercial variants often add on their own products or provide bundles for you. Each offers unique capabilities and each has a vision of what their customers should focus on, so we will cover some of the current offerings. We will also offer some advice on the application of security to non-Hadoop platforms. While Hadoop is the most commonly used platform, there are specialized flavors of NoSQL that are eminently appropriate for certain business challenges and are in wide use. Some even use HDFS or other Hadoop components that allow the use of the same security controls across different clusters. We will close out this section discussing where the security controls we have already discussed can be deployed in non-Hadoop environments where appropriate. As with our original paper, this is not intended to be an exhaustive look at all potential security options, but to get the IT and development teams who run these clusters basic security controls in place. Up next, Hadoop Architecture and Assembly. Share:

Read Post

The EIGHTH Annual Disaster Recovery Breakfast: Clouds Ahead

Once again Securosis and friends are hosting our RSA Conference Disaster Recovery Breakfast. It’s really hard to believe this is the eighth year for this event. Regardless of San Francisco’s February weather, we expect to be seeing clouds all week. But we’re happy to help you cut through the fog to grab some grub, drinks, and bacon. Kidding aside, we are grateful that so many of our friends, clients, and colleagues enjoy a couple hours away from the show that is now the RSAC. By Thursday we’re all disasters, and it’s very nice to have a place to kick back, have some conversations at a normal decibel level, and grab a nice breakfast. Did we mention there will be bacon? With the continued support of Kulesa Faul, we’re honored to bring two new supporters in this year. If you don’t know our friends at CHEN PR and LaunchTech, you’ll have a great opportunity to say hello and thank them for helping support your habits. As always the breakfast will be Thursday morning from 8-11 at Jillian’s in the Metreon. It’s an open door – come and leave as you want. We will have food, beverages, and assorted non-prescription recovery items to ease your day. Yes, the bar will be open – Mike has acquired a taste for Bailey’s in his coffee. Please remember what the DR Breakfast is all about. No marketing, no spin, no t-shirts, and no flashing sunglasses – it’s just a quiet place to relax and have muddled conversations with folks you know, or maybe even go out on a limb and meet someone new. After three nights of RSA Conference shenanigans, we are confident you will enjoy the DRB as much as we do. See you there. To help us estimate numbers, please RSVP to rsvp (at) securosis (dot) com. Share:

Read Post

Security is Changing. So is Securosis.

Last week Rich sent around Cockroaches Versus Unicorns: The Golden Age Of Cybersecurity Startups, by Mahendra Ramsinghani over at TechCrunch, for us to read. It isn’t an article every security professional needs to read, but it is certainly mandatory reading for anyone who makes buying decisions, tracks the security market, or is on the investment or startup side. It also nearly perfectly describes what we are going through as a company. His premise is that ‘unicorns’ are rare in the security industry. There are very few billion-dollar market cap companies, relative to the overall size of the market. But security companies are better suited to survive downturns and other challenging times. We are basically ‘cockroaches’, which persist through every tech Armageddon, often due to our ability to fall back on services. Many security startups are not unicorns; rather, they are cockroaches – they rarely die, and  in tough times, they can switch into a frugal/consulting mode. Like cockroaches, they can survive long nuclear winters. Security companies can be capital-efficient, and typically consume ~$40 million to reach break-even. This gives them a survival edge – but VCs are looking for a “growth edge.” The security market also appears much smaller than it should be considering the market dynamics, although it is very possible that is changing thanks to the hostile world out there. The article also postulates that the entire environment is shifting, with carriers and managed services providers jumping into acquisitions while large established players struggle. Yet most of the startups VCs see are just more of the same, fail to differentiate, and rely far too much on really poor FUD-based sales dynamics. With increasing hacks, the CISO’s life has just become a lot messier. One CISO told me, “Between my HVAC vendor and my board of directors, I am stretched. And everyday I get a hundred LinkedIn requests from vendors. Their FUD approach to security sales is exhausting.” And “I have seen at least 40 FireEye killers in the past 12 months,” one Palo Alto-based VC told me. Clearly he was exhausted. Some sub-sectors are overheated and investors are treading cautiously. We certainly see the same thing. How many threat intel and security analytics startups does the industry need? We get a few briefing requests a week, from another new company doing exactly the same things. And all our CISO friends hate vendor sales techniques. These senior security folks get upwards of 500 emails and 100 phone calls a week from sales people trying to get meetings. All this security crap looks the same. This combination inevitably leads to a contraction of seed capital, and that is where our story starts. DisruptOPS Most of you have noticed that over the past few years our research has skewed strongly toward cloud security, automation, and DevOps. This started with our initial partnership with the Cloud Security Alliance to build out the CCSK training class around 6 years ago. Rich had to create all the hands-on labs, which augered him down the rabbit hole of Amazon Web Services, OpenStack, Azure, and all the supporting tools. As analysts we like to think it’s our job to have a good sense of what’s coming down the road. We made a bet on the cloud and it paid off, transitioning from a hobby to generate beer money to a major source of ongoing revenue. It also opened us up to a wider client base, especially among end-user organizations. Three years ago Rich realized that in all his cloud security engagements, and all the classes we taught, we heard the same problems over and over. The biggest unsolved problem seemed to be cloud security automation. The next year was spent writing some proof-of-concept code merely to support conference presentations because there were no vendor examples, but at every talk attendees kept asking for “more… faster”. This demand became too great to ignore, and nearly 2 years ago we decided to start building our own platform. And we did … we built our own cloud security platform. Don’t worry, we don’t have anything to sell you – this is where Ramsinghani’s article comes in.   Our initial plan was to self fund development (Securosis is an awesome business) until we had a solid demo/prototype. Then we assumed it would be easy to get seed cash from some of our successful friends and build a new company in parallel with Securosis to focus on the product. We didn’t just want to start up a software company and jettison Securosis because our research is an essential driver to maintain differentiation, and we wanted to build the company without going the traditional VC route. We also have some practical limitations on how we can do things. We are older, have families to support, and have deep roots where we live that preclude relocation. The analogy we use is that we can’t go back to eating ramen for dinner every night in a coding flophouse. The demo killed when we showed it to people, we are really smart, and people like us. Our future was bright. Then we got hit with the reality clue bat. Everything was looking awesome last year at RSA when we started showing people and talking to investors. By summer all our options fell apart. We didn’t fit the usual model. We weren’t going to move to the Bay Area. We couldn’t take pay cuts to ‘normal’ founder levels and still support our families. And to be honest, we still didn’t want to go the normal VC route. We just weren’t going to play that game, given the road rash both Mike and Adrian have from earlier in their careers. Just like the article said, we couldn’t find seed funding. At least not the way we wanted to build the company. We even had a near-miss on an acquisition, but we couldn’t line everything up to hit everyone’s goals and expectations. Yet while this all went on, the Securosis business you see every day continued to boom. We

Read Post

Totally Transparent Research is the embodiment of how we work at Securosis. It’s our core operating philosophy, our research policy, and a specific process. We initially developed it to help maintain objectivity while producing licensed research, but its benefits extend to all aspects of our business.

Going beyond Open Source Research, and a far cry from the traditional syndicated research model, we think it’s the best way to produce independent, objective, quality research.

Here’s how it works:

  • Content is developed ‘live’ on the blog. Primary research is generally released in pieces, as a series of posts, so we can digest and integrate feedback, making the end results much stronger than traditional “ivory tower” research.
  • Comments are enabled for posts. All comments are kept except for spam, personal insults of a clearly inflammatory nature, and completely off-topic content that distracts from the discussion. We welcome comments critical of the work, even if somewhat insulting to the authors. Really.
  • Anyone can comment, and no registration is required. Vendors or consultants with a relevant product or offering must properly identify themselves. While their comments won’t be deleted, the writer/moderator will “call out”, identify, and possibly ridicule vendors who fail to do so.
  • Vendors considering licensing the content are welcome to provide feedback, but it must be posted in the comments - just like everyone else. There is no back channel influence on the research findings or posts.
    Analysts must reply to comments and defend the research position, or agree to modify the content.
  • At the end of the post series, the analyst compiles the posts into a paper, presentation, or other delivery vehicle. Public comments/input factors into the research, where appropriate.
  • If the research is distributed as a paper, significant commenters/contributors are acknowledged in the opening of the report. If they did not post their real names, handles used for comments are listed. Commenters do not retain any rights to the report, but their contributions will be recognized.
  • All primary research will be released under a Creative Commons license. The current license is Non-Commercial, Attribution. The analyst, at their discretion, may add a Derivative Works or Share Alike condition.
  • Securosis primary research does not discuss specific vendors or specific products/offerings, unless used to provide context, contrast or to make a point (which is very very rare).
    Although quotes from published primary research (and published primary research only) may be used in press releases, said quotes may never mention a specific vendor, even if the vendor is mentioned in the source report. Securosis must approve any quote to appear in any vendor marketing collateral.
  • Final primary research will be posted on the blog with open comments.
  • Research will be updated periodically to reflect market realities, based on the discretion of the primary analyst. Updated research will be dated and given a version number.
    For research that cannot be developed using this model, such as complex principles or models that are unsuited for a series of blog posts, the content will be chunked up and posted at or before release of the paper to solicit public feedback, and provide an open venue for comments and criticisms.
  • In rare cases Securosis may write papers outside of the primary research agenda, but only if the end result can be non-biased and valuable to the user community to supplement industry-wide efforts or advances. A “Radically Transparent Research” process will be followed in developing these papers, where absolutely all materials are public at all stages of development, including communications (email, call notes).
    Only the free primary research released on our site can be licensed. We will not accept licensing fees on research we charge users to access.
  • All licensed research will be clearly labeled with the licensees. No licensed research will be released without indicating the sources of licensing fees. Again, there will be no back channel influence. We’re open and transparent about our revenue sources.

In essence, we develop all of our research out in the open, and not only seek public comments, but keep those comments indefinitely as a record of the research creation process. If you believe we are biased or not doing our homework, you can call us out on it and it will be there in the record. Our philosophy involves cracking open the research process, and using our readers to eliminate bias and enhance the quality of the work.

On the back end, here’s how we handle this approach with licensees:

  • Licensees may propose paper topics. The topic may be accepted if it is consistent with the Securosis research agenda and goals, but only if it can be covered without bias and will be valuable to the end user community.
  • Analysts produce research according to their own research agendas, and may offer licensing under the same objectivity requirements.
  • The potential licensee will be provided an outline of our research positions and the potential research product so they can determine if it is likely to meet their objectives.
  • Once the licensee agrees, development of the primary research content begins, following the Totally Transparent Research process as outlined above. At this point, there is no money exchanged.
  • Upon completion of the paper, the licensee will receive a release candidate to determine whether the final result still meets their needs.
  • If the content does not meet their needs, the licensee is not required to pay, and the research will be released without licensing or with alternate licensees.
  • Licensees may host and reuse the content for the length of the license (typically one year). This includes placing the content behind a registration process, posting on white paper networks, or translation into other languages. The research will always be hosted at Securosis for free without registration.

Here is the language we currently place in our research project agreements:

Content will be created independently of LICENSEE with no obligations for payment. Once content is complete, LICENSEE will have a 3 day review period to determine if the content meets corporate objectives. If the content is unsuitable, LICENSEE will not be obligated for any payment and Securosis is free to distribute the whitepaper without branding or with alternate licensees, and will not complete any associated webcasts for the declining LICENSEE. Content licensing, webcasts and payment are contingent on the content being acceptable to LICENSEE. This maintains objectivity while limiting the risk to LICENSEE. Securosis maintains all rights to the content and to include Securosis branding in addition to any licensee branding.

Even this process itself is open to criticism. If you have questions or comments, you can email us or comment on the blog.