It’s no secret that we are currently working on a new software platform to deliver actionable security research to a broader market, engage folks, and… umm… feed our families. As you might expect, like any software project, it’s running about 30% late and 70% over budget. I just can’t seem to stop making our developers find exactly the right imagery and user experience to best represent the Securosis brand. Mike has coined a new term, ‘analness’, to describe the gyrations we’ve gone through, but I’m okay with that because we have spent years building our reputation and aren’t about to roll out a huge steaming pile of crap just to hit a delivery date.
As we close in on the finish line, we faced a huge decision on how to host this. Our current provider is pretty good, but we ran into some issues earlier this year that prompted us to look at alternatives. And we are co-hosted, which won’t work once we start loading sensitive content into a paid service. So we began the long evaluation process of picking the right architecture and host. Well, that and satisfying our paranoia regarding site security. Despite being heavy cloud folks, we eventually decided on a dedicated server model offered by a specialized hosting company. Yes, we understand that’s probably counterintuitive, so here’s why we didn’t go that way.
Co-hosting and VPS
For the most part our current site is totally fine with our current load, and our hosting provider is a lot more security-conscious than most. I launched securosis.com as a blog over at Bluehost, on a WordPress co-host. It worked totally fine, but as we started expanding it was clear that platform couldn’t meet our growing needs. We decided to switch to a better content management system (ExpressionEngine), and while we could technically run it there, we decided to go with a more specialized provider (enginehosting.com).
We have been mostly happy with the change, even though EH is considerably more expensive, because we get a lot more for what we pay. They also have excellent growth options to expand to a Virtual Private Server or even dedicated boxes if needed. But it’s still a co-host model.
The one problem we hit earlier this year appeared after a major platform upgrade. Our back end became nearly unusable due to performance problems, and when I submitted a support request they kept blaming our configuration or plugins. We are big boys, and willing to accept when we screw up. We turned our system upside down and couldn’t find anything that would kill the performance of the admin console. As it turned out we were right. Another client in our cluster over-used resources – as I had initially suggested. We were bothered by their lack of investigation, and by the (realized) potential for another customer to impact us. That convinced us we need to get off co-hosting, and into VPS or cloud. We also had to factor in all the security reasons to drop a co-hosted model once we have content we want to protect.
VPS vs. Cloud
We quickly ruled out VPS. As our knowledge and experience working with various cloud services grew, we saw no reason to pick VPS over a pure cloud model. To be honest, while I see co-hosting surviving for a while, I definitely see the allure of VPS cratering in the next few years, as customers keep comparing VPS offerings against the rapidly evolving public cloud offerings.
I decided we would go completely cloud. Aside from the lack of advantages to VPS, we were conscious of the importance of eating our own dogfood, now that we are working so deeply with the Cloud Security Alliance and advising people on cloud projects.
Our criteria for a cloud provider including a security conscious shop, judged on both what they publish and checks with various industry connections. We wanted some IPS/firewall and patch management support options to improve our baseline security and reduce our management overhead. As our IT guy, I simply don’t have the time to manage all our patches/fixes myself. If I were caught on an international flight when we needed to block and fix a critical 0day, we could be screwed. That was unacceptable.
Other factors included our plan to use a cloud-based WAF. Not that it could block everything, but the combination of blocking basic scans and providing better analytics was attractive. We also factored in performance, as we know our potential audience is self-limiting, and what we are delivering isn’t very CPU intensive. We need a little beef, and more importantly the capability to grow, but we couldn’t forsee a need for anything too crazy. It’s not like we are Netflix or anything (yet).
So there we were – I thought we were all set, until…
From Cloud to Dedicated
I wasn’t fully satisfied with the options I found (all of which cost a heck of a lot more than a basic AWS deployment), but I felt confident that we could get what we need at a reasonable price.
Then we mentioned what we were doing to a trusted friends in the industry.
For now I won’t mention who we are working with, but someone we highly respect offers dedicated hosting in a special section of a major data center they lease (their own cage). I am not sure they expected us to take them up on the offer. It’s not like they were soliciting our business – this came up over beer. These folks are as paranoid as we are (maybe more), and aside from hosting the site they will implement some stringent and unusual security controls we couldn’t possibly get anywhere else for any reasonable price. Normally they don’t use this model even with their existing clients, and we are going to be their first test case beyond internal infrastructure. As a bonus, their data center guarantees 100% infrastructure uptime. In writing. (Note: this doesn’t mean our boxes, just their network and power).
Trusted host. Extensive security. Friends and family discount.
Our costs will be 4X what we initially planned, but we are getting far more than we could anywhere else, and we would expect to pay twice this much if we could find it somewhere.
We talked about using their architecture with an IaaS provider. While they think it is technically possible, they don’t have experience with it yet.
Old-School Winner
Here’s what we finally decided and why:
- Security is really important to us. We know that our credibility will go out the window if our platform is pwned by something stupid. Not that we don’t expect issue, or have plans for how to cope. But if we are going to get nailed, we will make the attackers work for it. Call it the embarrassment factor.
- We are going with our friends, using dedicated servers in their secure co-location facility within a major data center.
- They will secure our servers and applications beyond our own knowledge and capabilities, and maintain the security over time. Later I will try and do a post on the model, after we have gone through the process and figured out what works and what doesn’t (and gotten their permission).
- We will replicate the work in the cloud as our recovery option. We have a lot of redundancy built in, but this will help both us and our provider gain experience in implementing their secure model in the cloud, and provide us some excellent resiliency. By ‘recovery’ I mean we would expect to be back up within 24 hours. RPO (Recovery Point Objective) is full functionality, and RTO (Recovery Time Objective) is 24 hours. This is in a worst-case scenario – RTOs for localized failure are shorter.
Does this make us hypocrites? Not at all. We performed a risk assessment, evaluated our options, and are going with traditional hosting first and cloud backup model. To be clear, I fully believe the cloud can be secure enough for what we will deploy, but we chose to go with the best people we could find to keep secure, and they aren’t yet comfortable deploying the specialized model we will use in the cloud. The model isn’t ‘magic’ or secret, it’s just not a route most people would be willing to take (or need).
We will eventually move completely to the cloud, but we have an amazing opportunity to work with some top-tier folks and host our application in a far more secure way than any other provider could present at any price we could afford. We are spending 4X or more than we need to for this, with our discount. So we are putting our money where our mouths are, and that meant we had to work within our provider’s constraints.
- Our top priority is security. Someone made an offer we couldn’t refuse. We’d be idiots to turn it down so we could host in the cloud – especially because we need the security more than we need the cloud advantages. We are doing this in a way that enables a cloud migration when we are all ready, with immediate resiliency benefits from the cloud.
The technical term is “slam dunk”.
Reader interactions
10 Replies to “Why We Didn’t Pick the Cloud (Mostly), and That’s Okay”
Actually, I take that back… our filter is supposed to put everything in the main queue (we changed platforms). Thus we haven’t blocked anything you put in. I see 4 comments ever submitted, all approved.
Dude- what are you talking about? I’ve approved whatever comments you sent in. If something didn’t post it means our spam filter nailed it, usually if it has a lot of links. We don’t monitor that queue since it fills so quickly…
We do not now nor have we ever, filtered for anything other than spam or personal attacks.
It’s evident that Securosis is now filtering comments based on content to their liking. It would be wise to fully disclose this to your readers since you’ve stated in the past that this was not your practice.
Well, getting people to listen to me isn’t a problem. Even Mr. Anonymous is reading 🙂
While I won’t even begin to say I agree or disagree with Anonymous, he does illustrate a big problem with IT and security these days and the level of knowledge you need in order to get some people to begin to listen to you.
It’s all about cost, both in dollars, maintenance, and people-skills to keep it all working. Not many small companies, even those staffed with technically-minded people can do things that would make security geeks satisfied. Hell, large companies with deep pockets can’t even do that.
I will say, though, that rather than throw peanuts, definitely offer positive suggestions. They don’t have to be listened-to and may be declined due to costs or being too much effort, but others may be decent suggestions.
But hey, you can offer 100 security persons (of various skills) the question of how to build a secure SMB, and you’ll get 100 different answers based on their experience and risk analyses. Have them debate amongst themselves to figure out *the* best answer, and I doubt you’ll find even minor consensus beyond the absolute generics that aren’t actionable (patch your systems, run as least priv, unique passwds everywhere…blah blah blah, like saying, “be a good person” but not offering any guidance beyond that).
Rich, please don’t feed the trolls
Heh- amusing.
Let’s see- there’s the 2 other layers of email security after using Reflexion as the front end filter. There’s the multiple boxes for the new system, much higher costs than the traditional alternatives, and, like I said, 4X the cost of the cloud hosts we were evaluating. Cost savings my ass.
And that we’re hiring people to crawl and clean our code even after selecting a CMS with a good security track record? You know, since we *know* we aren’t hip-deep with actual code on a day to day basis and thus don’t trust ourselves?
Yeah, we don’t know shit. You should stop reading us. Or go pop EasyDNS and show us how stupid we are. I *totally* know all these other 7 person companies spend *way* more on security than we do. And all those enterprises that keep calling us back year after year? Yeah, none of them have *real* enterprise experience or they’d know better than to trust us.
It’ll be a fun day when Securosis goes dark because of your awesome architecture. Sounds like it’s one DC in one cage. Hardly resilient. And anyone that guarantees you 100% network uptime is 1) lying, because 99.9999% is hard enough to provide at a reasonable cost. And we’re guessing since you went with ‘friends’ you’re actually more concerned with cost than security.
Just another one that Securosis is showing its lack of finger on the pulse. Hypocrisy in cloud, outourcing your own email ‘security’ (reflexion and easydns < LULZ), and disappearing context around Door (because, you’ve never actually dealt with it on an enterprise scale and it shows in the ‘analysis’. Patiently waiting your overview and the framework choices you’ve made. It’ll be fun to see more hypocrisy bleeding through the we’re-not-developers-or-architects-but-we-think-we-got-security-figured-out-so-we’ll-cobble-it-together-ourself.
Great post and a well reasoned and justifiable argument. Hopefully everyone understands that sometimes theres an OR and sometimes an AND in “cloud … be damned”. As with everything, and its good to see you highlighting the point here, its a risk based decision based on all possible options and eventualities. Me? I’m intrigued as to the proposition you went for and how unique it is.
Alan,
I didn’t say the cloud couldn’t deliver the level of security that we want, but that the people we want to work with don’t have the experience with it yet.
We could have asked them to build it out for the cloud but that wouldn’t have probably doubled our costs again.
The cloud can deliver the security. Our provider has a model we like, at an exceptional discount. But they haven’t worked it for the cloud yet.
It’s all a matter of timing and opportunity. If it wasn’t for that offer we would have chosen a cloud provider and moved forward… and been very secure. This is a great opportunity to take things to a higher level, but meant not hosting on cloud for now (or delaying longer than we have time for).