It’s no secret that we are currently working on a new software platform to deliver actionable security research to a broader market, engage folks, and… umm… feed our families. As you might expect, like any software project, it’s running about 30% late and 70% over budget. I just can’t seem to stop making our developers find exactly the right imagery and user experience to best represent the Securosis brand. Mike has coined a new term, ‘analness’, to describe the gyrations we’ve gone through, but I’m okay with that because we have spent years building our reputation and aren’t about to roll out a huge steaming pile of crap just to hit a delivery date.
As we close in on the finish line, we faced a huge decision on how to host this. Our current provider is pretty good, but we ran into some issues earlier this year that prompted us to look at alternatives. And we are co-hosted, which won’t work once we start loading sensitive content into a paid service. So we began the long evaluation process of picking the right architecture and host. Well, that and satisfying our paranoia regarding site security. Despite being heavy cloud folks, we eventually decided on a dedicated server model offered by a specialized hosting company. Yes, we understand that’s probably counterintuitive, so here’s why we didn’t go that way.
Co-hosting and VPS
For the most part our current site is totally fine with our current load, and our hosting provider is a lot more security-conscious than most. I launched
securosis.com as a blog over at Bluehost, on a WordPress co-host. It worked totally fine, but as we started expanding it was clear that platform couldn’t meet our growing needs. We decided to switch to a better content management system (ExpressionEngine), and while we could technically run it there, we decided to go with a more specialized provider (
We have been mostly happy with the change, even though EH is considerably more expensive, because we get a lot more for what we pay. They also have excellent growth options to expand to a Virtual Private Server or even dedicated boxes if needed. But it’s still a co-host model.
The one problem we hit earlier this year appeared after a major platform upgrade. Our back end became nearly unusable due to performance problems, and when I submitted a support request they kept blaming our configuration or plugins. We are big boys, and willing to accept when we screw up. We turned our system upside down and couldn’t find anything that would kill the performance of the admin console. As it turned out we were right. Another client in our cluster over-used resources – as I had initially suggested. We were bothered by their lack of investigation, and by the (realized) potential for another customer to impact us. That convinced us we need to get off co-hosting, and into VPS or cloud. We also had to factor in all the security reasons to drop a co-hosted model once we have content we want to protect.
VPS vs. Cloud
We quickly ruled out VPS. As our knowledge and experience working with various cloud services grew, we saw no reason to pick VPS over a pure cloud model. To be honest, while I see co-hosting surviving for a while, I definitely see the allure of VPS cratering in the next few years, as customers keep comparing VPS offerings against the rapidly evolving public cloud offerings.
I decided we would go completely cloud. Aside from the lack of advantages to VPS, we were conscious of the importance of eating our own dogfood, now that we are working so deeply with the Cloud Security Alliance and advising people on cloud projects.
Our criteria for a cloud provider including a security conscious shop, judged on both what they publish and checks with various industry connections. We wanted some IPS/firewall and patch management support options to improve our baseline security and reduce our management overhead. As our IT guy, I simply don’t have the time to manage all our patches/fixes myself. If I were caught on an international flight when we needed to block and fix a critical 0day, we could be screwed. That was unacceptable.
Other factors included our plan to use a cloud-based WAF. Not that it could block everything, but the combination of blocking basic scans and providing better analytics was attractive. We also factored in performance, as we know our potential audience is self-limiting, and what we are delivering isn’t very CPU intensive. We need a little beef, and more importantly the capability to grow, but we couldn’t forsee a need for anything too crazy. It’s not like we are Netflix or anything (yet).
So there we were – I thought we were all set, until…
From Cloud to Dedicated
I wasn’t fully satisfied with the options I found (all of which cost a heck of a lot more than a basic AWS deployment), but I felt confident that we could get what we need at a reasonable price.
Then we mentioned what we were doing to a trusted friends in the industry.
For now I won’t mention who we are working with, but someone we highly respect offers dedicated hosting in a special section of a major data center they lease (their own cage). I am not sure they expected us to take them up on the offer. It’s not like they were soliciting our business – this came up over beer. These folks are as paranoid as we are (maybe more), and aside from hosting the site they will implement some stringent and unusual security controls we couldn’t possibly get anywhere else for any reasonable price. Normally they don’t use this model even with their existing clients, and we are going to be their first test case beyond internal infrastructure. As a bonus, their data center guarantees 100% infrastructure uptime. In writing. (Note: this doesn’t mean our boxes, just their network and power).
Trusted host. Extensive security. Friends and family discount.
Our costs will be 4X what we initially planned, but we are getting far more than we could anywhere else, and we would expect to pay twice this much if we could find it somewhere.
We talked about using their architecture with an IaaS provider. While they think it is technically possible, they don’t have experience with it yet.
Here’s what we finally decided and why:
- Security is really important to us. We know that our credibility will go out the window if our platform is pwned by something stupid. Not that we don’t expect issue, or have plans for how to cope. But if we are going to get nailed, we will make the attackers work for it. Call it the embarrassment factor.
- We are going with our friends, using dedicated servers in their secure co-location facility within a major data center.
- They will secure our servers and applications beyond our own knowledge and capabilities, and maintain the security over time. Later I will try and do a post on the model, after we have gone through the process and figured out what works and what doesn’t (and gotten their permission).
- We will replicate the work in the cloud as our recovery option. We have a lot of redundancy built in, but this will help both us and our provider gain experience in implementing their secure model in the cloud, and provide us some excellent resiliency. By ‘recovery’ I mean we would expect to be back up within 24 hours. RPO (Recovery Point Objective) is full functionality, and RTO (Recovery Time Objective) is 24 hours. This is in a worst-case scenario – RTOs for localized failure are shorter.
Does this make us hypocrites? Not at all. We performed a risk assessment, evaluated our options, and are going with traditional hosting first and cloud backup model. To be clear, I fully believe the cloud can be secure enough for what we will deploy, but we chose to go with the best people we could find to keep secure, and they aren’t yet comfortable deploying the specialized model we will use in the cloud. The model isn’t ‘magic’ or secret, it’s just not a route most people would be willing to take (or need).
We will eventually move completely to the cloud, but we have an amazing opportunity to work with some top-tier folks and host our application in a far more secure way than any other provider could present at any price we could afford. We are spending 4X or more than we need to for this, with our discount. So we are putting our money where our mouths are, and that meant we had to work within our provider’s constraints.
- Our top priority is security. Someone made an offer we couldn’t refuse. We’d be idiots to turn it down so we could host in the cloud – especially because we need the security more than we need the cloud advantages. We are doing this in a way that enables a cloud migration when we are all ready, with immediate resiliency benefits from the cloud.
The technical term is “slam dunk”.