An interesting discussion popped up on Slashdot this Saturday afternoon about Preventing My Hosting Provider From Rooting My Server. ‘hacker’ is claiming that when he accuses his hosting provider of service interruption, they assume root access on his machines without permission.
“I have a heavily-hit public server (web, mail, cvs/svn/git, dns, etc.) that runs a few dozen OSS project websites, as well as my own personal sites (gallery, blog, etc.). From time to time, the server has ‘unexpected’ outages, which I’ve determined to be the result of hardware, network and other issues on behalf of the provider. I run a lot of monitoring and logging on the server-side, so I see and graph every single bit and byte in and out of the server and applications, so I know it’s not the OS itself. When I file ‘WTF?’-style support tickets to the provider through their web-based ticketing system, I often get the response of: ‘Please provide us with the root password to your server so we can analyze your logs for the cause of the outage.’ Moments ago, there were three simultaneous outages while I was logged into the server working on some projects. Server-side, everything was fine. They asked me for the root password, which I flatly denied (as I always do), and then they rooted the server anyway, bringing it down and poking around through my logs. This is at least the third time they’ve done this without my approval or consent. Is it possible to create a minimal Linux boot that will allow me to reboot the server remotely, come back up with basic networking and ssh, and then from there, allow me to log in and mount the other application and data partitions under dm-crypt/loop-aes and friends?”
Ignoring for a moment the basic problem of requesting assistance while not wishing to provide access, how do you protect the servers from remote hosting administrators? If someone else has physical access to your machine, even if you machine is virtual, a skilled attacker will gain access to your data regardless. It’s not clear if the physical machine is owned by ‘hacker’ or if it is just leased server capacity, but it seems to me that if you want to keep remote administrators of average skill from rooting your server and then rummaging around in your files, disk encryption would be an effective choice. You have the issue of needing to supply credentials remotely upon reboot, but this would be effective in protecting log data. If you need better security, place the server under your physical control, or all bets are off.
Reader interactions
6 Replies to “Hosting Providers and Log Security”
Oh come on:
“It takes the logs and copies them to a user directory that the hosting provider can access via an unprivileged user account else it gets the hose again.”
If you want to get cool points, use rsync to drop them someplace on a hosting provider machine with a cron job to update the ISP.
I think the poster is playing “penis envy games” with his ISP and needs to just leave for greener pastures. This is why we can’t have nice things–kids nowadays just don’t understand how to do this stuff.
I also think the problem is that Mr System Administrator didn’t tune his apache correctly–that will overspawn http daemons and drag down your stuff pretty rapidly as the box starts thrashing.
Adrian,
I think you and Alan have mostly resolved this question.
As said, if the goal is just keeping the hosting or service provider out, the answer is contractual, and immutable audit logs that provide forensically defensible evidence can certainly make sure that agreements are kept. 🙂
By extension, this problem forms the crux of cloud security, and right now I believe the consensus is that encryption is not cost effective enough yet to avoid erasing the economic advantages of going to the cloud.
Adrian, first of all I used the wrong to (too) in my first comment, not sure it can be corrected. So the difference between the old power, ping and pipe co-lo model versus some of the newer models is more of an element of managed service? I think by definition the managed services are going to contractually and actually require a higher level of access. The other factor is the rise of virtual environments. This opens a whole new level of access I think. Especially when you get virtuals within virtuals and stuff like that.
Again not sure technology is the answer though. You can build a better mouse trap but some mice will just get smarter. If the goal is keeping the hosting or service provider out, the answer is contractual. If the goal is overall security of the hosted application or server, then technology is the way to go and I agree that encryption is a great start!
Alan – well stated. I think you are right … for most companies, a clear contractual description of when access is appropriate solves this issue. But a subtle change in mind set I am starting to see with small shop IT admins and developers, now that their server is no longer under their control, is how to secure it. Seemed that the feeling was when the machine was physically under their control it was safer, even when that means it was in a locked cage at a co-lo facility. And many of the co-lo’s offered some monitoring and basic protections that is not there with many new providers. From the comments I am seeing on web developer blogs, these old questions are popping up with renewed interest.
-Adrian
Adrian, this is a contractual issue that you are in search of a technological solution
tooto. The real issue is does the hosting provider have the authority to root into the machine without the explicit approval of the hosting customer? If this is specifically allowed under the T&Cs; of the hosting contract any technological barriers would put the customer in default of the terms of his contract. If it is not specifically allowed, I would think the hosting provider is violating their terms of use. I strong letter threatening legal action the next time they root in without permission should take care of that.Also what about just giving them the logs instead of them just going in and taking it?
Happy Holidays to the securosis gang!
there is always the option of encrypting the data on disk. that doesn’t solve the remote root access problem, but at least you can protect your data and data integrity. i guess an alternative is to virtualize your ‘virtual machine. ie: use your provider’s pre-installed OS as the host for your own virtual instance.