I remember sitting at lunch with a friend and well-respected member of our security community as I described the architecture we used to protect our mail server. I’m not saying it’s perfect, but this person responded with, “that’s insane – I know people selling 0-days to governments that don’t go that far”. On another occasion I was talking with someone with vastly more network security knowledge and experience than me; someone who once protected a site attacked daily by very knowledgeable predators, and he was… confused as to why I architected the systems like I did.

Yesterday, it saved my ass.

Now I’m not 100% happy with our current security model on mail. There are aspects that are potentially flawed, but I’ve done my best to reduce the risk while still maintaining the usability we want. I actually have plans to close the last couple holes I’m not totally comfortable with, but our risk is still relatively low even with them.

Here’s what happened.

Back when we first set up our mail infrastructure we hit a problem with our VPN connections. Our mail is on a fully segregated network, and we had some problems with our ISP and IPSec-based VPNs even though I tried multiple back-end options.

Timing-wise we hit a point where I had to move forward, so I set up PPTP and mandated strong passwords (as in I reviewed or set all of them myself). Only a handful of people have VPN access anyway, and, at the time, a properly-constructed PPTP password was still very secure.

That delusion started dying this summer, and was fully buried yesterday thanks to a new, cloud-based, MS-CHAP cracking tool released by Moxie Marlinspike and David Hulton.

The second I saw that article I shut down the VPN. But here’s how my paranoia saved my ass.

As a fan of hyper-segregation, early on I decided to never trust the VPN. I put additional security hardware behind the VPN, with extremely restrictive policies. Connecting via the VPN gave you very little access to anything, with the mail server still completely walled off (a UTM dedicated only to the mail server). Heck, the only two things you could try to hack behind the VPN were the VPN server itself and the UTM… nothing else is directly connected to that network, and all that traffic is monitored and filtered.

When I initially set things up people questioned my choice to put one security appliance behind another like that. But I didn’t want to have to rely on host security for the key assets if someone compromised anyone connected to our VPN.

In this case, it worked out for me. Now I set all this up pre-cloud, but you can set up a similar architecture in many VPC or private cloud environments (you need two virtual NICs and a virtual UTM, although even simple firewall rules can go a long way to help).

This is also motivation to finish the next part of my project, which involves yet another UTM and server.