For the record, yes, those hazmat suits are really freaking hot and sweaty. I guess that’s what they mean by, “vapor barrier”.Random 021_2.jpg

No, nothing freaky is going on; that’s just a picture from an old practice. And that’s pretty much how I’m spending this week- training, practicing, and cleaning bathrooms. I’ve talked about the value of training before, and it’s one reason we’re constantly practicing those critical skills until they become second nature. At this point, putting on a hazmat suit (level A, B, or C) is second nature. That’s the only way to survive if I ever have to wear one during a real incident. It’s an opportunity I highly doubt I’ll ever experience, but it’s also the kind of thing you can only screw up once.

One of the classes I’m taking this week is Basic Disaster Life Support. It’s a fairly new class that focuses on medical management in massive incidents from the natural (earthquakes) to the man made (blowing stuff up). The biggest lesson I’m taking away from this class isn’t some specific technique for managing a specific injury but a single general principle with direct applications in the IT world-

What’s next?

When donning a hazmat suit it means what’s the next step? Boots, mask, hood? Then, when something fails (and it will) what do you do next? In a disaster it means what happens after you’ve exceeded your plans. Finished getting all those patients out of your hospital when the big storm is coming in? Great, where are you going to send them next? Oh, the ambulances. Right, um, how many of them are there? Where are they going?

When we plan for disasters that’s the one question we need to ask at every step, and keep asking. Forever. We need contingency plans for our contingency plans.

It really isn’t any different in IT. The parallels to the business continuity side are easy to draw. What happens when the power goes out? Okay, the generators just ran out of gas, what next? The roads are flooded so you can’t get more gas, so what’s next?

Same thing for security, except usually we’re talking defenses. Web application firewall? Great, what happens when some bad guy gets past it or they skip it by hitting the database from a compromised internal machine? How about if they had an 0day you didn’t know about and now own the machine?

And eventually you’ll run out of answers, because at that point there’s either nothing to do or it’s time to just turn it all off, or let it burn and collect the insurance money. But through the process of constantly asking that question you’ll develop a methodical, mechanical approach to solve seemingly insurmountable problems. You’ll even learn that sometimes it isn’t just having the right answer, but continuously moving (or appropriately pausing) that eventually gets you past those obstacles.

What’s next?

Never assume.

React faster, and better.

Stay in school. Don’t do drugs.