Business Continuity Planning

What’s Next?

By Rich

For the record, yes, those hazmat suits are really freaking hot and sweaty. I guess that's what they mean by, "vapor barrier".

No, nothing freaky is going on; that's just a picture from an old practice. And that's pretty much how I'm spending this week- training, practicing, and cleaning bathrooms. I've talked about the value of training before, and it's one reason we're constantly practicing those critical skills until they become second nature. At this point, putting on a hazmat suit (level A, B, or C) is second nature. That's the only way to survive if I ever have to wear one during a real incident. It's an opportunity I highly doubt I'll ever experience, but it's also the kind of thing you can only screw up once.

One of the classes I'm taking this week is Basic Disaster Life Support. It's a fairly new class that focuses on medical management in massive incidents from the natural (earthquakes) to the man made (blowing stuff up). The biggest lesson I'm taking away from this class isn't some specific technique for managing a specific injury but a single general principle with direct applications in the IT world-

What's next?

When donning a hazmat suit it means what's the next step? Boots, mask, hood? Then, when something fails (and it will) what do you do next? In a disaster it means what happens after you've exceeded your plans. Finished getting all those patients out of your hospital when the big storm is coming in? Great, where are you going to send them next? Oh, the ambulances. Right, um, how many of them are there? Where are they going?

When we plan for disasters that's the one question we need to ask at every step, and keep asking. Forever. We need contingency plans for our contingency plans.

It really isn't any different in IT. The parallels to the business continuity side are easy to draw. What happens when the power goes out? Okay, the generators just ran out of gas, what next? The roads are flooded so you can't get more gas, so what's next?

Same thing for security, except usually we're talking defenses. Web application firewall? Great, what happens when some bad guy gets past it or they skip it by hitting the database from a compromised internal machine? How about if they had an 0day you didn't know about and now own the machine?

And eventually you'll run out of answers, because at that point there's either nothing to do or it's time to just turn it all off, or let it burn and collect the insurance money. But through the process of constantly asking that question you'll develop a methodical, mechanical approach to solve seemingly insurmountable problems. You'll even learn that sometimes it isn't just having the right answer, but continuously moving (or appropriately pausing) that eventually gets you past those obstacles.

What's next?

Never assume.

React faster, and better.

Stay in school. Don't do drugs.

–Rich

Train Like You Fight

By Rich

Ah, Monday. And not just the usual Monday, but a Monday after a perfect 5-day trip with my wife to Sonoma. A Monday where, right after we get back, the hot water heater in our old house (that we now rent) dies. Sigh. I really don't like this whole "real world" thing.

On the plus side we set two records on our wine tour: fewest wineries visited, and most time spent at a single winery. On our second stop at a small, 300 case a year winery we ended up polishing off a few bottles with the owner (and sole operator) over nearly 5 hours, making our guide late for his dinner. It was a total blast, not pretentious at all (I'm still pretty blue collar), and the wine was excellent. It did blow our stomachs for the entire next day, but that was a cost worth paying.

One of the lasts posts before I left was about the philosophy of REACT FASTER and BETTER I partially stole from Mike Rothman. In a response, Cutaway brought up a second, no less important issue, as almost a side note. He refers back to his Marine days and the importance of keeping your head up, even when you're down in the trenches responding to something else or stuck in the routine daily grind. When teaching martial arts I refer to this as situational awareness, which is what I think the military and law enforcement also call it. Know what's going on around you, even if you're bored off your rocker with tedium.

But that's not what I want to talk about today. Early in the post, Cutaway says,

All of this got me thinking about how we react to situations as a whole. I started thinking about how through training and effort we can begin to overcome hardships. I started thinking about how diligent practice can instill good habits and create muscle memory in any individual. ... "Yes, yes," you are thinking to yourself right now. We have heard this all before. Practice makes perfect. Practice your incident response. Practice your backup procedures. Practice your disaster recovery. Practice makes perfect. Practice, Practice, Practice. Blah, blah, blah. Yes, I am tell you that. But what I want to emphasize is that you can train yourselves all day long and still make mistakes.

Yep, we're absolutely going to make mistakes, and how we respond to those mistakes is just as important, maybe more important, than minimizing them. The only way we can do this is if you "train like you fight". In training, you need to run practical scenarios that emulate, as closely as possible, the chaos of the real world.

How many of you can honestly say your incident response, disaster recovery, or business continuity tests come close to emulating the real world? It's why I despise over-reliance on tabletop tests that prove nothing. It's why I really like programs like the DefCon Capture the Flag that test real attack and defense response skills.

If you are in incident response or disaster recovery/BCP, make sure you make heavy use of scenarios and practical tests as part of your training. Make them as real as possible, and throw in the unexpected to train people on how to respond to the chaotic. Tedious, rote training builds the "muscle memory" for tasks, while scenarios build the "muscle memory" for the unknown.

–Rich