Incident Management

What’s Next?

By Rich

For the record, yes, those hazmat suits are really freaking hot and sweaty. I guess that's what they mean by, "vapor barrier".

No, nothing freaky is going on; that's just a picture from an old practice. And that's pretty much how I'm spending this week- training, practicing, and cleaning bathrooms. I've talked about the value of training before, and it's one reason we're constantly practicing those critical skills until they become second nature. At this point, putting on a hazmat suit (level A, B, or C) is second nature. That's the only way to survive if I ever have to wear one during a real incident. It's an opportunity I highly doubt I'll ever experience, but it's also the kind of thing you can only screw up once.

One of the classes I'm taking this week is Basic Disaster Life Support. It's a fairly new class that focuses on medical management in massive incidents from the natural (earthquakes) to the man made (blowing stuff up). The biggest lesson I'm taking away from this class isn't some specific technique for managing a specific injury but a single general principle with direct applications in the IT world-

What's next?

When donning a hazmat suit it means what's the next step? Boots, mask, hood? Then, when something fails (and it will) what do you do next? In a disaster it means what happens after you've exceeded your plans. Finished getting all those patients out of your hospital when the big storm is coming in? Great, where are you going to send them next? Oh, the ambulances. Right, um, how many of them are there? Where are they going?

When we plan for disasters that's the one question we need to ask at every step, and keep asking. Forever. We need contingency plans for our contingency plans.

It really isn't any different in IT. The parallels to the business continuity side are easy to draw. What happens when the power goes out? Okay, the generators just ran out of gas, what next? The roads are flooded so you can't get more gas, so what's next?

Same thing for security, except usually we're talking defenses. Web application firewall? Great, what happens when some bad guy gets past it or they skip it by hitting the database from a compromised internal machine? How about if they had an 0day you didn't know about and now own the machine?

And eventually you'll run out of answers, because at that point there's either nothing to do or it's time to just turn it all off, or let it burn and collect the insurance money. But through the process of constantly asking that question you'll develop a methodical, mechanical approach to solve seemingly insurmountable problems. You'll even learn that sometimes it isn't just having the right answer, but continuously moving (or appropriately pausing) that eventually gets you past those obstacles.

What's next?

Never assume.

React faster, and better.

Stay in school. Don't do drugs.

–Rich

The Best Incident Response Training You Can Buy. For Free.

By Rich

Next week I'll be out of the office on one of my occasional stints as a federal emergency responder. I haven't had the opportunity to do much since we responded to Katrina, and, to be honest, am surprised the team still lets me hang on (it's in Colorado, I'm in Arizona, and I don't get to train much anymore). Who knows how much longer I'll get to put a uniform on- the politics of domestic response are a freaking mess these days, with all the cash funding the war, and I won't be surprised if some of the more expensive (and thus capable) parts of the system are dismantled. Hopefully we can hang on through the next election.

Anyway, enough of my left wing liberal complaints about domestic security and on to incident management.

Although I haven't written much about it on the blog (just the occasional post), one area I talk a lot about is incident response and disaster management. Translating my experiences as a 9-1-1 and disaster responder into useful business principles. I'm frequently asked where people can get management level training on incident management. While SANS and others have some technology-oriented incident response courses, the best management level training out there is from FEMA.

Yes, that FEMA.

For no cost you can take some of their Incident Command Systems (ICS) courses online. I highly recommend ICS 100 and ICS 200 for anyone interested in the topic. No, not all of it will apply, but the fundamental principles are designed for ANY kind of incident of ANY scale. If nothing else, it will get you thinking.

And while I'm at it, here's a definition of "Incident" that I like to use:

An incident is any situation that exceeds normal risk management processes.

Although I've sat through a lot of the training before, I never actually went through the program and test. I'm fairly impressed- these are some of the better online courses I've seen.

–Rich