Incident Response

My Personal Security Guiding Principles

By Rich

Fall of 2009 marks the 20th anniversary of the start of my professional security career. That was the first day someone stuck a yellow shirt on my back and sent me into a crowd of drunk college football fans at the University of Colorado (later famous for its student riots). I'm pretty sure someone screwed up, since it was my first day on the job and I was assigned a rover position -- which normally goes to someone who knows what the f&%$ they are doing, not some 18 year old, 135-lb kid right out of high school. And yes, I was breaking up fights on my first day (the stadium wasn't dry until a few years later).

If you asked me then, I never would have guessed I'd spend the next couple decades working through the security ranks, eventually letting my teenage geek/hacker side take over. Over that time I've come to rely on the following guiding principles in everything from designing my personal security to giving advice to clients:

1. Don't expect human behavior to change. Ever.
2. You cannot survive with defense alone.
3. Not all threats are equal, and all checklists are wrong.
4. You cannot eliminate all vulnerabilities.
5. You will be breached.

There's a positive side to each of these negative principles:

1. Design security controls that account for human behavior. Study cognitive science and practical psychology to support your decisions. This is also critical for gaining support for security initiatives, not just design of individual controls.
2. Engage in intelligence and counter-threat operations to the best of your ability. Once an attack has started, your first line of security has already failed.
3. Use checklists to remember the simple stuff, but any real security must be designed using a risk-based approach. As a corollary, you can't implement risk-based security if you don't really understand the risks; and most people don't understand the risks. Be the expert.
4. Adopt anti-exploitation wherever possible. Vulnerability-driven security is always behind the threat.
5. React faster and better. Incident response is more important than any other single security control.

With one final piece of advice -- keep it simple and pragmatic.

And after 20 years, that's all I've got...

–Rich

What’s Next?

By Rich

For the record, yes, those hazmat suits are really freaking hot and sweaty. I guess that's what they mean by, "vapor barrier".

No, nothing freaky is going on; that's just a picture from an old practice. And that's pretty much how I'm spending this week- training, practicing, and cleaning bathrooms. I've talked about the value of training before, and it's one reason we're constantly practicing those critical skills until they become second nature. At this point, putting on a hazmat suit (level A, B, or C) is second nature. That's the only way to survive if I ever have to wear one during a real incident. It's an opportunity I highly doubt I'll ever experience, but it's also the kind of thing you can only screw up once.

One of the classes I'm taking this week is Basic Disaster Life Support. It's a fairly new class that focuses on medical management in massive incidents from the natural (earthquakes) to the man made (blowing stuff up). The biggest lesson I'm taking away from this class isn't some specific technique for managing a specific injury but a single general principle with direct applications in the IT world-

What's next?

When donning a hazmat suit it means what's the next step? Boots, mask, hood? Then, when something fails (and it will) what do you do next? In a disaster it means what happens after you've exceeded your plans. Finished getting all those patients out of your hospital when the big storm is coming in? Great, where are you going to send them next? Oh, the ambulances. Right, um, how many of them are there? Where are they going?

When we plan for disasters that's the one question we need to ask at every step, and keep asking. Forever. We need contingency plans for our contingency plans.

It really isn't any different in IT. The parallels to the business continuity side are easy to draw. What happens when the power goes out? Okay, the generators just ran out of gas, what next? The roads are flooded so you can't get more gas, so what's next?

Same thing for security, except usually we're talking defenses. Web application firewall? Great, what happens when some bad guy gets past it or they skip it by hitting the database from a compromised internal machine? How about if they had an 0day you didn't know about and now own the machine?

And eventually you'll run out of answers, because at that point there's either nothing to do or it's time to just turn it all off, or let it burn and collect the insurance money. But through the process of constantly asking that question you'll develop a methodical, mechanical approach to solve seemingly insurmountable problems. You'll even learn that sometimes it isn't just having the right answer, but continuously moving (or appropriately pausing) that eventually gets you past those obstacles.

What's next?

Never assume.

React faster, and better.

Stay in school. Don't do drugs.

–Rich

The Best Incident Response Training You Can Buy. For Free.

By Rich

Next week I'll be out of the office on one of my occasional stints as a federal emergency responder. I haven't had the opportunity to do much since we responded to Katrina, and, to be honest, am surprised the team still lets me hang on (it's in Colorado, I'm in Arizona, and I don't get to train much anymore). Who knows how much longer I'll get to put a uniform on- the politics of domestic response are a freaking mess these days, with all the cash funding the war, and I won't be surprised if some of the more expensive (and thus capable) parts of the system are dismantled. Hopefully we can hang on through the next election.

Anyway, enough of my left wing liberal complaints about domestic security and on to incident management.

Although I haven't written much about it on the blog (just the occasional post), one area I talk a lot about is incident response and disaster management. Translating my experiences as a 9-1-1 and disaster responder into useful business principles. I'm frequently asked where people can get management level training on incident management. While SANS and others have some technology-oriented incident response courses, the best management level training out there is from FEMA.

Yes, that FEMA.

For no cost you can take some of their Incident Command Systems (ICS) courses online. I highly recommend ICS 100 and ICS 200 for anyone interested in the topic. No, not all of it will apply, but the fundamental principles are designed for ANY kind of incident of ANY scale. If nothing else, it will get you thinking.

And while I'm at it, here's a definition of "Incident" that I like to use:

An incident is any situation that exceeds normal risk management processes.

Although I've sat through a lot of the training before, I never actually went through the program and test. I'm fairly impressed- these are some of the better online courses I've seen.

–Rich

Train Like You Fight

By Rich

Ah, Monday. And not just the usual Monday, but a Monday after a perfect 5-day trip with my wife to Sonoma. A Monday where, right after we get back, the hot water heater in our old house (that we now rent) dies. Sigh. I really don't like this whole "real world" thing.

On the plus side we set two records on our wine tour: fewest wineries visited, and most time spent at a single winery. On our second stop at a small, 300 case a year winery we ended up polishing off a few bottles with the owner (and sole operator) over nearly 5 hours, making our guide late for his dinner. It was a total blast, not pretentious at all (I'm still pretty blue collar), and the wine was excellent. It did blow our stomachs for the entire next day, but that was a cost worth paying.

One of the lasts posts before I left was about the philosophy of REACT FASTER and BETTER I partially stole from Mike Rothman. In a response, Cutaway brought up a second, no less important issue, as almost a side note. He refers back to his Marine days and the importance of keeping your head up, even when you're down in the trenches responding to something else or stuck in the routine daily grind. When teaching martial arts I refer to this as situational awareness, which is what I think the military and law enforcement also call it. Know what's going on around you, even if you're bored off your rocker with tedium.

But that's not what I want to talk about today. Early in the post, Cutaway says,

All of this got me thinking about how we react to situations as a whole. I started thinking about how through training and effort we can begin to overcome hardships. I started thinking about how diligent practice can instill good habits and create muscle memory in any individual. ... "Yes, yes," you are thinking to yourself right now. We have heard this all before. Practice makes perfect. Practice your incident response. Practice your backup procedures. Practice your disaster recovery. Practice makes perfect. Practice, Practice, Practice. Blah, blah, blah. Yes, I am tell you that. But what I want to emphasize is that you can train yourselves all day long and still make mistakes.

Yep, we're absolutely going to make mistakes, and how we respond to those mistakes is just as important, maybe more important, than minimizing them. The only way we can do this is if you "train like you fight". In training, you need to run practical scenarios that emulate, as closely as possible, the chaos of the real world.

How many of you can honestly say your incident response, disaster recovery, or business continuity tests come close to emulating the real world? It's why I despise over-reliance on tabletop tests that prove nothing. It's why I really like programs like the DefCon Capture the Flag that test real attack and defense response skills.

If you are in incident response or disaster recovery/BCP, make sure you make heavy use of scenarios and practical tests as part of your training. Make them as real as possible, and throw in the unexpected to train people on how to respond to the chaotic. Tedious, rote training builds the "muscle memory" for tasks, while scenarios build the "muscle memory" for the unknown.

–Rich

React Faster, And Better, With The A B Cs

By Rich

I've had a bit of a weird week. As I mentioned on Monday, I was driving to physical therapy (physio for my Australian and European friends) when there was an accident in front of me and I stopped to help out. Wednesday night I was coming home from PT and there was another accident right as I was going through the intersection.

This one was far more serious. As soon as I heard the smash and saw the impact out of the corner of my eye, I pulled into the median, hit my hazard lights, and called 9-1-1. One of the advantages of working in the field for so long is that you learn an economy of words to describe a complex situation in just a sentence or two of the crucial information. My first call was:

I'm on-scene of an injury accident at the corner of [x and y]. Two vehicles, with an unconscious unresponsive patient with a compromised airway. Patient is entrapped in the passenger side of the vehicle with access through the driver's side door. I'm a former paramedic and need to go manage her airway

There was a bit more jargon, but not much. The patient was unrestrained in the car with the airbag deployed, which probably meant she hit her head on the passenger window or strut since it was a side impact. There were a bunch of other bystanders and one came out and identified himself as a flight nurse. Her head was slumped over, which caused her difficulty breathing. The nurse jumped in the back of the car, we tilted her head to a normal position and stabilized her neck (one of the few times you're allowed to move the neck after an accident). Her breathing got better, and she slowly started waking up, but clearly had a head injury, which we reported to 9-1-1. The fire department showed up a few minutes later, we got out of the way, and she was being loaded into the chopper as I drove off.

That might be one of the only times I've stopped to help at an accident where my assistance may have mattered. Truth is, unless you're on the ambulance or have advanced equipment with you, the most useful thing you can do is calm the patient and make sure there isn't any more damage. The kinds of injuries you sustain in a major accident are rarely something even a highly trained bystander can help with. I didn't even bother evaluating anything more than her breathing, since nothing else mattered. All you EMTs can skip that full survey if you're helping as a bystander in an urban area.

In this case her head position was keeping her from breathing well, making the situation worse. Just moving it so she could breathe more normally might have oxygenated her noggin a bit more and helped her wake up.

Why the heck am I talking about this on a security geek blog?

Because it's one of those times where there are direct lessons we can apply to our world, and often forget.

I'm a big fan of Rothman's philosophy of REACT FASTER. The idea is that it's more about how you respond to an incident than having the incident in the first place. Truth is in IT, as in life, bad stuff will happen no matter what you do. Systems will crash, hard drives will die, and hackers will break in. David Mortman is one of the other major proponents of this philosophy- incident response is just as important, if not more important, than incident prevention. That's why I'm adding REACT BETTER.

Emergency services are just like programming- a series of algorithms in a structured program flow. It all comes down to the A B Cs- Airway, Breathing, Circulation- in meat-space. Patient have any airway? Nope? Then nothing else matters until you fix that. Breathing? Check. Circulation okay? Then move on to spinal immobilization. It's a recognition that you can't jump from A to C and expect success. It's exactly what we did to help that girl in the car, rather than focusing on the blood or other distractions.

Don't just react- have a response plan with specific steps you don't jump over until they're complete. Take the most critical thing first, fix it, move to the next, and so on until you're done. Evaluate, prioritize, contain, fix, and clean. (You OODA fans should love this).

And always remember the loudest patient is rarely the most important. If they're screaming their head off, their airway is fine. It's the quiet ones you have to watch out for.

–Rich