In our last post we covered organizational structure options for incident response. Aside from the right org structure and incident response process, it’s important to have a few infrastructure pieces (tools) in place, and take some preparatory steps ahead of time. As with all our recommendations in this series, remember that one size doesn’t fit all, and those of you in smaller companies will probably skip some of the tools or not need some of the prep steps.

Incident Response Support Tools

The following tools are extremely helpful (sometimes essential) for managing incidents. This isn’t a comprehensive list, but an overview of the major categories we see most often used by successful organizations:

  • Multiple secure communications channels: It’s bad to run all your incident response communications over an pwned email server, or to lose the ability to communicate if a cell tower is out. Your incident response team should have multiple communications options – landlines, mobile phones (on multiple carriers if possible), encrypted online tools (via secure systems), and possibly even portable mobile radios. For smaller organizations this might be as simple as GPG or S/MIME for encrypted email (and a list of backup email accounts on multiple providers), or a collaboration Web site and some backup cell phones.
  • Incident management system: Many organizations use their trouble ticket systems to manage incidents, or handle them manually. There are also purpose-built tools with improved security and communications options. As long as you have some central and secure place to keep track of an incident, and a backup option or two, you should be covered.
  • Analysis and forensics tools: As we will discuss later in the series, one of the most critical elements of incident response is the investigation. You need a mix of forensics tools to figure out what’s going on – including tools for analyzing network packet captures, endpoints and mobile devices, and various logs (everything from databases to network devices). This is a very broad category that depends on your skill set, the kinds of incidents you are involved with, and budget.
  • Secure test environment/lab: This is clearly more often seen in larger organizations and those with larger budgets and higher incident rates, but even in a smaller organization it is extremely helpful to have some test systems and a network or two – especially for analysis of compromised endpoints and servers.
  • Network taps and capture/analysis laptops: At some point during an investigation, you’ll likely need to physically go to a location and plug in an inline or passive network tap to analyze part of a network – sometimes even the communications from a single system. This kind of monitoring may not be possible on your existing network – not all routers let you plug in and capture local traffic (heck, you might simply be out of ports), so we recommend you have a small tap and spare laptop (with a large hard drive, possibly external) available. These are very cost-effective and useful even for smaller organizations.
  • Data collection and monitoring infrastructure: As previously discussed.

Preparatory Steps

Hopefully the idea that tools are only a small part of every security process is starting to set in. Once again, tools are a means to an end. The following steps help set up your infrastructure to support the response process and make the best use of your investment in tools. They cost little aside from time, but will determine the success and/or failure of your response efforts:

  • Define a communications plan: As we mentioned above, it’s important to have multiple communications methods. It’s even more important to have a calling list with all the various numbers, emails, and other addresses you need. Don’t forget to include key contacts outside your team – such as management, key business units, and outside resources like local law enforcement contacts (even for federal agencies) or an outside incident response firm in case something exceeds your own capabilities.
  • Establish a point of contact, promote it, and staff it: It is truly surprising how many organizations fail to provide contact options for users or other IT staff for when something goes wrong. Set up a few options, including phone/email, make sure someone is always there to respond, and promote them. Many organizations route everything through the help desk, in which case you need to educate them on how to identify a potential incident, when to escalate and how to contact you if something looks big enough that adding it to your ticket queue might be a tad too passive.
  • Liaise with key business units: Lay the groundwork for working with different business units before an incident occurs. Let them know who you are, that you are there to help, and what their responsibilities will be if they get involved in an incident (either because it’s affecting their unit, or because they are an outside resource).
  • Liaise with outside resources: If you are on a large dedicated incident response team this might mean meeting with local federal law enforcement representatives, auditors, and legal advisors. For a smaller organization it might mean researching and interviewing some response and investigation firms in case something exceeds your internal response capability and getting to know the folks so they’ll call you back when you need them. You don’t want to be calling someone cold in the middle of an incident.
  • Document your incident response plan: Even if your plan is a single page with a bullet list of steps, have it in writing ahead of time. Make sure all of the folks with responsibility to act in an incident understand what exactly they need to do. In any incident, checklists are your friends.
  • Train and practice: Ideally run some full scenarios on top of training on tools and techniques. Even if you are a single part-time responder, create some practice scenarios for yourself. And practice. And practice. And practice again. It’s a bad time to find a hole in your process, while you are responding to a real incident.

Again, we could write an entire paper on building your incident response infrastructure, but these key elements will get you on the right path.

Share: