Incident Response Fundamentals: Roles and Organizational Structure

In our last post we introduced some of the key principles of incident response. Today we will focus on the major roles and organizational structure. Organizational Structure As we return to our IT security focus, the incident response organization consists of two major kinds of resources: those dedicated completely to response, and those with other primary functions who get pulled into incidents as needed depending on the scope or nature. For example, the legal team isn’t necessarily involved in every incident, but clearly plays an important role in anything with legal or regulatory consequences. Also, a smaller organization might have no dedicated resources, while a larger one may have a full time team with defined roles, which deals with multiple overlapping incidents. That’s okay because the structure and system can expand and contract as needed if you follow the ICS principles. Resources These individuals and roles may not spend all their time on incident response, but are the key roles to fill when an incident occurs. One person can fill multiple roles, especially for a smaller incident or organization, but only if they have the right skill set. Team Lead/Incident Commander: The person with overall responsibility and accountability for the direct management of incidents. Typically reports to the CISO, CIO, or even CEO, but following unity of command, should definitely only be accountable to a single manager. When an incident triggers, the first person to respond is the incident lead until they hand off responsibility to someone of equal or higher authority. That way someone is always in charge, even if only for the first few minutes. Command is then handed off to higher and higher levels as needed. When you have a full-time team, the team lead/manager is also responsible for ongoing training, program development, and so on. Network Analysts: Experts in analyzing network packets/traffic, including forensic captures. Analysis includes ongoing monitoring, as well as deeper investigation during incidents. Systems Analysts: Experts in analyzing endpoints and servers. Forensics Analysts: Often a subspecialty of systems analyst, these individuals have deeper training in forensic investigation – which includes both the technical skills for the forensics examination of a system, and the legal training to properly handle evidence if there may be legal considerations (keep in mind that merely firing someone may lead to civil legal action). SIEM/Log Management Analysts: Individuals experienced in monitoring SIEM output and log analysis. Network, Systems, Database, and Application Administrators: Those individuals responsible for the maintenance of systems and networks. It is their responsibility to implement defensive mitigations during and after an incident, and to clean up affected systems. A firewall/IPS administrator might be responsible for closing the entry or egress points being used by the attacker. Systems administrators might roll out patches or configuration changes to host firewalls. A DBA might change account permissions or close out connection methods. This is a rather large bucket, and in most organizations these people operate at the direction of dedicated incident responders or other members of the security team. Legal, Human Resources, and Risk: Any time an incident might involve legal action, employees, or a material costs, you should involve any required combination of these business units. Communications/PR: If an incident has public impact, such as breach notifications, it’s critical to involve those responsible for organizational communications. Accounting/Finance: Incident response costs money. It’s important to include the bean counters early, even if only to pay for the pizza and Red Bull. They can also take responsibility for tracking ongoing incident costs so those of you responsible for stopping and cleaning the problem don’t have to spend your time spinning accounting spreadsheets. Logistics: This role can be a bit nebulous, but includes those responsible for getting the things you need during an incident. It may be someone from finance, the purchasing team, or the security team. Basically it’s someone with a credit card and the authority to use it. They keep people fed, purchase needed hardware and software, and hire outside experts. Communications: Those responsible for making sure responders (and management) can communicate. You might only need this role in a big incident, but make sure you identify people ahead of time who can keep you talking – via cell phones, landlines, email, IM, or whatever other mechanism isn’t totally pwned. Executive Management: We list them last, but they are ultimately responsible for everything in the organization – including incidents. Except in the organization’s very largest incidents, they probably won’t be involved directly. Yes, that is a large number of potential roles, but remember that not all are needed for every incident, and the same person might fill multiple roles based on organization and incident size. For example, in a small or mid-sized organization it isn’t unusual to have the team lead also be the network and systems analyst, and possibly also responsible for cleaning systems or reconfiguring the firewall. In terms of structure, here is one approach:   Finally, don’t forget our key concepts for the organizational structure: People should only report to a single manager. Any manager should only command 3 to 7 other people, ideally 5. The organizational structure fills in resources as needed. You don’t need everything, and what you do need you don’t need all the time. This is a scaffold to build on, not a permanent building. Share:

Read Post

The Thing about Espionage

Imagine you’re a young, skilled techie just starting your career. Maybe you’re fresh out of school, or still in an internship program. Or maybe you’ve been out of school for a few years, working your way up through various companies in the industry. You came from a normal background – possibly you thought about the military at some point, but the allure of working in technology drew you into the private sector. Your skills are solid, you produce at work, and you don’t get into any trouble beyond the usual for your age. Then one day you’re contacted by someone in the government who was sent your way by a buddy from school, or maybe an old professor. They need someone with your skills to help them out with a project. Perhaps it’s to join their agency directly. Or maybe they merely ask you to take a look at something for them – sort of steering you toward a bit of a grey area you wouldn’t normally explore because you don’t want to get in trouble. They tell you it’s a matter of national security, and this is finally your opportunity to give back to your country without having to get shot at. Heck, maybe you spent time in the military and this is a great opportunity to continue your service on a volunteer basis without getting stuck with crappy military pay and travel/deployment requirements. Perhaps you already work for a foreign company your government friends are worried may be a risk to national security. All they want is for you to provide a little information, or maybe plug a USB drive into a system in the office for a few minutes. Or maybe you’ve been working for them on some projects for a while, even if they don’t really pay you and merely “suggest” things for you to look at. You’ve done a good job and they ask you to apply for some work or study abroad in another country. Or for a foreign company in your country. Either way, all they’re asking for is you to further your education and career, maybe helping your country out a little along the way. Ethically this is no different than joining the military, an intelligence agency, or working for a private contractor or university on government projects. You are serving your country while advancing you career – pretty much the best of both worlds. You can’t talk much, if at all, about it with your friends and family, but you sleep at night with the satisfaction that you’re able to blend the needs of your nation with your own personal development goals. Did I mention you grew up outside Shanghai? The thing about espionage is that there are no good guys or bad guys. Merely patriotic individuals living in different places who believe, with complete conviction, that they are doing the right thing and serving the public good. Share:

Read Post

Friday Summary: October 29, 2010

What a wild few weeks. Talk about been there, done that, got the t-shirt. It all started October 9th, when I finally achieved a goal I’ve been chasing for well over a decade, and completed my first Olympic-distance triathlon. (1.5K swim, 40K bike, 10K run – those are distances, not dollar values). I first learned about triathlon when I was working as a medic for a race in Boulder – probably back in 1992. Being the young, aggressive type, I thought any sport where you write your number on your arms and legs in permanent ink had to be hard core. I spent most of those years competing in a sport where you hit people in the face a lot (I guess that’s kind of hard core too), but in the late 90’s I started traveling a lot for work, which made staying competitive at the level I was at pretty much impossible. Getting frustrated by not being able to make it to the next level (I was competing nationally, but only winning locally), and spending a lot of time injured due to overtraining, I decided to give tri a shot. At least I could run, and often swim or bike, when on the road. But then I got sick… really sick. As in people started calling me “liver boy” because some virus attacked my third favorite part of my body and I couldn’t drink for over a year, never mind sustain hard workouts. But I recovered, started working with a swim coach, and then got distracted by getting married and traveling even more. And then I tore my rotator cuff and had surgery. And then had a kid. And… you get the idea. About 4-5 months ago I was finally injury-free and working out regularly again, and decided to give it another shot. Started riding with a bike group and then joined a masters swim program. I figured another 3 months of training and I’d be ready, but my swim coach pushed me to race and I gave it a shot. I may have finished near the back, but I finished. Easily. And now I’m hooked. Next up is a marathon, and maybe a half-Ironman in a year or so. Then back to the booze. The day after the tri I boarded a plane and headed off to London for RSA Europe. Chris Hoff and I spent a bunch of (platonic) private time together, and it turns out we’ve been working on some extremely complementary research that we’re going to combine for our joint RSA presentation this year. I was also really happy my work passed the sniff test, because Chris spends a heck of a lot more time on cloud than I ever will, and if the research holds up for him I know it’s solid. Then back home for 3 days, and back on a plane to China. I was again presenting with Hoff, and we managed to sneak out for a few hours to visit the Forbidden City. Which is quite welcoming, if you buy a ticket. They have beer. All reds for some reason. On a sour note, the day before the tri I got word that a very good friend died of cancer. Jim launched my technology career and changed the course of my life in ways that are hard to describe. A little over a year ago we started on some collaborative smart grid research, soon after which he found out about the cancer he never recovered from. Jim deserved better. On to the Summary: Webcasts, Podcasts, Outside Writing, and Conferences Rich in China. In Chinese. Mike quoted in Dark Reading on SIEM and cloud. Dave Lews and David Mortman get a mention in an article on SecTor. Rothman again, this time on consolidation. Rich talks about China and Europe on The Network Security Podcast Favorite Securosis Posts Mike Rothman: The Thing about Espionage. Clearly a fine line between good and bad. But I do think there is right and wrong. And regardless of how you slice it, if it’s called espionage, it’s probably wrong. Adrian Lane: React Faster and Better: Incident Command Principles. Rich: Can we ever break IT?. (We’re light on posts this week, so we’ll leave it at that.) Other Securosis Posts React Faster and Better: Roles and Organizational Structure. SunSec Rises on November 3rd. Incite 10/27/2010: Traffic Ahead. NSO Quant: The Report and Metrics Model. Everything You Ever Wanted to Know about DLP. Favorite Outside Posts Adrian Lane: Robert Graham’s FireSheep analysis. Mike Rothman: Cloud Creates SIEM Blind Spot. Keep in mind the cloud changes the rules for how we do things like monitoring. And I’m quoted. Enjoy the gratuitous pat on my own back… Chris Pepper: iPhone Jailbreak Tool Sets Stage for Mobile Malware. Eric Monti demonstrates that “jailbreak” = “remote root exploit”. Gunnar Peterson: Paypal enables billing and payments on Azure cloud. Project Quant Posts NSO Quant: Index of Posts. NSO Quant: Health Metrics – Device Health. NSO Quant: Manage Metrics – Monitor Issues/Tune IDS/IPS. Research Reports and Presentations Network Security Operations Quant Metrics Model. Network Security Operations Quant Report. Understanding and Selecting a DLP Solution, v2.0. White Paper: Understanding and Selecting an Enterprise Firewall. Understanding and Selecting a Tokenization Solution. Top News and Posts Koobface Worm Targets Java. NSA Declassified Documents. Interesting stuff. Adobe Flash Bug. Perhaps we should leave a permanent reference in the Friday summary for Flash vulnerabilities and just update the link du jour. Idiocy tool. Just to remind people they are insecure. Firesheep launched. Critical Firefox Bug. LinkedIn Drive-by Malware Attack. 19 Arrested in Zeus Malware Bank Heists. Oracle claims Google directly copied Java code. Silver Tail Systems gets In-Q-Tel funding. Banks weak against skimming attacks. PCI Council releases a “sort of” update. Blog Comment of the Week Remember, for every comment selected, Securosis makes a $25 donation to Hackers for Charity. This week’s best comment goes to Mike Fratto, in response to The Thing About Espionage. Rich, based on your definition, the good guys are us and the bad guys are them for any definition of “us”

Read Post

Totally Transparent Research is the embodiment of how we work at Securosis. It’s our core operating philosophy, our research policy, and a specific process. We initially developed it to help maintain objectivity while producing licensed research, but its benefits extend to all aspects of our business.

Going beyond Open Source Research, and a far cry from the traditional syndicated research model, we think it’s the best way to produce independent, objective, quality research.

Here’s how it works:

  • Content is developed ‘live’ on the blog. Primary research is generally released in pieces, as a series of posts, so we can digest and integrate feedback, making the end results much stronger than traditional “ivory tower” research.
  • Comments are enabled for posts. All comments are kept except for spam, personal insults of a clearly inflammatory nature, and completely off-topic content that distracts from the discussion. We welcome comments critical of the work, even if somewhat insulting to the authors. Really.
  • Anyone can comment, and no registration is required. Vendors or consultants with a relevant product or offering must properly identify themselves. While their comments won’t be deleted, the writer/moderator will “call out”, identify, and possibly ridicule vendors who fail to do so.
  • Vendors considering licensing the content are welcome to provide feedback, but it must be posted in the comments - just like everyone else. There is no back channel influence on the research findings or posts.
    Analysts must reply to comments and defend the research position, or agree to modify the content.
  • At the end of the post series, the analyst compiles the posts into a paper, presentation, or other delivery vehicle. Public comments/input factors into the research, where appropriate.
  • If the research is distributed as a paper, significant commenters/contributors are acknowledged in the opening of the report. If they did not post their real names, handles used for comments are listed. Commenters do not retain any rights to the report, but their contributions will be recognized.
  • All primary research will be released under a Creative Commons license. The current license is Non-Commercial, Attribution. The analyst, at their discretion, may add a Derivative Works or Share Alike condition.
  • Securosis primary research does not discuss specific vendors or specific products/offerings, unless used to provide context, contrast or to make a point (which is very very rare).
    Although quotes from published primary research (and published primary research only) may be used in press releases, said quotes may never mention a specific vendor, even if the vendor is mentioned in the source report. Securosis must approve any quote to appear in any vendor marketing collateral.
  • Final primary research will be posted on the blog with open comments.
  • Research will be updated periodically to reflect market realities, based on the discretion of the primary analyst. Updated research will be dated and given a version number.
    For research that cannot be developed using this model, such as complex principles or models that are unsuited for a series of blog posts, the content will be chunked up and posted at or before release of the paper to solicit public feedback, and provide an open venue for comments and criticisms.
  • In rare cases Securosis may write papers outside of the primary research agenda, but only if the end result can be non-biased and valuable to the user community to supplement industry-wide efforts or advances. A “Radically Transparent Research” process will be followed in developing these papers, where absolutely all materials are public at all stages of development, including communications (email, call notes).
    Only the free primary research released on our site can be licensed. We will not accept licensing fees on research we charge users to access.
  • All licensed research will be clearly labeled with the licensees. No licensed research will be released without indicating the sources of licensing fees. Again, there will be no back channel influence. We’re open and transparent about our revenue sources.

In essence, we develop all of our research out in the open, and not only seek public comments, but keep those comments indefinitely as a record of the research creation process. If you believe we are biased or not doing our homework, you can call us out on it and it will be there in the record. Our philosophy involves cracking open the research process, and using our readers to eliminate bias and enhance the quality of the work.

On the back end, here’s how we handle this approach with licensees:

  • Licensees may propose paper topics. The topic may be accepted if it is consistent with the Securosis research agenda and goals, but only if it can be covered without bias and will be valuable to the end user community.
  • Analysts produce research according to their own research agendas, and may offer licensing under the same objectivity requirements.
  • The potential licensee will be provided an outline of our research positions and the potential research product so they can determine if it is likely to meet their objectives.
  • Once the licensee agrees, development of the primary research content begins, following the Totally Transparent Research process as outlined above. At this point, there is no money exchanged.
  • Upon completion of the paper, the licensee will receive a release candidate to determine whether the final result still meets their needs.
  • If the content does not meet their needs, the licensee is not required to pay, and the research will be released without licensing or with alternate licensees.
  • Licensees may host and reuse the content for the length of the license (typically one year). This includes placing the content behind a registration process, posting on white paper networks, or translation into other languages. The research will always be hosted at Securosis for free without registration.

Here is the language we currently place in our research project agreements:

Content will be created independently of LICENSEE with no obligations for payment. Once content is complete, LICENSEE will have a 3 day review period to determine if the content meets corporate objectives. If the content is unsuitable, LICENSEE will not be obligated for any payment and Securosis is free to distribute the whitepaper without branding or with alternate licensees, and will not complete any associated webcasts for the declining LICENSEE. Content licensing, webcasts and payment are contingent on the content being acceptable to LICENSEE. This maintains objectivity while limiting the risk to LICENSEE. Securosis maintains all rights to the content and to include Securosis branding in addition to any licensee branding.

Even this process itself is open to criticism. If you have questions or comments, you can email us or comment on the blog.