Critical Response Teamwork

A Computer Security Incident Response Team is as necessary to your enterprise these days as door locks and employee name badges. How big a CSIRT do you need, and how do you justify the expense to top management?

• By Roberta Bragg
• 12/01/2001

—www.forensics-intl.com/info.html

Suddenly, you're faced with yet another intrusion, attack, abuse, virus or worm. Who should you call? What will you do immediately and long-term? Will your company even know it's been violated?

Sadly, many companies these days don't have a clue—despite the recent call by new cyberspace security chief Richard Clarke to IT leaders to secure their systems. They lack any formal policy, procedure or program to handle "computer incidents." When one occurs—whether Code Red, Nimda, a distributed denial of service attack, or e-mail-born worms or viruses—they suddenly need to do something—anything—to mitigate the effects, clean the systems, recover the data, reassure users and management, and spend some money so it won't happen again. Then it's back to business as usual … until the next time.

Of course, we can't protect against all attacks and intrusions, including downright stupidity, and these sorts of incidents won't ever be eliminated. But there are disciplined ways of preparing for, guarding against and responding to computer security incidents. Having a response plan can reduce your exposure and diminish the impact of attacks, and it's especially important in these times of heightened security.

In some large organizations, a formal team is often available. Users not only know whom to call, but what types of events should trigger such a call. IT relies on the team to provide advance notice of new risk; to screen the steady avalanche of announced vulnerabilities; and to provide direction in secure practices and assistance, leadership and resolution, during and after an incident. A member of the team sits on the management committees that produce computer security policies.

This team is the Computer Security Incident Response Team (CSIRT). Every organization needs one.

A Few Definitions
The formation and practice of computer security incident handling is a young science. The first known example of a formal organization for this purpose was the formation of the CERT Coordination Center at the Software Engineering Institute at Carnegie Melon University (www.cert.org) in 1988 in response to the first Internet worm, the Morris Worm. (CERT is a registered trademark, incidentally, that belongs to Carnegie Melon University.) Because CSIRT organizations don't have a long history, variances in definitions and duties exist. The definitions I provide here are based on those provided by CERT/CC and tempered by conversations with members of other CSIRTs.

A CSIRT is a team that provides services and support dedicated to preventing and responding to computer security incidents. It may provide services for a single organization, such as a single company or governmental body; or broad services for the public. Its range of services may be restricted to cataloging and providing informational resources, responding directly to quell the attack or preserve evidence for possible legal sanctions against the intruders, or providing every possible proactive, responsive and preventative measure. Other names for these organizations exist, including the copyrighted CERT/CC, Computer Security Incident Response Capability (CSIRC), Incident Response Team (IRT) and others.

Computer security incidents are any actions such as system compromise, probes and scans, denials of service, viruses, worms, data deletions, data thefts, data manipulations, unauthorized uses of systems, networks and services—whether or not they result in or were intended to result in negative effects. Each incident may be identified as a single action or a series of related actions that produce negative activity depending on the defined reporting structure of the CSIRT.

Attack is an attempt to compromise a system, while an intrusion is the result of a successful attack.

User community is the entire body of the organization, in other words, anyone who uses computer services including IT personnel, management and staff.

What Does a CSIRT DO?
While computer security in your organization may be formalized in policy, implemented by administrators and evaluated by formal security departments, your CSIRT's role may incorporate some or all of the following activities:

• Rapid response and recovery efforts in the face of an active computer security incident.
• Coordination of response and/or recovery efforts.
• Sharing of strategies and successful efforts with other CSIRTs and the public.
• Providing early alerts to new vulnerabilities and related attacks.
• Advising system developers and infrastructure architects on how to create secure programs and network designs.
• Identify vulnerabilities in practices, policies and procedures and participate in the development of security-minded replacements.
• Perform incident detection.
• Provide security awareness training.
• Act as a single point of contact for incident reporting.
• Investigate the nature of the vulnerability and extent of the attack.
• Preserve evidence for possible legal follow-up.
• Promote awareness of computer and network security.
• Encourage the use of secure network protocols, software and practices.
• Provide security services.
• Provide security advice.

Notice the important triplet here? The CSIRT should act as the single point of contact for incident reporting. It should coordinate, if not perform response and recovery, and push back knowledge gained to the user community. CSIRT members don't sit around waiting for their charges to be attacked; they provide proactive and preventative services as well.

Planning Your Work and Working Your Plan
Every organization needs a CSIRT, but can every organization afford one? Can any organization afford to be without one? A CSIRT won't generate revenue, bring in customers or directly reduce costs. Its role, however, is as necessary as locks on doors, employee name badges and other defensive measures. The question becomes then, how big a CSIRT do you need? What should it do? Where should it be? What does it need to operate?

Plan to Scale
It's all a matter of scale. The Australian Computer Emergency Response Team (AUSCERT) estimates that one full-time technical person can handle one new incident per day, assuming another 20 incidents are open and under investigation. In some organizations, the CSIRT function—like other IT-related functions—may be outsourced. In a very small business, one person might serve part-time as a clearinghouse for incident notification and determine at what point outside resources are called into play. As organization size increases, a one- or two-member CSIRT can adequately manage. Medium businesses may find a small department, including administrative and clerical staff, to be necessary, while large, distributed organizations will benefit from distributed CSIRT offices that provide more local access for far-flung locations. In addition to organization size, the size of the CSIRT will always be relative to the number of functions it fulfills—and, of course, the number of incidents and the extent with which they're dealt.

Mistakes to Avoid
As you begin your preliminary planning for constructing a CSIRT, be aware that you need a clear reason for the CSIRT's existence, one that your organization can understand.

It seems a simple thing, doesn't it? Surely everyone realizes the need to respond to and proactively work to prevent intrusions. But that's not the case. Some may not realize there's a true and identifiable problem; after all, many businesses with connections to the Internet don't even have a firewall. Others believe their current security practices protect them—the firewall or intrusion detection system as silver bullet syndrome. Many think they can just call on the federal or local police forces for assistance in the event of an attack. And some wouldn't recognize an attack if it left them black and blue—instead, they'd ascribe it to faulty software.

Once there's an understanding of the need for a response team, make sure to evaluate the possibility that this is being handled in some other manner—or could be. Is there a public sector organization that can help? Is someone or some department within your organization already performing these functions? In either case, this doesn't mean you don't need a CSIRT—it simply means you need to determine what forming a CSIRT will offer over what you have already. Will it serve as a coordinator of activities by disparate groups? Is there some function not currently being performed? Can a local group more readily respond to local issues, in a more timely fashion?

After you've established the need for some form of response, make sure you don't build unreasonable expectations. Clearly state your goals. It's highly unlikely that your proactive operations (security awareness, advice on vulnerabilities and security advice) will prevent all attacks from being successful. You simply won't be able to "catch" all intruders and bring them to justice—so don't promise to others that you can. Stating objectives in a manner that everyone can understand helps CSIRT staff as well. Think of it like a menu at a restaurant. If the menu doesn't offer filet mignon, the wait staff won't take such an order and cooks don't have to try to cobble something together while ignoring other orders on the grill. In the CSIRT goals, don't forget to include items such as which technologies the team will deal with, and how deep an analysis will be made. Make your duties and expectations as clear as you can. The continued success of the CSIRT depends on it.

And don't forget to budget for CSIRT staff training! Technical knowledge doesn't remain static. The technical staff needs a steady source of new information and skills, as well as time for attendance at conferences and classes, in addition to self study, tool development, discussion and information sharing.

Power and Direction
Typically, a CSIRT isn't an enforcement body; the role is one of advisor rather than commander. Many people find this surprising. In fact, if you ask members of your IT management staff what a CSIRT is and what it does, you may find out that they see a CSIRT as an outside force imposing its rules on their freedom of activity. Obviously, some PR is in order here. The rationale behind the CSIRT as advisor is simple: People are more likely to report incidents to an advisory body than to an individual. Without the cooperation of your organization's employees, you'll miss a great deal of important intelligence. Without a "security partnership" relationship, your security awareness messages will be difficult, if not impossible to promote.

CSIRTs haven't been around long enough to establish a generally agreed-upon chain of command. Some CSIRTs report to IT, others to Information Security, while still others exist higher in the chain of command, perhaps directly under a Chief Security Officer. While they generally don't need direct authority to enforce security policy, placing a CSIRT high in the pecking order lends them authority of another sort: Their function is recognized as critical to the function of the company.

Tools of the Trade
In addition to appropriate staffing, important tools and services must be arranged. The types of tools are determined by the goals and duties of the CSIRT. Will they be involved in attempts at recovering data, determining an exacting analysis of the attack or collecting evidence for prosecution? Forensic tools may be required. How will details of attacks be stored; will they later be analyzed for patterns and relationships? A database of some sort should be prepared. Are specialized software tools available for purchase or need to be built?

It's obvious that ordinary phones, e-mail services, faxes and other communication tools are necessary, but what secure methods of communication need to be provided? At a minimum, the ability to encrypt sensitive communications should be mandated. For internal communications, existing encryption facilities may be adequate. PGP is currently one of the standards used for sharing with external CSIRTs.

Will you provide 24-hour contact? If your information services are 24x7, your incident-response capabilities should be too. An answering service, call forwarding, paging or other alerting service may be configured if the CSIRT won't be staffed 24 hours.

Nothing can destroy credibility faster than the compromise of the CSIRT systems—systems that loom large on any attacker's list of targets. Every effort should be made to identify and put into place known security practices—including, but not limited to—the use of firewall, intrusion detection, physical security and strong authentication and authorization controls. As an internal department that is constantly working with sensitive information, the CSIRT will benefit from the ability to segment its portion of the network from the rest of the organization, and to protect via firewall any connections from external (outside CSIRT) sources.

A test network where images of compromised systems can be analyzed is essential. The range of test equipment for this network should model that used throughout the organization.

Rules of the Game
A set of detailed policies and procedures should be drafted that reflect every action and potential activity that may be part of the CSIRT's operation. Three areas should be considered: First, security policies are critical. What better way to advertise sound security practices than to practice them? Lucid policies communicate more than staff direction; many of them can be available to the whole organization and help to identify the scope and operations of the CSIRT. Second, internal operations of the CSIRT should be clearly defined so that every member knows what to do in a wide range of situations. This can assure consistency of approach, avoid mistakes that destroy evidence or further compromise systems and make operations more efficient. Any laws that impact CSIRT operations and communications should be addressed within the policies. Finally, appropriate responses to inquiries by the press, individuals and other CSIRTs and law enforcement agencies should be outlined.

Reactive Maneuvers
Of course, it's extremely difficult ahead of time to define the exact specifics of a response. However, Carnegie Melon's CERT lists these activities as appropriate:

1. Triage: Identify, categorize and assign incoming information. It's the same old story: No two incidents are created alike. You, or someone senior on your staff to whom you've assigned the responsibility, is going to have to determine which incidents need to be handled first. Indeed, by evaluating all current requests rather than taking them on a first-come, first-served basis, you may recognize trends and see a bit more of the big picture—a vantage that will allow you to respond to the most important incident first and appropriately. This does involve some preliminary analysis, but is primarily a prioritizing function.

2. Analyze: Examine the report and identify actions to be taken. Here's where further investigation takes place. Does the earlier estimate of seriousness of the problem and size of its impact hold up or need to be changed? What resources can be brought to bear? Are there fixes, workarounds or other tasks that can be immediately accomplished? Who needs to be contacted and what needs to be done? How much time and effort will you spend? What more information do you require? What tools will you use to analyze it?

3. Respond: Will your team report to other CSIRTs? Request that systems be disconnected from the network? Clean up compromised systems? When and how will the team communicate results to the organization and recommend preventative action?

Proactive Activities
A CSIRT can never have enough staffing or funding to effectively manage and respond to all incoming incidents, but can certainly fully expend all staff and funding in an attempt. The only sure way to slow the number of incidents that require response is to participate in activities that prevent them from happening. I know I'm stating the obvious here, but if you're considering the implementation of a CSIRT, you have a unique opportunity to build into its charter the requirements to perform proactive activities. Staff engaged in these activities may be recalled to more reactive efforts when you're faced with current crises.

Some activities that are good fits for CSIRT staff include vulnerability analysis, which means finding or further elucidating known vulnerabilities, and provides an opportunity to protect systems from them. Part of this process may be the constant review and screening of vendor- and other-produced reports of vulnerabilities. This should lead to hot lists of patching activities for system administrators to perform, and can reduce some of the hyperactivity that's generated each time a new alert is issued.

A CSIRT staff can also take on incident post mortem. After an incident has been responded to, can its cause be determined? What can be done in the future to prevent it from happening again?

A CSIRT can also handle education and training. There will be a wealth of knowledge in a CSIRT, including that gained by handling incidents and through constant study. This information can be communicated in the form of advisories, conference presentation, workshops, articles, internal meetings and training sessions for all classes of employees.

Call to Action
According to the January 2001 Internet Domain Survey of the Internet Software Consortium, there's an estimated 110 million host computer systems on the Internet. That's a lot of targets for malcontents, curious script kiddies, hactivists and cyber terrorists. Incidents are getting more sophisticated, widespread and coordinated, and the tools of mass system destruction are becoming more easily available. Isn't it time you rallied your troops and joined against the enemy? A CSIRT can be the foundation of your engagement and a way of communication and coordination with the rest of the good guys.