Incident response must be highly organized in order to be effective. You must have a response plan developed ahead of time. There are six stages of activity in a formal incident response. This is a brief description of these stages. For a more involved description, contact CERT at http://www.cert.org/
Organizations should outline the objectives for incident handling. You must determine the minimal acceptable level of security controls for systems and networks, and implement these controls. Ensuring that all systems and network components have at least a minimum level of security often prevents incidents from becoming widespread. Security response team should be available 24 hours per day.
The preparation stage of incident response should entail the installation and testing of software that will be used when an incident occurs. Intrusion detection software can be very effective, for example, as can software that verifies the integrity of programs and data, such as tripwire, or the Red Hat package manager. Waiting to obtain useful software can be a costly mistake.
You can find more information on detecting signs of intrusion at CERTs Security Improvement site http://www.cert.org/security-improvement/modules.html
It is important to be able to recognize suspicious signs that an incident has occurred. Becuase detection must often rely on extremely subtle signs, the use of incident-detecting software is a standard practice in systems security efforts. System accounting logs are usually a dependable source of information about possible incidents, if they are configured correctly initially. In many organizations, the system administrator is required to inspect each system's log data on a daily basis. There are also programs available that can scan log data for the most common types of exploits, and provide a summary for the administrator to track and investigate.
Every detail about a possible intrusion should be recorded. Preserving as many details as possible will help the incident response team to understand how the attack occurred and how it affected the victim system.
Most organizations now employ the use of firewall systems to increase the security of the internal networks. Examining the firewall logs can lead to intrusion attempts. There are also programs available to process firewall logs, and produce a report of possible successful and failed intrusions.
Be sure to see the Intrusion Detection section of this document as well.
The third stage of incident response is containment. Once you have realized there is an attack going on, you need to be sure it does not spread further to other systems, or produce further damage to your system. Spotting a security compromise under way can be a tense undertaking. How you react can have large consequences. Hasty actions can cause more harm than the attacker would have.
If the compromise you are seeing is a physical one, odds are you have spotted someone who has broken into your home, office or lab. You should notify your local authorities. In a lab setting you might have spotted someone trying to open a case or reboot a machine. Depending on your authority and procedures, you might ask them to stop, or contact your local security people.
If you have detected a local user trying to compromise your security, the first thing to do is confirm they are in fact who you think they are. Check the site they are logging in from. Is it the site they are normally in from? no? Then use a non electronic means of getting in touch. For instance, call them on the phone or walk over to their office/house and talk to them. If they agree that they are on, you can ask them to explain what they were doing or tell them to cease doing it. If they are not on, and have no idea what you are talking about, odds are this incident requires further investigation. Look into such incidents , and have lots of information before making any accusations.
If you are unable to disconnect the network (if you have a busy site,
or you do not have physical control of your machines), the next best
step is to use something like TCP Wrappers or ipfwadm to deny
access from the intruders site.
If you can't deny all people from the same site as the intruder, locking the users account will have to do. Note that locking an account is not an easy thing. You have to keep in mind .rhosts files, FTP access, and a host of backdoors).
After you have done one of the above (disconnected network, denied access from their site, and/or disabled their account), you need to kill all their user processes and log them off.
You should monitor your site well for the next few minutes, as the attacker will try and get back in. Perhaps using a different account, and/or from a different network address.
The first priority of containment is to determine what is at risk if the incident spreads. If at all possible, a backup of the existing status of the machines should be made. There are several reasons for this, including keeping evidence of an attack for legal reasons, as well as keeping the data in the event the exploit deletes data.
Containing a network attack is often a matter of shutting the system down, which is in many cases, the safest response. If the system contains sensitive information, you might consider disconnecting the system from the network, booting to single user mode, or configuring the firewall to deny incoming requests.
In some cases, allowing an attacker to continue is an effective way to track the attacker's actions. Obviously, this should only be done with prior arrangements being made with the incident advisory group. Several people have tracked intruders in the past, and written their reports for others to learn from. Consider reading ``UNIX Backdoors'' or ``Protecting Your Site By Breaking Into It''.
If you are able to determine what means the attacker used to get into your system, you should try and close that hole. For instance, perhaps you see several FTP entries just before the user logged in. Disable the FTP service and check and see if there is an updated version or any of the lists know of a fix.
Check all your log files, and make a visit to your security lists and pages and see if there are any new common exploits you can fix. You can find Caldera security fixes here http://www.caldera.com/tech-ref/security/. Red Hat has not yet seperated their security fixes from bugfixes, but their distribution errata is available at http://www.Red Hat.com/errata It is very likely that if one vendor has released a security update, that most other Linux vendors will as well.
If you don't lock the attacker out, they will likely be back. Not just back on your machine, but back somewhere on your network. If they were running a packet sniffer, odds are good they have access to other local machines.
So you have either detected a compromise that has already happened or you have detected it and locked (hopefully) the offending attacker out of your system. Now what?
The fourth stage of incident response includes getting rid of the problem. Obviously in order to eradicate the problem, you need to know where the source of the problem is. Generally it is difficult to find the exact cause of the exploit.
Network intrusions are generally more difficult to eradicate, because attackers can use any system on a network to launch an attack on other addressable systems. Network-based exploits may require patches to the operating system, or routers on the network, which will take time to find and fix.
An excellent document describing what steps to take upon finding out you've been compromised is available by CERT at http://www.cert.org/tech_tips/root_compromise.html
The fifth stage of intrusion detection is rebuilding, and coming back online after the exploit. Restoration entails returning a system to its normal operational status, or ensuring that the system and the data are exactly as they were before the incident occurred. Ensuring that every aspect of the system is the same as before a security incident occurred is typically a labor-intensive activity. It requires that the integrity of every file in the compromised system be examined and restored.
For this reason, administrators typically backup important data, and reinstall the operating system from CDROM. Performing an integrity check or restoring the services from a backup is only the first step. The response team should then verify the integrity of services with nonproduction data in a test environment before they are allowed to resume in production mode.
The first thing is to assess the damage. What has been compromised? If you are running an Integrity Checker like Tripwire you can make a tripwire run and it should tell you. If not, you will have to look around at all your important data.
Since Linux systems are getting easier and easier to install, you might consider saving your config files and then wiping your disk(s) and reinstalling, then restoring your user files from backups and your config files. This will insure that you have a new clean system. If you have to backup files from the compromised system, be especially cautious of any binaries that you restore as they may be trojan horses placed there by the intruder.
Re-installation should be considered mandatory upon an intruder obtaining root access. Additionally, you'd like to keep any evidence there is, so having a spare disk in the safe may make sense.
Then you have to worry about how long ago the compromise happened, and whether the backups hold any damaged work.
Having regular backups is a godsend for security matters. If your system is compromised, you can restore the data you need from backups. Of course some data is valuable to the attacker to, and they will not only destroy it, they will steal it and have their own copies, but at least you will still have the data.
You should check several backups back into the past before restoring a file that has been tampered with. The intruder could have compromised your files long ago, and you could have made many successful backups of the compromised file!!!
Of course, there are also a raft of security concerns with backups. Make sure you are storing them in a secure place. Know who has access to them. (If an attacker can get your backups, they can have access to all your data without you ever knowing it.)
The final stage of incident response is the follow-up involved in reviewing the incident that has transpired and the action taken to handle it.
This should be done to make sure not only that it won't happen again, but also to see if the procedure can be improved. Total cost of the incident in terms of employee time spent, loss of critical data, legal costs, computer time, etc, should be evaluated. This information should be compiled into a document to be used to determine what the risks of loss in the future, similiar incidents will be.
It is also highly recommended that CERT be notified, and the proper documentation be completed, in order to prevent others from being afflicted by the same attack. CERT will be more than willing to help you with your attempt at finding the intruder.
You should report the attack to the admin contact at the site where the attacker attacked your system. You can look up this contact with "whois" or the internic database. You might send them an email with all applicable log entries and dates and times. If you spotted anything else distinctive about your intruder, you might mention that too. After sending the email, you should (if you are so inclined) follow up with a phone call. If that admin in turn spots your attacker, they might be able to talk to the admin of the site where they are coming from and so on.
Good crackers often use many intermediate systems. Some (or many) of which may not even know they have been compromised. Trying to track a cracker back to their home system can be difficult. Being polite to the admins you talk to can go a long way to getting help from them.
You should also notify any security organizations you are a part of (CERT or similar), as well as your Linux system vendor. If you decide to report the intrusion, which is strongly advised, you should be ready for the types of questions that will be asked. Disclosure of the information is not mandatory, and most documents allow you to specify which information can be disclosed and which cannot. To be prepared for the types of questions that will be asked, you should document the working condition of your systems, including host information, administrative contact for that network, and the type of incident (probe, scan, prank, scam, email spoofing or bombardment, sendmail attack, etc)
Incident response is still a relatively new subject, and has not been studied as extensively as other subjects, such as contingency planning and risk analysis. However, there is a great deal of resources available to help in such circumstances. Two organizations, CERT/CC and FIRST are well-equipted to help in such events.
You can find the appropriate documents for reporting security exploits near the end of this document.