Containment Phase - Incident Response

Saturday, December 19, 2009

Mark Bennett

Hello Everyone,

This week I want to talk about the Containment Phase in Incident Response. As Always at the end of the blog I have my Tech Tip of the week.

The goal of the containment phase is to stop the damage being done. Fairly straight forward, right? Wrongo!

Many moons ago when I was a wee lad, I remember finding an ant hill. So I took a stick and poked a hole in it. At that time ants came pouring out of the hole I made, so I poked a few more, each time ants came flowing out of the new air conditioning I had just made for them. Needless to say the ants were mad and were bent on finding the attacker and neutralizing him. Fortunately for me of course I was hovering over them all and they did not see me, as they do not know how to "look up". Anyhow I left and went back a few days later and the holes were subsequently plugged back up again.

OK, where am I taking you?

Having a lack of Incident Response guidelines creates a very similar situation. There was a company not to long ago that got some nasty bit of malware in their environment and it was quite easily spread. When that happened tech people and managers seemed to pour out of the woodwork, all running to and fro bent on stopping this attack. Each one had loose ideas on what to do and they all ran around reloading machines in an attempt to plug the holes, all the while the machines reloaded became infected again.

On reflection I began to realize how much they all reminded me of the ants. All of them bent on stopping the attacker but not having a specific plan of action, a process, or set of guidelines on what to do, they all just flowed out in an attempt to overtake it all by mass of numbers.

Now don't get me wrong the approach does work, as it did for them, eventually they figured it all out and they are happily back up and running. However from and Incident Handler's perspective it was not well done. When you consider the time and effort and mistakes done by effectively re-inventing the wheel on what to do, they cost a lot more money by doing it that way instead of having a specific action plan. This is of course the whole point of Incident Response..Having a Plan! In the containment phase of Incident Response you want to prevent the attacker from getting any further into the organization or spreading to other systems.

In the case of the company I mentioned above they were creating the exact same environment before the attack occurred, which of course simply ensured that they would become reinfected. At a certain point when they all realized that this is not working they knew they needed to do some quick analysis of the infection (Identification phase), bring their anti-virus company on-board and see about what to do. Once this occurred they were able to begin to make progress as they figured out what it was and with the help of the vendor got the anti-virus engines updated and subsequently they began to clean their environment.

So instead of Red Alert, Red Alert, Oh My the Sky is falling, Prepare to enter Panic Phase! (Ants Running around not really knowing what to do) You already know what to do. Again don't get me wrong during an outbreak you definitely want to be as industrious as an Ant, it is definitely not the time for a lunch break! But by having a methodical plan of action you can perform short term containment while performing analysis to get a picture of what is occurring. That means not blindly destroying evidence to perform a quick short term containment. Had the company I mentioned above done this, they would have seen that reloading their machines would not be enough and all their hard industrious work would all be wasted. ( I talked about this in my blog on the Eradication phase, read about it here )

So how does one perform containment? Well there is a short and long term containment. I will focus a minute on the Short Term. The short term is to stop the attacker from causing more damage. Pull the power cord, apply filters to network/security equipment, null routing, changing a dns entry etc... (It is interesting to note that while the company mentioned above was running around all their machines were connecting to a single source for Command and Control. A simple change in their network would have prevented them from doing so defusing some of the situation while they worked the internal issue. )

These all can perform short term containment while your team is still analyzing in the Identification Phase. It is very important at this point to absolutely Remain Calm! This is a difficult thing to do when it seems like your environment is burning down around you. However more mistakes and sometimes costly ones are done when people forget the Incident Response phases and go directly to Panic Phase (This is NOT and Incident Response phase by the way). This almost always elicits a knee jerk, shotgun it, Cowboy type of response, and it never goes well.

Perhaps I have an intruder and not necessary malware. This could be a compromised system that is being controlled. You might have an attacker live on the box. Needless to say you don't want to alert the attacker to your presence. So don't get on the system and ping back to the attacker. You don't want them to know you are aware of what is going on. Above all DON'T HACK BACK! Remember two wrongs do not equal a right, and you don't want to break any laws.

Isolate the system. To do so try to image it when it is live, unplug the nic or better just put the box in a dirty vlan, as some things are link state aware.

Get a memory dump using Memoryze (my favorite) before anything so you have the image in the most pristine way without you doing much of anything else of the system that will show up.
You want to be able to be able to make a forensic image of the drive(s) that are affected to perform your analysis. If you must bring the machine down, Don't do a graceful shutdown, windows for example writes some 200+ files to the drive every-time you do that, just pull the power cord!

Here is where it is important for your Incident Responder to practice, practice, practice. It never ceases to amaze me how I can make backup after backup in my sleep and then when an Incident occurs this becomes a bit of stress for me. Just recently I was given a drive that I needed to perform a bit image on. Well no problem I know what to do, and I get the drive and whoa...It is smaller than a credit card with a serial ATA connector that is mini in size. Well that just killed my cabling to usb as I did not have such a small connector. It was the first time I seen one like this. OK so I will put it back into the box and boot up with Helix. Get the box install the drive, boot to helix... Nope. The Linux o/s just crashes on this. GRRRrrrrr! Now at this time I am starting to feel the pressure, as I am keenly aware of the clock now. I have found these things happen again and again, something totally unexpected during an incident. I had one where the o/s was so dorked up that imaging it was a real challenge, we tried multiple ways including using netcat to pipe it's output over the network. After realizing that based upon the rate of speed it would take some 15 days to get the image this was intolerable! The bottom line is making images on a machine that is fine and not having any issues is easy, try doing it on one that has a messed up o/s, or real new hardware that you don't have drivers for in Linux.

Now that backup needs to be a bit-by-bit image. dd is a great tool that will work in Linux and there is a windows version as well. I however prefer dcfldd as it gives you a byte count as it is performing the image dump. This is all good practice for the incident responder. Every time I encounter a new difficulty I have it written done in my notes (incident responders know how to take notes) so that when I encounter it again I don't have the same difficulty.

Now lets talk about Long Term Containment. The reason for doing long term containment is to keep the system in production while you are busy building another clean system. This is usually because it is a critical system and it simply cannot be taken down.

Once we have our backup for analysis we can start to make changes to the compromised system. This usually involves patching, the system as well as other systems to get rid of the vulnerability that an attacker or other malware took advantage of. Change the routes to null route the attackers address, alter firewall rules, remove the accounts that the attacker used (assuming not root) to bust the box.

Be sure to keep the lines of communication open! You need to be talking to the business unit that owns the machine. Let them know where you are in the stream of time with regular updates. Have a person that can interface well with management. Be aware that whether you perform short or long term containment you many have a business impacting outage. The business unit needs to sign off on what they want to do.

This is where having a good plan already in place as to how we will handle given situations. Make sure they have been discussed with management and they approve on the different courses of action for given situations, but then that is what Incident Response is all about...HAVING A PLAN! Remember the ant analogy? Talk about scenarios, play the what Then when an incident occurs remain calm, and work the problem based upon those plans. This way you can minimize damage, minimize lost productivity and maximize your eficencency in dealing with the issues.

OK. Enough already! I know you want the Tech Tip. As always when I write these things I give a tech tip of the week. Well here it is:

Data Recovery - Using a command called fls (From The Sleuth Kit) we can perform basic data recovery. Most times if you have read my other blogs I talk about the Metadata Layer, or the Data layer. Well today we will look at the File Name Layer. The file name layer is looking at the structures of the directory that has files in it. So it has pointers to the names of the files contained within it (metadata)

Let's try it:
Using an image that you have carved out type fls you will see the entire root directory (assuming you are working on the root slice) and it will produce output d/d inode number /bin etc... notice if there are any files as well they will have an inode number too. If they have an * (asterisk) that means they are deleted files.

You have the inode number (metadata layer) you can use icat to simply carve out that inode and re-write the file.

Now that is the root directory. How about the rest of the drive. Try this command fls -rpd this will recursively search the image for all deleted files and directories.

Now I talked about linux, this works in Microsoft as well fls -rpd

You simply need to carve it out using icat. Type icat > name of file you want to call it.

*Note if you are using the FAT filesystem you need to use the -r flag with icat.*

Well that is this weeks blog and tech tip of the week. As always be good, be safe, hack legally and responsibly, and share the knowledge...I'm Out.
Possibly Related Articles:
Enterprise Security
Incident Response Security Management
Post Rating I Like this!
The views expressed in this post are the opinions of the Infosec Island member that posted this content. Infosec Island is not responsible for the content or messaging of this post.

Unauthorized reproduction of this article (in part or in whole) is prohibited without the express written permission of Infosec Island and the Infosec Island member that posted this content--this includes using our RSS feed for any purpose other than personal use.