Last week I wrote up my near epic fail on Amazon Web Services where I ‘let’ someone launch a bunch of Litecoin mining instances in my account.
Since then I received some questions on my forensics process, so I figure this is a good time to write up the process in more detail. Specifically, how to take a snapshot and use it for forensic analysis.
I won’t cover all the steps at the AWS account layer – this post focuses on what you should do for a specific instance, not your entire management plane.
The first step, which I skipped, is to collect all the metadata associated with the instance. There is an easy way, a hard way (walk through the web UI and take notes manually), and the way I’m building into my nifty tool for all this that I will release at RSA (or sooner, if you know where to look).
The best way is to use the AWS command line tools for your operating system. Then run the command
aws ec2 describe-instances --instance-ids i-5203422c (inserting your instance ID). Note that you need to follow the instructions linked above to properly configure the tool and your credentials.
I suggest piping the output to a file (e.g.,
aws ec2 describe-instances --instance-ids i-5203422c > forensic-metadata.log) for later examination.
You should also get the console output, which is stored by AWS for a short period on boot/reboot/termination.
aws ec2 get-console-output --instance-id i-5203422c. This might include a bit more information if the attacker mucked with logs inside the instance, but won’t be useful for a hacked instance because it is only a boot log. This is a good reason to use a tool that collects instance logs outside AWS.
That is the basics of the metadata for an instance. Those two pieces collect the most important bits. The best option would be CloudTrail logs, but that is fodder for a future post.
Now on to the instance itself. While you might log into it and poke around, I focused on classical storage forensics. There are four steps:
- Take a snapshot of all storage volumes.
- Launch an instance to examine the volumes.
- Attach the volumes.
- Perform your investigation.
If you want to test any of this, feel free to use the snapshot of the hacked instance that was running in my account (well, one of 10 instances). The snapshot ID you will need is
Snapshot the storage volumes
I will show all this using the web interface, but you can also manage all of it using the command line or API (which is how I now do it, but that code wasn’t ready when I had my incident).
There is a slightly shorter way to do this in the web UI by going straight to volumes, but that way is easier to botch, so I will show the long way and you can figure out the shorter alternative yourself.
- Click Instances in your EC2 management console, then check the instance to examine.
- Look at the details on the bottom, click the Block Devices, then each entry. Pull the Volume ID for every attached volume.
- Switch to Volumes and then snapshot each volume you identified in the steps above.
- Label each snapshot so you remember it. I suggest date and time, “Forensics”, and perhaps the instance ID.
You can also add a name to your instance, then skip direct to Volumes and search for volumes attached to it.
Remember, once you take a snapshot, it is read-only – you can create as many copies as you like to work on without destroying the original. When you create a volume from an instance it doesn’t overwrite the snapshot, it gets another copy injected into storage.
Snapshots don’t capture volatile memory, so if you need RAM you need to either play with the instance itself or create a new image from that instance and launch it – perhaps the memory will provide more clues. That is a different process for another day.
Launch a forensics instance
Launch the operating system of your choice, in the same region as your snapshot. Load it with whatever tools you want. I did just a basic analysis by poking around.
Attach the storage volumes
- Go to Snapshots in the management console.
- Click the one you want, right-click, and then “Create Volume from Snapshot”. Make sure you choose the same Availability Zone as your forensics instance.
- Seriously, make sure you choose the same Availability Zone as your instance. People always mess this up. (By ‘people’, I of course mean ‘I’).
- Go back to Volumes.
- Select the new volume when it is ready, and right click/attach.
- Select your forensics instance. (Mine is stopped in the screenshot – ignore that).
- Set a mount point you will remember.
Perform your investigation
- Create a mount point for the new storage volumes, which are effectively external hard drives. For example,
sudo mkdir /forensics.
- Mount the new drive, e.g.,
sudo mount /dev/xvdf1 /forensics. Amazon may change the device mapping when you attach the drive (technically your operating system does that, not AWS, and you get a warning when you attach).
- Remember, use
sudo bash(or the appropriate equivalent for your OS) if you want to peek into a user account in the attached volume.
And that’s it. Remember you can mess with the volume all you want, then attach a new one from the snapshot again for another pristine copy. If you need a legal trail my process probably isn’t rigorous enough, but there should be enough here that you can easily adapt.
Again, try it with my snapshot if you want some practice on something with interesting data inside. And after RSA check back for a tool which automates nearly all of this.