Note that some of the tools used in this writeup are no longer supported.
The techniques used, though, still apply.
Check out The Slueth Kit and Autopsy

Honeynet Scan of the Month 15

May 2001

Brian Carrier (carrier at

1 Introduction

1.1 Environment

All analysis work is done in one directory, which contains the following subdirectories:

The image is mounted in loopback mode on the mnt directory:

# mount -o ro,loop,nodev,noexec images/honeypot.hda8.dd mnt

The MD5 hash of all allocated files in the image is saved:

# find mnt -type f -exec md5sum {} \; > data/md5.all

The following forensics tools are used for the analysis:

The images directory is used as the morgue directory for Autopsy. The fsmorgue file contains the following:

# cat images/fsmorgue
honeynet.tar.gz       /

1.2 Validation

During a forensic investigation, it is important to verify that the images have not been tampered with. The MD5 hash of the compressed and uncompressed files will be compared to the values given on the Honeynet site.

# md5sum images/honeynet.tar.gz
0dff8fb9fe022ea80d8f1a4e4ae33e21 images/honeynet.tar.gz
# md5sum images/honeypot.hda8.dd
5a8ebf5725b15e563c825be85f2f852e images/honeypot.hda8.dd

2 Examination

The goal of this analysis is to identify deleted files from a rootkit. The difficulty of this task depends greatly on what OS and file system type was used. As this analysis is being performed on an EXT2FS image from a Redhat Linux system, the task maybe easy.

When Linux deletes a file, the file name and inode value are not deleted in the directory entry list and the block pointers are not deleted from the inode contents. Other systems, such as Solaris, delete the inode value from the directory entry and delete the block pointers from the inode. Therefore, recovering a recently deleted file is much easier on a Linux system than on a Solaris system.

2.1 Timeline Creation

The first step is to make a timeline from the unallocated inodes and unallocated directory entries, using TCT and TCTUTILs. The timeline details will be explored using the Autopsy Forensic Browser.

The timeline is created by listing the unallocated inodes, using ils (TCT), and converting them to the mactime format, using ils2mac (TCT). mactime (TCT) is used to create the timeline, and fls (TCTUTILs) and mac_merge (TCTUTILs) are then used to add deleted file names to it.

# ils images/honeypot.hda8.dd | ils2mac > data/body.ils
# mactime -p mnt/etc/passwd -g mnt/etc/group -b data/body.ils \
> 03/14/2001 > data/mactime.tmp
There is a slight error in the timeline because there is an entry for February 8, 2002 at the top of the timeline, which is out of order. This will cause mac_merge to not work, so open the timeline and move it to the end of the timeline. Next, run fls and mac_merge to integrate deleted file names.
# fls -m "mnt/" images/honeypot.hda8.dd 2 > data/hda8.fls
# mac_merge data/mactime.tmp data/hda8.fls > data/mactime.txt

2.2 Timeline Analysis

The timeline now contains entries for all unallocated inodes (from ils) and some deleted file names (from fls). Inodes that were unallocated due to regular system maintenance can be identified easily. For example, at 06:19:37 (CST) the following 3 unallocated inodes and 3 fls entries can be correlated and there is no need to examine them right now.

16 ma. lrwxrwxrwx root root < honeypot.hda8.dd-dead-12107>
16 ma. lrwxrwxrwx root root < honeypot.hda8.dd-dead-20883>
16 ma. lrwxrwxrwx root root < honeypot.hda8.dd-dead-28172>
16 ma. lrwxrwxrwx 0     0     mnt/etc/rc.d/rc4.d/K83ypbind (deleted)
16 ma. lrwxrwxrwx 0     0     mnt/etc/rc.d/rc2.d/K83ypbind (deleted)
16 ma. lrwxrwxrwx 0     0     mnt/etc/rc.d/rc5.d/K83ypbind (deleted)

The timeline contains many administrative file deletions until 20:36:48 (CST). At this point, we see a modification time for inode 23, which is identified by fls as /lk.tgz. Based on the name and size (500kB), this is probably the installation file for the rootkit. This file is recovered using icat (TCT).

# icat images/honeypot.hda8.dd 23 > recover/

At 20:44:50, there are six unallocated inodes that were accessed (2039, 2040, 2043, 2048, 2050, 2052), in addition to inode 23. The inode contents will be examined using the inode browsing mode of Autopsy.

When inode 2039 is entered into Autopsy, the inode and block details are listed for the 600kB file (screen shot). The view as file option is selected and it is obvious that this file is an executable because it starts with "ELF". By selected the strings option, the file is run through the strings(1) UNIX utility and is much easier to read. A quick scan of the output indicates that this is an ssh client (screen shot). It is unknown at this point if this was the original binary that was replaced by a trojan version or if this is a copy of a trojan version. It will be recovered for further analysis.

# icat images/honeypot.hda8.dd 2039 > recover/i2039.ssh.elf

A similar process is performed on the other inodes listed in the timeline. Each executable file is run through strings to identify what its original function was. The following is found:
inodefls NameType Description (from strings)
23/lk.tgzgzip tar Rootkit installation file
2039ELF ssh client
2040 All zeros
2041sh script installs rootkit
2042ELF unknown
2043bash script sauber: removes log entries
2044text /etc/inetd.conf file
2045sh script runs mkxfs and linsniffer
2046text /etc/services file
2047perl script sorts output of linsniffer
2048text ssh client config file
2049binary SSH private key for
2050binary ssh key file
2051binary unknown data
2052text ssh server config file
2053ELF port scanner
2054ELF executes commands via an http server
2058ELF top
2059script restarts linsniffer
2060text ssh server config file
2061ELF sshd
30188ELF netstat
30191ELF ps
48284ELF ifconfig
56231 Nothing, all zeros

Each of the above inodes is recovered and saved to the recover directory. This set of inodes represents all unallocated inodes with a MAC value after March 14. Although, there maybe additional deleted files whose inode has been reallocated. Further analysis will be performed to determine if this happened.

2.3 lk.tgz

The lk.tgz file is assumed to be the rootkit installation file. It will be opened to identify the files that are contained in it.

# md5sum recover/
115f438631de8d0a7c03c9d458eb7257 recover/
# gunzip recover/
# md5sum recover/
a68523a575b6e6b35738103d6d120b94 recover/
# cd recover
# tar xf
# cd ..
md5sum will be used to correlate the unallocated inodes with files in the rootkit. The MD5 value of the recovered inodes and the MD5 value of the rootkit files will be generated and uniq will be used to identify the files that were deleted but not in the rootkit and the files that were in the rootkit but not deleted.
# md5sum recover/* recover/last/* | sort | uniq -u -w 32
086394958255553f6f38684dad97869e recover/last/ifconfig
2b07576213c1c8b942451459b3dc4903 recover/last/netstat
5e1725f2734365fef9e55398785f3033 recover/
6c0f96c1e43a23a21264f924ae732273 recover/last/linsniffer
7728c15d89f27e376950f96a7510bf0f recover/last/ps
928c5f9a4b4068a5db47dfdc65ea6cde recover/i2042.exe
a68523a575b6e6b35738103d6d120b94 recover/
f174e862d00d0998c3fa4ccd632019b5 recover/i30188.netstat.exe

This shows that many of the recovered inodes were part of the rootkit. Inodes 30191, 2042, and 30188 were not part of the lk.tgz file. From the rootkit files that were not found in the unallocated inodes and the strings analysis, it can be assumed that inode 30188 is the original copy of netstat, which was replaced by the rootkit version. Similarly, inode 30191 is the original version of ps, which was replaced by the rootkit version.

This assumption will be verified by comparing the MD5 value of the /bin/netstat, and /bin/ps binaries with the ones from the rootkit installation. /sbin/ifconfig will also be compared.

# md5sum mnt/bin/netstat recover/last/netstat
2b07576213c1c8b942451459b3dc4903 mnt/bin/netstat
2b07576213c1c8b942451459b3dc4903 recover/last/netstat
# md5sum mnt/bin/ps recover/last/ps
7728c15d89f27e376950f96a7510bf0f mnt/bin/ps
7728c15d89f27e376950f96a7510bf0f recover/last/ps
# md5sum mnt/sbin/ifconfig recover/last/ifconfig
086394958255553f6f38684dad97869e mnt/sbin/ifconfig
086394958255553f6f38684dad97869e recover/last/ifconfig

Therefore, the rootkit was installed and the above binaries were replaced with trojan versions.

To identify if other system files were replaced by the rootkit, md5sum will again be used. As shown in Section 1.1, the MD5 hash values of all allocated files were saved to data/md5.all. The following commands search the file of MD5 hashes for ones that match those in the rootkit.

# for i in recover/last/*
> do echo $i;
> grep `md5sum $i` data/md5.all;
> done;
data/md5.all:086394958255553f6f38684dad97869e mnt/sbin/ifconfig
data/md5.all:b63485e42035328c0d900a71ff2e6bd7 mnt/etc/inetd.conf
data/md5.all:6c0f96c1e43a23a21264f924ae732273 mnt/dev/ida/.drag-on/linsniffer
data/md5.all:6c0f96c1e43a23a21264f924ae732273 mnt/dev/ida/.. /linsniffer
data/md5.all:5f22ceb87631fbcbf32e59234feeaa5b mnt/dev/ida/.drag-on/logclear
data/md5.all:5f22ceb87631fbcbf32e59234feeaa5b mnt/dev/ida/.. /logclear
data/md5.all:18a2d7d3178f321b881e7c493af72996 mnt/dev/ida/.drag-on/mkxfs
data/md5.all:18a2d7d3178f321b881e7c493af72996 mnt/dev/ida/.. /mkxfs
data/md5.all:2b07576213c1c8b942451459b3dc4903 mnt/bin/netstat
data/md5.all:68b329da9893e34099c7d8ad5cb9c940 mnt/etc/at.deny
data/md5.all:7728c15d89f27e376950f96a7510bf0f mnt/bin/ps
data/md5.all:06d04fa3c4941b398756d029de75770e mnt/dev/ida/.drag-on/s
data/md5.all:06d04fa3c4941b398756d029de75770e mnt/dev/ida/.. /s
data/md5.all:464dc23cac477c43418eb8d3ef087065 mnt/dev/ida/.drag-on/sense
data/md5.all:464dc23cac477c43418eb8d3ef087065 mnt/dev/ida/.. /sense
data/md5.all:54e41f035e026f439d4188759b210f07 mnt/etc/services
data/md5.all:4cfae8c44a6d1ede669d41fc320c7325 mnt/dev/ida/.drag-on/sl2
data/md5.all:4cfae8c44a6d1ede669d41fc320c7325 mnt/dev/ida/.. /sl2
data/md5.all:c2c1b08498ed71a908c581d634832672 mnt/dev/ida/.drag-on/ssh_host_key
data/md5.all:c2c1b08498ed71a908c581d634832672 mnt/dev/ida/.. /ssh_host_key
data/md5.all:ad265d3c07dea3151bacb6930e0b72d3 mnt/dev/ida/.. /ssh_random_seed

This data shows that the rootkit files were also copied to '/dev/ida/.. /' and '/dev/ida/.drag-on/'. After this step, all recovered inodes except 2042 have been identified.

2.4 /last Directory

Inode 2038 is identified by fls as the deleted /last directory. This directory will be examined for further information. When a directory is deleted, Linux sets the size to 0 and therefore icat can not be used to view the directory contents. ils though, will always show the first two blocks that an inode uses.

# ils images/honeypot.hda8.dd 2038
Block 8481 contains the directory entry contents. Its contents can be viewed in Autopsy, using Block Browsing mode (screen shot). For the sake of this document, bcat (TCTUTILs) will be used on the command line.
# bcat -h images/honeypot.hda8.dd 8481 512
0f60700000c0001022e00000002000000 ............ ....
16f40302022e2e0000f70700000c000301 ............ ....
3273736800f80700001000070170696466 ssh......... pidf
48696c6500f907000010000701696e7374 ile......... inst
64616c6c00fa07000014000801636f6d70 all......... comp
807574657265720000fb07000010000701 uterer...... ....
96636c65616e657200fc07000014000a01 cleaner..... ....
112696e6574642e636f6e660000fd070000 inetd.conf.. ....
128100006016c73617474720000fe070000 ....lsattr.. ....
144200008017365727669636573ff070000 ....
1601000050173656e736500000000080000 ....sense... ....
17628000a017373685f636f6e6669670000 (...ssh_conf ig..
1920108000014000c017373685f686f7374 ........ssh_ host
2085f6b657902080000300010017373685f _key....0... ssh_
224686f73745f6b65792e70756203080000 ....
24018000f017373685f72616e646f6d5f73 ....ssh_rand om_s
2566565640004080000fc020b0173736864 eed......... sshd
2725f636f6e66696700050800000c000301 _config..... ....
288736c320006080000dc0208016c617374 sl2......... last
3042e636769070800002c00020170730000 .cgi....,... ps..
32008080000200007016e65747374617400 .... ...nets tat.
33609080000100008016966636f6e666967 ........ifco nfig
3520a0800000c000301746f70000b080000 ....
368100008016c6f67636c6561720c080000 ....logclear ....
38484020101730000000d08000078020501 ....s....... x...
4006d6b7866730000000000000000000000 mkxfs....... ....
41600000000000000000000000000000000 ............ ....

This block contains a list of directory entry structures. Each structure contains the inode value, the size, and the file name. By parsing the structures (and converting hex to decimal), the following data is determined:
File NameInode

When this list is compared to the results found in Section 2.2, many of the original assumptions are verified. According to this data, inode 2042 was allocated by the /last/computer file. It would seem that all inodes have been accounted for and the analysis is over. Upon rootkit examination however, this is not true. The install file shows that the computer file should be ASCII text, but the previous strings analysis shows that it is an ELF executable. Therefore, either the inode or block was reallocated to a new file. Further analysis shows that inode 2042 points to block 9124, as does inode 2053. Inode 2053 has been identified as the sl2 executable.

3 Conclusion

3.1 Answers

1. The recovery steps are shown above.

2. The following files make up the deleted rootkit (lk.tgz from inode 23):

The installation of the rootkit creates a file named computer. While this file could not be recovered, its contents were found in blocks 90417 and 90418 by performing a search in Autopsy for the string "Bogomips".

The analysis also found the original ps, netstat, and ifconfig executables.

3.2 Bonus Question

The bonus question is if the rootkit was actually installed. The answer is yes, based on the existence of the '/dev/ida/.drag-on/' and '/dev/ida/.. /' directories, the recovered computer file, the replaced binaries (ps, netstat, and ifconfig), and the replaced /etc/ files.
Brian Carrier []