Note that some of the tools used in this writeup are no longer supported.
The techniques used, though, still apply.
Check out The Slueth Kit and Autopsy

Honeynet Scan of the Month 15

May 2001

Brian Carrier (carrier at cerias.purdue.edu)

1 Introduction

1.1 Environment

All analysis work is done in one directory, which contains the following subdirectories:

data: location of analysis tool output (i.e. grave-robber, mactime, & fls).
images: location of forensic images and the morgue directory for Autopsy.
mnt: location where root of compromised system is mounted.
recover: location of recovered inodes and blocks.

The image is mounted in loopback mode on the mnt directory:

# mount -o ro,loop,nodev,noexec images/honeypot.hda8.dd mnt

The MD5 hash of all allocated files in the image is saved:

# find mnt -type f -exec md5sum {} \; > data/md5.all

The following forensics tools are used for the analysis:

The Coroners Toolkit (TCT) [Farmer and Venema] Available at: www.porcupine.org/forensics/tct.html. TCT provides command line based UNIX utilities for post-mortem data gathering, and file system utilities for FFS and EXT2FS partitions that allow block- and inode-level analysis. Some features include making a timeline based on file modification, access, and change (MAC) times, analyzing blocks of the file system in an HTML browser, and tools to extract the contents of files based on the inode value. This set of tools is very powerful, but requires low-level knowledge of the compromised system.
TCTUTILs [Carrier] Available at: www.sleuthkit.org [NOTE: TCTUTILs is no longer supported]. TCTUTILs is based on TCT and adds more file system analysis tools. Some of its tools provide file name-level analysis and mapping between blocks and inodes, and between inodes and names. The tools in this package are also command line based and require low-level knowledge of a system.
Autopsy Forensic Browser [Carrier] Available at: www.sleuthkit.org/autopsy/ Autopsy is an HTML based graphical interface that integrates the tools from TCT, TCTUTILs, and standard UNIX utilities (strings, md5sum, grep etc.). It allows an investigator the ability to "browse" a forensic image from the file, inode, or block level. Autopsy makes the typical, tedious tasks easier to perform, while allowing the investigator to use the individual tools for more complex scenarios. The Autopsy package does not include any forensic analysis tools, just the "glue" to integrate the tools from other packages.

The images directory is used as the morgue directory for Autopsy. The fsmorgue file contains the following:

# cat images/fsmorgue
honeynet.tar.gz /

1.2 Validation

During a forensic investigation, it is important to verify that the images have not been tampered with. The MD5 hash of the compressed and uncompressed files will be compared to the values given on the Honeynet site.

# md5sum images/honeynet.tar.gz
0dff8fb9fe022ea80d8f1a4e4ae33e21 images/honeynet.tar.gz
# md5sum images/honeypot.hda8.dd
5a8ebf5725b15e563c825be85f2f852e images/honeypot.hda8.dd

2 Examination

The goal of this analysis is to identify deleted files from a rootkit. The difficulty of this task depends greatly on what OS and file system type was used. As this analysis is being performed on an EXT2FS image from a Redhat Linux system, the task maybe easy.

When Linux deletes a file, the file name and inode value are not deleted in the directory entry list and the block pointers are not deleted from the inode contents. Other systems, such as Solaris, delete the inode value from the directory entry and delete the block pointers from the inode. Therefore, recovering a recently deleted file is much easier on a Linux system than on a Solaris system.

2.1 Timeline Creation

The first step is to make a timeline from the unallocated inodes and unallocated directory entries, using TCT and TCTUTILs. The timeline details will be explored using the Autopsy Forensic Browser.

The timeline is created by listing the unallocated inodes, using ils (TCT), and converting them to the mactime format, using ils2mac (TCT). mactime (TCT) is used to create the timeline, and fls (TCTUTILs) and mac_merge (TCTUTILs) are then used to add deleted file names to it.

# ils images/honeypot.hda8.dd | ils2mac > data/body.ils
# mactime -p mnt/etc/passwd -g mnt/etc/group -b data/body.ils \
> 03/14/2001 > data/mactime.tmp

There is a slight error in the timeline because there is an entry for February 8, 2002 at the top of the timeline, which is out of order. This will cause mac_merge to not work, so open the timeline and move it to the end of the timeline. Next, run fls and mac_merge to integrate deleted file names.

# fls -m "mnt/" images/honeypot.hda8.dd 2 > data/hda8.fls
# mac_merge data/mactime.tmp data/hda8.fls > data/mactime.txt

2.2 Timeline Analysis

The timeline now contains entries for all unallocated inodes (from ils) and some deleted file names (from fls). Inodes that were unallocated due to regular system maintenance can be identified easily. For example, at 06:19:37 (CST) the following 3 unallocated inodes and 3 fls entries can be correlated and there is no need to examine them right now.

16 ma. lrwxrwxrwx root root < honeypot.hda8.dd-dead-12107>
16 ma. lrwxrwxrwx root root < honeypot.hda8.dd-dead-20883>
16 ma. lrwxrwxrwx root root < honeypot.hda8.dd-dead-28172>
16 ma. lrwxrwxrwx 0 0 mnt/etc/rc.d/rc4.d/K83ypbind (deleted)
16 ma. lrwxrwxrwx 0 0 mnt/etc/rc.d/rc2.d/K83ypbind (deleted)
16 ma. lrwxrwxrwx 0 0 mnt/etc/rc.d/rc5.d/K83ypbind (deleted)

The timeline contains many administrative file deletions until 20:36:48 (CST). At this point, we see a modification time for inode 23, which is identified by fls as /lk.tgz. Based on the name and size (500kB), this is probably the installation file for the rootkit. This file is recovered using icat (TCT).

# icat images/honeypot.hda8.dd 23 > recover/i23.lk.tar.gz

At 20:44:50, there are six unallocated inodes that were accessed (2039, 2040, 2043, 2048, 2050, 2052), in addition to inode 23. The inode contents will be examined using the inode browsing mode of Autopsy.

When inode 2039 is entered into Autopsy, the inode and block details are listed for the 600kB file (screen shot). The view as file option is selected and it is obvious that this file is an executable because it starts with "ELF". By selected the strings option, the file is run through the strings(1) UNIX utility and is much easier to read. A quick scan of the output indicates that this is an ssh client (screen shot). It is unknown at this point if this was the original binary that was replaced by a trojan version or if this is a copy of a trojan version. It will be recovered for further analysis.

# icat images/honeypot.hda8.dd 2039 > recover/i2039.ssh.elf

A similar process is performed on the other inodes listed in the timeline. Each executable file is run through strings to identify what its original function was. The following is found:

inode fls Name Type Description (from strings)

23 /lk.tgz gzip tar Rootkit installation file

2038 /last directory

2039 ELF ssh client

2040 All zeros

2041 sh script installs rootkit

2042 ELF unknown

2043 bash script sauber: removes log entries

2044 text /etc/inetd.conf file

2045 sh script runs mkxfs and linsniffer

2046 text /etc/services file

2047 perl script sorts output of linsniffer

2048 text ssh client config file

2049 binary SSH private key for root@dil2.datainfosys.net

2050 binary ssh key file

2051 binary unknown data

2052 text ssh server config file

2053 ELF port scanner

2054 ELF executes commands via an http server

2058 ELF top

2059 script restarts linsniffer

2060 text ssh server config file

2061 ELF sshd

30188 ELF netstat

30191 ELF ps

48284 ELF ifconfig

56231 Nothing, all zeros

inode	fls Name	Type	Description (from strings)
23	/lk.tgz	gzip tar	Rootkit installation file
2038	/last	directory
2039		ELF	ssh client
2040			All zeros
2041		sh script	installs rootkit
2042		ELF	unknown
2043		bash script	sauber: removes log entries
2044		text	/etc/inetd.conf file
2045		sh script	runs mkxfs and linsniffer
2046		text	/etc/services file
2047		perl script	sorts output of linsniffer
2048		text	ssh client config file
2049		binary	SSH private key for root@dil2.datainfosys.net
2050		binary	ssh key file
2051		binary	unknown data
2052		text	ssh server config file
2053		ELF	port scanner
2054		ELF	executes commands via an http server
2058		ELF	top
2059		script	restarts linsniffer
2060		text	ssh server config file
2061		ELF	sshd
30188		ELF	netstat
30191		ELF	ps
48284		ELF	ifconfig
56231			Nothing, all zeros

Each of the above inodes is recovered and saved to the recover directory. This set of inodes represents all unallocated inodes with a MAC value after March 14. Although, there maybe additional deleted files whose inode has been reallocated. Further analysis will be performed to determine if this happened.

2.3 lk.tgz

The lk.tgz file is assumed to be the rootkit installation file. It will be opened to identify the files that are contained in it.

# md5sum recover/i23.lk.tar.gz
115f438631de8d0a7c03c9d458eb7257 recover/i23.lk.tar.gz
# gunzip recover/i23.lk.tar.gz
# md5sum recover/i23.lk.tar
a68523a575b6e6b35738103d6d120b94 recover/i23.lk.tar
# cd recover
# tar xf i23.lk.tar
# cd ..

md5sum will be used to correlate the unallocated inodes with files in the rootkit. The MD5 value of the recovered inodes and the MD5 value of the rootkit files will be generated and uniq will be used to identify the files that were deleted but not in the rootkit and the files that were in the rootkit but not deleted.

# md5sum recover/* recover/last/* | sort | uniq -u -w 32
086394958255553f6f38684dad97869e recover/last/ifconfig
2b07576213c1c8b942451459b3dc4903 recover/last/netstat
5e1725f2734365fef9e55398785f3033 recover/i30191.ps.exe
6c0f96c1e43a23a21264f924ae732273 recover/last/linsniffer
7728c15d89f27e376950f96a7510bf0f recover/last/ps
928c5f9a4b4068a5db47dfdc65ea6cde recover/i2042.exe
a68523a575b6e6b35738103d6d120b94 recover/i23.lk.tar
f174e862d00d0998c3fa4ccd632019b5 recover/i30188.netstat.exe

This shows that many of the recovered inodes were part of the rootkit. Inodes 30191, 2042, and 30188 were not part of the lk.tgz file. From the rootkit files that were not found in the unallocated inodes and the strings analysis, it can be assumed that inode 30188 is the original copy of netstat, which was replaced by the rootkit version. Similarly, inode 30191 is the original version of ps, which was replaced by the rootkit version.

This assumption will be verified by comparing the MD5 value of the /bin/netstat, and /bin/ps binaries with the ones from the rootkit installation. /sbin/ifconfig will also be compared.

# md5sum mnt/bin/netstat recover/last/netstat
2b07576213c1c8b942451459b3dc4903 mnt/bin/netstat
2b07576213c1c8b942451459b3dc4903 recover/last/netstat
# md5sum mnt/bin/ps recover/last/ps
7728c15d89f27e376950f96a7510bf0f mnt/bin/ps
7728c15d89f27e376950f96a7510bf0f recover/last/ps
# md5sum mnt/sbin/ifconfig recover/last/ifconfig
086394958255553f6f38684dad97869e mnt/sbin/ifconfig
086394958255553f6f38684dad97869e recover/last/ifconfig

Therefore, the rootkit was installed and the above binaries were replaced with trojan versions.

To identify if other system files were replaced by the rootkit, md5sum will again be used. As shown in Section 1.1, the MD5 hash values of all allocated files were saved to data/md5.all. The following commands search the file of MD5 hashes for ones that match those in the rootkit.

# for i in recover/last/*
> do echo $i;
> grep `md5sum $i` data/md5.all;
> done;
recover/last/cleaner
recover/last/ifconfig
data/md5.all:086394958255553f6f38684dad97869e mnt/sbin/ifconfig
recover/last/inetd.conf
data/md5.all:b63485e42035328c0d900a71ff2e6bd7 mnt/etc/inetd.conf
recover/last/install
recover/last/last.cgi
recover/last/linsniffer
data/md5.all:6c0f96c1e43a23a21264f924ae732273 mnt/dev/ida/.drag-on/linsniffer
data/md5.all:6c0f96c1e43a23a21264f924ae732273 mnt/dev/ida/.. /linsniffer
recover/last/logclear
data/md5.all:5f22ceb87631fbcbf32e59234feeaa5b mnt/dev/ida/.drag-on/logclear
data/md5.all:5f22ceb87631fbcbf32e59234feeaa5b mnt/dev/ida/.. /logclear
recover/last/lsattr
recover/last/mkxfs
data/md5.all:18a2d7d3178f321b881e7c493af72996 mnt/dev/ida/.drag-on/mkxfs
data/md5.all:18a2d7d3178f321b881e7c493af72996 mnt/dev/ida/.. /mkxfs
recover/last/netstat
data/md5.all:2b07576213c1c8b942451459b3dc4903 mnt/bin/netstat
recover/last/pidfile
data/md5.all:68b329da9893e34099c7d8ad5cb9c940 mnt/etc/at.deny
recover/last/ps
data/md5.all:7728c15d89f27e376950f96a7510bf0f mnt/bin/ps
recover/last/s
data/md5.all:06d04fa3c4941b398756d029de75770e mnt/dev/ida/.drag-on/s
data/md5.all:06d04fa3c4941b398756d029de75770e mnt/dev/ida/.. /s
recover/last/sense
data/md5.all:464dc23cac477c43418eb8d3ef087065 mnt/dev/ida/.drag-on/sense
data/md5.all:464dc23cac477c43418eb8d3ef087065 mnt/dev/ida/.. /sense
recover/last/services
data/md5.all:54e41f035e026f439d4188759b210f07 mnt/etc/services
recover/last/sl2
data/md5.all:4cfae8c44a6d1ede669d41fc320c7325 mnt/dev/ida/.drag-on/sl2
data/md5.all:4cfae8c44a6d1ede669d41fc320c7325 mnt/dev/ida/.. /sl2
recover/last/ssh
recover/last/ssh_config
recover/last/ssh_host_key
data/md5.all:c2c1b08498ed71a908c581d634832672 mnt/dev/ida/.drag-on/ssh_host_key
data/md5.all:c2c1b08498ed71a908c581d634832672 mnt/dev/ida/.. /ssh_host_key
recover/last/ssh_host_key.pub
recover/last/ssh_random_seed
data/md5.all:ad265d3c07dea3151bacb6930e0b72d3 mnt/dev/ida/.. /ssh_random_seed
recover/last/sshd_config
recover/last/top

This data shows that the rootkit files were also copied to '/dev/ida/.. /' and '/dev/ida/.drag-on/'. After this step, all recovered inodes except 2042 have been identified.

2.4 /last Directory

Inode 2038 is identified by fls as the deleted /last directory. This directory will be examined for further information. When a directory is deleted, Linux sets the size to 0 and therefore icat can not be used to view the directory contents. ils though, will always show the first two blocks that an inode uses.

# ils images/honeypot.hda8.dd 2038
[...]
2038|f|1031|100|984707105|984707105|984707105|984707169|40755|0|0|8481|0

Block 8481 contains the directory entry contents. Its contents can be viewed in Autopsy, using Block Browsing mode (screen shot). For the sake of this document, bcat (TCTUTILs) will be used on the command line.

# bcat -h images/honeypot.hda8.dd 8481 512

0 f6070000 0c000102 2e000000 02000000 .... .... .... ....

16 f4030202 2e2e0000 f7070000 0c000301 .... .... .... ....

32 73736800 f8070000 10000701 70696466 ssh. .... .... pidf

48 696c6500 f9070000 10000701 696e7374 ile. .... .... inst

64 616c6c00 fa070000 14000801 636f6d70 all. .... .... comp

80 75746572 65720000 fb070000 10000701 uter er.. .... ....

96 636c6561 6e657200 fc070000 14000a01 clea ner. .... ....

112 696e6574 642e636f 6e660000 fd070000 inet d.co nf.. ....

128 10000601 6c736174 74720000 fe070000 .... lsat tr.. ....

144 20000801 73657276 69636573 ff070000 ... serv ices ....

160 10000501 73656e73 65000000 00080000 .... sens e... ....

176 28000a01 7373685f 636f6e66 69670000 (... ssh_ conf ig..

192 01080000 14000c01 7373685f 686f7374 .... .... ssh_ host

208 5f6b6579 02080000 30001001 7373685f _key .... 0... ssh_

224 686f7374 5f6b6579 2e707562 03080000 host _key .pub ....

240 18000f01 7373685f 72616e64 6f6d5f73 .... ssh_ rand om_s

256 65656400 04080000 fc020b01 73736864 eed. .... .... sshd

272 5f636f6e 66696700 05080000 0c000301 _con fig. .... ....

288 736c3200 06080000 dc020801 6c617374 sl2. .... .... last

304 2e636769 07080000 2c000201 70730000 .cgi .... ,... ps..

320 08080000 20000701 6e657473 74617400 .... ... nets tat.

336 09080000 10000801 6966636f 6e666967 .... .... ifco nfig

352 0a080000 0c000301 746f7000 0b080000 .... .... top. ....

368 10000801 6c6f6763 6c656172 0c080000 .... logc lear ....

384 84020101 73000000 0d080000 78020501 .... s... .... x...

400 6d6b7866 73000000 00000000 00000000 mkxf s... .... ....

416 00000000 00000000 00000000 00000000 .... .... .... ....

[...]

This block contains a list of directory entry structures. Each structure contains the inode value, the size, and the file name. By parsing the structures (and converting hex to decimal), the following data is determined:

File Name Inode

ssh 2039

pidfile 2040

install 2041

computer 2042

cleaner 2043

inetd.conf 2044

lsattr 2045

services 2046

sense 2047

ssh_config 2048

ssh_host_key 2049

ssh_host_key.pub 2050

ssh_random_seed 2051

sshd_config 2052

sl2 2053

last.cgi 2054

ps 2055

netstat 2056

ifconfig 2057

top 2058

logclear 2059

s 2060

mkxfs 2061

File Name	Inode
ssh	2039
pidfile	2040
install	2041
computer	2042
cleaner	2043
inetd.conf	2044
lsattr	2045
services	2046
sense	2047
ssh_config	2048
ssh_host_key	2049
ssh_host_key.pub	2050
ssh_random_seed	2051
sshd_config	2052
sl2	2053
last.cgi	2054
ps	2055
netstat	2056
ifconfig	2057
top	2058
logclear	2059
s	2060
mkxfs	2061

When this list is compared to the results found in Section 2.2, many of the original assumptions are verified. According to this data, inode 2042 was allocated by the /last/computer file. It would seem that all inodes have been accounted for and the analysis is over. Upon rootkit examination however, this is not true. The install file shows that the computer file should be ASCII text, but the previous strings analysis shows that it is an ELF executable. Therefore, either the inode or block was reallocated to a new file. Further analysis shows that inode 2042 points to block 9124, as does inode 2053. Inode 2053 has been identified as the sl2 executable.

3 Conclusion

3.1 Answers

1. The recovery steps are shown above.

2. The following files make up the deleted rootkit (lk.tgz from inode 23):

cleaner
ifconfig
inetd.conf
install
last.cgi
linsniffer
logclear
lsattr
mkxfs
netstat
pidfile
ps
s
sense
services
sl2
ssh
ssh_config
ssh_host_key
ssh_host_key.pub
ssh_random_seed
sshd_config
top

The installation of the rootkit creates a file named computer. While this file could not be recovered, its contents were found in blocks 90417 and 90418 by performing a search in Autopsy for the string "Bogomips".

The analysis also found the original ps, netstat, and ifconfig executables.

3.2 Bonus Question

The bonus question is if the rootkit was actually installed. The answer is yes, based on the existence of the '/dev/ida/.drag-on/' and '/dev/ida/.. /' directories, the recovered computer file, the replaced binaries (ps, netstat, and ifconfig), and the replaced /etc/ files.

Brian Carrier [carrier@cerias.purdue.edu]

0	f6070000	0c000102	2e000000	02000000	....	....	....	....
16	f4030202	2e2e0000	f7070000	0c000301	....	....	....	....
32	73736800	f8070000	10000701	70696466	ssh.	....	....	pidf
48	696c6500	f9070000	10000701	696e7374	ile.	....	....	inst
64	616c6c00	fa070000	14000801	636f6d70	all.	....	....	comp
80	75746572	65720000	fb070000	10000701	uter	er..	....	....
96	636c6561	6e657200	fc070000	14000a01	clea	ner.	....	....
112	696e6574	642e636f	6e660000	fd070000	inet	d.co	nf..	....
128	10000601	6c736174	74720000	fe070000	....	lsat	tr..	....
144	20000801	73657276	69636573	ff070000	...	serv	ices	....
160	10000501	73656e73	65000000	00080000	....	sens	e...	....
176	28000a01	7373685f	636f6e66	69670000	(...	ssh_	conf	ig..
192	01080000	14000c01	7373685f	686f7374	....	....	ssh_	host
208	5f6b6579	02080000	30001001	7373685f	_key	....	0...	ssh_
224	686f7374	5f6b6579	2e707562	03080000	host	_key	.pub	....
240	18000f01	7373685f	72616e64	6f6d5f73	....	ssh_	rand	om_s
256	65656400	04080000	fc020b01	73736864	eed.	....	....	sshd
272	5f636f6e	66696700	05080000	0c000301	_con	fig.	....	....
288	736c3200	06080000	dc020801	6c617374	sl2.	....	....	last
304	2e636769	07080000	2c000201	70730000	.cgi	....	,...	ps..
320	08080000	20000701	6e657473	74617400	....	...	nets	tat.
336	09080000	10000801	6966636f	6e666967	....	....	ifco	nfig
352	0a080000	0c000301	746f7000	0b080000	....	....	top.	....
368	10000801	6c6f6763	6c656172	0c080000	....	logc	lear	....
384	84020101	73000000	0d080000	78020501	....	s...	....	x...
400	6d6b7866	73000000	00000000	00000000	mkxf	s...	....	....
416	00000000	00000000	00000000	00000000	....	....	....	....