1

Dave SchippersTechnology Director

Hudsonville Public Schools10/16/2009

dschippe@hpseagles.nethttp://hudsonvillepublicschools.org

Digital Forensics:Internal and Formal Investigations

2

Agenda• Basic Digital Forensics Information• Differences in Staff and Student

Investigations• Real Life Examples

– Researching an incident - Internal Case– Researching an incident- Formal Case

• Conclusion

3

Where does data reside?

• Data can exist in many places– Workstation– Network (firewalls, etc.)– Servers– Internet (webpages, newsgroups, ftp servers,

etc.)

4

Digitial Evidence - Internet Browsers

• History• Browser Cache• Login names & passwords• Authenticated Sessions• Cookies

5

Digitial Evidence - Unix/Linux Systems

• acct - every command typed by every user • lastlog - lists each user's most recent login• loginlog - records failed logins• syslog/messages - often contain a large

amount of informaiton

6

Digitial Evidence - Unix/Linux Systems

• sulog - attempts or login as administrator/superuser

• utmp/utmpx - all users currently logged in• wtmp/wtmpx - all past and current logins and

records system startups and shutdowns• vold.log - external media media errors• xferlog - files transferred using FTP• Applies Mac OS X - based on Unix

7

Digitial Evidence - Windows Systems

• appevent.evt - applications usage• secevent.evt -activities with security

implications(ie logins)• sysevent.evt - system events (ie shutdown)• event viewer - recent events (usually clears

in hour)

8

Digital Evidence - Temp/Swap Files

• Temporary Files- Files that the OS stores temporarily on the hard drive

• Swap Files/Virtual Memory - Files moved from RAM/Memory when additional Memory space is needed

9

File Deletion - FAT & NTFS

• Deleting a file on Windows deletes the reference to the file, not the file itself.

• Deleted partitions can also be scanned and files recovered if they have not been overwritten.

• Quick Recovery FAT and NTFS is a do-it-yourself Non-Destructive NTFS Data Recovery Software..

10

File Deletion

• Shadow Files– After files are deleted, if the write head does

not overwrite exactly over the previous file, it is possible to extract at least portions of it.

11

Digital Evidence-Network

• Packet sniffers can be utilized to listen to traffic on a network when attempting to monitor or gather real time information– Most effective on wireless networks– Wired network packet sniffing requires port

cloning in switched networks

12

Digital Evidence - Networks

• Server logs• Server files• Firewall logs• General Logs

13

As digital evidence is found it should be:

• Collected• Documented• Preserved• Classified• Compared with other samples• Individualized

14

Considerations

• Copy Evidence without Changing it • Must Corroborate Event

15

Student vs StaffInvestigations

16

• Student– Internet Violations

• Inappropriate Content (music, pictures etc.)• Copyright Violations

• Staff– AUP Violations - Assume Outside Review– Outside Review - Civil or Criminal

16

17

Record Examples

• Firewall Record -

• Ssh session results-

Apr 29 2009 16:14:06: %PIX-5-304001: 10.162.70.50 (unresolved)  Accessed URL 74.125.95.101 (iw-in-f101.google.com) :/videosearch?q=sexy%20girls%20naked%20dansn&hl=en&emb=0&output=results

hto-501-01:~ administrator$ ssh root@10.162.70.50The authenticity of host '10.162.70.50 (10.162.70.50)' can't be established.RSA key fingerprint is ba:75:50:59:21:40:9d:88:a5:ec:a3:cc:5c:bc:b1:df.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added '10.162.70.50' (RSA) to the list of known hosts.Password:BBB-107-01:~ root# lastroot ttys000 10.162.198.11 Fri May 1 13:21 still logged inbrubble console Fri May 1 10:43 still logged inbrubble console Fri May 1 08:24 - 09:44 (01:19)reboot ~ Fri May 1 08:21 shutdown ~ Thu Apr 30 14:42 brubble console Thu Apr 30 08:14 - 14:41 (06:27)reboot ~ Thu Apr 30 08:13 shutdown ~ Wed Apr 29 16:17 brubble console Wed Apr 29 08:18 - 16:16 (07:58)reboot ~ Wed Apr 29 08:17 shutdown ~ Tue Apr 28 15:31

18

Investigation Examples

19

Email Complaint• Contacted on Feb 6th, 2005 by Betty• Betty insistent an email was authored without her

creating it. • Betty contacted a tech (Fred) on Feb 3rd. • Fred assured Betty that I had been involved and

that Wilma had created the email• (At the bottom of the email that was created

without her consent, Fred’s auto signature had been added.)

20

Investigation• After the initial contact, the assumption - Sys

Admin (Fred) accessing other accounts.• The email was reviewed

– Fred’s signature was on the email– Statement concerning Fred’s ex-wife Wilma

• The auto-signature could have been added by hand as well as the Groupwise client

• Investigation Status - “Not Actionable”

21

Investigation• Groupwise Server maintains a log of all user login

access with IP, Date & Time• Log File Example• 06:13:31 3F7 C/S Login mac ::GW Id=fflinstone ::

10.162.138.52 (Building1 IP)• 06:47:46 2FA C/S Login Web ::GW Id=fflinstone ::

24.11.18.198[10.162.5.1] (Outside IP)

22

Investigation

• Logs showed Fred’s office computer accessed other acounts.

• People could have used his computer.• More examination necessary• Investigation Status - “Not Actionable”

23

Investigation

• Further log research over multiple days showed:– Fred and the other accounts being accessed

from an outside IP– The outside IP was tracked back to a local ISP.– We contemplated asking the ISP to provide the

user information, but presumed a court order would be necessary

• Investigation Status - “Not Actionable”

24

Investigation

• Interviewing some of the system’s administrators revealed that Fred had called George stating that someone had gotten his login and password.

• George connected via VPN and changed Fred’s password after hours.

• Investigation Status - “Not Actionable”

25

Investigation

• Log review focused on the approximate dates of the account reset

• Logs revealed:– On 02/03/2005-George reset Fred’s password

and Fred immediately logged in using the outside IP

– On 02/22/2005-Unauthorized access corresponds to the same outside IP

• Investigation Status - “Actionable”

26

Investigation Recap• 56 occurrences of unauthorized access• 6 accounts were accessed from an

unauthorized source• 12 occurrences of unauthorized access

occurred from an outside IP adress• 44 occurrences of unauthorized access

occurred from within the organization• Every occurrence matches access from the

IP address as Fred’s login.

27

Investigation Lessons• All user accounts had been compromised

– Fred has access to all user accounts logins and passwords

– Email was a digital “forgery” not created by the user

• Recommendations:– Restructure sys admin account access to user

accounts (limit high level user’s to a very limited group of sys admins)

– Reset all passwords organization wide– Dismiss Fred based on Acceptable Use Policy

violations

28

Filter Testing Case

• Suspicious hits - on the firewall log - different issue - real time.

• Inappropriate content hits • Backtracked - laptop - presumed to be a

student. • Deciding Factor - laptop was a staff

checkout.• Recap - potential staff surfing inappropriate

content during work time28

29

Filter Testing Case

• SSH’d into computer - generic userid & deleted Internet history

• Searched Firewall & DHCP logs - multiple Google searches for inappropriate content

• Searches continued for a week and stopped - beginning of summer break

• Possibility staff member was a principal• Investigation Status - “Not Actionable”

29

30

Filter Testing Case

• Interesting Tidbits - – Principal’s laptop had been stolen in a break-in– Assigned temporary check-out– Primary laptop returned from repair -

• Contents could have created by thieves - content checks unreliable

• Informed building tech - potential issue of a staff member - informed principal

• Job Loss - External Ramifications

30

31

Filter Testing Case• After summer break -

– searches began - normal laptop used -generic userid– Internet history being cleared - session frequency lower

• Used firewall logs, remote desktop and SSH for research– Millions of firewall records to search– Distinct pattern of search terms utilized arose to avoid

filter blocks - elaborate combinations & searches– Eastern European sites were being hit– Anybody know how to block foreign words?

31

32

Filter Testing Case

• Created Digital Image– Matched several website visits - primary -

checkout– Repeating pattern of searches & sites - both

• Contacted Super– Possible Staff Member Issue– Identified staff member– Indicated we needed more time/evidence - job

impact

• Investigation Status - “Not Actionable”32

33

Honey Pot

• Requested -– Utilize LanRev - Screen Grabs - User Picture– Granted Screen Grabs - Assigned Desktop &

Laptop– User Logs In

• Every X time a screen grab is taken w/Time Stamp & placed on server for review

33

34

Honey Pot

• Grabs ran a week– Gathered a Sunday evening login - – Assigned Desktop with document creation

under user’s ID– Log out– Login with a minute on laptop - generic user– Inappropriate activity

• Investigation Status - “Actionable”

34

35

Investigation Recap

• Staff Member– Resigned– LEO investigated for legal implications– Staff member cleared

35

36

Lessons I’ve Learned• Process/protocols

– Potential Student Issues• 2 Sys Admins Receive Alerts• Principal Notification Process

– Potential Staff Issues• 2 Sys Admins Receive• 2 Supers can handle

– In case one goes bad

• Assume all issues will go public/litigation– Sys Admins reviewed by LEO

– Student & Staff Issues• Log & Track by UserID

36

37

Lessons I’ve Learned• Internet Content Filtering

– Firewall Email Alerts• Expect ~10,000 notifications• ~95-99% False Positive• All traffic should be monitored• Naughty Notifications spike the day before vacations

• AUPs– Account for what is/is not acceptable– Sys Admins should sign an AUP/filed– Never Compromise Sys Admin Expectations

37

38

Lessons I’ve Learned

• Understand the differences between – Males– Females

38

39

Lessons I’ve Learned

• Rational & respected people do irrational things.

• They do them even when they are warned• Be prepared

39

40

Questions