Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian...
Transcript of Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian...
![Page 1: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/1.jpg)
Research Overview: Virtualization-Based Malware Defense
Assistant ProfessorDepartment of Computer Science
N.C. State University4/21/2009
Xuxian Jiang
![Page 2: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/2.jpg)
Outline
q Motivations and research overviewq Virtualization-based malware defenseQ New virtualization mechanism: OBSERVQ New capabilities enabledv Invisible system loggingv Stealth malware detectionv OS Kernel integrity protection
q Future workq Summary
![Page 3: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/3.jpg)
q Internet malware remains a top threatQ Malware: viruses, worms, rootkits, spyware, bots…
Motivations
![Page 4: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/4.jpg)
Research Goals
q Research goalsQ Malware-free Cyberspace (long-term)Q Gaining the upper hand over malware (short-term)
![Page 5: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/5.jpg)
Honeyfarm
Virtualization Technology
Research Agenda
Malware PlaygroundMalware Profiling &
Protocol Reverse Engineering
Malware Contamination Tracking*
Past Present Future
USENIX Sec’04, NDSS’06, JPDC’06
TPDS’07, ICDCS’06
RAID’05 NDSS’08, RAID’08,WORM’06
Kernel/HypervisorRootkit Defense
Botnet Defense*
Other Applications
OBSERV Mechanism & Applications (CCS’07, RAID’07)*
![Page 6: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/6.jpg)
q State-of-the-art malware defenseQ Running anti-malware software inside the monitored
systemv Advantage: They can see everything (e.g., files,
processes…)v Disadvantage:
VirusScanFirefoxIE
OS Kernel
…
Why OBSERV?
They may not see anything!
![Page 7: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/7.jpg)
Why OBSERV?q Current approach fundamentally flawedQ Malware running in the same system space with anti-
malware software at the same privileged levelQ No clear winner in the arms race between them
q Solution: Going out of the box
FirefoxIE
OS Kernel
…
VirusScan
Virtual Machine Monitor (VMM)
![Page 8: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/8.jpg)
The “Semantic-Gap” Challenge
q What we can observe:Q Low-level states v Memory pages, disk blocks…
Q Low-level eventsv Privileged instructions,v Interrupts, I/O…
q What we want to observe:Q High-level semantic statesv Files, processes…
Q high-level semantic eventsv System calls, context switches…
Virtual Machine Monitor (e.g., VMware, Xen)
Guest OSSemantic Gap
VirusScan
![Page 9: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/9.jpg)
Our Solution: OBSERVq OBSERV: “Out-of-the-Box” with SEmantically
Reconstructed ViewQ A new mechanism missing in all current VMMs
FirefoxIE
OS Kernel
…
Virtual Machine Monitor (VMM)
OBSERV
![Page 10: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/10.jpg)
New Capabilities
Capability II: Malware detection by
view comparison
Capability I: Invisible system logging
FirefoxIE
OS Kernel
…
Virtual Machine Monitor (VMM)
OBSERV
protection
Capability III: OS kernel integrity
protection
OBSERV View In-the-boxView Diff
![Page 11: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/11.jpg)
OBSERV: Bridging the Semantic Gap
q Step 1: Procuring low-level VM states and eventsQ Disk blocks, memory pages, registers…Q Traps, interrupts…
q Step 2: Reconstructing high-level semantic viewQ Files, directories, processes, and kernel modules…Q System calls, context switches…
VM Introspection
Guest View Casting
![Page 12: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/12.jpg)
Step 1: VM Introspection
Raw VMM Observations
Virtual Machines (VMs)
VMware Academic Program
VM disk image
VM hardware state (e.g., registers)
VM physical memory
VM-related low-level events (e.g., interrupts)
![Page 13: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/13.jpg)
Step 2: Guest View Casting
Virtual Machine Monitor (VMM)
Guest OS
Key observation: The guest OS provides all semantic “templates” of data structures and functions to reconstruct VM’s semantic view
OBSERVSemantic Gap
![Page 14: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/14.jpg)
Guest View Casting
Raw VMM Observations Casted Guest Functions & Data Structures
Reconstructed Semantic View
Device drivers, file system drivers
Memory translation,task_struct, mm_struct
CR3, MSR_SYSENTER_CS,MSR_SYSENTER_EIP/ESP
Event semanticsSyscalls,
context switches, ....
Event-specific arguments…
VM disk image
VM hardware state (e.g., registers)
VM physical memory
VM-related low-level events (e.g., interrupts)
![Page 15: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/15.jpg)
Guest View Casting on Memory StateProcess List
Process Memory Layout
![Page 16: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/16.jpg)
Guest Memory Addressing
q Traditional memory addressingQ MMU translates VA to PAQ OS image mapped to known PAv Linux: VA 0xc0000000 == PA 0x0v Windows: VA 0x80000000 == PA 0x0
q VM complicates the translation
Q Guest virtual -> guest physical
Q Guest physical -> host physicalVM IntrospectionReverse Address Translation
Emulated Address Translation
Kernel Symbols
![Page 17: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/17.jpg)
Trap
Guest View Casting on System Calls
q System call instructionsQ int 0x80; sysenter
q System call conventionQ EAX, EBX, ECX, EDX, ESI, EDI, EBP, …
Instr Handler
1. int 0x80, sysenter
2. Trap Generation
5. Continue the Execution
Trap Handler
3. Trap and Emulate
Guest
VMM
User
Kernel
4. Emulate Instruction
![Page 18: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/18.jpg)
Related Workq Virtual machine introspection (Livewire[Garfinkel03],
IntroVirt[Joshi05], HyperSpector[Kourai05])Q Focusing on targeted attacks for specialized
IDSesq Secure monitors (CoPilot[Petroni04], Terra[Garfinkel03],
sHype[Sailer05], SecVisor[Perrig07])Q Missing a basic mechanism similar to OBSERV
![Page 19: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/19.jpg)
Outline
q Motivations and research overviewq Virtualization-based malware defenseQ New VMM mechanism: OBSERVQ New capabilities enabledv Invisible system loggingv Stealth malware detectionv OS Kernel integrity protection
q Future workq Summary
![Page 20: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/20.jpg)
New Capability I: Invisible System Loggingq Trusted logging: an essential function for
honeypotsq Two current approaches Q External (e.g., tcpdump, ethereal, etc)v Only monitoring network traffic
Q Internal (e.g., sebek, syslog, etc)v Can be compromised!
Internal
External
Tamper-Resistance
Deep Inspection
![Page 21: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/21.jpg)
q Sebek: de-facto honeypot logging toolq Can be detected, disabled, or bypassed by
NoSEBrEaK [Holz+, BlackHat’04/Defcon 12]
Invisible System Logging
![Page 22: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/22.jpg)
q Demo Clip (2.5 minutes):Q http://www.cs.ncsu.edu/faculty/jiang/research/vms
cope/sebek.swf
Invisible System Logging
[Holz+, Blackhat’04/Defcon 12]
OBSERV-based logging
Sebek-based logging
![Page 23: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/23.jpg)
Invisible System Logging
Opera profile
Firefox profile
![Page 24: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/24.jpg)
New Capabilities II & III
Capability II: Malware detection by
view comparison
Capability I: Invisible system logging
FirefoxIE
OS Kernel
…
Virtual Machine Monitor (VMM)
OBSERVprotection
Capability III: OS kernel integrity
protection
OBSERV View In-the-boxView Diff
![Page 25: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/25.jpg)
q Experiment setupQ Guest VM: Windows XP (SP2)v Windows Fu rootkit
Q Host OS: Scientific Linux 4.4Q VMM: VMware Server 1.0.1
View Comparison on Volatile Memory State
“In-the-box” viewOBSERV view
Diff
![Page 26: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/26.jpg)
q Experiment setupQ Guest VM: A Redhat 7.2-based honeypotv Linux SHv4 rootkit
Q Host OS: Windows XP (SP2)Q VMM: VMware Server 1.0.1
View Comparison on Persistent Disk State
“In-the-box” viewOBSERV view
Diff
Symantec AntiVirusSymantec AntiVirus
![Page 27: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/27.jpg)
![Page 28: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/28.jpg)
External Run of COTS Anti-Malware Softwareq Experiment setupQ Both guest OS and host OS run Windows XP (SP2)Q VMM: VMware Server 1.0.1
q Running Symantec AntiVirus twiceQ InsideQ Outside
Hacker Defender
NTRootkit
![Page 29: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/29.jpg)
External Scanning Result
Internal Scanning Result
Diff
![Page 30: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/30.jpg)
OBSERV Capability III: OS Kernel Integrity Protection
q High-assurance OS kernel Q No malicious kernel codeQ No kernel rootkit attacks
q Two main tasks:Q Tracking run-time kernel code layoutQ Enforcing the following propertiesv Only loading authenticated kernel codev Only executing authenticated kernel code
R. Riley, X. Jiang, D. Xu, "Guest-Transparent Prevention of Kernel Rootkits with VMM-based Memory Shadowing", RAID’08, Boston, MA, September 2008
![Page 31: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/31.jpg)
OBSERV
NICKLE: “No Instruction Creeping into Kernel Level Executed”
NICKLE
Standard memory
Kernel Code
Shadow memory
VMM
Guest OS
q Step 1: Create two memory spacesQ Standard memoryQ Shadow memory
q Step 2: Authenticate and copy kernel code to shadow memory
q Step 3: Memory access dispatchQ Kernel code fetch ->
shadow memoryQ All other accesses ->
standard memoryKernel Code
![Page 32: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/32.jpg)
Demonstration of Effectiveness
Successfully preventing 23 real-world kernel rootkits!
![Page 33: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/33.jpg)
Other OBSERV-enabled Capabilities
q Tamper-resistant malware profiling and analysisQ Contamination tracking [TPDS’07, ICDCS’06]Q Protocol reverse engineering [NDSS’08, WORM’06]
q “Out-of-the-box” policy enforcement [SACMAT’07]
q Other opportunities
![Page 34: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/34.jpg)
Future Work
Honeyfarm
Virtualization Technology
Malware PlaygroundMalware Profiling &
Protocol Reverse Engineering
Malware Contamination Tracking*
Past Present Future
USENIX Sec’04, NDSS’06, JPDC’06
TPDS’07, ICDCS’06
RAID’05 NDSS’08, RAID’08, WORM’06
Kernel/Hypervisor Rootkit Defense
Botnet Defense*
Other Applications
OBSERV Mechanism and Applications (CCS’07, RAID’07)*
![Page 35: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/35.jpg)
Rootkit Defenseq Reality: rampant rootkits
Source: McAfee Avert Lab Report (April 2006)
400% growth400% growth
Q1 of 2005
700% growth700% growth
Viruses/worms/bots, …
![Page 36: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/36.jpg)
Rootkit Defense -- Challenges
q A fundamental questionQ How to grab the upper hand?
q ChallengesQ How to secure the lowest level access?v Rethinking VMM design
Q How to defeat rootkit infection?v Rethinking OS kernel design (e.g., NX protection)
Q How to balance protection and performance?v Rethinking guest OS-VMM interactions
![Page 37: Research Overview: Virtualization-Based Malware Defense · N.C. State University 4/21/2009 Xuxian Jiang. Outline qMotivations and research overview ... QInternal (e.g., sebek, syslog,](https://reader035.fdocuments.us/reader035/viewer/2022071006/5fc3cdb03a969d2f8b0ca381/html5/thumbnails/37.jpg)
Summaryq OBSERV enables “out-of-the-box” malware defenseQ Eliminating semantic gapQ Enabling new malware defense capabilitiesQ A step towards malware-free Cyberspace
FirefoxIE
OS Kernel
…
Virtual Machine Monitor (VMM)
OBSERV OK kernel integrity protection
Invisible system logging
Malware detection by view comparison