All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All...

36
About Why VMI In-VM vs. Out-of-VM Hypervisor Definition All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang Lin Department of Computer Sciences The University of Texas at Dallas August 24 th , 2015

Transcript of All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All...

Page 1: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

All You Ever Wanted to Know About Virtual Machine Introspection:

Introduction

Zhiqiang Lin

Department of Computer SciencesThe University of Texas at Dallas

August 24th, 2015

Page 2: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Outline

1 About

2 Why VMI

3 In-VM vs. Out-of-VM

4 Hypervisor Definition

Page 3: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

Outline

1 About

2 Why VMI

3 In-VM vs. Out-of-VM

4 Hypervisor Definition

Page 4: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

About the Instructor

Assistant Professor at UTDallas

Founding Director of S3

Lab

Working with 6 PhDs, 4MS, 4 Undergraduate

Research is supported byDARPA, AFOSR, NSF,VMware

Goal

Building new systems and automatedtechniques to secure modern computersystems, including OS kernels and therunning software.

Research Interests

Software security (or binary codeanalysis)

Systems security, cloud computing(Virtualization, Introspection)

Memory or disk data analysis (forintrospection, digital forensics, andintrusion detection)

Page 5: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

About the Instructor

Assistant Professor at UTDallas

Founding Director of S3

Lab

Working with 6 PhDs, 4MS, 4 Undergraduate

Research is supported byDARPA, AFOSR, NSF,VMware

Goal

Building new systems and automatedtechniques to secure modern computersystems, including OS kernels and therunning software.

Research Interests

Software security (or binary codeanalysis)

Systems security, cloud computing(Virtualization, Introspection)

Memory or disk data analysis (forintrospection, digital forensics, andintrusion detection)

Page 6: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

About the Instructor

Assistant Professor at UTDallas

Founding Director of S3

Lab

Working with 6 PhDs, 4MS, 4 Undergraduate

Research is supported byDARPA, AFOSR, NSF,VMware

Goal

Building new systems and automatedtechniques to secure modern computersystems, including OS kernels and therunning software.

Research Interests

Software security (or binary codeanalysis)

Systems security, cloud computing(Virtualization, Introspection)

Memory or disk data analysis (forintrospection, digital forensics, andintrusion detection)

Page 7: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

About This Lecture

This lecture aims to provide the students with the necessaryknowledge of VMI. It starts from the basic concept, to theprinciples behind, and the enabled applications.

In particular, based on years of experiences, the instructor willdiscuss how VMI works essentially, what the challenges are,and how to develop the practical VMI tools.

Hands on labs with pre built VM with the corresponding toolsets will also be provided in this lecture.

Page 8: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

The Road Map

Guest-OS

CPU-Reg Phy-MEM H-Event Instruction

What hypervisor observes

…DISK

Guest-OSAssisted

Debug-Info Source-CodeData-structureAssisted

Binary-Code

With different constraints

e Se

man

tic G

ap

Manual Debugging-Tool Compiler-AssistedThe Binary Analysis

Approaches

What we want

…Data-Type Interrupts ExceptionsObjects Sys-call Lib-callK-events

I i K l OSMalware Analysis

Intrusion Detection

Intrusion Prevention

Kernel Integrity

Memory Forensics

OSManagement

Applications

Deployment

Guest-OS, HypervisorHardwareHypervisor Hypervisor, Hardware Guest-OS, Hardware

Page 9: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

Outline

1 About

2 Why VMI

3 In-VM vs. Out-of-VM

4 Hypervisor Definition

Page 10: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Today’s Cyber

http://to-day2.blogspot.com/2013/08/cloud-mobile-social-big-data-monetizing.html

Page 11: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Today’s Cyber

http://to-day2.blogspot.com/2013/08/cloud-mobile-social-big-data-monetizing.html

Page 12: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

About Virtualization

Hardware Layer

Virtualization Layer

Product‐VM Product‐VM Product‐VM

Linux Win‐7

..Windows XP

Virtualization (i.e.,hypervisor) [Popek and

Goldberg, 1974] has pushedour computing paradigmfrom multi-tasking tomulti-OS.

Multiplexing, Isolation,Migration, ...

Page 13: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

About Virtualization

Hardware Layer

Virtualization Layer

Product‐VM Product‐VM Product‐VM

Linux Win‐7

..Windows XP

Virtualization (i.e.,hypervisor) [Popek and

Goldberg, 1974] has pushedour computing paradigmfrom multi-tasking tomulti-OS.

Multiplexing, Isolation,Migration, ...

Page 14: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

About Virtualization

Hardware Layer

Virtualization Layer

Product‐VM Product‐VM Product‐VM

Linux Win‐7

..Windows XP

Virtualization (i.e.,hypervisor) [Popek and

Goldberg, 1974] has pushedour computing paradigmfrom multi-tasking tomulti-OS.

Multiplexing, Isolation,Migration, ...

Page 15: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

About Virtualization

Hardware Layer

Virtualization Layer

Product‐VM Product‐VM Product‐VM

Linux Win‐7

..Windows XP

Virtualization (i.e.,hypervisor) [Popek and

Goldberg, 1974] has pushedour computing paradigmfrom multi-tasking tomulti-OS.

Multiplexing, Isolation,Migration, ...

Page 16: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

On Anti-Virus Software Evolution

Operating Systems Linux Kernel

Virtualization Layer

Page 17: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

On Anti-Virus Software Evolution

Operating Systems Linux Kernel

Virtualization Layer

Page 18: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

On Anti-Virus Software Evolution

Operating Systems Linux Kernel

Virtualization Layer

Page 19: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

On Anti-Virus Software Evolution

Operating Systems Linux Kernel

Virtualization Layer

Page 20: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Virtual Machine Introspection (VMI) [Garfinkel et al, NDSS’03]

Hardware Layer

Virtualization Layer

Product‐VM Product‐VM

Linux Win‐7

Introspection

Using a trusted,dedicated virtualizationlayer program to monitorthe running VMs

Intrusion DetectionMalware AnalysisMemory Forensics

Page 21: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Virtual Machine Introspection (VMI) [Garfinkel et al, NDSS’03]

Hardware Layer

Virtualization Layer

Product‐VM Product‐VM

Linux Win‐7

Introspection

Using a trusted,dedicated virtualizationlayer program to monitorthe running VMs

Intrusion DetectionMalware AnalysisMemory Forensics

Page 22: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Virtual Machine Introspection (VMI) [Garfinkel et al, NDSS’03]

Hardware Layer

Virtualization Layer

Product‐VM Product‐VM

Linux Win‐7

Introspection

Using a trusted,dedicated virtualizationlayer program to monitorthe running VMs

Intrusion DetectionMalware AnalysisMemory Forensics

Page 23: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

Outline

1 About

2 Why VMI

3 In-VM vs. Out-of-VM

4 Hypervisor Definition

Page 24: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Security monitoring system

M(S,P) → {True,False} (1)

where M denotes the security enforcing mechanism, S denotesthe current system state, and P denotes the predefined policy.

If the current state S satisfies the security policy P, then itis in a secure state (True)If M is an online mechanism, it can allow continuedexecution; Otherwise, snapshot based approach (e.g., forforensics).

Page 25: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Security monitoring system

M(S,P) → {True,False} (1)

where M denotes the security enforcing mechanism, S denotesthe current system state, and P denotes the predefined policy.

If the current state S satisfies the security policy P, then itis in a secure state (True)If M is an online mechanism, it can allow continuedexecution; Otherwise, snapshot based approach (e.g., forforensics).

Page 26: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

In-VM Inspection

Operating Systems (In‐VM) 

AdvantagesRich Abstractions. Plentyof abstractions for in-VMmonitors to extract the OSand process state.Fast Speed. In-VM statecan be directly accessed,and in-VM enforcement canbe instantly executedwithout any trapping intohypervisor.

DisadvantagesThe security monitoringsystem can be disabledFalse State can begenerated to mislead thedetection. Log files, procfilesThe Security Policy and TheSecurity EnforcingMechanism can betampered

Page 27: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

In-VM Inspection

Operating Systems (In‐VM) 

AdvantagesRich Abstractions. Plentyof abstractions for in-VMmonitors to extract the OSand process state.Fast Speed. In-VM statecan be directly accessed,and in-VM enforcement canbe instantly executedwithout any trapping intohypervisor.

DisadvantagesThe security monitoringsystem can be disabledFalse State can begenerated to mislead thedetection. Log files, procfilesThe Security Policy and TheSecurity EnforcingMechanism can betampered

Page 28: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

In-VM Inspection

Operating Systems (In‐VM) 

AdvantagesRich Abstractions. Plentyof abstractions for in-VMmonitors to extract the OSand process state.Fast Speed. In-VM statecan be directly accessed,and in-VM enforcement canbe instantly executedwithout any trapping intohypervisor.

DisadvantagesThe security monitoringsystem can be disabledFalse State can begenerated to mislead thedetection. Log files, procfilesThe Security Policy and TheSecurity EnforcingMechanism can betampered

Page 29: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Out-of-VM Introspection

Operating Systems

Out‐of‐VMOut of VM

Advantages

Strong Isolation (TamperResilient)Transparent DeploymentComplete ViewHigh Cost-SavingLess Vulnerability

DisadvantagesNo Abstractions. No guestOS abstractions, no systemcalls, no file descriptor, novariables.Slow Speed. Additionaladdress translation, andworld switching.

Page 30: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Out-of-VM Introspection

Operating Systems

Out‐of‐VMOut of VM

Advantages

Strong Isolation (TamperResilient)Transparent DeploymentComplete ViewHigh Cost-SavingLess Vulnerability

DisadvantagesNo Abstractions. No guestOS abstractions, no systemcalls, no file descriptor, novariables.Slow Speed. Additionaladdress translation, andworld switching.

Page 31: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Out-of-VM Introspection

Operating Systems

Out‐of‐VMOut of VM

Advantages

Strong Isolation (TamperResilient)Transparent DeploymentComplete ViewHigh Cost-SavingLess Vulnerability

DisadvantagesNo Abstractions. No guestOS abstractions, no systemcalls, no file descriptor, novariables.Slow Speed. Additionaladdress translation, andworld switching.

Page 32: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

Outline

1 About

2 Why VMI

3 In-VM vs. Out-of-VM

4 Hypervisor Definition

Page 33: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Hypervisor Definition: Bare metal

Type 1 (bare metal) hypervisors, which run directly on thehost’s hardware to control the hardware and monitor the guestOS. Typical examples of such hypervisors include Xen,VMware ESX, and Microsoft Hyper-V.

Page 34: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Hypervisor Definition: Hosted

Type 2 (hosted) hypervisors, which run within a traditional OS.That is, a hosted hypervisor adds a distinct software layer atopthe host OS, and the guest OS becomes a third software layerabove the hardware. Well-known examples includeKVM/VMware Workstation/VirtualBox/QEMU.

Page 35: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Hypervisor Definition: Native

Native hypervisors that directly push the guest code toexecute natively on the hardware with hardware virtualization.

Page 36: All You Ever Wanted to Know About Virtual Machine Introspection: Introduction · 2015-12-19 · All You Ever Wanted to Know About Virtual Machine Introspection: Introduction Zhiqiang

About Why VMI In-VM vs. Out-of-VM Hypervisor Definition

Hypervisor Definition: Emulation

Emulation hypervisors that translate each guest instructionfor an emulated execution with software virtualization.