SPLIT PERSONALITY MALWARE DETECTION AND DEFEATING IN POPULAR VIRTUAL MACHINES

Alwyn Roshan Pais

Alwyn.pais@gmail.com

1

Department of Computer Science & Engineering National Institute of Technology, Karnataka

2

OBJECTIVE

To study the VM detection techniques in popular Virtual machines.

Develop strategy to counter the detection. Prevent analysis aware malwares from

detecting VM.

3

PLAN OF ACTION

Introduction VM detection techniques Detection techniques in VMware, VirtualBox

and VirtualPC. Related Work Prevent Analysis aware malwares from

detecting VM. VMDetectGuard – Tool to mask VM detection :

Windows Optimization of VMDetectGuard Results

4

INTRODUCTION

5

MALWAREMalware: It is a collective term for any

malicious software which enters the system without the authorization of the user of the system.

Anti-virus/anti-malware products do not guarantee complete protection.

6

PRESENT SCENARIO

Security researchers use malware analysis tools to build defenses against the unknown malware forms.

They then build patches for the newly discovered vulnerabilities and exploits.

Virtualization has emerged as a very promising technology.

Malware analyst use Virtual Machine Environment (VME), debuggers and sandboxes in their analysis work. 7

VIRTUALIZATION A software based representation of a

computer that executes programs in the same way as a real computer.

Examples, VMware, Virtual PC, VirtualBox. Advantages

Reduced capital and operational costs through more efficient use of hardware resources.

Simplifies maintenance . Improves scalability and deployment agility. Improves reliability.

8

BENEFITS OF VIRTUALIZATION TO SECURITY RESEARCHERS

Researchers can intrepidly execute potential malware samples without having their systems affected.

If a malware destabilizes the OS, analyst just needs to load in a fresh image on a VM.

Reduces time and cost. Increases productivity.

9

ANALYSIS AWARENESS FUNCTIONALITY

Malware developers have added a new functionality to malware.

Detect the presence of analysis tools such as VMs, debuggers and sandboxes.

Hide their malicious behavior on detection. Analysis Aware / Split Personality malware.

10

RELATED WORK

Carpenter (Carpenter et al., 2007) proposes two mitigation techniques.

They aim at tricking the malware by1. Changing the configuration settings of the .vmx

file present on the host system and,2. Altering the magic value to break the guest-

host communication channel.

11

DRAWBACKS OF THE FIRST APPROACH

The configuration options break the communication channel between guest and host not just for the program trying to detect the VM, but for all the programs.

Moreover the authors claim that these are undocumented features and that they are not aware of any side effects.

12

RELATED WORK

The work by Guizani (Guizani et al., 2009) provides an effective solution for Server-Side Dynamic Code Analysis.

Small part of the solution deals with tricking the Split Personality malware that employ Memory Detection and VM Communication Channel Detection techniques.

13

RELATED WORK

Kalpa Vishnani et. al. 2011: Masks all the detection techniques used in Vmware.

14

RELATED WORK

Other works concentrate Detecting this category of malwares Running in host machine

Save the current state quickly restore to previous state

Virtual machines in the order of market share VMware, Virtual PC, and Virtual Box.

15

VM DETECTION TECHNIQUES

Hardware fingerprinting Registry Check Process and File Check Memory Check Timing Analysis Communication Channel Check Invalid Instruction Check

16

HARDWARE FINGERPRINTING

Involves looking for specific virtualized hardware.

VMs give an abstracted view of many hardware components.

Querying for such components reveals VM presence.

For Example: BIOS, Motherboard, SCSI Controllers, USB Controllers, etc.

17

HARDWARE FINGERPRINTING RESULTS

18

REGISTRY CHECK

The registry entries contain hundreds of references to the string containing the name of the VM, Ex. “Vmware”, VirtualPC and VirtualBox.

Checking the registry values for certain keys clearly reveals the VM presence.

19

REGISTRY CHECK

For Example:

HKEY_LOCAL_MACHINE\HARDWARE\DEVICEMAP\Scsi\Scsi Port1\Scsi Bus 0\Target Id 0\Logical Unit Id 0\IdentifierVMware, VMware Virtual S1.0 HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class\{4D36E968-E325-11CE-BFC1-08002BE10318}\0000\DriverDesc VMware SCSI Controller HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Class\{4D36E968-E325-11CE-BFC1-08002BE10318}\0000\ProviderName VMware, Inc.

20

PROCESS AND FILE CHECK

Check - VM specific processes and files presence

Eg. VBoxService.exe : In VirtualBox for

synchronization with host

drivers like “vboxhook.dll” and “vpcbus” driver present in %SYSDIR%/drivers

21

MEMORY CHECK This involves looking for values of critical

operating system data structures. These data structures are relocated on a virtual

machine so that they do not conflict with the host system's copies.

Store Interrupt Descriptor Table (SIDT), Store Local Descriptor Table( SLDT), Store Global Descriptor Table (SGDT), Store Task Register (STR), Store Machine Status Word (SMSW)

Redpill.exe, ScoopyNG.exe use this method.

22

TIMING ANALYSIS

Obvious yet rare attack. Involves looking at a local Time Stamp

Counter (TSC) value. By noting down the time difference VM

presence is detected.

23

VM COMMUNICATION CHANNEL CHECK

This check involves detecting the presence of a host-guest communication channel.

IN instruction and a magic number ‘VMXh’ VmDetect.exe uses this check. Not applicable to VirtualPC and VirtualBox. Runs in VMware without exception.

24

INVALID OPCODE CHECK

Specific to VirtualPC

Uses certain opcodes for guest host communication

In host system raise exception and no exception in VirtualPC.

25

VMWARE DETECTION

hardware details motherboard serial number, graphics card and

network adapter captions

Windows Management Instrumentation (WMI) contains classes hardware, display, registry etc.

Check for VM specific strings

HARDWARE FINGERPRINTING

26

REGISTRY CHECK

Windows Registry stores configuration settings low-level operating system components Applications running

Check for Strings like “VirtualPC”, “VBOX”, “VirtualBox” value that is specific to the corresponding virtual

machine being testing on.

27

PROCESS AND FILE CHECK

Check - VM specific processes and files presence

Eg. VBoxService.exe : In VirtualBox for

synchronization with host

drivers like “vboxhook.dll” and “vpcbus” driver present in %SYSDIR%/drivers

28

MEMORY CHECK

involves looking at the values of specific memory locations

STR (Store Task Register) stores the selector segment of the TR register

(Task Register) in the specified operand (memory or other general purpose register).

Value specific in Virtual Machine

29

INVALID OPCODE CHECK

Specific to VirtualPC

Uses certain opcodes for guest host communication

In host system raise exception.

30

DETECTION OF VM RUNNING LINUX

Techniques: (tested on Vmware)

Hardware Fingerprinting

Dmesg check - prints the message buffer of the kernel

/proc file system check - interface to internal data structures in the kernel.

Communication channel check31

DMESG AND /PROC FILE SYSTEM CHECK

Dmesg - prints the message buffer of the kernel

Shows diagnostic message showing presence of hardware during boot contain strings like “VMware”,

/proc file system - an interface to internal data structures in the kernel Contains system dependent information

32

COMMUNICATION CHANNEL CHECK

IN instruction

Raises exception ““EXCEPTION PRIV INSTRUCTION” in host

Runs in VMware without exception initiates guest to host communication by calling the

“IN” instruction.

33

VMWAREDETECT

Is the proof of concept tool. It employs the various VM detection

techniques to detect the presence of VMware virtual machine. Memory Check VM Communication Channel Check Hardware Fingerprinting Registry Check Timing Analysis

34

VMWAREDETECT

35

VIRTUALMACHINEDETECT - VIRTUALPC Check using all the methods

In VirtualPC In Native Machine

Hardware Fingerprinting

BIOS American Megatrenda L900781

Graphics Card Virtual PC Integration Components S3 Trio32/64

NVDIA GeForce 310

Baseboard Manufacturer Microsoft co-orporation LENOVO

System Name VIRTUALXP User-think

USB Controller USB Virtualisation Bus Driver Intel® 5 Series /3400 …

Registry Check

SCSI: HARDWARE\\DEVICEMAP\\Scsi\\Scsi Port 0\\Scsi Bus 0\\Target Id 0\\Logical Unit Id 0

Virtual HD Hitachi HDS721050CLA362

Control class for usb : SYSTEM\\ControlSet001\\Control\\Class\\{36FC9E60-C465-11CF-8056-444553540000}\\0000

USB Virtualisation Bus Driver Intel® 5 Series /3400 …

Control class for graphics: SYSTEM\\ControlSet001\\Control\\Class\\{4D36E968-E325-11CE-BFC1-08002BE10318}\\0000

Virtual PC Integration Components S3 Trio32/64

NVDIA GeForce 310

Controlset for cd/dvd drive: SYSTEM\\CurrentControlSet\\Enum\\IDE

Disk Virtual_HD____1._1__ Registry not found

Invalid Opcode Did not raise exception Raised exception

File Check

Vpcubus Driver (Virtual USB Bus Driver) Present Not Present

Vpcgbus Driver (Virtual PC Guest Bus Driver) Present Not Present

Vpcuhub Driver (Virtual USB Hub Driver) Present Not Present

36

VIRTUALMACHINEDETECT - VIRTUALBOX

Virtual Box running windows Host Windows Machine

Hardware Fingerprinting

BIOS 0 L900781

Graphics Card Virtual Box Graphics Adapter NVDIA GeForce 310

N/W adapter AMD PCNET Family PCI Ethernet Adapter WAN Miniport(SSTP) …

Processor Null CPU1

USB Controller Std Open HCD USB Host Controller Intel® 5 Series /3400 …

Registry Check

Dsdt: : HARDWARE\\ACPI\\DSDT VBOX__ Registry not present

Scsi P0 : HARDWARE\\DEVICEMAP\\Scsi\\Scsi Port 0\\Scsi Bus

0\\Target Id 0\\Logical Unit Id 0

VBOX HARDDISK Hitachi HDS721050CLA362

Scsi P1: HARDWARE\\DEVICEMAP\\Scsi\\Scsi Port 1\\Scsi Bus

0\\Target Id 0\\Logical Unit Id 0

VBOX CD-ROM Null

Vedio Bios Version: HARDWARE\\DESCRIPTION\\System\

VideoBiosVersion

Oracle VM VirtualBox Version 4.1.2 VGA Bios Version 70.18.3E.00.05

System Bios Version: HARDWARE\\DESCRIPTION\\System\

SystemBiosVersion

VBOX-1 LENOVO-133

Instruction Check

STR (store task register) 28 0 40 00

File Check

VBOXHook.exe Present Not Present

VBOXTray Present Not Present

VBOXService.exe Present Not Present

37

VIRTUAL MACHINE DETECT

In VB

38

REMOTE DETECTION

Scenario There is access to the terminal of a system

need not be administrator access

WMIC ( Windows management instrumentation command line) is used

39

MASKING DETECTION OF VM

Using PIN API provided by Pin tool.

Can get all the instructions, the arguments and return value

Steps followed for masking Get each call made by binary. Check if matches a predefined list of calls. E.g.

RegEnumValueA Str LoadLibraryA __emit

40

MASKING DETECTION OF VM

Provide false values if VM specific values are read (matched from

predefined list) Eg.

Registry read returns the value “VBOX” Pin Tool gets the return value and modifies it in

runtime. Registry read function returns modified value

41

MASKING DETECTION OF VM

Binary does not detect – manipulated value received.

This currently supports 64 and 32 bit OS 64 and 32 bit applications

42

MASKING DETECTION OF VMLoad Binary Detect if the binary

is 64 or 32 bit.Display the detection and give option to user to change it.

Detect the OS as 64/32 bit.

Detect the Underlying VM

Virtual PC VirtualBox

Register Check masking

Invalid Opcode Check Masking

File Check Masking

File Check Masking

Register Check Masking

Instruction Check Masking

Feedback

Save to db for further analysis

Execution of loaded binary completed

43

OUR APPROACH

44

OUR APPROACH

STEP 1: Maintain a list of all the hardware as well as registry querying API calls. Also maintain a list of all the VM specific instructions such as SIDT, SLDT, SGDT, STR, IN.

45

OUR APPROACH Following is a partial list of API calls to be monitored. Hardware Querying APIs

SetupDiEnumDeviceInfo SetupDiGetDeviceInstanceId SetupDiGetDeviceRegistryProperty

Registry Querying APIs RegEnumKey RegEnumValue RegOpenKey RegQueryInfoKeyValue RegQueryMultipleValues RegQueryValue 46

OUR APPROACH

Step 2:Perform dynamic binary instrumentation of the sample under test in order to obtain its low level information as well as to intercept all the API calls made by it.

We hook into the sample under test by means of .dll injection.

This is achieved using the pin framework. 47

OUR APPROACH

Step3:Check to see if the sample under test makes a call or executes any of the monitored API calls or instructions respectively. If a match is found, set the OUTPUT to “Split Personality Malware Detected”. Also, log the activity and provide fake values to the sample so as to make it feel that it is running on a host system.

48

IMPLEMENTATION

Designed, implemented and tested VMDetectGuard.

Implemented in the framework provided by the Pin tool released by Intel Corporation.

Pin is a tool for the instrumentation of programs.

We made use of its framework to intercept the various API calls and low level instructions executed by the sample under test. 49

COUNTERING HARDWARE FINGERPRINTING Hardware emulation. APIs that query for BIOS, Motherboard,

Processor, Network Adapter. Ex. VM returns a value “none” for motherboard

serial number. VMDetectGuard returns a more appropriate string such as “.16LV3BS.CN70166983G1XF” instead.

50

COUNTERING REGISTRY CHECK

VMDetectGuard monitors registry querying APIs such as the following:RegEnumKey RegEnumValueRegOpenKeyRegQueryInfoKeyValueRegQueryMultipleValuesRegQueryValue

If the output contains the string "VMware", our tool replaces this string with a more appropriate value that would have been returned on a non virtual system. 51

COUNTERING MEMORY CHECK

SIDT, SLDT, and SGDT and STR instructions are monitored.

The values of the target registers are then changed appropriately with the values that would have been obtained on a host OS.

52

COUNTERING MEMORY CHECK

53

COUNTERING VM COMMUNICATION CHANNEL CHECK

Monitor execution of the IN instruction. We change the value of the magic number . This leads to generation of “EXCEPTION PRIV

INSTRUCTION” exception.

54

COUNTERING TIMING ANALYSIS

Instructions such as CPUID and RDTSC (Read Time Stamp Counter) are monitored.

The tool maintains a log of each type of instruction executed.

If the threshold value for a particular type of instruction is exceeded, it logs this activity too.

Sample is tricked by deleting the CPUID instruction and modifying the values of ebx, ecx, and edx.

55

VMDETECTGUARD

VMDetectGuard is our solution tool to counter Split Personality Malware.

VMDetectGuard runs in two different modes. VM Guard Mode Non VM Guard Mode

56

VMDETECTGUARD

Output Generated by VMDetectGuardResult: Split Personality malware

detected/not detected.VM Specific Log Instruction TraceSystem Call TraceRegistry TraceOpcode Mix Instruction CountDiff Tool Feature

57

VMDETECTGUARD

58

RESULTS & ANALYSIS

59

REDPILL

Red Pill is a very well known VM detection tool by Rutkowska J.

Runs a single machine language instruction SIDT and analyses its result.

60

61

SCOOPYNG

ScoopyNG is a very well known tool for VM detection developed by Klein T.

More reliable tool for VM detection in comparison to Red Pill.

It performs the following checksSIDT checkSLDT checkSGDT checkSTR check IN check (VMware communication channel)

62

63

VMDETECT

This is another well known proof of concept VM detecting sample that makes use of the VMware communication channel to detect VMware Presence.

64

65

BACKDOOR.WIN32.SDBOT.FMN

Captured this malware from the internet.

Employs Memory check and Timing Analysis mechanisms .

In the absence of VMDetectGuard: “This application cannot run under a Virtual Machine.”

In the presence of VMDetectGuard, it behaved malicious.

66

67

68

VMDETECTGUARD

Running VMDetect in VirtualPC

Running VMDetect under masking tool

69

VMDETECTGUARD

Running DetectionChecks in VirtualBox

Running DetectionChecks under masking tool

70

OPTIMIZATION

Before (sec) After (sec) % decrease in time taken

VirtualBox 167.310 112.411 32.08%

VirtualPC 294.786 205.953 30.13%

VMware 418.642 299.158 28.54%

Running Firefox binary under masking tool, in all the three virtual machines.

71

RESULTS

Tested VMDetectGuard Malwares captured from internet Proof of concept tools

The results obtained after testing is given in table.

72

RESULTSBinary Detection Technique Used Run without tool Run under tool

Virtual Box

VBDetect: calls others binaries for individual checks within.

Registry Check File and Process Check Instruction Check

Detected VirtualBox Did not detect VirtualBox

Rebhip File and Process Check Runs benignly Runs maliciously

VirtualPC

VPCDetect: calls others binaries for individual checks within.

Registry Check File and Process Check Invalid Opcode Check

Detected VirtualPC Did not detect VirtualPC

Backdoor.Win32.SdBot.fmn File and Process Check Invalid Opcode Check

Displays a message, “This application cannot run under a Virtual Machine

Ran maliciously

VMDetect Invalid Opcode Check Detects VirtualPC Does not detect VirtualPC

Trojen.Karsh-252 Invalid Opcode Check Displays a message, “This application cannot run under a Virtual Machine

Ran Maliciously

73

CONCLUSION

Split Personality malware is on a gradual rise.

Lack of academic research in this field.

There does not exist any full-fledged tool to counter Split Personality Malware.

We have designed, implemented and tested VMwareDetect, a proof of concept tool that detects the presence of Vmware.

74

CONCLUSION

We also successfully designed and implemented VMDetectGuard, a tool to counter Split Personality malware.

It detects as well as tricks the split personality binaries.

Leads to the effective analysis of malware in the virtualized environment.

Increases productivity.

75

SCOPE FOR FUTURE WORK

Further testing of more number of malware. Tool is currently built for Vmware, VPC and

VB. Providing solutions for other analysis tools

such as debugger, sandbox etc. The work currently aims at Native binaries

Can be extended to Managed binaries Extended to other operating systems.

76

REFERENCES Rutkowska J. (2004). “Red Pill”. http://invisiblethings.org/papers/redpill.html (Nov

20, 2010) Quist D, Smith V. (2005). “Detecting the Presence of Virtual Machines Using the

Local Data Table”. http://www.offensivecomputing.net/files/active/0/vm.pdf, (Nov 14, 2010)

  Klein, T. (2005) “Scoopy Doo” .

http://www.trapkit.de/research/vmm/scoopydoo/index.html (Nov 4, 2010) P. Ferrie. “Attacks on Virtual Machines”. In Proceedings of the Association of

Anti-Virus Asia Researcher Conference, 2007. Zhu D. and Chin E. (2007). “Detection of VM-Aware Malware.”

http://radlab.cs.berkeley.edu/w/uploads/3/3d/Detecting_VM_Aware_Malware.pdf (Dec 1, 2010)

Carpenter M., Liston T., Skoudis E. (2007). "Hiding Virtualization from Attackers and Malware". IEEE Security and Privacy, June 2007

Lau B, Svajcer V. (2008). “Measuring virtual machine detection in malware using DSD tracer”. In the Proceedings of Virus Bulletin, 2008

Balzarotti D., Cova M., Karlberger C., Kruegel C, Kirda E, Vigna G. (2010).“Effcient Detection of Split Personalities in Malware”. In the Proceedings of 17th Annual Network and Distributed System Security Symposium (NDSS 2010),San Diego, February 2010

77

REFERENCES VMware Inc. (2011), “VMware KB: Changing a MAC address in a Windows

virtual machine”. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008473 (Jan 15, 2010)

Pin (2004). “Pin - A Dynamic Binary Instrumentation Tool”. http://www.pintool.org/ (Jan 10, 2010)

Liston T. and Skoudis E. (2006). “On the Cutting Edge: Thwarting Virtual Machine Detection”. http://handlers.sans.org/tliston/ThwartingVMDetection_Liston_Skoudis.pdf (Nov 1, 2010)

Tiga, 2007. “Sourpill”, http://www.woodmann.com/collaborative/tools/index.php/SourPill_VM_Detector (Nov 4, 2010)

VMDetect (2005). “VmDetect, Detect if your program is running inside a Virtual Machine”. http://www.codeproject.com/KB/system/VmDetect.aspx (Jan 4, 2010)

Guizani, W., Marion, J.-Y., Reynaud-Plantey, D., & Bp, C. S. (2009). “Server-Side Dynamic Code Analysis”. Analysis, 2009

Omella A. (2006). “Methods for Virtual Machine Detection”. http://www.s21sec.com (Nov 24, 2010)

OECD, “Malicious Software (Malware): A Security Threat to Internet economy”, (2007) http://www.oecd.org/dataoecd/53/34/40724457.pdf (Oct 20, 2010)

78

Thank You!

79