Secure and Flexible Monitoring of Virtual Machines

Bryan D. Payne Martim D. P. de A. Carbone Wenke LeeGeorgia Institute of Technology

{bdpayne,mcarbone,wenke}@cc.gatech.edu

Abstract

The monitoring of virtual machines has many applica-tions in areas such as security and systems management. Amonitoring technique known as introspection has receivedsignificant discussion in the research literature, but theseprior works have focused on the applications of introspec-tion rather than how to properly build a monitoring archi-tecture. In this paper we propose a set of requirements thatshould guide the development of virtual machine monitor-ing solutions. To illustrate the viability of these require-ments, we describe the design of XenAccess, a monitoringlibrary for operating systems running on Xen. XenAccessincorporates virtual memory introspection and virtual diskmonitoring capabilities, allowing monitor applications tosafely and efficiently access the memory state and disk ac-tivity of a target operating system. XenAccess’ efficiencyand functionality are illustrated through a series of perfor-mance tests and practical examples.

1 IntroductionOn the wave of its renewed popularity, researchers are

identifying many new applications that leverage the abstrac-tions and isolation provided by virtualization. One area thathas received significant attention is security. Security ap-plications benefit from virtualization by running in isolatedvirtual machines (VMs) and building smaller trusted com-puting bases (TCBs). This technique has been used in avariety of research projects [21, 9, 10, 18, 7, 15, 2].

Among the security research that uses virtualization, anincreasingly common practice is to setup a secure VM thatis used to monitor the other VMs running on the same vir-tual machine monitor (VMM)1. These monitors are used inintrusion detection systems (IDS), integrity checking, hon-eypot systems and forensic analysis, among others. We be-lieve that this idea is sound, yet many previous efforts inthis space have focused more on the applications of intro-

1Throughout this paper, the term VMM refers to the virtual machinemonitor on top of which VM’s run, whereas the term monitor refers to theactual VM monitoring applications that we focus on in this work.

spection than building a proper architecture to support thetechnique.

In this paper, we present the XenAccess monitoring li-brary and our lessons learned from the implementation pro-cess. XenAccess provides virtual memory introspection andvirtual disk monitoring capabilities. Our experience de-signing and implementing this library has shown that im-plementing introspection in a secure and efficient manneris non-trivial. However, our architecture demonstrates howone can achieve these goals without losing monitoring func-tionality. Monitoring with XenAccess requires no changesto the VMM or to the VM being monitored. In addition,no changes are required to the OS being monitored, so Xe-nAccess is not restricted to monitoring open source OSes.While our current implementation focuses on monitoringXenoLinux the XenAccess Library can be extended to mon-itor any OS that runs on the VMM. XenAccess incurs anegligible performance penalty for typical monitor appli-cations.

We designed the XenAccess architecture based on sixhigh-level requirements. In a general sense, these require-ments can be seen as typical good programming guidelines,or good security guidelines. For example, some of our re-quirements could be seen as specialized versions of Saltzerand Schroeder’s classic security design principles [28]. Thisis intentional, as our goal was to leverage known designprinciples in order to build a robust monitoring architec-ture. With this in mind, we identify the following six re-quirements for monitoring VMs:

1. No superfluous modifications to the VMM. TheVMM should remain as small and simple as possiblesince it is part of the TCB. If a VMM includes the nec-essary primitives to support the monitoring architec-ture, then it should not be modified. If a VMM lacksthe necessary primitives, then the modifications madeshould be what is minimally required to support themonitoring architecture.

2. No modifications to the VM or the target OS. Mod-ifications to the target OS (i.e., the OS being moni-tored), are problematic. The target OS can tamper with

this code, and changes to the target OS may require ac-cess to the target OS source code, which is not alwaysavailable. One of the key reasons why virtualization isattractive for monitoring is the isolation between VMs.Placing monitoring code within the same OS that isbeing monitored bypasses this isolation, negating thiskey benefit. Therefore, this requirement encourages allmonitoring code to remain in an isolated VM unlesssuch a restriction makes it impossible for a monitor togather the necessary information.

3. Small performance impact. An excessive perfor-mance impact can render a monitoring architectureworthless. This requirement ensures that the monitor-ing architecture does not prevent the target OS fromperforming its intended functions. The performanceimpact is measured as any reduction in performance ofan application caused by the monitoring software. Ide-ally this impact is both small and consistent, but someinitialization costs may be required.

4. Rapid development of new monitors. New monitorsmay be needed to address new types of attacks. Fur-thermore, it is advantageous to keep the monitor codesimple to limit the opportunity for introducing errorsinto the monitors. The monitoring architecture shouldprovide APIs that are used to develop new monitors.Therefore, satisfaction of this requirement means thatthe APIs should be designed in a way that simplifiesthe job of the monitor developer.

5. Ability to monitor any data on target OS. Monitorsshould have a full view into the target OS. The moni-toring architecture should not be limited to providinginformation about a small part of the target OS. Forexample, an ideal memory monitor should be able toview all memory on the target OS. Likewise, an idealdisk monitor should be able to view all data going toand from the disk device. While this ideal may not al-ways be possible, the more information a monitor canview, the harder it is for an attacker to evade detection.

6. Target OS cannot tamper with monitors. If the tar-get OS can tamper with the monitors, then the pos-sibility exists for malicious code to tamper with themonitors. For this reason, all of the monitors shouldbe isolated or protected from the target OS. This is re-lated to requirement (2), above. However, here we re-quire that all monitor code, regardless of its location,be protected from attack. If all monitor code is in anisolated VM, then this is not difficult. If some moni-tor code must be placed outside of the TCB, then addi-tional measures must be taken to protect that code. Theextent of these measures will depend on the nature ofthe code being protected.

Our main contribution is the XenAccess monitoring ar-chitecture that satisfies the above requirements. It is im-portant to emphasize that this paper addresses an architec-ture for security and not actual security techniques (such asIDS algorithms). These are topics for future papers. Theremainder of this paper focuses on the XenAccess archi-tecture, its implementation, and some example applicationsthat demonstrate the performance and flexibility of XenAc-cess. Section 2 discusses the related work. Section 3 pro-vides background information on the components in Xenused to build XenAccess. Section 4 presents the architec-ture and implementation details for XenAccess. Section5 shows the results of our performance testing along withsome example applications. Section 6 discusses future di-rections in this research space and we conclude with Section7.

2 Related WorkVMMs first came into use over 35 years ago [11]. While

Madnick and Donovan identified the security benefits ofVMMs in the early 70s [20], research that explicitly lever-aged these benefits did not take place until nearly 20 yearslater [17, 16]. More recently, virtualization is being used indifferent ways to address a variety of systems management,and security problems. In the security space, we have seeninnovative work in intrusion detection systems [10, 18, 15],workload isolation [21, 9], attack investigation and debug-ging [7], and system monitoring [13, 2]. Each of these ap-plications have one thing in common: they each require theability to monitor data from a target OS. However, the me-chanics of how to properly do such monitoring have notbeen adequately addressed in the literature. Through thedetails provided in this paper, and by making XenAccess anopen source project, we are exposing these mechanics forthe benefit of future work in this space.

The technique of virtual memory introspection was in-troduced by Garfinkel and Rosenblum [10]. While thiswork laid out how introspection could be used to build anintrusion detection system, the underlying mechanics of in-trospection were not discussed. Joshi et al presented a sys-tem called IntroVirt [15] that uses introspection and replayto test if a system was previously attacked through a knownvulnerability. Similar to the first effort, only limited detailswere given regarding the introspection mechanism. Morerecently, several projects have provided details about theirintrospection techniques, only to reveal suboptimal securitydecisions in their architecture. The Hyperspector project[18] is a virtual distributed monitoring environment usedfor intrusion detection. The Hyperspector approach to in-trospection is to provide access to a few specific pieces ofinformation (processes, sockets, etc). This limited view intothe target OS violates property (5) of our requirements for arobust monitoring solution, and Hyperspector also violates

property (1) by extensively modifying the VMM, and (6) bysharing OS kernels between VMs. Asrigo et al presented asystem for monitoring honeypots [2], but they violate prop-erty (2) by requiring hooks in the target OS kernel, property(3) by causing a substantial performance impact, and prop-erty (4) by incorporating kernel code in new monitor hooks.Finally, the Antfarm system [13] tracks only OS-level pro-cesses, violating property (5), and performs the monitor-ing from within the VMM, violating property (1). Each ofthese virtual memory introspection systems were built toprovide monitoring capabilities for a security system. How-ever, none of these systems meet our six requirements for amonitoring solution, making it much more likely for an in-truder to compromise, evade or disable the monitors.

Monitoring in a virtualized environment is not the onlyapproach. Petroni et al developed Copilot [23], a secure co-processor used to monitor the memory of a host. In practice,this approach is very similar to virtual memory introspec-tion from a VM, but it requires extra hardware and cannotbe generalized to monitoring other data such as disk I/O.Looking into the commercial world, many monitoring ap-plications sold today simply run within the target OS. Forexample, anti-virus software typically runs in the same OSthat it is protecting. However, this architecture is flawed be-cause malicious software can simply disable the anti-virussoftware [3].

Monitoring at the disk level has traditionally taken placeas part of a research trend focused on creating smarter,more semantically-aware devices. This has applications inboth systems optimization and security. Sivathanu et alhas shown how smart disks can employ gray-boxing tech-niques [1] to infer the semantics of the underlying filesys-tem and use this knowledge to enable various performanceimprovements and features like secure file deletion [29].Researchers at Carnegie-Mellon University have leveragedthe physical isolation of such systems to enable intrusiondetection [22, 12] and recovery capabilities [30]. These sys-tems are able to perform their functions in a tamper-resistantmanner, regardless of an OS compromise. This approach,however, has the obvious downside of requiring additionalhardware support and the need for a special infrastructurefor communication between the management tools insidethe OS and the disk IDS. XenAccess leverages virtualiza-tion to provide the same level of monitoring functionalitywithout either of these limitations.

More recently, disk monitoring has started to receiveattention in the context of virtual machines. Hyper-Spector’s approach is to mount a shadow version of themonitored filesystem and execute integrity checkers (e.g.,tripwire). Not only does this require significant modifi-cations to the VMM, violating property (1) and increasingthe chances of a VMM compromise; it also limits access tothe disk data by providing an exclusively static and high-

level view of it, violating property (5) and making it veryeasy to evade the monitor. Elango et al [8] and Jones et al[14] have applied some of the principles of semantically-smart disk systems and gray-boxing [29, 1] to the perfor-mance improvement of Xen virtual machines. Their resultsshow how monitoring and active control of virtual machinescan have a wide variety of applications outside the securityarea.

XenAccess is designed to work with Xen [4], but theideas of virtual memory introspection and disk monitoringare not unique to Xen. Our architecture could be ported toany of today’s virtualization solutions. In the past, manyresearchers choose to work with User Mode Linux (UML)[6], a virtualization solution that allows you to boot a Linuxkernel as a process in a running version of Linux. The ear-liest work with introspection used VMWare [31], a full fea-tured commercial virtualization product. Looking forward,interest is now growing in the kernel-based virtualizationdriver (KVM) [24] that is built into the Linux kernel startingwith version 2.6.20. While our techniques are viable on anyof these platforms, a virtualization solution designed as anindependent and lightweight software layer running directlyon the hardware, such as Xen, offers a solid foundation to asecurity-oriented solution.

3 Xen Hypervisor BackgroundThe XenAccess monitoring library is based on Xen [4], a

popular open-source virtual machine monitor (VMM). Thissection gives an overview of Xen’s architecture, followedby a discussion of its memory management and block de-vice I/O subsystems. The discussion of these subsystemsis central to the understanding of XenAccess’ monitoringcomponents.

3.1 Overview

Xen has traditionally used a paravirtualized approach toimplement virtualization. This technique consists of alter-ing the guest OSes by replacing sensitive instructions thatcannot be virtualized with special hypercalls, that is, callsthat are made directly to the VMM. This approach has theadvantage of providing good performance, since no trap-ping is done, and also allowing virtual machines to run ontop of non-virtualizable architectures (such as x86) [25].Nevertheless, one drawback of paravirtualization is that theguest OSes must be modified. Recent versions of Xen havethe capability to run unmodified OSes by using the new IntelVT-x and AMD-V technologies. XenAccess uses paravir-tualized domains without violating property (2) because thechanges required for paravirtualization are not strictly partof the XenAccess architecture and do not make it any easierfor the target OS to tamper with the monitoring code.

Xen uses a split domain architecture, meaning that reg-

Figure 1: Blktap disk I/O architecture.

ular guest OSes are kept in unprivileged domains (domU),whereas a single administrative domain exists as Domain0 (dom0). Dom0 can be seen as a domain-level extensionof Xen in which all of the management functionalities arelocated. It has complete access rights to all virtual ma-chines being run and also works as a device driver proxyfor domU’s virtual devices.

The VMM itself is a simple and thin software layerwhose main job is to guarantee proper isolation betweenvirtual machines, performing minimal resource manage-ment. This isolation is quite robust, since Xen relies di-rectly on hardware-level protection mechanisms and has amuch narrower interface than a standard operation system(e.g., Linux).

3.2 Memory Management

One of the key tasks for a VMM is to partition the sys-tem memory between each VM. Xen achieves this usingthree levels of memory: machine, physical, and virtual ad-dresses. The machine addresses are the actual addressesused by the hardware and managed by the VMM. The phys-ical addresses are what each paravirtualized OS uses. Thisfirst abstraction allows the VMM to assign non-contiguousmemory regions to a paravirtualized VM. As far as the VMis concerned, the physical addresses are the addresses usedby the hardware. In reality, these addresses are translatedby a lookup table in the VMM into machine addresses. Thethird type of address is a virtual address, which is used thesame way in the paravirtualized OS as traditionally used inOSes.

Both machine and physical addresses are often referredto in terms of a machine frame number (MFN) and a phys-ical frame number (PFN). These numbers refer to a singlepage of memory, which are 4k bytes each2. A complete ad-dress is given as both an MFN or PFN and an offset into

2Here we refer to the x86 architecture where memory pages are usually4k, but can also be 4M.

that page of memory. Using this scheme, Xen provides ta-bles to convert from MFN to PFN and PFN to MFN. Thesetables are called M2P and P2M, respectively. Similarly, therunning OSes use a page table (PT) to convert between vir-tual addresses and machine addresses. Xen protects thesePTs in order to ensure the memory isolation between VMs.A paravirtualized OS must invoke a hypercall to modify itsPT.

3.3 Device I/O

Xen’s device I/O architecture is based on a split drivermodel. In this model, there is a frontend driver insidedomU’s kernel that communicates with a backend driver in-side dom0’s kernel. Inter-domain communication relies onshared asynchronous I/O rings, shared memory pages and acontrol framework called XenBus. This architecture is cur-rently used for block and network devices.

For block devices, system calls are issued by domU ap-plications, which are translated into block-level operationrequests by the kernel. Traditionally, the backend driver indom0’s kernel receives the request from the frontend, andsends them directly to the disk. Xen 3.0.3 introduced anew architecture for block device I/O, illustrated in Fig-ure 1, with some interesting new properties. A block tapwas introduced, allowing disk drivers to be implemented asuserspace applications. These userspace drivers tap into thebackend driver (blktap) and can directly manage disk activ-ity with relatively small performance costs [32]. This archi-tectural change added substantial flexibility to disk driverdevelopment, greatly simplifying it and at the same timeallowing more powerful functionalities to be implemented.The fact that tapdisk drivers are regular userspace applica-tions allows virtual disk I/O to be implemented with simplefile-manipulation system calls and/or library calls.

4 XenAccess Monitoring Library4.1 Architecture

The primary goal for the XenAccess architecture is tosatisfy the six requirements stated in the Introduction. Wechose Xen as a virtualization solution because it is a TypeI VMM; it runs directly on the hardware, allowing for asolid foundation to the TCB. It also already includes an in-frastructure suitable to satisfy our monitoring needs, so thatchanges to the VMM are unnecessary (property (1)). Like-wise, by building on top of Xen’s infrastructure, we wereable to design the monitoring architecture to work withoutchanges to the target OS, allowing us to satisfy property (2).To prevent the target OS from tampering with the monitorsand satisfy property (6), we place the monitors in a differ-ent VM than the target OS. Xen provides sufficient isola-tion between VMs for this to be a viable solution. And, for

Figure 2: The XenAccess architecture leverages existing ca-pabilities in Xen to reduce complexity and improve overallperformance.

environments that require more explicit isolation, manda-tory access control is built into the Xen VMM [27]. Fi-nally, we desire an architecture that can monitor any dataon the target OS in order to satisfy property (5). XenAccesscurrently provides monitoring capabilities for both memoryand disk I/O. However, the architecture can easily be ex-tended to monitor additional information such as networktraffic, CPU context, and static disk contents. We examineXenAccess’ adherence to properties (3) and (4) in Section5.

Figure 2 shows the overall software architecture fromtwo perspectives. On the right, we show the location of thecritical components in relation to the VMM and VMs. Cur-rently, XenAccess runs in Domain 0 as this simplifies ac-cess to the Blktap Architecture and the XenControl Library.XenAccess could also run in a user domain after sufficientprivileges are given for that domain to perform the monitor-ing. The left side of Figure 2 shows how XenAccess fits intothe Xen software stack. Here we emphasize that XenAccessis a library intended for use by monitor applications. Somesimple example applications are discussed in Section 5.2.

Virtual memory introspection requires accessing thememory of one VM from another. Xen provides a func-tion in the XenControl Library that is used for this pur-pose and this functionality could be added to other VMMsusing a small amount of additional code. In Xen, thefunction xc map foreign range(), maps the memoryfrom one VM into another. After the memory is mapped, itcan be treated as local memory, providing for fast monitor-ing capabilities. In order to convert a XenAccess API callinto a call to xc map foreign range(), XenAccessmust perform several memory address translations. Thisrequires additional information about the target OS whichcan be obtained from the XenStore, a database of informa-tion about each VM, and interpreted using some knowledgeof the target operating system’s implementation. The steps

needed to convert a kernel symbol or virtual address into amemory mapped page are discussed in Section 4.2.1.

Whereas virtual memory introspection monitors the cur-rent state of memory pages, the virtual disk monitoring cap-tures data traveling to and from the disk. This data is cap-tured by placing code that directly intercepts the data pathbetween the target OS and the hard drive it uses for datastorage. We chose the Blktap Architecture for this capabil-ity due to its good performance [32] and its ability to de-liver low-level information to user-space software such asthe XenAccess Library.

The XenAccess architecture utilizes functionality in-cluded with Xen in order to reduce the implementation over-head and adhere to property (1). At this point it is importantto emphasize that while we acknowledge that XenAccess’functionality and its adherence to the principles establishedin the Introduction are significantly based on the infrastruc-ture already provided by Xen, the XenAccess architectureand the principles supporting it could be implemented onother VMMs as well. The core functionality needed inthe VMM includes mapping memory between virtual ma-chines, viewing VM-specific metadata (e.g., running kernelversion), and tapping into the data between a device (e.g.,the hard disk drive) and the associated device driver. Thisfunctionality could be added to any modern virtualizationenvironment, if it is not already there, allowing for sup-port of the XenAccess monitoring architecture. The onlycaution that must be taken is to implement at the VMMlevel only that which is strictly necessary, and do it verycarefully, so as to minimize the probability of introducingbugs in the TCB. All the remaining functionality should beimplemented in a special security or management domain(such as Xen’s dom0) taking the appropriate performanceconsiderations.

4.2 Implementation

XenAccess is implemented in C as a shared library with1935 source lines of code (SLOC). XenAccess makes useof libxc, libxenstore, and the Blktap Architecture.The current version is built to monitor a paravirtualized ver-sion of Linux 2.6.16 running on Xen 3.0.4 1, however thetechniques here can be extended to work with other OSes.The two primary monitoring functionalities in XenAccessare virtual memory introspection and virtual disk monitor-ing. Implementation details for each of these techniques arediscussed in the sections below.

4.2.1 Virtual Memory Introspection

XenAccess uses the xc map foreign range() func-tion, provided through the XenControl Library (libxc),to view the memory of another VM. Using this function

eliminates the need to modify the VMM or the target OS,satisfying properties (1) and (2). This function can be usedto map a memory page from the target OS using its MFN.XenAccess uses this function for raw memory access andthen builds up from there using address translation tables inthe VMM and the target OS. For example, to convert a PFNto a MFN, XenAccess uses lookup tables that are providedby Xen. Similarly, to convert a virtual address to a MFN,XenAccess uses the PTs.

Memory Introspection API

• xa init(): Initializes access to a specific domUgiven a domain ID. This function takes a domain IDand returns a structure that holds cached informa-tion related to accessing that domain. All calls toxa init() must eventually call xa destroy().

• xa destroy(): Destroys an instance by freeingmemory and closing any open handles.

• xa access kernel symbol(): Memory mapsone page from domU to a local address range. Thememory to be mapped is specified with a kernel sym-bol (e.g., from System.map). This memory must beunmapped manually with munmap().

• xa access virtual address(): Memorymaps one page from domU to a local address range.The memory to be mapped is specified with a kernelvirtual address. This memory must be unmappedmanually with munmap().

• xa access user virtual address(): Mem-ory maps one page from domU to a local addressrange. The memory to be mapped is specified witha virtual address inside a process address range. Thisfunction also requires a process ID. This memory mustbe unmapped manually with munmap().

We provide an overview of the implementation of each ofthese functions.

All users of the introspection library must begin witha call to xa init(). This function initializes thexa instance struct which holds information that isused throughout the introspection process. Any work thatcan be done “up front” and cached is held in this structure.This includes locating the address of the kernel page direc-tory, initializing a handle to libxc, initializing a pointerto a PFN to MFN lookup table, determining if the domainis paravirtualized or fully virtualized, and more. Once auser is done with the library, a call should be made toxa destroy() to free any memory associated with thexa instance struct.

After initializing the xa instance struct,one can use any of the three access functions

Figure 3: The steps needed to map a kernel memory pagebased on a kernel symbol using virtual memory introspec-tion.

listed above. Starting with the simplest, thexa access virtual address() takes a kernelvirtual address and returns a pointer to the memory pageholding that address along with the offset to the specifiedaddress within the memory page. This address translationrequires a PT lookup, which requires XenAccess to loadthree memory pages. First, the page directory is loadedto find the location of the PT. Next, the PT is loaded tofind the location of the address. Finally, the memory pageholding the address is loaded and this page, along withan offset to the address, is returned to the user. Returninga shared memory page contributes to the good inter-VMmemory copy performance shown in Section 5.1, which isa requirement for property (3).

The xa access kernel symbol() function,shown in Figure 3, requires one extra step beyond thevirtual address translation described above. This step isto convert a kernel symbol to a virtual address. XenAc-cess performs this conversion using the System.mapfile associated with the kernel from domU. If thisfile is not available, then this function will fail. TheSystem.map file is essentially a large table of symbolsand addresses. XenAccess scans this file until it findsthe symbol provided. It then proceeds with a virtualaddress access using the address associated with the kernelsymbol. Since this operation requires performing a lookupfrom a file on disk, it is considerably slower than thexa access virtual address() function, but theresults are cached so the average case is fast as discussed inSection 5.1. Further performance improvements could beachieved by memory mapping the file, moving the costlyfile read operations into the xa init function. However,most monitoring applications will repeatedly view the samememory location, using the cached information.

The final function in the virtual memory introspectionAPI is xa access user virtual address(). Thisfunction provides access to user space memory. Page tablelookups for a virtual address in user space are essentiallythe same as kernel space. The main difference is that wemust lookup the location of the page directory associatedwith the process. Recall that for kernel space, the locationof the page directory is cached during library initialization,but the page directory locations for each process can changeas processes come and go. To lookup the page directory fora process, XenAccess scans the kernel task list looking fora process with the given process ID. Upon finding a match,the page directory can be obtained from the task structin kernel memory. Using this page directory, the remainderof the virtual address translation is the same as previouslydescribed for the kernel.

HVM Support XenAccess has preliminary support toperform memory introspection on fully virtualized (HVM)VMs. In HVM VMs, physical addresses and machine ad-dresses are the same. Therefore, XenAccess will automati-cally detect HVM domains and not attempt to perform thistranslation in those cases. In practice, the P2M translationis a simple table lookup, so omitting this step does not mea-surably improve performance. Since memory introspectionsupport for HVM VMs is in its early stages, there is somereduced functionality. This reduced functionality is the rea-son why there is no HVM performance data available forthe user address function in Figure 5.

Improving Performance Since XenAccess must usememory from the target OS and the VMM to perform ad-dress translations, these operations can be costly. Therefore,XenAccess uses a least recently used (LRU) cache to storethe results of the address translations. This is similar to atranslation lookaside buffer (TLB). However, in the case ofXenAccess, we also cache kernel symbol names since diskaccess is always a slow operation. This caching is critical toachieving acceptable performance and satisfying property(3), as discussed in Section 5.1.

Use of OS-specific Information A virtual memory ad-dress can be converted to a MFN without any knowledgeof the OS in domU. This is because the address conversionis specific to the processor architecture and not to the OS. APT lookup, which is required to perform this address con-version, starts by obtaining the address of the page direc-tory. This information is stored in one of the control reg-isters, CR3, of the domU CPU context. Starting with thepage directory, one can complete a PT lookup and, there-fore, find the MFN associated with any virtual address on ahost. However, it can be difficult to determine what virtualaddress to access.

Identifying virtual addresses that are interesting requiressome knowledge about the OS. One artifact of compiling aLinux kernel is the System.map file. This file is a listingof symbols exported from the kernel along with the virtualaddress of each symbol. Using this file, combined with theability to access arbitrary virtual addresses, one can viewand modify data such as the system call table, interrupt de-scriptor table, Linux kernel module (LKM) list, task list,and more. In Microsoft Windows, exported symbols areavailable in debugging libraries and in ntdll.dll. Ofcourse, making use of these data structures requires knowl-edge of the data layout inside each structure. In Linux, thisis determined by inspecting the source code and using tech-nical references such as the kernel books by Bovet and Ce-sati [5] or Love [19]. In Windows, much of this informationis available in technical references as well [26].

The memory introspection implementation in XenAc-cess provides logical separations between the OS-specificcode and the general code that is OS-neutral. This is doneto permit rapid integration of new target OSes. For example,the current code is only designed to monitor a Linux 2.6 OS,but extending XenAccess to monitor Windows or FreeBSDwould only require adding the specific knowledge for eachOS.

4.2.2 Virtual Disk Monitoring

XenAccess introspects into low-level disk traffic, just as it isable to map raw memory pages. It therefore satisfies prop-erty (5) by providing full and complete access to data. Xe-nAccess also includes an inference engine which is able todynamically infer the high-level filesystem operations ex-ecuted inside a domain based on the intercepted low-leveldisk traffic. To this end, we have decided to leverage theBlktap architecture described earlier since it simplifies theimplementation of the interception mechanism and avoidsmaking modifications to the VMM, which is encouraged byproperty (1). The biggest challenge, however, is faced bythe inference engine which must somehow overcome thesemantic gap between the low-level view and the desiredhigher-level, filesystem-oriented view that will be given asoutput. It does this combining pre-programmed filesystemstructure knowledge with dynamic inference techniques.

Whereas the interception mechanism (which is roughlyequivalent to the introspection memory-mapping) is inde-pendent of the current OS and filesystem by only providingraw access to disk traffic, the inference engine is dependanton knowledge of the filesystem in use. So far, knowledgehas been included in the inference engine to be able to deter-mine only file/directory creation/removal operations underthe ext2 filesystem, although knowledge about other filesys-tems can be incorporated.

Figure 4: The XenAccess disk monitoring functionality. (1)A mkdir() system call is executed by an application insidethe user VM. This call is translated into low-level operationsby the kernel and sent to the backend driver at the monitoringVM. (2) The block data flow is intercepted by the wrapperdriver before it is processed by the tapdisk driver and sentthrough the tap FIFO. (3) Blocks are read from the FIFO andhashed into the table. (4) If a record is found for a block X(which holds part of /tmp contents), it is parsed and sorted.If not, it is discarded. (5) The new and old version of blockX are compared, its differences are translated into a directorycreation.

Disk Monitoring API

• xadisk init(): Opens the FS image file and ini-tializes the library’s main data structures;

• xadisk destroy(): Closes the image file anddeallocates all global structures;

• xadisk set watch(): Sets a watch-point in oneof the filesystem’s directories;

• xadisk unset watch(): Removes the watch as-sociated with a directory;

• xadisk activate(): Creates a thread that runsthe main monitoring and inference engine;

• xadisk deactivate(): Finishes the monitoringthread associated with a monitoring instance.

Wrapper drivers As illustrated in Figure 4, wrapperdrivers intercept the disk data flow before it is processed bythe tapdisk driver, sending all the data received to the infer-ence engine. Their functionality is simple: disk block dataand metadata received from kernel space are marshalledinto a buffer and sent to the inference engine through aFIFO. This architecture allows the inference routines to ex-ecute asynchronously with the actual disk reads and writes,which reduces performance impact.

Initialization and Watchpoints The disk monitoring li-brary is initialized by the xadisk init() function. Itcreates and initializes all relevant data structures, and opensall FIFO and file descriptors that will be used. It also readsand parses important filesystem metadata contained in thefilesystems’s superblock and the group descriptors. It gath-ers all the essential information to bootstrap the monitor-ing instance, allowing watchpoints to be set and the in-ference engine to start. This information is stored in axadisk t struct, which is returned by the function.The xadisk destroy() function ends a monitoring in-stance, closing the image file descriptor and deallocating allassociated data structures.

Watchpoints are set by the xadisk set watch()function. It receives as an argument the full path of adirectory that is to be monitored for file/directory cre-ation/removal. This function initially performs a recursivesearch through the filesystems’s structures (inodes and di-rectory entries) with the goal of finding the disk block(s)which store the directory’s content. The correspondingblocks are then read from disk, parsed, and have their en-tries sorted. Finally, they are inserted into a global hashtable. Watchpoints are erased by removing the correspond-ing block records from the hash table. This is done by thexadisk unset watch() function.

Inference Engine The inference engine constitutes thecore of the virtual disk monitoring library and its opera-tion is illustrated in Figure 4. It is activated by the APIfunction xadisk activate(), which subsequently cre-ates a new thread running the engine itself. At each iter-ation, the engine reads a new record from the tap FIFO,sent by the wrapper driver. Next, the record’s block num-ber is hashed into the global hash table and the existenceof an older version is checked. If it is found, file/directorycreations/removals are inferred by determining the differ-ences between the old and new versions of the block. Thename and type of the object created or removed can be de-termined by looking at the type field in the correspondingentry. Results are written to a per-application FIFO that isaccessible by a program using XenAccess. The functionxadisk deactivate() stops the inference engine bykilling the thread executing it and closing all associated de-scriptors.

Limitations The current implementation has some limita-tions. Although it has access to all data sent to the disk, fornow the inference engine is only able to infer file/directorycreation/deletion. It is certainly possible, nevertheless, toimplement the inference of more intricate operations likefile read/write, object renaming and file truncating. Moreelaborate algorithms will be required, but the same moni-toring architecture can be used.

Since we directly rely on the Blktap Architecture whichis exclusive to paravirtualized environments, disk monitor-ing of fully-virtualized guests (and therefore, OSes otherthan Linux) is not supported for now. However, we feel thatimplementing this support would not require major effort,since the inference engine would not require any structuralchanges. It would be a matter of finding out at which pointof the disk data flow a tap must be introduced and a newwrapper driver implemented. And of course, the inclusionof specific knowledge of the filesystem being used is alsonecessary.

Similarly, although the current implementation of theinference engine is dependant on detailed knowledge ofthe ext2 filesystem, its architecture could be used withoutchanges to monitor other filesystems that adhere to the samegeneral design principles as ext2 (such as the use of direc-tory structures). Besides, we were careful enough through-out its design and implementation to compartmentalize al-most all filesystem-specific implementation, making it easyfor the integration of new ones.

5 Experimental ResultsThis section focuses on the performance and qualita-

tive evaluation of our prototype. We evaluate the perfor-mance of each technique separately through a series ofmicro-benchmarks. This is followed by example applica-tions of each monitoring technique, showing their useful-ness in monitoring an OS’s internal structures and disk ac-tivity.

5.1 Performance Results

The performance figures show that XenAccess intro-duces minimal overhead. We performed the testing on Xen3.0.4 1 running Fedora Core 6 in both dom0 and domU.This software was run on a 2.33 GHz Intel Core Duo pro-cessor with 2 MB L2 cache, 2 GB RAM, and an 80 GB7200 RPM disk. Dom0 was assigned 2 processor cores anddomU was assigned one processor core.

5.1.1 Virtual Memory Introspection

Each of the performance measurements shown in this sec-tion were done using the gettimeofday() function,which has a micro-second granularity. Times were mea-sured by recording the time immediately before and afterthe function being measured. The difference between thetwo times was recorded. This measurement was repeatedfor 1000 times for each test. We choose 1000 measure-ments because this was sufficient to minimize the standarddeviation for a given set of measurements under this setup.Additional measurements did not improve the precision.

The data in Figure 5 show the average time to com-plete the specified function call. The cache hit columns

0

10

20

30

40

50

60

70

80

90

Virtual Address Kernel Symbol User Address

Tim

e in

micr

osec

onds

212136881 541PV-MPV-H

HVM-MHVM-H

Figure 5: Performance of the three memory access functionsin XenAccess for a paravirtualized target domain.

0

1

2

3

4

5

4000300020001000500100

Tim

e in

mic

rose

co

nd

s

Data size in bytes

fc6-pvfc6-hvm

Figure 6: Time for monitor to read memory through intro-spection.

represent the results with the LRU cache enabled. Thecache miss columns represent the results with LRU cachedisabled. The simplest case is shown on the left of thisgraph. The xa access virtual address() func-tion must map three memory pages on a cache missand one on a cache hit. This difference explains theimprovement seen with the LRU cache. The time forxa access kernel symbol() is dominated by the op-eration to lookup the kernel symbol. This operation isa lookup inside a file on disk, which is costly. Witha cache hit, the symbol to machine address mapping isstored in the cache, making the performance similar toxa access virtual address(). The last accessfunction is xa access user virtual address().This function must traverse the task list in the domUkernel to locate the page directory for the process vir-tual address. This explains the slower performancefor the cache miss. On a cache hit, this traversalis not needed, performance is essentially the same asxa access virtual address().

After the memory is accessed, the next step is to readfrom or write to that memory. As seen in Figure 6, this op-eration is fast compared to mapping the memory. These per-formance results show the time required to memcpy() data

0

500

1000

1500

2000

10 50 100 200 500 1000 2000

Tim

e in

milli

seco

nds

Number of Files Created

Mode 1Mode 2Mode 3

Figure 7: Performance of three different scenarios run-ning a disk benchmark. Mode 1: Disk monitoring enabledand watchpoints set on each benchmarking directory; Mode2: Disk monitoring enabled and no watchpoints set on thebenchmarking directories; Mode 3: Standard block-aiotapdisk driver being used (no monitoring).

from kernel memory in the target OS. In general, we foundthat data is copied into a data monitor at a rate of approxi-mately 1kB / µsec. Figure 6 shows that memcpy() perfor-mance for PV and HVM VMs is essentially the same. Thevariance in these measurements can be attributed to exper-imental noise given the precision of our timing mechanismand the small measurement times. Looking at the cachehit values in Figure 5 and the memory copy performance,the memory introspection capabilities in XenAccess per-form well enough to have a negligible impact on the overallsystem performance, satisfying property (3) from the Intro-duction.

5.1.2 Virtual Disk Monitoring

XenAccess’ disk monitoring performance was evaluatedthrough the execution of a benchmarking shell script. Thisscript tests exhaustively one of the operations monitored byXenAccess: file creation. It works by measuring the time ittakes to create a variable number of files inside ten differentdirectories. The total number of files created is equally dis-tributed throughout the ten directories. The timing measure-ments include both the execution of the file creation com-mands and the manual flushing of the changes to the disk(through the sync command). Manual flushing was nec-essary so that the actual performance impact would not behidden by the operating system’s buffer cache.

The script was executed in three different modes and theresults were compared. A sample size of 50 was used foreach mode, which was enough to minimize the standard de-viation. Measurements were made using the Linux timecommand, which has millisecond precision. A descriptionof the modes is included in Figure 7 caption.

Figure 7 shows that the difference between the different

timing measurements for each test case is negligible. Evenin the case where 2000 files were created, the differencesbetween the measurements were not statistically significant.The conclusion is that the performance overhead added byXenAccess’ disk monitoring capabilities are minimal, andtherefore obeys property (3) as defined in the Introduction.

The explanation for this negligible overhead lies in thedesign of the Blktap architecture, as well as XenAccess’own design. Most of XenAccess’ disk monitoring enginecode is executed asynchronously with regard to the actualdisk I/O. The wrapper driver is basically the only extra codeadded by XenAccess to the disk I/O critical path. It does asimple data marshaling followed by a memory copy oper-ation to the tap FIFO. These are not expensive operations.The asynchronism created by the use of a FIFO allows theinference engine, which is the most performance-intensivecomponent of the architecture, to execute in parallel withthe actual disk operations. In addition, we ran our bench-marks on a dual-core platform, enabling real parallelizationof the tasks.

5.2 Example Applications

XenAccess is straight forward to use, allowing for rapiddevelopment of new monitors to satisfy property (4) fromthe Introduction. In this section we show several exam-ple applications to demonstrate our monitoring capabilities.While reading through these examples, keep in mind thatwe are deliberately showing simple use cases as an intro-duction to the library. However, XenAccess provides com-plete read/write access to a VM’s memory space and hascomplete access to the disk I/O. With this level of power, thepotential applications are only limited by the user’s imagi-nation.

5.2.1 Virtual Memory Introspection

Using introspection, XenAccess can view and modify datain memory of a running OS. The example below shows howto use XenAccess to view the LKMs. Additional examplesin the open source release show how to list running pro-cesses and view memory pages of a particular process onthe target OS. These examples utilize information directlyfrom a running Linux kernel.

List Linux Kernel Modules This example uses thexa access kernel symbol() function to list theLKMs installed into the domU kernel and is 44 SLOC. Pro-gram 5.1 shows the code for this example. The code followsa linked list in the domU kernel memory using introspec-tion. It starts by loading the memory page containing thehead of the list, which is found using the modules kernelsymbol. This address points to a module struct. This

structure contains a circular doubly linked list that points tothe rest of the modules. Therefore, the code proceeds byloading the memory page addressed by the next pointer allthe way down the list. For each structure, the module nameis accessed by creating a pointer to its offset, and then itis printed to stdout. Since the linked list is circular, thecode ends when it finds a pointer back to the head of the list.

Program 5.1 Source code for an example that lists all run-ning LKMs in the domU kernel. All error checking codehas been removed for clarity.xa_init(dom, &xai);memory = xa_access_kernel_symbol(&xai,

"modules", &offset);memcpy(&next_module, memory + offset, 4);list_head = next_module;munmap(memory, XA_PAGE_SIZE);while (1){

memory = xa_access_virtual_address(&xai,next_module, &offset);

memcpy(&next_module, memory + offset, 4);if (list_head == next_module){

break;}name = (char *) (memory + offset + 8);printf("%s\n", name);munmap(memory, XA_PAGE_SIZE);

}xa_destroy(&xai);if (memory) munmap(memory, XA_PAGE_SIZE);

Since this example is accessing and displaying OS-specific information, it requires OS-specific knowledge. Inthis case, the knowledge falls into two categories. First, wemust know that the modules symbol points to the begin-ning of a linked list that will provide the information that weneed. Second, we must know the offsets within the modulestruct needed to access information such as the nextpointer and the module name. Requiring this type of infor-mation is common for introspection applications. For thisexample, the information needed was available in both theLinux source code, and Bovet and Cesati’s kernel book [5].

Additional Examples The example above is straightfor-ward and provides a quick understanding of XenAccess’sintrospection capabilities in operation. Other monitors arenot much more complex. For example, we developed anapplication that monitors for changes in the system call ta-ble (110 SLOC) and an application that monitors the in-tegrity of an installed LKM (172 SLOC). The security ap-plications of these types of monitoring are clear in areas likeintrusion detection and integrity checking, and have beenwell explored in literature. XenAccess makes these typesof applications possible by providing memory access at theproper levels of abstraction. Compared to other virtualiza-tion monitoring architectures, such as the work by Asrigo

et al [2], XenAccess allows for rapid monitor developmentsince monitors are small user-space applications rather thankernel hooks. Based on our experience building XenAccessintrospection monitors, we feel that our architecture satis-fies property (4).

5.2.2 Virtual Disk Monitoring

In this example, three file/directory creation/deletion com-mands are executed inside domU followed by the synccommand, which flushes the changes to disk. In the man-agement VM, a monitoring program is run which catcheschanges to the /root directory in domU for 30 secondsusing XenAccess disk monitoring capabilities. This is doneby first initializing the engine and setting a watchpoint in/root by using the xadisk set watch() function.Next, the engine is activated and its output is directed tothe standard output. The monitor’s source code is shownin Program 5.2 and a sample execution is shown below forboth domU (left) and dom0 (right).

domU ˜ # mkdir foo dom0 ˜ # ./monitor /rootdomU ˜ # touch dummy MKDIR: /root/foodomU ˜ # rm bar MKFILE: /root/dummydomU ˜ # sync RMFILE: /root/bar

Program 5.2 This disk monitoring application outputs tothe standard output all file/directory creation/removals hap-pening in domU’s /root directory as soon as they are com-mitted to disk.xadisk_t *x;xadisk_obj_t *obj;x = xadisk_init(1, FILE_IMAGE);obj = xadisk_set_watch(x, argv[1]);xadisk_activate(x, "/dev/xen/tapfifo0\0");dup2(1, x->fifo_fd);sleep(30);xadisk_unset_watch(x, obj);xadisk_destroy(x);

The simplicity of this example shows how XenAccesscan be used to enable rapid development of similar moni-toring applications, satisfying property (4).

From a security perspective, one application of thisengine is a disk-based intrusion detection system [22,12] whose goal is to detect suspicious file/directory cre-ation/deletion commonly done by rootkits. In this case,the watchpoints would most likely be set in privileged sys-tem directories such as /bin and /usr/bin. An IDS de-ployed in this fashion has the obvious advantage of not re-quiring additional hardware in the hard drive, as done bytraditional disk-based IDSes to achieve isolation. In ourcase, the isolation is provided by the VMM.

The need to manually perform a disk flush through thesync command in the example above illustrates one of the

problems involved in this type of monitoring: the fact thatchanges made to the filesystem are not immediately com-mitted to the disk in modern OSes. This can have seriousimplications for real-time disk-based IDSes, as it opens awindow for evasion attacks. One way of addressing thisproblem is to use disk monitoring together with memoryintrospection: disk flushes could be externally enforced byactively manipulating the flushing timers in the guest OSmemory through introspection. This approach is currentlybeing investigated, along with others in which memory in-trospection and disk monitoring can be associated.

6 Discussion and Future WorkStepping back to look at the six requirements given for

a robust monitoring solution, we note that XenAccess sat-isfies each of these requirements. (1) The XenAccess Li-brary uses an unmodified version of Xen as a VMM plat-form. (2) Using the capabilities provided by Xen, no spe-cial code needs to be inserted into the target OS. This isespecially useful as it allows XenAccess monitors to workwith both open and closed source target OSes. (3) Our per-formance testing shows that our address translation, mem-ory copying, and disk I/O monitoring functions have smalloverheads, making these capabilities effective for a varietyof monitoring applications. (4) Our example applicationsshow that developing monitors with XenAccess is straight-forward, with a minimal learning curve. (5) While our exist-ing library implementation can view memory and disk I/O,the XenAccess architecture is easily extensible to collectany type of data from the target OS. (6) Finally, leveragingthe protections provided by the VMM, XenAccess is suf-ficiently isolated from the target OS and any possibility oftampering by malicious software.

XenAccess currently provides a solid foundation formonitoring in a virtualized environment. Yet, our experi-ences working with virtual memory introspection and vir-tual disk monitoring highlighted some areas that would ben-efit from additional research. Introspection requires useof OS-specific information, as discussed in Section 4.2.1.This means that it is possible for an OS upgrade, hotfix, orpatch to break the monitors. Ideally, XenAccess should pro-vide an abstraction layer that dynamically adapts to thesechanges and provides a consistent interface to monitor ap-plications. Finding techniques to enable this approach isstill an open research problem.

For reasons of backwards compatibility, changes infilesystem structure and layout are very rare. So disk moni-toring is not prone to the types of problems discussed abovefor introspection. Instead, we envision the future work inthis space to focus on scalability, functionality, and HVMsupport. The current issues with scalability and function-ality were discussed in Section 4.2.2. HVM support willrequire changes to tap into the QEMU device driver mech-

anism used by Xen.

7 ConclusionThis paper described XenAccess, a monitoring library

for Xen virtual machines. XenAccess’ development wasguided by a set of design principles aimed at providinga solid foundation for secure and flexible virtual machinemonitoring. XenAccess implements virtual memory intro-spection and virtual disk monitoring capabilities by leverag-ing Xen’s existing infrastructure. By using it to access thetarget VM’s raw memory pages and disk I/O, XenAccess isable to infer OS data structures and filesystem operations ata useful abstraction level.

Our evaluation revealed that XenAccess imposes a min-imal performance overhead to the target OS memory anddisk operation. We also showed practical examples of thetype of information that memory introspection and diskmonitoring can gather, illustrating the potential of eachtechnique.

AcknowledgmentsThis material is based upon work supported in part by

the National Science Foundation under Grant No. CNS-0627430. Any opinions, findings, and conclusions or rec-ommendations expressed in this material are those of theauthor(s) and do not necessarily reflect the views of the Na-tional Science Foundation.

References[1] A. C. Arpaci-Dusseau and R. H. Arpaci-Dusseau. Informa-

tion and control in gray-box systems. In Proceedings of the18th Symposium on Operating System Principles, 2001.

[2] K. Asrigo, L. Litty, and D. Lie. Using VMM-based sensors tomonitor honeypots. In Proceedings of the 2nd ACM/USENIXInternational Conference on Virtual Execution Environments,2006.

[3] P. Barford and V. Yegneswaran. An inside look at botnets. InAdvances in Information Security. Springer, 2006.

[4] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho,R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art ofvirtualization. In Proceedings of the 19th ACM Symposiumon Operating Systems Principles, 2003.

[5] D. P. Bovet and M. Cesati. Understanding the Linux Kernel.O’Reilly & Associates, Inc., 3rd edition, 2005.

[6] J. Dike. User Mode Linux. Prentice Hall, 1st edition, April2006.

[7] G. W. Dunlap, S. T. King, S. Cinar, M. Basrai, and P. M.Chen. Revirt: Enabling intrusion analysis through virtual-machine logging and replay. In Proceedings of the 2002 Sym-posium on Operating Systems Design and Implementation,December 2002.

[8] P. Elango, S. Krishnakumaran, and R. H. Arpaci-Dusseau.Design choices for utilizing the disk idleness in a virtual ma-chine environment. In Workshop on the Interaction betweenOperating Systems and Computer Architecture, June 2006.

[9] T. Garfinkel, B. Pfaff, J. Chow, M. Rosenblum, and D. Boneh.Terra: A virtual machine-based platform for trusted comput-

ing. In Proceedings of ACM Symposium on Operating Sys-tems Principles (SOSP), October 2003.

[10] T. Garfinkel and M. Rosenblum. A virtual machine intro-spection based architecture for intrusion detection. In Pro-ceedings of the 2003 Network and Distributed System Sym-posium, 2003.

[11] R. Goldberg. Survey of virtual machine research. IEEEComputer Magazine, 7:34 – 45, June 1974.

[12] J. L. Griffin, A. Pennington, J. S. Bucy, D. Choundappan,N. Muralidharan, and G. R. Ganger. On the feasibility of in-trusion detection inside workstation disks. Technical ReportCMU-PDL-03-106, Carnegie Mellon University, 2003.

[13] S. T. Jones, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. Antfarm: Tracking processes in a virtual machineenvironment. In Proc. of the USENIX Annual Technical Con-ference, June 2006.

[14] S. T. Jones, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau. Geiger: Monitoring the buffer cache in a virtualmachine environment. In Architectural Support for Program-ming Languages and Operating Systems (ASPLOS XII), SanJose, CA, October 2006.

[15] A. Joshi, S. T. King, G. W. Dunlap, and P. M. Chen. Detect-ing past and present intrusions through vulnerability-specificpredicates. In Proceedings of ACM Symposium on OperatingSystems Principles (SOSP), pages 1–15, Oct 2005.

[16] P. A. Karger, M. E. Zurko, D. W. Bonin, A. H. Mason, andC. E. Kahn. A retrospective on the VAX VMM security ker-nel. IEEE Transactions on Software Engineering, 17(11),November 1991.

[17] N. L. Kelem and R. J. Feiertag. A separation model for vir-tual machine monitors. In Proceedings of the 1991 IEEESymposium on Research in Security and Privacy, pages 78– 86, 1991.

[18] K. Kourai and S. Chiba. Hyperspector: Virtual distributedmonitoring environments for secure intrusion detection. InProceedings of the 1st ACM/USENIX International Confer-ence on Virtual Execution Environments, 2005.

[19] R. Love. Linux Kernel Development. Novell Press, 2nd edi-tion, 2005.

[20] S. E. Madnick and J. J. Donovan. Application and analysisof the virtual machine approach to information system secu-rity and isolation. In Proceedings of the Workshop on VirtualComputer Systems, pages 210 – 224, March 1973.

[21] R. Meushaw and D. Simard. Nettop: A network on yourdesktop. Tech Trend Notes (NSA), 9(4):3 – 11, Fall 2000.

[22] A. G. Pennington, J. D. Strunk, J. L. Griffin, C. A. N. Soules,G. R. Goodson, and G. R. Ganger. Storage-based intrusiondetection: Watching storage activity for suspicious behav-ior. In Proceedings of the 12th USENIX Security Symposium,2003.

[23] N. L. Petroni, Jr., T. Fraser, J. Molina, and W. A. Arbaugh.Copilot - a coprocessor-based kernel runtime integrity moni-tor. In Proceedings of the 13th USENIX Security Symposium,August 2004.

[24] Qumranet, Inc. Kvm: Kernel-based virtualization driver.[25] J. S. Robin and C. E. Irvine. Analysis of the Intel pentium’s

ability to support a secure virtual machine monitor. In Pro-ceedings of the 9th USENIX Security Symposium, 2000.

[26] M. E. Russinovich and D. A. Solomon. Microsoft WindowsInternals. Microsoft Press, 4th edition, 2004.

[27] R. Sailer, T. Jaeger, E. Valdez, R. Caceres, R. Perez,S. Berger, J. Griffin, and L. van Doorn. Building a MAC-based security architecture for the Xen opensource hypervi-sor. In Proceedings of the 21st Annual Computer SecurityApplications Conference (ACSAC), December 2005.

[28] J. H. Saltzer and M. D. Schroeder. The protection of infor-mation in computer systems. Communications of the ACM,17(7), July 1974.

[29] M. Sivathanu, V. Prabhakaran, F. I. Popovici, T. E.Denehy, A. C. Arpaci-Dusseau, and R. H. Arpaci-Dusseau.Semantically-smart disk systems. In Proceedings of FAST:2nd USENIX Conference on File and Storage Technologies,2003.

[30] J. D. Strunk, G. R. Goodson, M. L. Scheinholtz, C. A. N.Soules, and G. R. Ganger. Self-securing storage: Protect-ing data in compromised systems. In Proceedings of the 4thSymposium on Operating Systems Design and Implementa-tion, 2000.

[31] C. A. Waldspurger. Memory resource management invmware esx server. In Proceedings of the 5th Symposium onOperating Systems Design and Implementation (OSDI ’02),volume 36, pages 181 – 194, 2002.

[32] A. Warfield. Virtually persistent data. In Xen Developer’sSummit (Fall 2006), 2006.