Heterogeneous Live Migration of Virtual Machines

Pengcheng Liu, Ziye Yang, Xiang Song, Yixun Zhou, Haibo Chen, and Binyu Zang

Parallel Processing Institute, Fudan Universityhttp://ppi.fudan.edu.cn

The market of Virtualization is booming

• Virtual data-center, grid computing, and cloud computing

• “By 2010, at least 30%of non-desktop IT infrastructure will be virtualized (up from less than 5 percent in 2007).”– www.cio.com/article/170653

• What can virtualization do?– Sever consolidation– Mobility

• Virtual machine live migration– …

VM Live Migration

• Benefits– Load balancing– Online maintenance – Fault tolerance

23/4/18 PPI, Fudan University 3

VirtualMachine

Virtual Machine

VMMHardware Hardware

VMM

War between different VMM venders

• The battle in virtualization area is and will be continuing – Vmware, Microsoft, kvm, …

– They will likely co-exisit in a long time• Linux kernel 2.6.23 embeds different vmm support

– Xen, lguest, kvm

• Cloud Computing• More resources brings inevitably heterogeneity

Virtual Machine

VirtualMachine

Motivation

• Heterogeneity of different VMMs– Break uniformity and mobility– Downgrade the applicability of live migration

23/4/18 PPI, Fudan University 5

VMM 2VMM 1Hardware Hardware

Contribution: Vagrant

• A preliminary study aim at– Bridging the heterogeneity of different VMMs and

support live migration

• Goal– Provide common migration protocol– Use common format of VM abstraction– Provide a pool of memory migration algorithms

• Preliminary results– Support VM migration across Xen and KVM– Accept able performance

Outline

• Heterogeneity among VMMs

• Vagrant Design and Implementation

• Evaluation

• Conclusion

23/4/18 PPI, Fudan University 7

Heterogeneity in Abstracting Resources

• CPU– Internal virtual CPU format

• Memory– Memory management information

• I/O Devices– Virtual I/O Device format

23/4/18 PPI, Fudan University 8

Heterogeneity in Migration Algorithms

• Stop-and-copy

• Pre-copy

• Push and pull

23/4/18 PPI, Fudan University 9

Heterogeneity in Migration Protocols

• Different set of control commands

• Example – Xen VMM initiates migration by sending a string

”receive” to the target– KVM sends a special integer to start migration

23/4/18 PPI, Fudan University 10

Outline

• Heterogeneity among VMMs

• Vagrant Design and Implementation

• Evaluation

• Conclusion

23/4/18 PPI, Fudan University 11

Architecture of Vagrant

• Vagrant acts as a middle man between different VMMs

23/4/18 PPI, Fudan University 12

VMM A

VM

Management VM

VMM B

Vagrant Agency

StateTransformer

Algorithm pool

Management VM

Vagrant Agency

StateTransformer

Algorithm pool

Bridging the Heterogeneity

• CPU State– Intercept the state from local virtual CPU module

– Essential CPU state is transformed into Vagrant format

– Requirements :• VMMs can expose same CPU features

23/4/18 PPI, Fudan University 13

Bridging the Heterogeneity (2/2)

• I/O Devices– Intercept the state from local device module• H/W based virtualization relies on device emulation

– Eg. QEMU used by Xen, KVM and VirtualBox

– Requirements:• VMMs can support all I/O devices existing in the

migrated VM

23/4/18 PPI, Fudan University 14

Migration Protocol

• Initiate and Control the migration process– Common migration protocol and commands– Negotiation before the migration• Ensure the same capability of pre- and post- migration

environment– CPU features, supported I/O devices

23/4/18 PPI, Fudan University 15

VMM A VMM B

Hey,I want a RTL8169 NIC,

Can you support?

Migration Algorithms

• Each VMM has its own migration algorithm

23/4/18 PPI, Fudan University 16

Stop and Copy

23/4/18 PPI, Fudan University 17

VirtualMachine

Virtual Machine

VMMHardware Hardware

VMM

Simply stop the VM and send

VM state

May cause very long

downtime, but total

time is short

Pre-copy Based Algorithm

• Sending memory pages while the VM is still running

23/4/18 PPI, Fudan University 18

VirtualMachine

Virtual Machine

VMMHardware Hardware

VMM

Sending memory pages

VM is still running

Stop the VM

Send whole VM state to destination

Pre-copy Reduces

downtime

Algorithm Pool

• Algorithm pool– At least one common algorithm is needed– Choose the algorithm by negotiation

• Ideally, support a set of different algorithms– Choose appropriate algorithm in different

scenarios– eg, the user wants to finish the migration as soon

as possible, stop and copy is better

23/4/18 PPI, Fudan University 19

Implementation

• A prototype based on– X86 with VT-x extensions– Xen 3.1.0 and KVM-48

– CPU• Both expose most CPU features

– I/O Device• Both rely on QEMU• VMM support for some devices, such as PIC/APIC/IOAPIC

23/4/18 PPI, Fudan University 20

Outline• Heterogeneity among VMMs

• Vagrant Design and Implementation

• Evaluation– Migration time– Application performance

• Conclusion23/4/18 PPI, Fudan University 21

Experimental Setup(1/2)

• Machine setup:– Migration between pair of PCs with

• 2.33 GHz Intel Core Duo CPU, 2GB RAM• RTL8169 NIC• 250 GB 7200 RPM SATA disk

– NFS server• 3.0 GHz Pentium IV with 1GB RAM• Intel Pro 100/1000 Ethernet NIC• 250 GB 7200 RPM SATA disk

– Machines were connected via switched Gigabit Ethernet

23/4/18 PPI, Fudan University 22

Experimental Setup (2/2)

• VM configuration– 128 MB of RAM– 5GB disk image with installation of Fedora Core 6

• Migration time under different workload– Idle, kernel-compile, memtest86, apache benchmark (ab)

• Application performance– Migrating a running Web Server

• Continuously serves a 256 KB file to a remote client

23/4/18 PPI, Fudan University 23

Questions for Evaluation

• Whether Vagrant has an acceptable total migration and minimal downtime?

• Whether applications during the migration can still provide acceptable performance?

• Is Vagrant’s performance comparable to other migration framework, eg. Xen and KVM?

23/4/18 PPI, Fudan University 24

Total Migration Time

23/4/18 PPI, Fudan University 25

Vagrant adopts batched memory transfer while KVM uses one-by-one manner

Migration Downtime

23/4/18 PPI, Fudan University 26

Downtime in Vagrant in comparable to homogeneous live migration

Migrating a Running Web Server

23/4/18 PPI, Fudan University 27

Memory Pre-copy, about 10% downgrade

Less than 1 second

Related Work

• Homogenous live migration over LAN– Vmotion, Xen (Clark et al. ’05) and KVM

• Live wide-area migration– Bradford et al. (vee’07)– “VM Turntable” (FGCS’06)

• Tons of virtualization related work

23/4/18 PPI, Fudan University 28

Conclusion

• Heterogeneous live VM migration framework– General abstraction of VMs and migration protocol

– No changes to hosted OS and application

– Minor changes to VMM abstraction

– Comparable downtime and end-to-end time

23/4/18 PPI, Fudan University 29

Questions?

• Thank you!

3023/4/18 PPI, Fudan University

Heterogeneous Live Migration of Virtual Machines

Pengcheng Liu, Ziye Yang, Xiang Song, Yixun Zhou, Haibo Chen, and Binyu Zang

Parallel Processing Institute, Fudan Universityhttp://ppi.fudan.edu.cn

Backup Slides

• Control commands in Vagrant

• CPU state in Vagrant

• I/O devices configured for the VM

• Memory migrating algorithm

23/4/18 PPI, Fudan University 32

Control Commands in Vagrant

• Common set of control commands– Each has a 3 seconds timeout

23/4/18 PPI, Fudan University 33

CPU State in Vagrant

• The necessary information to be transferred includes– GPR, control regs, segment regs, debug regs, msr,

fpu regs, sysenter registers– Time stamp counter– Pending events

23/4/18 PPI, Fudan University 34

I/O devices configured for the VM

• The migrated VM is configured with– Keyboard and mouse– Rtc– PIC/APIC/IOAPIC– PIT– IDE disk– NIC– Serial port– VGA

23/4/18 PPI, Fudan University 35

Memory migrating algorithm

• Algorithm implemented by Vagrant– Pre-copy in batched manner• Batch size is 1024

23/4/18 PPI, Fudan University 36