Predicting The Performance Of Virtual

Machine Migration

Presented by : Eli Nazarov

Sherif Akoush, Ripduman Sohan, Andrew W.Moore, Andy HopperUniversity of Cambridge

Agenda Introduction. How to migrate? Defining migration performance. Performance prediction. The AVG & HIST models. Evaluation. Conclusions.

Why performance prediction matters?

Provision and control computing capacity.

Guarantee performance levels . Efficient management.

Better VM placement. Better resource utilization (e.g. load balancing).

How to migrate? Stop-and-Copy

Minimizes total migration time. Highest downtime.

On-Demand. Short downtime. Very high total migration time.

Pre-Copy migration Pre-Copy migration involves 6 steps:

Initialization Pre-select a target for migration.

Reservation Reserve resources on the destination host.

Iterative Pre-Copy First Iteration : Send all RAM. Each iteration : Send modified pages.

Stop-and-Copy Stop VM for final transfer.

Commitment Destination host acknowledges that the copy finished correctly.

Activation Re-attachment of resources to VM on the destination host.

Pre-copy phase

Copy phase

Xen Stop Conditions Less then 50 pages were dirtied during

the last pre-copy iteration. Guarantees short downtime.

29 pre-copy iteration have been carried out.

Already copied more then 3*|VM|. At iteration N-1 we copied 3*|VM|-1page

Forces Stop-and-Copy stage.

Migration & Down times

TotalMigrationTime=

Pre-copy Stop-and-Copyi

i

Post-migration Overhead

+ Commitment + Activation

TotalDowntime= Stop-and-Copy

Post-migration Overhead

+ Commitment + Activation

Pre-migration Overhead

Inialization+Reservation

How To Predict? Calculate Bounds.

TotalMigrationTimeVMSizeOverheads + LinkSpeed

5• VMSize - 1• pageOverheads + LinkSpeed

TotalDowntimePost-migrationOverheads Post-migrationOverheads

VMSize LinkSpeed

1

1

0

3 11

Pre-copy Pre-copy Stop-and-Copyi

VMSizeV

n

ni

VMSizeM

P eSize

ag

Bounds are not enough! Don’t give accurate prediction.

Reason: Significant differences in lower and upper bounds due to link speed and VM size correlation.

Example: For VM Size=1,024 MB

MT =Total Migration Time, DT=Total Downtime, LB=Lower Bound, UB=Upper Bound

For big VM memory sizes even larger differences. We need something more accurate.

Speed100

Mbps96.3 s 459.1 s 0.314 s 91.4945 s

1 Gbps 13.3 s 49.9 s 0.314 s 9.4978 s10 Gbps 5.3 s 10.1 s 0.314 s 1.5187 s

LBMT UBMT LBDT UBDT

Parameters affecting migration Migration link bandwidth.

Higher speed links allow faster transfers. Pre and Post migration overheads.

Operations that aren’t part of the actual transfer. Examples:

Initializing container in destination host . Reattaching device drivers to the new VM. etc.

Example: 10 Gbps, VM size = 512MB Pre-overhead = 77%

Parameters affecting migration (cont.)

Page dirty rate. The rate at which memory pages in VM are

modified. Affects the number of pages transferred in

each pre-copy integration. Page dirty rate and performance relation is

not linear Reason: Link speed.

Page dirty rate and link speed

Downtime at low page dirty rate is almost constant and close to lower bound.

Downtime increases to upper bound when page dirty rate is high (reaches link capacity).

10Gbps – Total downtime

Page dirty rate and link speed (cont.)

Total migration time increases with page dirty rate.

Total migration time goes back to lower bound for extremely high page dirty rate.

Back to pure Stop-and-Copy.

10Gbps – Total migration time

100Mbps – Total migration time

What's next? Prediction using all parameters affecting

migration. Link speed. Page dirty rate. VM memory size. Overheads.

AVG - Average Page Dirty RateHIST – History Based Page Dirty

The AVG model Based on the migration logic. Assumes constant or average page dirty

rate. Useful when the dirty page rate is stable. Follow the core functionality of migration

in Xen.

The AVG model (cont.) Input parameters:

Link Speed. Page Dirty Rate.

Analytically determinable. Pre\Post overheads.

Time spent during actual transfer – Time to migrate idle VM VM Size.

Xen functionality: sim_clean(): returns the set of dirty pages + sets state

to “all clean”. sim_peek(): returns bitmap of dirty pages (no state

change).

Algorithm - the AVG model Each Pre-Copy phase:

Get dirty bitmap – sim_peek(). Skip the pages re-dirtied in this iteration Collect at most 1024 pages – batch. migration_time += if (last_iteration)

downtime_time += Clean pages status – sim_clean().

Calculate the total times: total_migration_time = migration_time +

pre_overheads + post_overheads. total_downtime = downtime + post_overheads.

BatchLinkSpeed

BatchLinkSpeed

The HIST model Used in cases where the dirty page rate

is a function of time. Depends on the history log of page dirty

rate.

The HIST model (cont.)

Given the start time of migration – t Predict migration times based on:

t+1,t+2, …, t+N Changed sim_clean() and sim_peek() to return

#dirty pages at the above points in time for log.

Use AVG algorithm with these function.

Observation:For deterministic processes the set of dirtied pages at any point in time will be approximately the same as for previous runs of the same workload running in a similar environment.

Evaluation Test-bed:

Xenserver 5.5.0 (Xen 3.3.1) on 3 servers. 1 pool master, 2 hosts for migration.

Each server: 2 Intel® Xeon™ 2.13 GHZ, 6GB DDR3.

SAN – IBM eserver xSeries 336. 2 GB DIMM. Ultra320 SCSI. Ubuntu 2.6.27-7 kernel.

Compared to: Actual migration using 2 SolarFlare10Gbps NICs.

Evaluation (Cont.) Page Modification Micro-Benchmark

Can be used both for AVG & HIST. Deterministic application. Writes to memory pages at fixed rates. High resolution of page modification

Up to pages/sec. Over 25,000 live migrations.

610

Evaluation (cont.) - Results

AVG v.s Real migration

HIST v.s Real migration

Results (Cont.) - Results For |VM|=1024MB , LinkSpeed=10Gbps: HIST mean deviation from the

measurements : 3.3% - total migration time. 6.2% - total downtime.

AVG mean deviation from the measurements: 2.6% - total migration time. 3.3% - total downtime.

Evaluation(cont.) – Industry workloads

Comparing against a set of industry-standard workloads. SPEC CPU

For CPU bounds workloads. SPECweb

WebServer workloads. SPECsfs

I/O, MapReduce & non-interactive workloads.

Industry workloads - Results

CPU 5.8 s 5.7 s 2.4% 0.317 s

0.314 s

2.4%

WEB 7.5 s 7.4 s 2.0% 0.449 s

0.42 s 6.4%

SFS 14.8 s 14.9 s 1.5% 0.2176 s

0.2177 s

0.1%

MR 14.9s 15.13s 1.4% 0.348 s

0.384 0.2%MT =Total Migration Time, DT=Total Downtime, A=Actual Measurements P=HIST Prediction

AMT PMT Err ADT PDT Err

Comments

Presented an accurate model for prediction.

Performed a large scale evaluation.

Very specific to Xen implementation. Didn’t perform evaluation comparing to

other prediction methods. Didn’t state how to predict with bounds.

Questions?

?