1

PLAN: Joint Policy- and Network-Aware VMManagement for Cloud Data Centers

Lin Cui, Member, IEEE, Fung Po Tso, Dimitrios P. Pezaros, Senior Member, IEEE ,Weijia Jia, Senior Member, IEEE , and Wei Zhao, Fellow, IEEE

Abstract—Policies play an important role in network configuration and therefore in offering secure and high performance servicesespecially over multi-tenant Cloud Data Center (DC) environments. At the same time, elastic resource provisioning throughvirtualization often disregards policy requirements, assuming that the policy implementation is handled by the underlying networkinfrastructure. This can result in policy violations, performance degradation and security vulnerabilities. In this paper, we define PLAN,a PoLicy-Aware and Network-aware VM management scheme to jointly consider DC communication cost reduction through VirtualMachine (VM) migration while meeting network policy requirements. We show that the problem is NP-hard and derive an efficientapproximate algorithm to reduce communication cost while adhering to policy constraints. Through extensive evaluation, we show thatPLAN can reduce topology-wide communication cost by 38% over diverse aggregate traffic and configuration policies.

Index Terms—Data centers, Policy, Virtual Machine, Migration.

F

1 INTRODUCTION

N Etwork configuration and management is a complextask often overlooked by research that focuses on

improving resource usage efficiency. However, providingsecure and balanced distributed services while maintain-ing high application performance is a major challenge forproviders. In Cloud Data Centers (DC)s in particular, thischallenge is amplified by the collocation of diverse servicesover a centralized infrastructure, as well as by virtualizationthat decouples services from the physical hosting platforms.Applications over DC networks have complex communica-tion patterns which are governed by a collection of networkpolicies regarding security and performance. In order to im-plement these policies, network operators typically deploya diverse range of network appliances or “middleboxes”,including firewalls, traffic shapers, load balancers, IntrusionDetection and Prevention Systems (IDS/IPS), and applica-tion enhancement boxes [1]. Across all network sizes, thenumber of middleboxes is on par with the number of routersin a network, hence such deployments are large and requirehigh up-front investment in hardware totalling thousandsto millions of dollars [2] [3].

• Lin Cui is with Department of Computer Science, Jinan University,Guangzhou, China.E-mail: tcuilin@jnu.edu.cn

• Fung Po Tso is with Department of Computer Science, Liverpool JohnMoores University, L3 3AF, UK.E-mail: p.tso@ljmu.ac.uk.

• Dimitrios P. Pezaros is with School of Computing Science, University ofGlasgow, G12 8QQ, UK.E-mail: dimitrios.pezaros@glasgow.ac.uk.

• Weijia Jia is with Department of Computer Science and Engineering,Shanghai Jiao Tong University, Shanghai China.E-mail: jia-wj@cs.sjtu.edu.cn.

• Wei Zhao is with Department of Computer and Information Science,University of Macau, Macau SAR, China.E-mail: weizhao@umac.mo.

Network policies demand that traffic traverse a sequenceof specified middleboxes. As a result, network administra-tors are often required to manually install middleboxes inthe data path of end points or significantly alter networkpartition and carefully craft routing in order to meet policyrequirements [3]. There is a consequent lack of flexibilitythat makes DC networks prone to misconfiguration, and itis no coincidence that there is emerging evidence demon-strating that up to 78% of DC downtime is caused bymisconfiguration [2] [4].

In order to combat the complexity of middleboxes man-agement, a body of research works have been proposedto dynamically manage network policies. These works canbe broadly classified into the two categories. Virtualiza-tion and Consolidation: Software-centric middlebox applica-tions, including network function virtualization (NFV) [5],have been proposed to separate policy from reachability(i.e., virtualization) [3] [4] and middlebox functions can beconsolidated [3] dynamically. SDN-based policy enforcement:Software-Defined Networking (SDN) [6] has enabled a newparadigm for enforcing middlebox policies [7]. SDN ab-stracts a logically centralized global view of the network andcan be exploited to programmatically ensure correctness ofmiddlebox traversal [8] [9] [10].

On the other hand, Cloud applications can be rapidlydeployed or scaled on-demand, fully exploiting resourcevirtualization. Consolidation is the most common techniqueused for reducing the number of servers on which VMs arehosted to improve server-side resource fragmentation, andis typically achieved through VM migration. When a VMmigrates, it retains its IP address, and the standard 5-tuple(source and destination addresses, source and destinationports, protocol) used to describe a flow remains the same.This implies that migrating a VM from one server to anotherwill inevitably alter the end-to-end traffic flow paths, requir-ing subsequent dynamic change or update of the affectedpolicy requirements [11]. Clearly, the change of the point

2

of network attachment as a result of VM migrations sub-stantially increases the risk of breaking predefined sequenceof middlebox traversals and leads to violations of policyrequirements. It has been demonstrated that, in PACE [1],VM placements in Cloud DC without considering networkpolicies may lead to up to 91% policy violations.

It is common in DCs that a multi-tier application in-volving multiple VMs (e.g., indexing, document, web, etc.)is hosted in non-collocated servers. The underlying trafficflows need to traverse distinct firewalls and IPSes that areattached to different switches and routers, making the trueend-to-end paths longer than the shortest paths due tomiddlebox traversals (see Fig. 1), incurring redundant crosstraffic between switches. Therefore, when deciding whereto migrate any one of these VMs, not only dependency ofVMs but also the locations of these middleboxes have to betaken into consideration. Failing to do so will not only leadto sub-optimal performance due to much longer middleboxtraversal paths, but will also cause service disruption andunreachability as a result of being unable to follow a prede-fined sequence of middlebox traversal rules.

Using the SDN+NFV paradigm described above, such asOpenNF [8] and FlowTags [9], may be able to implement thecorrect sequence of traversal even when VMs migrate, butthey neither ensure shortest traversal paths nor reduce networkcommunication cost. S-CORE [12] has demonstrated that theshortest path between any two communicating VMs givesminimal communication cost in data center network envi-ronments. PACE [1] jointly considers middlebox traversaland VM placement in a Cloud DC environment, howeverit only considers static placement and does not provideany reliable mechanisms to facilitate subsequent dynamicVM migration. In comparison, we focus on dynamic VMmanagement, and our initial effort has shown that policy-aware dynamic VM consolidation can remarkably improvenetwork utilization [13].

In this paper, we explore the joint policy-aware andnetwork-aware VMs migration problem, and present an ef-ficient PoLicy-Aware and Network-aware VM management(PLAN) scheme, which, (a) adheres to policy requirements,and (b) reduces network-wide communication cost in DCnetworks. The communication cost is defined with respect topolicies associated to each VM. In order to attain both goals,we model the utility (i.e., the reduction ratio of commu-nication cost) of VM migration under middlebox traversalrequirements and aim to maximize it during each migrationdecision. To the best of our knowledge, this is the first jointstudy on policy-aware performance optimization throughelastic VM management in DC networks [13].

In short, the contributions of this paper are three-fold:1) The formulation of the policy-aware VM management

problem (PLAN), the first study that jointly considerspolicy-aware VM migration and performance optimiza-tion in DC networks;

2) An efficient distributed algorithm to optimize networkcommunication cost and guarantee network policycompliance; and

3) A real-life implementation of PLAN1 and an extensive

1. The source code for our implementation is available at:https://github.com/posco/policy-aware-plan

performance evaluation demonstrating that PLAN caneffectively reduce communication cost while meetingpolicy requirements.

The remainder of this paper is organized as follows.Section 2 describes the model of policy-aware VM man-agement (PLAN), and defines the communication cost andutility for VM migration. An efficient distributed algorithmis proposed in Section 3, followed by presentation of ourpython-based testbed implementation in Section 4. Section 5evaluates the performance of PLAN. Section 6 outlines re-lated work on VM migration and policy implementations.Finally, Section 7 concludes the paper and discusses futureworks.

2 PROBLEM MODELING

2.1 Motivating Example

We describe a common DC Web service application as anexample to demonstrate that migrating VMs without policy-awareness will lead to unexpected results and applicationperformance degradation.

2.1.1 Topology and Application

Fig. 1 depicts a typical Fat-tree DC network topology [14]that consists of a number of network switches and severaldistinct types of middleboxes. Firewall F1 will filter un-wanted or malicious traffic and protect tenants’ networks inthe DC from the Internet. Intrusion Prevention Systems (IPS),e.g., IPS1 and IPS2, are configured with a ruleset, moni-toring the network for malicious activity, and subsequentlylog and block/stop it. They also provide a detailed viewand checking of how well each middlebox is performingfor the traffic flow. A Load Balancer, e.g., LB1, providesone point of entry to the web service, but forwards trafficflows to one or more hosts, e.g. v1, which provide the actualservice. In this example, v1 is a web server, which acceptsHTTP requests from an Internet client (denoted by u). Afterreceiving such requests, v1 will query data server v2 (i.e., adatabase), perform some computation based on the fetcheddata, and feed results back to the client.

2.1.2 Policy Configurations

Polices are identified through a 5-tuple and a list of mid-dleboxes (A more formal definition is given in Section 2.2).The following policies are configured through the PolicyController to govern traffic related to the web applicationin this example:• p1 = {u, LB1, ∗, 80, HTTP} → {F1, LB1}• p2 = {u, v1, ∗, 80, HTTP} → {IPS1}• p3 = {v1, v2, 1001, 1002, TCP} → {LB2, IPS2}• p4 = {v2, v1, 1002, 1001, TCP} → {IPS2, LB2}• p5 = {v1, u, 80, ∗, HTTP} → {IPS1, LB1}• p6 = {LB1, u, 80, ∗, HTTP} → {}Policy p1: The Internet client first sends an HTTP request

to the public IP address of LB1. All traffic from the Internetmust traverse firewall F1, which is in charge of the first lineof defense and configured to allow only HTTP traffic.

Policy p2: LB1 will load-balance among several webservers and change the destination to web server v1 in the

3

Internet

CR

AS

ES

CR

AS

ES

v1

LB1

F1

IPS1

Networking components:

CR: Core Router

AS: Aggregate Switch

ES: Edge Switch

Middleboxes:

F: Firewall

LB: Load Balance

IPS: Intrusion Prevention

System

Servers:

s1~s3

Virtual machines:

v1: Web server

v2: Data server

Flows:

Original flows

Migrated flows

Migration decisions

flow 1

flow 2

flow 3

LB2

s1 s3

v2

AS

ES

AS

ES

AS

ES

AS

ES

AS

ES

AS

ES

CR CR

IPS2

v2'

s2

Middlebox and

Policy Controller

Fig. 1: Flows traversing different sequences of middleboxes in DC networks. Without policy-awareness, v2 will be migratedto s1, resulting in longer paths for flow 1 and wasting network resources.

example. Traffic will need to traverse IPS1, which protectsweb servers.

Policy p3, p4: v1 will communicate with a data server tofetch the required data, which is in turn protected by IPS2.This traffic will be forwarded toLB2 for load-balancing first,and then reach the data server v2 after traversing IPS2. Theresponse traffic from v2 to v1 also needs to traverse bothIPS2 and LB2.

Policy p5, p6: Upon obtaining the required data from thedata server, the web server will send computed results tothe client. The reply traffic is sent to LB1 first, traversingIPS1, and then forwarded to the Internet client by LB1.Any traffic originating from v1 and destined to an Internetclient needs no further checks, and hence does not need totraverse F1.

2.1.3 Migration Rule

The DC network is often increasingly oversubscribed frombottom to core layers in a bid to reduce total investment. Inorder to reduce congestion in the core layers of DC network,effective VM management schemes cluster VMs to confinetraffic in lower layers of the network such that as muchtraffic as possible is routed only over the edge layer (whichis not oversubscribed) [15] [12]. As a result, VMs as wellas middleboxes, which communicate and exchange packetsmore often and intensively, are collocated in order to keeptraffic within the edge layer boundaries.

Consider the migration of v2 in the above example appli-cation. v2 was originally hosted by server s2. A large trafficvolume needs to be exchanged between web server v1 anddata server v2. This would cost precious bandwidth on corerouters. Without policy awareness, in order to consolidateVMs on servers and keep traffic in the edge layer, v2 may bemigrated to s1 so that v1 and v2 are close to each other.However, it will increase the route length of flow 3 andwaste more network bandwidth. This is because that alltraffic between v1 and v2 need to traverse LB2 and IPS2,

according to the policy rules (i.e., p3 and p4). Consideringpolicy configurations and traffic patterns in this example,when migrating v2, it should be migrated to server s2 toreduce the cost generated between v2 and IPS2.

Clearly, policy-aware VM migration will require findingan optimal placement whilst satisfying network bandwidthand policy requirements. Unless stated otherwise, our dis-cussion and problem formulation in the rest of this paperfocus on policy-aware VM migration with an aim to reducecommunication cost.

2.2 Communication Cost with PoliciesFor simplicity, in the following, we use a multi-tier DCnetwork which is typically structured under a multi-roottree topology (canonical [16] or fat-tree [14]) as example.However, with appropriate link weight assignments (seeSection 4.2.3), the model and solution described in this papercan be extended to any other data center topologies.

Let V = {v1, v2, . . .} be the set of VMs in the DCnetwork hosted by the set of servers S = {s1, s2, . . .}. Letλk(vi, vj) denote the traffic load (or rate) in data per time unitexchanged between VM vi and vj (from vi to vj) followingpolicy pk. Here, λk(vi, vj) can be the average pairwise trafficrates over a certain temporal interval, which can be setsuitably long to match the dynamism of the environmentwhile not responding to instantaneous traffic bursts. This isreasonable that many existing DC measurement studies sug-gest that DC traffic exhibits fixed-set hotspots that changeslowly over time [17] [18] [19].

For a group of middleboxes MB = {mb1,mb2, . . .},there are various deployment points in DC networks. Theycan be on the networking path or off the physical net-work [4]. Without loss of generality, we consider that mid-dleboxes are attached to switches for improved flexibilityand scalability of policy deployment [4]. These middleboxesmay belong to different applications, deployed and config-ured by a Middlebox Controller, see Fig. 1. The centralized

4

Middlebox Controller monitors the liveness of middleboxesand informs the switches regarding the addition or fail-ure/removal of a middlebox. Network administrators canspecify and update policies, and reliably disseminate themto the corresponding switches through the Policy Controllerin Fig. 1.

The set of policies is P = {p1, p2, . . .}. Each policy pi isdefined in the form of {flow → sequence}. flow is rep-resented by a 5-tuple: {srcip, dstip, srcport, dstport, proto}(i.e., source and destination IP addresses and port num-bers, and protocol type). sequence is a list of middleboxesthat all flows matching policy pi should traverse them inorder: pi.sequence = {mbi1,mbi2, . . .}. We denote pini andpouti to be the first (ingress) and last (egress) middleboxesrespectively in pi.sequence. Let P (vi, vj) be the set of allpolicies defined for traffic from vi to vj , i.e., P (vi, vj) ={pk|pk.src = vi, pk.dst = vj}.

We denote L(ni, nj) to be the routing path betweennodes (e.g., VM, middlebox or switch) ni and nj . l ∈L(ni, nj) if link l is on the path. If a flow from VM vi tovj is matched to policy pk, its actual routing path is:

Lk(vi, vj) = L(vi, pink )

+∑

mbks 6=poutk

L(mbks ,mbks+1)

+ L(poutk , vj)

(1)

Not all DC links are equal, and their cost depends on theparticular layer they interconnect. High-speed core routerinterfaces are much more expensive (and, hence, oversub-scribed) than lower-level ToR switches [15]. Therefore, inorder to accommodate a large number of VMs in the DCand at the same time keep investment cost low from aproviders perspective, utilization of the “lower cost” switchlinks is preferable to the “more expensive” router links. Letci denote the link weight for li. In order to reflect the in-creasing cost of high-density, high-speed (10 Gb/s) switchesand links at the upper layers of the DC tree topologies,and their increased over-subscription ratio, we can assigna representative link weight ωi for an ith-level link per dataunit. Without loss of generality, in this case ω1 < ω2 < ω3.

Hence, the Communication Cost of all traffic from VM vito vj is defined as

C(vi, vj) =∑

pk∈P (vi,vj)

λk(vi, vj)∑

ls∈Lk(vi,vj)

cs

=∑

pk∈P (vi,vj)

(Ck(vi, pink ) + Ck(p

ink , p

outk )

+ Ck(poutk , vj))

(2)

where Ck(vi, pink ) = λk(vi, vj)

∑ls∈L(vi,pin

k )

cs is the commu-

nication cost between vi and pink for flows which matchedpk. Similarly, Ck(p

outk , vj) is the communication cost be-

tween poutk and vj for pk, and Ck(pink , p

outk ) is the communi-

cation cost between pink and poutk .Since we jointly consider compliance of network poli-

cies and minimization of network communication costthrough VM migration, which will affect both Ck(vi, p

ink )

and Ck(poutk , vj) above. The value of Ck(p

ink , p

outk ) in (2)

remain unchanged when migrating either vi or vj , and can

be ignored as it makes no contribution to the minimizationof the communication cost during VM migration.

For policy-free flows, which are not governed by anypolicies, their communication cost can be calculated directlybetween the source and destination VMs. Unless otherwisestated, we only consider policy flows for ease of discussionin the rest of the paper.

2.3 Policy-Aware VM Allocation ProblemWe denote MBin(vi) to be the set of ingress middleboxes ofall outgoing flows from vi, i.e., MBin(vi) = {mbj |mbj =pink , pk.src = vi}. Similarly, MBout(vi) = {mbj |mbj =poutk , pk.dst = vi} is the set of egress middleboxes of allincoming flows to vi.

As each server is connected to an edge switch, and eachedge switch can retrieve the global graph of all middleboxesfrom the Policy Controller, we define all the servers thatcan reach middlebox mbk as S(mbk). Thus, to preserve thepolicy requirements, the acceptable servers that a VM vi canmigrate to are:

S(vi) =⋂

mbk∈MBin(vi)∪MBout(vi)

S(mbk) (3)

Hence, for traffic not governed by any policies, S(vi) is allservers that can be reached by vi, i.e., possible destinationswhere vi can be migrated to.

The vector Ri denotes the physical resource require-ments of VM vi. For instance, Ri could have three compo-nents that capture three types of physical resources such asCPU cycles, memory size, and I/O operations, respectively.Accordingly, the amount of physical resource provisioningby host server sj is given by a vector Hj . And Ri � Hj

means all types of resource of sj are enough to accept vi.We denote A to be an allocation of all VMs. A(vi) is

the server which hosts vi in A, and A(sj) is the set ofVMs hosted by sj . Considering a migration for VM vifrom its current allocated server A(vi) to another server s:A(vi) → s, the feasible space of candidate servers for vi ischaracterized by:

Si = {s|(∑

vk∈A(s)

Rk +Ri) � Hj ,∀s ∈ S(vi) \A(vi)} (4)

Let Ci(sj) be the total communication cost induced by vibetween sj and MBin(vi) ∪MBout(vi), where sj = A(vi).

Ci(sj) =∑

pk∈P (vi,∗)

Ck(vi, pink ) +

∑pk∈P (∗,vi)

Ck(vi, poutk ) (5)

Migrating a VM also generates network traffic betweenthe source and destination hosts of the migration, as itinvolves copying the in-memory state and the content ofCPU registers between the hypervisors. The live migrationallows moving a continuously running VM from one phys-ical host to another. To enable that, modern DC networksuse a technique called pre-copy [20], which is comprised ofthree phases: pre-copy phase, pre-copy termination phaseand stop-and-copy phase.

According to [21], the amount of traffic generated duringmigration depends on the VMs image sizeMs, its page dirtyrate Pr , and the available bandwidth Ba. Xmin and Tmax

are user setting for minimum required progress for each

5

pre-copy cycle and the maximum time for the final stop-and-copy cycle respectively [21]. Let Nj denote the trafficon the jth pre-copy cycle. The first iteration will result inmigration traffic equal to the entire memory size N0 = Ms,and time Ms/Ba. During that time, Ms

Pr

Baamount of mem-

ory becomes dirty. Then, the second iteration results inN2 = Ms

Pr

Baamount of traffic. It is easy to obtain that

Nj = Ms(Pr

Ba)j−1 for j ≥ 1. Thus if there are n pre-

copy cycles and one final stop-and-copy cycle, the estimatedmigration cost, i.e., total traffic generated during migration,for VM vi is [21]:

Cm(vi) =Ms ·1− (Pr/Ba)

n+1

1− (Pr/Ba)(6)

There are two conditions for stopping the pre-copycycle corresponding to the minimum required progressfor each pre-copy cycle and the maximum time for thefinal stop-copy cycle respectively [21]: (1) Ms(

Pr

Ba)n <

Tmax · Ba, and (2) Ms(Pr

Ba)n−1 −Ms(

Pr

Ba)n < Xmin. Thus,

we can derive that the number of pre-copy cycles n =min(dlogPr/Ba

Tmax·Ba

Mse, dlogPr/Ba

Xmin·Pr

Ms·(Ba−Pr)e).

Such migration overhead in (6) can be measured by thehypervisor hosting the VM and should not outweigh thereduction in the overall communication cost. We then con-sider the utility in terms of the expected benefit (of migratinga VM to a server) minus the expected cost incurred by suchoperation. The utility of the migration A(vi) → s, wheres ∈ Si, is defined as:

U(A(vi)→ s) = Ci(A(vi))− Ci(s)− Cm(vi) (7)

And the total utility UA→A is the summation of utilities forall migrated VMs from allocation A to A.

The PoLicy-Aware VM maNagement (PLAN) problem isdefined as follows:

Definition 1. Given the set of VMs V, servers S, policies P, andan initial allocation A, we need to find a new allocation A thatmaximizes the total utility:

max UA→A

s.t. UA→A > 0

A(vi) ∈ Si,∀vi ∈ V(8)

PLAN can be treated as a restricted version of the Gener-alized Assignment Problem (GAP) [22]. However, the GAPis APX-hard to approximate [22]. The existing centralizedapproximation algorithms are too complex and infeasibleto implement over a DC environment, which could includethousands or millions of servers, VMs, switches and trafficflows.

Theorem 1. The PLAN problem is NP-Hard.

Proof. To show the non-polynomial complexity of PLAN, wewill show that the Multiple Knapsack Problem (MKP) [23],whose decision version has already been proven to bestrongly NP-complete, can be reduced to this problem inpolynomial time. Consider a special case of allocation A0,in which all VMs are allocated to one server s0, then thePLAN problem is to find a new allocation A for migratingVMs that maximizes the total utility UA0→A. We denoteS′ = S \ {s0} to be the set of destination servers for

migration. For a VM vi, suppose the computed communi-cation cost induced by vi on all candidate servers is thesame, i.e., Ci(s) = δi,∀s ∈ S′, where δi is a constant.Consider each VM to be an item with size Ri and profitU(A(vi) → s) = Ci(A(vi)) − δi − Cm(vi), each serversj ∈ S′ to be knapsack with capacityHj . The PLAN problembecomes finding a feasible subset of VMs to be migrated toservers S′, maximizing the total profit. Therefore, the MKPproblem is reducible to the PLAN problem in polynomialtime, and hence the PLAN problem is NP-hard.

3 PLAN ALGORITHMS

The PLAN problem is a restricted version of the General-ized Assignment Problem (GAP), which has been provenAPX-hard to approximate [22]. We can use some existingcentralized algorithms to approximately maximize the totalgained utility by migration, e.g., [24], [25]. However, thecomputation times of those algorithms are unacceptable forDCs, especially considering the large scales of servers, VMs,switches and millions of traffic flows [12]. In this section, wedesign a decentralized heuristic scheme to perform policy-aware VMs migration.

3.1 Policy-Aware Migration AlgorithmsServer hypervisors (or SDN controller, if used, see Section 4)will monitor all traffic load for each collocated VM vi. A mi-gration decision phase will be triggered periodically duringwhich vi will compute the appropriate destination server sfor migration. If no migration is needed, U(A(vi)→ s) = 0.Otherwise, the total utility is increased after migration whenA(vi) 6= s.

Algorithm 1 and Algorithm 2 show the correspondingroutines for VMs (PLAN-VM) and servers (PLAN-Server),respectively. PLAN-VM is only triggered for a migrationdecision every Tm + τ time. PLAN-VM operations willbe suppressed for Tm time period if vi is migrated to anew server, avoiding too frequent migration or oscillationamong servers. The value of Tm depends on the trafficpatterns, e.g., smaller value for a DC with more stabletraffic. τ is a random positive value to avoid synchronizationof migrations for different VMs. Its value range can bedetermined by the scale of VMs, e.g., smaller range for fewnumber of VMs. PLAN-Server is designed for hypervisors onservers which can accept requests from VMs based on theresidual resources of the corresponding server and preparesfor migration of remote (incoming) VMs.

Several control messages will be exchanged for bothPLAN-VM and PLAN-Server. The interface sendMsg(type,destination, resource) sends a control message of a specifiedtype and resource declaration to the destination. The in-terface getMsg() reads such messages when received. Therequest message is a probe from VM to a destination serverfor migration. A server can respond by sending back anaccept or reject message, according to the residual resourceof the server and the requirements of the VM. If the serveraccepts the request from the distant VM, a migrate messagewill be sent back as confirmation.

For each VM vi, the PLAN-VM algorithm starts withchecking feasible servers, in a greedy manner, for improving

6

Algorithm 1 PLAN-VM for vi

/∗ Triggered every Tm + τ period∗/1: L = ∅2: DECISION-MIGRATION(vi, L)3: loop4: msg ← getMsg()5: switch msg.type do6: case reject7: L = L ∪ {msg.sender}8: DECISION-MIGRATION(vi, L)9: case accept

10: sendMsg(migrate, msg.sender, Ri)11: perform migration: vi → s

12: end switch13: end loop

14: function DECISION-MIGRATION(vi, L)15: s0 ← A(vi)16: Si ← feasible servers in Equation (4)17: X ← argmaxx∈Si\L U(A(vi)→ x)18: if X 6= ∅ && s0 6∈ X then19: s← the one with most residual resources in X20: sendMsg(request, s, Ri)21: else22: exit . exit whole algorithm if no migration23: end if24: end function

utility by calling the function Decision-Migration(), i.e., line 2and 7. The function Decision-Migration() will find a potentialdestination server for vi to perform migration. A blacklistL is maintained during each execution of PLAN-VM toavoid repeating request for the same servers which rejectvi previously. The function Decision-Migration() tries to finda set of servers with maximum migration utility, i.e., line17. If the current server, which hosts vi, is not includedin this set, the one with most residual resources will bechosen as a candidate server for migration, i.e., line 18 ∼ 20.Otherwise, no migration action will take place. This willavoid oscillation of VM migration among the same set ofservers. If a feasible server s accepts vi’s request, vi will bemigrated to s, e.g., line 10 ∼ 11.

For each server sj , the PLAN-Server algorithm keepslistening incoming migration requests from VMs. For arequest from vi, sj will check its residual resources and sendback an accept message if it has enough resource to host vi,i.e., line 5 ∼ 8. Otherwise, it will reject the migration requestof vi, i.e., line 16. For accepted VMs, sj will prepare for theincoming migration by provisionally reserve resources forthem, i.e., line 14.

The PLAN scheme described in Algorithms 1 and 2 candecrease the total communication cost and will eventuallyconverge to a stable state:

Theorem 2. Algorithms 1 and 2 will converge after a finitenumber of iterations.

Proof. The cost of each VM vi is determined by its hostingserver and related ingress/egress middleboxes in MBin(vi)and MBout(vi). Hence, under the policy scheme described

Algorithm 2 PLAN-Server for sj

1: loop2: msg ← getMsg()3: switch msg.type do4: case request5: vi = msg.sender6: Ri = msg.resouce7: if

∑vk∈A(sj)

Rk +Ri ≤ Hj then8: sendMsg(accept, vi)9: else

10: sendMsg(reject, vi)11: end if12: case migrate13: if

∑vk∈A(sj)

Rk +Ri ≤ Hj then14: provisionally resource reservation etc.15: else16: sendMsg(reject, vi)17: end if18: end switch19: end loop

in the previous section, the migrations of different VMs areindependent. Furthermore, each time a migration occurs inline 11 of Algorithms 1, say, A(vi) → s, the utility gainedfrom the migration is varied and always larger than zeroduring each migration, i.e., U(A(vi) → s) > 0. Thus, thetotal induced communication cost, which is always a pos-itive value, is strictly decreasing while VMs are migratingamong servers. Furthermore, the amount of decreased cost(i.e., utilities) is variable for each migration. So, the twoalgorithms will finally converge after a finite number ofsteps.

Suppose the total number of VMs is m and total numberof servers is n. In the worst case, each VM needs to send arequest to all the other servers and is rejected by all of themexcept the last one. Thus, the total message complexity ofPLAN is O(mn).

In Fig. 2, we use the same scenario in Fig. 1 as an exampleto show operations of PLAN algorithms. We consider thesame policies configuration as described in Section 2.1.2.Link weights are assigned to grow exponentially for eachlayer, i.e., e0 for edge layer links, e1 for aggregation layerlinks, e2 for core layer links respectively. Fig. 2a lists all ex-ample flows with src/dst, rates and applied policies. Fig. 2bis the initial placement. Numbers within brackets are totalresources (or requirements) of servers (or VMs). Initially, v1and v2 are hosted on s1 and s3 respectively. Based on theflow rates and route path, the total communication cost is10692.15. Assume v1 first called PLAN-VM in time 1, andfind its current hosted server s1 is the best choice. v1 exitPLAN-VM on line 22. At time 2, v2 called PLAN-VM. Boths1 and s2 can accept v2, but migrating to v2 receives moreutility (line 17). So, v2 sent a request to s2 which wouldaccept the request since s2 has enough space to host v2 (line8 in PLAN-Server). Finally, v2 is migrated to s2 and the totalcommunication cost becomes 6997.62.

7

Flow: src→dst(rate) Policiesf1 : u→ v1(30kbps) p1, p2f2 : v1 → v2(50kbps) p3f3 : v2 → v1(200kbps) p4f4 : v1 → u(100kbps) p5, p6

(a) Example flows

Server(res.) VMs(req.)s1(20) : v1(10)s2(15) : −s3(28) : v2(8)

(b) Initial placement (total cost =10692.15)

Server(res.) VMs(req.)s1(20) : v1(10)s2(15) : −s3(28) : v2(8)

(c) Time 1 (total cost = 10692.15)

Server(res.) VMs(req.)s1(20) : v1(10)s2(15) : v2(8)s3(28) : −

(d) Time 2 (total cost = 6997.62)

Fig. 2: Example of PLAN Algorithms

3.2 Initial Placement

Policy-aware initial placement of VMs is also critical fornew VMs in DC networks. When a VM instance, say vi,is to be initialized, the DC network controller needs tofind a suitable server to host the VM. Initially, predefinedapplication-specific policies should be known for vi. To-gether with vi’s resource requirement Ri and all servers’residual resources in the DC network, the feasible decisionspace Si can be obtained through Equation (4). Since theVM has just been initialized, its traffic load might not beavailable. However, we can still choose the best server tohost vi by considering traffic of all policies for vi equally,e.g., λk = 1,∀pk ∈ P (vi, ∗) ∪ P (∗, vi). In particular, themigration cost Cm(vi),∀vi ∈ V, is set to be zero duringinitial placement. Then, the destined server to host vi isargmaxs∈Si Ci(s).

4 IMPLEMENTATION

In this section, we discuss a real-world implementation ofthe PLAN scheme, highlighting the rationale as well as theoperational and design details of the individual compo-nents.

The foremost principle for the implementation of PLANis to be efficient and deployable in production DC - shouldwe adopt fully or partly decentralized implementation? Althougha fully decentralized PLAN implementation is possible, itrequires substantial modification to middleware for it tobe able to collect relevant VM and flow statistics. Givenmost of the required information such as temporal networktopology, per-flow and per-port based traffic statistics arereadily available in SDN paradigm, we have, hence, adopteda partially decentralized approach, i.e. computing com-munication cost –most computationally expensive part asdefined in Equation (7) – in the hypervisors and monitoringflow statistics using a centralized SDN-based orchestrationin [26].

4.1 Decentralized Modules

4.1.1 VM vs Hypervisor

While conceptually, PLAN relies on VMs to make their ownmigration decisions, in practice this is unsuitable becausethe hosts being virtualized should not be aware that they arein fact running within a virtualized environment preventingVMs from communicating directly with the underlyinghypervisor. Since migration is a facility provided by thehypervisor, we have decided to implement our solutionwithin Dom0, a privileged domain [20], of the Xen [27].

4.1.2 Distributed Cost ComputationUbuntu 14.04 was used for Dom0 and the Python-based xmwas used as the management interface. Open vSwitch [28] isenabled in Xen hypervisor as it only provides more flexibleretrieval of local traffic flow tables, but allows Dom0 tointeract with a SDN controller through OpenFlow protocol.Therefore, in order to compute the communication definedin Equation (7), this module needs to query SDN controllerfor the pair-wise flow statistics, λ, and the link weight, c,along the network path. Upon receiving requests, the SDNcontroller will return a JSON object containing all necessaryinformation.

4.1.3 Server Resource MeasurementSimilar to the distributed cost computation described above,Dom0’s privilege can be exploited to measure residualserver resources such as CPU utilization as well as availablememory size. The measurement should be reported to andaggregated in the controller as part of a JSON object beingexchanged between Open vSwitch and the controller.

4.1.4 Flow MonitoringIn order to keep track of communicating VMs, flow historygathering is required. However, Open vSwitch only main-tains flows for as long as they are active and discards anyinactive flows after 5 seconds, hindering the accumulationof any long-term history. To overcome this limitation, wehave extended the flow table for storing flow-level statistics.For the purposes of PLAN, the flow table must support thefollowing operations: Fast addition of new flows; Retrieval ofa subset of flows, by IP address; Access to the number of bytestransmitted per flow; Access to flow duration, for calculationof throughput. The flow table will be periodically updatedthrough polling Open vSwitch for datapath statistics allow-ing for the storage of flows for as long as it is required.Flows will be stored from when they start up till a migrationdecision is made for a VM.

4.2 Centralized Modules4.2.1 Policy ImplementationA centralized Policy Controller that stores and dissemi-nates policies to the corresponding switches/routers, andaccepts queries from them is a common deployment modelin both non SDN [4] and SDN-based implementation [9].We adopted SDN-based FlowTags [9] architecture to ensurecorrect sequence of middlebox traversals regardless of thelocation of VMs. In the following discussion we assumethat, as a result of FlowTags, network policies are strictlyfollowed no matter wherever VMs are migrated to andwhenever new flows are emitted to the DC network.

8

0 20 40 60 80 1000

0.2

0.4

0.6

0.8

1

Utility Improvement (%)

CD

F o

f U

tilit

y/C

ost

PLAN

PLAN−RIP

(a) CDF: ratio of utility to communi-cation cost

No Migration One Two Three0

10

20

30

40

50

60

70

Pe

rce

nta

te (

%)

Number of Migrations

PLAN

PLAN−RIP

(b) Number of migrations beforeconvergence

Fig. 3: Performance of PLAN

4.2.2 Topology DiscoveryThe knowledge of the network topology is crucial for PLANto assign link-cost each individual link. By utilising theOpenFlow Discovery Protocol (OFDP) and Link Layer Dis-covery Protocol (LLDP) we can easily construct the networktopology. Through the SwitchEnter and SwitchLeave eventsinvoked by OpenFlow enabled switches, active switcheswithin the topology can be discovered. Similarly, LinkAddand LinkDelete events are triggered on the addition andremoval of network links, allowing us to keep track ofinterconnected switches and physical ports.

4.2.3 Link WeightsIn order to reflect the higher communication cost for usinglinks in the higher layers of the topology, increasing weightswith respect to the topology are adopted. In our implemen-tation, link weights are set to grow exponentially for eachlayer, e.g., e0 for edge layer links, e1 for aggregation layerlinks, e2 for core layer links respectively. However, in thegeneral case, link weight assignment can be based on DCoperator’s policy to reflect diverse metrics, such as energyconsumption, performance, fault tolerance, and so on.

4.2.4 Flow StatisticsTo collected pairwise utilization, we used OpenFlow’sStatistic Request messages to periodically collect the numberof packets and bytes processed by the flow entry since theflow was installed. Both edge switches at the source anddestination have similar flow entries for a particular VM-to-VM communication, therefore during collection of flowstatistics, it is important to collect the metrics from thesame switch for a particular VM-to-VM flow. In the currentimplementation, the first time a new flow is discovered fromthe flow statistics, the Data Path ID (DPID) of the switch isstored and subsequent measurements must originate fromthe same switch or will otherwise be discarded.

5 EVALUATION

5.1 Simulation Setup

We have implemented PLAN in ns-3 [29] and evaluated itunder a fat-tree DC topology. In our simulation environ-ment, a single VM is modeled as a collection of socketapplications communicating with one or more other VMsin the DC network. For each server, we have implemented a

VM hypervisor application to manage all collocated VMs.The hypervisor also supports migration of VMs amongdifferent servers in the network. Fat-tree is a representativeDC topology and hence, results from this topology shouldextend to other types of DC networks without loss ofgenerality.

In order to model a typical DC server’s capability, wehave limited the CPU and memory resources for accom-modating a certain number of VMs. For example, a serverequipped with 16GB RAM and 8 cores can safely allow 8VMs running concurrently if each VM occupies one coreand 1GB RAM (the CPU and memory occupied by VMscan be varied). Throughout the simulation, we created 2320VMs on the 250 servers. Each VM has average of 10 randomoutgoing socket connections, which are CBR traffic witha randomly generated rate. We have considered practicalbandwidth limitations such that the aggregate bandwidthrequired by all VMs in a host does not exceed the networkcapacity of its physical interface. Therefore, a VM migrationis only possible when the target host has sufficient systemresources and bandwidth, i.e., a feasible server as defined inEquation (4).

We have also implemented the policy scheme describedin Section 3. In all experiments, we have set 10% of flowsto be policy-free, meaning that they are not subject to anyof the existing network policies in place. For the other 90%of flows, they have to traverse a sequence of middleboxesas required by policies before being forwarded to theirdestination [4]. Each policy-constrained flow is configuredto traverse 1∼3 middleboxes, including Firewall, IPS or LB.

To demonstrate the benefit of PLAN, we compare it withS-CORE [12], a similar but non policy-aware VM manage-ment scheme which has been shown to outperform otherschemes, e.g., Remedy [21]. S-CORE is a live VM migra-tion scheme that reduces the topology-wide communica-tion cost through clustering VMs without considering anyunderlying network policies. A VM migration takes placeso long as it yields a positive utility, the communicationcost reduction outweighs the migration cost, and the targetserver has sufficient resources to accommodate the newVM. In addition, PLAN by default is used with the initialplacement algorithm described in Section 3.2. In contrast,S-CORE initially starts with a set of randomly allocatedVMs. In order to offset such a bias, we have also simulatedPLAN without using the initial placement algorithm (whichis referred to as PLAN with Random Initial Placement orPLAN-RIP in the sequel).

Alongside the communication cost, we also considerthe impact of policies on average route length and linkutilization. Route length is defined as the number of hopsfor each flow, including the additional route for traversingmiddleboxes. Link utilization is calculated on each layer oflinks in the fat-tree topology, i.e., Edge Layer links intercon-nect servers and edge switches, Aggr Layer links intercon-nect edge and aggregation switches, and Core Layer linksinterconnect aggregation switches to core routers.

5.2 Simulation Results

We first evaluate the performance of PLAN. Fig. 3 demon-strates some unique properties of PLAN in progress towards

9

0 5 10 15 20 25 30 350

0.2

0.4

0.6

0.8

1

# of VMs per Server

CD

F o

f V

Ms p

er

Se

rve

r

Initial State

Converged State

Fig. 4: VMs clustering on servers at different states

convergence in terms of communication cost improvementas well as number of migrations. Fig. 3a depicts the im-provement of individual VM’s communication cost aftereach migration through calculating the ratio of utility tothe communication cost of that VM before migration. It canbe observed that each migration can reduce communicationcost by 39.06% on average for PLAN and 34.19% for PLAN-RIP, respectively. Nearly 60% of measured migrations caneffectively reduce their communication cost by as much as40%. Such improvements are more significant when PLAN isused without an initial placement scheme in which VMs areallocated randomly at initialization. Fig. 3b shows the num-ber of migrations per VM as PLAN converges. In PLAN, as aresult of initial placement, only 30% of VMs need to migrateonly once to achieve a stable state throughout the wholeexperimental run. In comparison, 60% of VMs in PLAN-RIPneed to migrate once when it converges. Nevertheless, inboth schemes (with and without initial placement), we ob-serve that very few (< 1%) VMs need to migrate twice andno VM needs to migrate three times or more. These resultsdemonstrate that low-cost, low-overhead initial placementcan significantly reduce migration overhead in general.

We also study the transitioning state behavior of PLANto reveal its intrinsic properties. Fig. 4 shows the snapshotof VM allocations at both the initial and the convergedstates of PLAN. Initially, before PLAN is running, VMs arenearly randomly distributed on servers, e.g., each serverhosts 5∼12 VMs. After PLAN converges, plenty of VMsare clustered into several groups of servers, e.g., nearly16% of servers host 56.55% of the total VMs. Moreover, animportant property we can exploit is that 3.2% of servers areidle when PLAN converges and they can be safely shutdownto, e.g., save power.

Next, we present performance results of PLAN whencompared to S-CORE. Fig. 5 shows the overall commu-nication cost reduction (measured in terms of number ofbytes using network links), average end-to-end route length,as well as link utilization for all layers, for all the threeschemes. Fig. 5a demonstrates that PLAN and PLAN-RIPcan efficiently converge to a stable allocation. PLAN reducesthe total communication cost by 22.42% while PLAN-RIPachieves an improvement of up to 38.27% which is a factorof nearly eight times better than S-CORE whose improve-ment is a mere 4.79%. The reason that PLAN-RIP has higherimprovement is that the initial random VM placement offersmore space for optimization than the already policy-awareinitial placement of PLAN. However, it is evident that thispotential is not exploited by S-CORE. More importantly,

as shown in Fig. 5b, by migrating VMs, the average routelength can be significantly reduced by as much as 20.12%and 10.08% for PLAN-RIP and PLAN, respectively, whileS-CORE only improves it by 4.22%. Being able to reducethe average route length is an important feature of PLANas it implies that flows can be generally completed fasterand are less likely to create congestion in the network. BothFig. 5a and 5b show that PLAN can effectively optimize thenetwork-wide communication cost by localizing frequentlycommunicated VMs as well as to reduce the length of theend-to-end path.

For the same reasons, Fig. 5c and 5d demonstrate thatPLAN can mitigate link utilization at the core and ag-gregation layers by 30.55% and 7.01%, respectively. ForPLAN-RIP, because it starts with random allocation of VMs,which is non-optimal and inefficient compared to PLANwith initial placement, it can reduce link utilization acrossthe core and aggregate layer links by 42.87% and 12.81%,respectively. The corresponding reduction for S-CORE isonly 4.6% and 4.8%, respectively. On the other hand, uti-lization improvement on edge links is marginal for all threeschemes, since they try to fully utilize lower-layer linkswhere bisection bandwidth is maximum. Mitigation of linkutilization at core and aggregation layers means that PLANcan effectively create extra topological capacity headroomfor accommodating larger number VMs and services. Mean-while, Fig. 5 also reveals that PLAN’s initial placement al-gorithm can greatly improve the communication cost, routelength and link utilization, and the algorithm itself can thencontinue to adaptively optimize network resource usage asit evolves.

In order to examine PLAN’s adaptability to dynamicchanges in policy configuration and traffic patterns, Fig. 6presents the algorithm’s performance results when policiesare changed at 50s, 100s, and 150s, respectively, and after thealgorithm had initially converged. Since S-CORE does notconsider the underlying network policies, its performance isindependent of policy configurations and is, thus, omitted.Throughout the experiments shown in Fig. 6, 10% of policiesare removed at 50s, making the corresponding flows policy-free. This leads the DC to an non-optimized state, leavingroom for further optimizing VMs allocations. Due to policy-awareness, PLAN can promptly adapt to new policy pat-terns, reducing the total communication cost, route lengthand link utilization to a great extent. The sudden dropat 50s is due to policy-free flows not needing to traversethrough any middleboxes, hence causing the total commu-nication cost to fall immediately. The same phenomenoncan be observed when new policies are added at 100s andexisting policies are modified at 150s. In particular, disablingsome policies produces new policy-free flows so PLAN canlocalize their hosting VMs, greatly improving bandwidth.So, core-layer link utilization is promptly reduced whensome policies are disabled at 50s. All the above resultsdemonstrate that PLAN is highly adaptive to dynamism inpolicy configuration.

5.3 Testbed Results on VM MigrationWe have also set up a testbed environment to assess thealgorithm’s footprint and the performance of actual VM mi-grations. The servers used all have an Intel Core i5 3.2GHz

10

5 10 15 20 253.5

4

4.5

5

5.5

6

6.5

7x 10

7

To

tal C

ost

Time(s)

PLAN

PLAN−RIP

S−CORE

(a) Total communication cost

5 10 15 20 256.5

7

7.5

8

8.5

Ave

rag

e R

ou

te L

en

gth

(h

op

s)

Time(s)

PLANPLAN−RIPS−CORE

(b) Average route length

5 10 15 20 250.2

0.25

0.3

0.35

0.4

Lin

k u

tiliz

ation

Time(s)

Edge Layer (PLAN−RIP)Aggr Layer (PLAN−RIP)Core Layer (PLAN−RIP)Edge Layer (S−CORE)Aggr Layer (S−CORE)Core Layer (S−CORE)

(c) Link utilization

5 10 15 20 250.18

0.2

0.22

0.24

0.26

0.28

0.3

0.32

Lin

k u

tiliz

ation

Time(s)

Edge Layer (PLAN)Aggr Layer (PLAN)Core Layer (PLAN)

(d) Link utilization

Fig. 5: Performance comparison of PLAN and S-CORE VM migration schemes

100

102

104

1060

100

200

300

400

500

No. of Flows

Mem

ory

Usa

ge (

MB

)

(a) Flow table memory usage

100

102

104

106

0

2

4

6

8

10

No. of Flows

Tim

e (s

)

Add

Lookup

Delete

(b) Flow table operation times for upto 1 million unique flows

100

1050

5

10

15

20

No. of Flows

CP

U U

sage

(%

)

1s polling interval2s polling interval3s polling interval4s polling interval5s polling interval

(c) CPU utilization when updatingflow table at varying polling intervals

0

1

2

3

4

5

6

16 (k=4) 54 (k=6) 128 (k=8) 250 (k=10) 432 (k=12) 686 (k=14)

Tim

e t

o c

olle

ct

flo

w s

tatistics (

s)

Number of hosts (k-value)

20 VMs (80 flows) per host16 VMs (64 flows) per host12 VMs (48 flows) per host

8 VMs (32 flows) per host4 VMs (16 flows) per host

(d) Controller’s flow collection time

Fig. 7: Testbed Results

50 100 150 2003.4

3.45

3.5

3.55

3.6

3.65

3.7

3.75

3.8

3.85x 10

8

Tota

l C

ost

Time(s)

Total Cost

(a) Change of total communicationcost

50 100 150 2007.2

7.3

7.4

7.5

7.6

7.7

7.8

7.9

Ave

rag

e R

ou

te L

en

gth

(h

op

s)

Time(s)

Avg Route Length

(b) Change of route length

Fig. 6: Performance of PLAN with dynamic policies.

CPU with 8GB RAM running Xen hypervisor Ver. 4.2 withUbuntu server 12.04 as Dom0. VMs are Ubuntu 12.04 with1GB RAM allocated to each VM. We mainly stress-tested ourdecentralized module to see if it will be the performance bot-tleneck. We omitted testing our SDN controller since mostfunctions are readily available in OpenFlow specificationand hence will not create bottlenecks.

We first created 1 million flows with all source IP ad-dresses being unique to examine flow tables (Section 4.1.4).This results in a new entry being created at the root of theflow table for each flow. As shown in Fig. 7a, the size ofthe flow table scales sub-linearly. With 10,000 and 100,000entries, the flow table has a memory footprint of only 16MBand 91MB respectively. However, a number of studies havereported that the total number of concurrently active flowsbetween VMs is much more contained: a busy server rarelytalks to servers outside the rack [19] and the number of

active concurrent flows going in and out of a machineis 10 [17]. With a more realistic scenario in which everyvirtual server concurrently sends or receives 10 flows, with100 in the worst case, we anticipate that actual memoryconsumption of the flow table will be between 24.75 KB -186.47 KB for a hypervisor hosting 16 VMs.

To understand the time taken to perform the differentoperations on the flow table, we have measured the time toadd, lookup and delete flows, summing the times over thenumber of flows, for the same sets of flows. Fig. 7b showsthe time to perform various flow table operations with dif-fering numbers of flows in a single operation. Nevertheless,addition, lookup and deletion operations will not need morethan 100ms for a realistic DC production workload of 100concurrent flows.

Next, we measured the CPU usage for manipulating theflow table. This experiment was done by measuring theCPU usage for gradually adding and looking up the flowtable with an interval between 1 second to 5 seconds, asshown in Fig. 7c. It is evident that the performance impactfor adding up to 10,000 flows is negligible for any pollinginterval accounting for less than 5% CPU utilization. In thebest case for 10,000 flows added or updated each time, CPUutilization was only around 1% at a polling rate of 5 seconds,while the worst case CPU utilization was 3.6% at a pollingrate of 1 second. For a more realistic load of 1,000 flows, thebest and worst cases are 0.002% and 0.01%, respectively.

Next, we test the centralized controllers performance oncollecting flow statistics from the network. Flow statisticsare collected from all software switches (Open vSwitch 2.3.1)operating at the hypervisors to be able to account for allVM communication as described in Sec. 4. The controller

11

periodically pulls OpenFlow flow statistics from all hosts toretrieve fine-grained statistics. According to our measure-ments, this is a reasonable solution with a single controllerfor mid-sized infrastructures, giving around 5.0s to retrieveflow statistics from 631 hosts each hosting 20 VMs, as shownin Fig. 7d. For larger infrastructures, for example, whenthere are > 1000 hosts, multiple SDN controllers can bedeployed to collect flows or sampling them using, e.g.,DevoFlow [30].

Similar to other DC management schemes, PLAN willinevitably impose control overhead on the network, such asquerying for flow statistics. An improperly designed controlscheme may overwhelm the network with additional –control– load, but how much overhead will PLAN create? As wedescribed in Section 4, fully distributed network is limited toa few control messages for triggering migration as describedin Algorithm 1 and 2. The partial centralized approachwill inevitably have higher overhead. With the best andworst scenario in a production DC (assuming 500 clusters)described above, the will be 5MB and 500MB respectivelyfor every Tm + τ (used to set sparse intervals).

6 RELATED WORKS

Network policy management research to date has eitherfocused on devising new policy-based routing/switchingmechanisms or leveraging Software-Defined Networking(SDN) to manage network policies and guarantee theircorrectness. Joseph et al. [4] proposed PLayer, a policy-awareswitching layer for DCs consisting of inter-connected policy-aware switches (pswitches). Vyas et al. [3] proposed a middle-box architecture, CoMb, to actively consolidate middleboxfeatures and improve middlebox utilization, reducing thenumber of required middleboxes for operational environ-ments. Abujoda et al. [31] present MIDAS, an architecturefor the coordination of middlebox discovery and selectionacross multiple network functions providers.

Recent developments in SDN enable more flexible mid-dlebox deployments over the network while still ensuringthat specific subsets of traffic traverse the desired set ofmiddleboxes [7]. Kazemian et al. [32] presented NetPlumber,a real-time policy-checking tool with sub-millisecond av-erage run-time per rule update, and evaluated it on threeproduction networks including Google’s SDN, the Stanfordbackbone and Internet2. Zafar et al. [10] proposed SIMPLE, aSDN-based policy enforcement scheme to steer DC traffic inaccordance to policy requirements. Similarly, Fayazbakhshet al. presented FlowTags [9] to leverage SDN’s globalnetwork visibility and guarantee correctness of policy en-forcement. Yaniv et al. [33] designed EnforSDN, a manage-ment approach that exploits SDN principles to decouple thepolicy resolution layer from the policy enforcement layerin network service appliances. Based on SDN, Chaithan etal. [34] tackled the problem of automatic, correct and fastcomposition of multiple independently specified networkpolicies by developing a high-level Policy Graph Abstrac-tion (PGA). Anat et al. [35] presented OpenBox, which de-couples the control plane of middleboxes from their dataplane, and unifies the data plane of multiple middleboxapplications using entities called service instances. Shameliet al. [36] studied the problem of placing virtual security

appliances within the data center in order to minimizenetwork latency and computing costs while maintainingthe required traversing order of virtual security appliances.However, these proposals are not fully designed with VMsmigration in consideration and may put migrated VMs onthe risk of policy violation and performance degradation.

Multi-tenant Cloud DC environments require more dy-namic application deployment and management as de-mands ebb and flow over time. As a result, there is con-siderable literature on VM placement, consolidation andmigration for server, network, and power resource opti-mization [12] [37] [38] [39] [40] [41]. However, none ofthese research efforts consider network policy in their de-sign. The closest work to PLAN is a framework for Policy-Aware Application Cloud Embedding (PACE) [1] to supportapplication-wide, in-network policies, and other realisticrequirements such as bandwidth and reliability. However,PACE only considers one-off VM placement in conjunctionwith network policies and hence fails to deal with andfurther improve resource utilization in the face of dynamicworkloads.

Some other research works focus on scheduling of mid-dleboxes. Duan et al. [42] studied latency behaviours ofsoftware middleboxes, and proposed Quokka which canschedule both traffic and middlebox positions to reducetransmission latencies of the network. Based on a Closnetwork design, Tu et al. [43] studied programmable mid-dlebox that can distribute traffic more evenly to improveQoS. A SDN controller is used to collect global informationto improve bandwidth utilization and reduce latency ofmiddleboxes. Tamas et al. [44] focused on theoretical anal-ysis of middleboxes placement problem. They presenteda deterministic and greedy approximation algorithm forincremental deployment scenarios and showed that it canachieve near optimal performance in the offline scenario.However, all those works above ignore the influence ofnetwork polices on VM migration. They present detailedtheoretical analysis on incremental middlebox deployment

Our another work in [45] proposed Sync, which alsostudies policy-awareness VMs management but focuses on adifferent problem. The PLAN proposed in this paper mainlyfocuses on VMs management with both policy-awarenessand network-awareness. However, Sync studies the syner-gistic between VMs and middleboxes and focuses on theconsolidation of both computing and networking resourcesin datacenters.

7 CONCLUSION AND FUTURE WORK

In this paper, we have studied the optimization of DC net-work resource usage while adhering to a variety of policiesgoverning the flows routed over the infrastructure. We havepresented PLAN, a policy-aware VM management schemethat meets both efficient DC resource management andmiddleboxes traversal requirements. Through the definitionof communication cost that incorporates policy, we havemodeled an optimization problem of maximizing the utility(i.e., reducing communication cost) of VM migration, whichis then shown to be NP-hard. We have subsequently deriveda distributed heuristic approach to approximately reducecommunication cost while preserving policy guarantees.

12

Our results show that PLAN can reduce network-wide com-munication cost by 38% over diverse aggregate traffic loadsand network policies, and is adaptive to changing policyand traffic dynamics.

In this paper, we mainly focus on VM migration andhardware middleboxes. For virtualized middlebox or NFV,new software MB instances can be easily launched oncommodity servers located within the network, and canbe scheduled dynamically. By utilizing the capability ofvirtualized Network Function, the communication cost andefficiency of VM migration can be further improved, whichis our following work currently.

ACKNOWLEDGMENTS

This work is partially supported by Chinese NationalResearch Fund (NSFC) Chinese National Research Fund(NSFC) Project No.: 61402200 and 61373125; NSFC KeyProject No. 61532013; National China 973 Project No.2015CB352401; Shanghai Scientific Innovation Act ofSTCSM No.15JC1402400 and 985 Project of Shanghai JiaoTong University with No. WF220103001; Macau FDCTproject 009/2010/A1 and 061/2011/A3; University ofMacau project MYRG112; US National Science Founda-tion (NSF) under grant 0963979; the UK Engineeringand Physical Sciences Research Council (EPSRC) grantsEP/N033957/1, EP/L026015/1, and EP/L005255/1.

REFERENCES

[1] L. E. Li, V. Liaghat, H. Zhao, M. Hajiaghayi, D. Li, G. Wilfong,Y. R. Yang, and C. Guo, “PACE: Policy-aware application cloudembedding,” in Proceedings of 32nd IEEE INFOCOM, 2013.

[2] J. Sherry, S. Hasan, C. Scott, A. Krishnamurthy, S. Ratnasamy, andV. Sekar, “Making middleboxes someone else’s problem: Networkprocessing as a cloud service,” ACM SIGCOMM Computer Commu-nication Review, vol. 42, no. 4, pp. 13–24, 2012.

[3] V. Sekar, N. Egi, S. Ratnasamy, M. K. Reiter, and G. Shi, “Designand implementation of a consolidated middlebox architecture.” inNSDI, 2012, pp. 323–336.

[4] D. A. Joseph, A. Tavakoli, and I. Stoica, “A policy-aware switchinglayer for data centers,” in ACM SIGCOMM Computer Communica-tion Review, vol. 38, no. 4. ACM, 2008, pp. 51–62.

[5] “Network functions virtualisation white paper #3.” [Online].Available: http://portal.etsi.org/NFV/NFV White Paper3.pdf

[6] N. Feamster, J. Rexford, and E. Zegura, “The road to SDN,” Queue,vol. 11, no. 12, p. 20, 2013.

[7] A. Gember, P. Prabhu, Z. Ghadiyali, and A. Akella, “Towardsoftware-defined middlebox networking,” in Proceedings of the 11thACM Workshop on Hot Topics in Networks. ACM, 2012, pp. 7–12.

[8] A. Gember, C. P. Raajay Viswanathan, R. Grandl, J. Khalid, S. Das,and A. Akella, “OpenNF: enabling innovation in network functioncontrol,” in Proceedings of the 2014 ACM conference on SIGCOMM.ACM, 2014, pp. 163–174.

[9] S. K. Fayazbakhsh, L. Chiang, V. Sekar, M. Yu, and J. C. Mogul,“Enforcing network-wide policies in the presence of dynamicmiddlebox actions using flowtags,” in Proc. USENIX NSDI, 2014.

[10] Z. A. Qazi, C.-C. Tu, L. Chiang, R. Miao, V. Sekar, and M. Yu,“SIMPLE-fying middlebox policy enforcement using SDN,” ACMSIGCOMM Computer Communication Review, vol. 43, no. 4, pp. 27–38, 2013.

[11] S. Sivakumar, G. Yingjie, and M. Shore, “A framework and prob-lem statement for flow-associated middlebox state migration,”2012.

[12] F. P. Tso, K. Oikonomou, E. Kavvadia, and D. Pezaros, “Scalabletraffic-aware virtual machine management for cloud data centers,”in IEEE International Conference on Distributed Computing Systems(ICDCS), 2014.

[13] L. Cui, F. P. Tso, D. P. Pezaros, W. Jia, and W. Zhao, “Policy-awarevirtual machine management in data center networks,” in IEEEInternational Conference on Distributed Computing Systems (ICDCS),2015.

[14] M. Al-Fares, A. Loukissas, and A. Vahdat, “A scalable, commoditydata center network architecture,” in ACM SIGCOMM ComputerCommunication Review, vol. 38, no. 4. ACM, 2008, pp. 63–74.

[15] A. Greenberg, J. Hamilton, D. A. Maltz, and P. Patel, “The costof a cloud: research problems in data center networks,” ACMSIGCOMM computer communication review, vol. 39, no. 1, pp. 68–73,2008.

[16] Cisco, “Data center: Load balancing data center services,” 2004.[17] A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri,

D. A. Maltz, P. Patel, and S. Sengupta, “VL2: a scalable and flexibledata center network,” in Proc. ACM SIGCOMM’09, 2009, pp. 51–62.

[18] T. Benson, A. Akella, and D. A. Maltz, “Network traffic character-istics of data centers in the wild,” in Proceedings of the 10th ACMSIGCOMM conference on Internet measurement. ACM, 2010, pp.267–280.

[19] S. Kandula, S. Sengupta, A. Greenberg, P. Patel, and R. Chaiken,“The nature of data center traffic: measurements & analysis,” inProc. ACM SIGCOMM Internet Measurement Conference (IMC’09),2009, pp. 202–208.

[20] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach,I. Pratt, and A. Warfield, “Live migration of virtual machines,” inProceedings of the 2nd conference on Symposium on Networked SystemsDesign & Implementation-Volume 2. USENIX Association, 2005, pp.273–286.

[21] V. Mann, A. Gupta, P. Dutta, A. Vishnoi, P. Bhattacharya, R. Pod-dar, and A. Iyer, “Remedy: Network-aware steady state vm man-agement for data centers,” in NETWORKING 2012. Springer, 2012,pp. 190–204.

[22] D. G. Cattrysse and L. N. Van Wassenhove, “A survey of algo-rithms for the generalized assignment problem,” European Journalof Operational Research, vol. 60, no. 3, pp. 260–272, 1992.

[23] H. Kellerer, U. Pferschy, and D. Pisinger, Knapsack problems.Springer Verlag, 2004.

[24] R. Cohen, L. Katzir, and D. Raz, “An efficient approximationfor the generalized assignment problem,” Information ProcessingLetters, vol. 100, no. 4, pp. 162–166, 2006.

[25] H. Ramalhinho and D. Serra, “Adaptive search heuristics forthe generalized assignment problem,” Mathware & soft computing,vol. 9, no. 3, pp. 209–234, 2008.

[26] R. Cziva, D. Stapleton, F. P. Tso, and D. Pezaros, “SDN-basedvirtual machine management for cloud data centers,” in CloudNetworking (CloudNet), 2014 IEEE 3rd International Conference on,Oct 2014, pp. 388–394.

[27] “The xen project.” [Online]. Available:http://www.xenproject.org/

[28] “Open vSwitch.” [Online]. Available: http://openvswitch.org/[29] “NS-3.” [Online]. Available: http://www.nsnam.org[30] A. R. Curtis, J. C. Mogul, J. Tourrilhes, P. Yalagandula, P. Sharma,

and S. Banerjee, “Devoflow: Scaling flow management for high-performance networks,” in ACM SIGCOMM Computer Communi-cation Review, vol. 41, no. 4. ACM, 2011, pp. 254–265.

[31] A. Abujoda and P. Papadimitriou, “MIDAS: Middlebox discoveryand selection for on-path flow processing,” IEEE COMSNETS,Bangalore, India, 2015.

[32] P. Kazemian, M. Chang, H. Zeng, G. Varghese, N. McKeown, andS. Whyte, “Real time network policy checking using header spaceanalysis,” in USENIX Symposium on Networked Systems Design andImplementation (NSDI), 2013.

[33] Y. Ben-Itzhak, K. Barabash, R. Cohen, A. Levin, and E. Raichstein,“EnforSDN: Network policies enforcement with SDN,” in 2015IFIP/IEEE International Symposium on Integrated Network Manage-ment (IM). IEEE, 2015, pp. 80–88.

[34] C. Prakash, J. Lee, Y. Turner, J.-M. Kang, A. Akella, S. Banerjee,C. Clark, Y. Ma, P. Sharma, and Y. Zhang, “PGA: Using graphsto express and automatically reconcile network policies,” in ACMSIGCOMM Computer Communication Review. ACM, 2015, pp. 29–42.

[35] A. Bremler-Barr, Y. Harchol, and D. Hay, “Openbox: Enablinginnovation in middlebox applications,” in Proceedings of the 2015ACM SIGCOMM Workshop on Hot Topics in Middleboxes and NetworkFunction Virtualization. ACM, 2015, pp. 67–72.

13

[36] A. Shameli-Sendi, Y. Jarraya, M. Fekih-Ahmed, M. Pourzandi,C. Talhi, and M. Cheriet, “Optimal placement of sequentially or-dered virtual security appliances in the cloud,” in 2015 IFIP/IEEEInternational Symposium on Integrated Network Management (IM).IEEE, 2015, pp. 818–821.

[37] M. Wang, X. Meng, and L. Zhang, “Consolidating virtual machineswith dynamic bandwidth demand in data centers,” in 2011 Pro-ceedings IEEE INFOCOM. IEEE, 2011, pp. 71–75.

[38] J. W. Jiang, T. Lan, S. Ha, M. Chen, and M. Chiang, “Joint vmplacement and routing for data center traffic engineering,” in 2012Proceedings IEEE INFOCOM. IEEE, 2012, pp. 2876–2880.

[39] A. Song, W. Fan, W. Wang, J. Luo, and Y. Mo, “Multi-objective vir-tual machine selection for migrating in virtualized data centers,”in Pervasive Computing and the Networked World. Springer, 2013,pp. 426–438.

[40] Z. Zhang, L. Xiao, M. Zhu, and L. Ruan, “Mvmotion: a metadatabased virtual machine migration in cloud,” Cluster Computing, pp.1–12, 2013.

[41] M. H. Ferdaus, M. Murshed, R. N. Calheiros, and R. Buyya,“Network-aware virtual machine placement and migration incloud data centers,” Emerging Research in Cloud Distributed Com-puting Systems, pp. 42–61, 2015.

[42] P. Duan, Q. Li, Y. Jiang, and S. T. Xia, “Toward latency-awaredynamic middlebox scheduling,” in 24th International Conferenceon Computer Communication and Networks (ICCCN), Aug 2015, pp.1–8.

[43] R. Tu, X. Wang, J. Zhao, Y. Yang, L. Shi, and T. Wolf, “Designof a load-balancing middlebox based on sdn for data centers,”in 2015 IEEE Conference on Computer Communications Workshops(INFOCOM WKSHPS), April 2015, pp. 480–485.

[44] T. Lukovszki, M. Rost, and S. Schmid, “It’s a match!: Near-optimaland incremental middlebox deployment,” SIGCOMM Comput.Commun. Rev., vol. 46, no. 1, pp. 30–36, Jan. 2016.

[45] L. Cui, R. Cziva, F. P. Tso, and D. P. Pezaros, “Synergistic policyand virtual machine consolidation in cloud data centers,” in IEEEInternational Conference on Computer Communications (INFOCOM),April 2016, pp. 217–215.

Lin Cui is currently with the Department of Com-puter Science at Jinan University, Guangzhou,China. He received the Ph.D. degree from CityUniversity of Hong Kong in 2013. He has broadinterests in networking systems, with focuseson the following topics: cloud data center re-source management, data center networking,software defined networking (SDN), virtualiza-tion, distributed systems as well as wireless net-working.

Fung Po Tso received his BEng, MPhil andPhD degrees from City University of Hong Kongin 2006, 2007 and 2011 respectively. He iscurrently Lecturer in the School of ComputerScience at the Liverpool John Moores Univer-sity (LJMU). Prior to joining LJMU, he workedas SICSA Next Generation Internet Fellow atthe School of Computing Science, University ofGlasgow. His research interests include: networkmeasurement and optimisation, cloud data cen-tre resource management, data centre network-

ing, software defined networking (SDN), virtualisation, distributed sys-tems as well as mobile computing and system.

Dimitrios P. Pezaros is Senior Lecturer at theEmbedded, Networked and Distributed Systems(ENDS) Group in the School of Computing Sci-ence, University of Glasgow, which he joined in2009. He is leading research on next generationnetwork and service management mechanismsfor converged and virtualised networked infras-tructures, and is director of the Networked Sys-tems Research Laboratory (netlab) at Glasgow.His research is being funded by the Engineeringand Physical Sciences Research Council (EP-

SRC), the London Mathematical Society (LMS), the University of Glas-gow, and the industry.

Weijia Jia is currently a full-time Zhiyuan ChairProfessor at Shanghai Jiaotong University. Heis leading currently several large projects onnext-generation Internet of Things, environmen-tal sensing, smart cities and cyberspace sensingand associations etc. He received BSc and MScfrom Center South University, China in 82 and84 and PhD from Polytechnic Faculty of Mons,Belgium in 1993 respectively. He worked in Ger-man National Research Center for InformationScience (GMD) from 93 to 95 as a research

fellow. From 95 to 13, he has worked in City University of Hong Kongas a full professor. He has published over 400 papers in various IEEETransactions and prestige international conference proceedings.

Wei Zhao is currently the rector of the Universityof Macau, China. Before joining the Universityof Macau, he served as the dean of the Schoolof Science, Rensselaer Polytechnic Institute. Be-tween 2005 and 2007, he served as the direc-tor for the Division of Computer and NetworkSystems in the US National Science Foundationwhen he was on leave from Texas A&M Univer-sity, where he served as senior associate vicepresident for research and professor of computerscience. As an elected IEEE fellow, he has made

significant contributions in distributed computing, real-time systems,computer networks, cyber security, and cyber-physical systems.