Towards Seamless Live Towards Seamless Live Migration in...

Towards Seamless Live Migration in SDN-Based Data Centers

Kyoomars Alizadeh Noghani

Kyoom

ars Alizadeh N

oghani | Towards Seam

less Live Migration in SD

N-B

ased Data C

enters | 2018:55


Live migration of Virtual Machines (VMs) has significantly improved the flexibility of modern Data Centers (DCs). Ideally, live migration ought to be seamless which requires a comprehensive support from the underlying network. However, legacy DC networks fall short to address the challenges of migration due to their inflexible and decentralized characteristics. In contrast, Software Defined Networking (SDN) is a new networking paradigm, which has the potential to improve the live migration thanks to its comprehensive view over the network, flexible structure, and its close integration with DC management infrastructures.

This thesis investigates networking challenges of short and long-haul live VM migration in SDN-based DCs. We propose solutions to make the intra- and inter-DC live migration procedures more seamless. Furthermore, our proposed SDN-based framework for inter-DC migration improves the management, enhances the performance, and increases the scalability of interconnections among DCs.

LICENTIATE THESIS | Karlstad University Studies | 2018:55

Faculty of Health, Science and Technology

Computer Science


ISSN 1403-8099

ISBN 978-91-7063-991-3 (pdf)

ISBN 978-91-7063-896-1 (print)

Print: Universitetstryckeriet, Karlstad 2018

Distribution:Karlstad University Faculty of Health, Science and TechnologyDepartment of Mathematics and Computer ScienceSE-651 88 Karlstad, Sweden+46 54 700 10 00

© The author

ISSN 1403-8099

urn:nbn:se:kau:diva-70166

Karlstad University Studies | 2018:55

LICENTIATE THESIS



WWW.KAU.SE

ISBN 978-91-7063-991-3 (pdf)

ISBN 978-91-7063-896-1 (print)

Towards Seamless Live Migration in SDN-BasedData CentersKyoomars Alizadeh Noghani

Department of Computer Science, Karlstad University, Sweden

AbstractLive migration of Virtual Machines (VMs) has significantly improved the flex-ibility of modern Data Centers (DCs). Ideally, live migration ought to beseamless which in turn raises challenges on how tominimize service disruptionand avoid performance degradation. To address these challenges, a compre-hensive support from the underlying network is required. However, legacyDCnetworks fall short to help as they take a reactive approach to livemigrationprocedure. Moreover, the complexity and inflexibility of legacy DC networksmake it difficult to deploy, manage, and improve network technologies thatDC providers may need to use for migration.

In this thesis, we explore the application of Software Defined Network-ing (SDN) paradigm for making live VM migration more seamless. Exploit-ing the characteristics of SDN such as its centralized view on network states,we contribute to the body of knowledge by enhancing the quality of intra-and inter-DC live migration. Firstly, for intra-DC migration, we provide anSDN-based solution which minimizes the service disruption by employingOpenFlow-based resiliency mechanisms to prepare a DC network for migra-tion proactively. Secondly, we improve the inter-DC live migration by acceler-ating the network convergence through announcing the migration in the con-trol plane usingMP-BGP protocol. Further, our proposed framework resolvesthe sub-optimal routing problem by conducting the gateway functionality atthe SDN controller. Finally, with the ultimate goal of improving the inter-DCmigration, we develop an SDN-based frameworkwhich automates the deploy-ment, improves themanagement, enhances the performance, and increases thescalability of interconnections among DCs.

Keywords: Data Center, Data Center Interconnection, EVPN, SDN, VMMigration.

iii

v

AcknowledgmentsPhD students, to be successful, should eat the PhD elephant one bite at atime. Quite sure that my licentiate thesis is not just half of the elephant as Iam preparing to defend my final PhD thesis approximately in a year. Howev-er, it does not diminish the benefits I will receive from presenting my work atthis current important stage. Not only I may receive valuable feedback, butalso the acknowledgement page provides me the opportunity to appreciate thepeople who helped me to have a better personal and professional life.

First and foremost, I would like to express my endless gratitude to mymain supervisor, Professor Andreas Kassler for his insightful advice, reliableguidance, and full support. Next, I would like to express my sincere thanksto my committed and punctual co-supervisor, Associate Professor Karl-JohanGrinnemo, who cared so much about my work. I am grateful to ProfessorAndrei Gurtov for reviewing my Licentiate proposal and accepting the roleof opponent in my Licentiate thesis defense. I would also like to thank all myco-authors, colleagues, and friends from theDepartment of Computer Scienceat Karlstad University.

My deep and sincere gratitude to my parents, brother, sister, and my in-laws for their continuous and unconditional love, encouragement, and sup-port.

I dedicate this milestone to my beloved wife, Farzaneh. Thank you foryour love and understanding. I am utterly blessed to have you in my life.

Karlstad University, December, 2018 Kyoomars Alizadeh Noghani

vii

List of Appended PapersThis thesis is based on the work reported in the following appended papers.

I. Cristian Hernandez Benet, Kyoomars Alizadeh Noghani, and AndreasKassler. Minimizing Live VM Migration Downtime Using OpenFlowbased ResiliencyMechanisms. In 5th IEEEConference on CloudNetwork-ing (Cloudnet), Pisa, Italy, October 3–5, 2016.

II. Kyoomars AlizadehNoghani, CristianHernandez Benet, Andreas Kassler,Antonio Marotta, Patrick Jestin, and Vivek Srivastava. Automating Eth-ernet VPNDeployment in SDN-basedData Centers. In 4th IEEEConfer-ence on Software Defined Systems (SDS), Valencia, Spain, May 8–11, 2017.

III. CristianHernandez Benet,Kyoomars AlizadehNoghani, Andreas Kassler,Ognjen Dobrijevic, and Patrick Jestin. Policy-based Routing and LoadBalancing for EVPN-based Data Center Interconnections. In IEEE Con-ference onNetwork FunctionVirtualization and SoftwareDefinedNetworks(NFV-SDN), Berlin, Germany, November 6–8, 2017.

IV. Kyoomars Alizadeh Noghani and Andreas Kassler. SDN EnhancedEthernet VPN for Data Center Interconnect. In 6th IEEE Conference onCloud Networking (Cloudnet), Prague, Czech Republic, September 25–27, 2017.

V. Kyoomars AlizadehNoghani, Andreas Kassler, and Prem SankarGopan-nan. EVPN/SDNAssisted LiveVMMigration betweenGeo-DistributedData Centers. In 4th IEEE Conference on Network Softwarization (Net-Soft), Montreal, Canada, June 25–29, 2018.

Note: Some of the appended papers have been subjected to minor editorialchanges.

Comments on my ParticipationPaper I The initial idea of the paper originated from discussions with mycolleague, Christian Hernandez Benet. I participated in developing all pro-posed resiliency solutions to address the challenges of intra-DC VM migra-tion. Additionally, I was actively involved in writing the paper except for theevaluation section.

Paper II I designed, developed, and implemented the proposed frameworkas well as conducted the experiments for evaluations. Moreover, I am theprincipal author of all parts of the paper. My co-authors assisted me in theevaluation section and writing the paper.

Paper III Christian Hernandez Benet, is the main author of this paper.I actively participated in developing the architecture and traffic engineering

viii

policies, as well as writing the initial draft of the paper.

Paper IV I am the main author of the paper. The idea of the paper camefrom reading the IETF documents about the EVPN technology. I proposedan SDN-based solution and conducted experiments for evaluations.

Paper V The idea of the paper came from watching Cisco summits on datacenter networks. I further investigated the problem, proposed an SDN-basedsolution, and carried out all the experimental evaluations presented in the pa-per. Furthermore, I authored all sections of the paper.

Other Publications• CristianHernandez Benet, RobayetNasim,Kyoomars AlizadehNoghani,and Andreas Kassler. OpenStackEmu - A Cloud Testbed CombiningNetwork Emulation with OpenStack and SDN. In 14th IEEE AnnualConsumer Communications Networking Conference (CCNC), Las Vegas,NV, USA, January 8–11, 2017.

• Cristian Hernandez Benet, Kyoomars Alizadeh Noghani, and JavidTaheri. SDN Implementations and Protocols. In Big Data and SoftwareDefined Networks, Chapter 2, Pages 27-48, The Institution of Engineer-ing & Technology, March 2018.

• Kyoomars Alizadeh Noghani, Cristian Hernandez Benet, and JavidTaheri. SDN helps Volume in Big Data. In Big Data and Software De-fined Networks, Chapter 9, Pages 185-205, The Institution of Engineer-ing & Technology, March 2018.

• Abdelmounaam Rezgui, Kyoomars Alizadeh Noghani, Javid Taheri,Amir Mirzaeinia, Hamdy Soliman, and Nikolas Davis. SDN helps BigData to Become Fault Tolerant. In Big Data and Software Defined Net-works, Chapter 15, Pages 319-336, The Institution of Engineering &Technology, March 2018.

ix

ContentsList of Appended Papers vii

Introductory Summary 1

1 Introduction 3

2 Background 52.1 Live VM Migration . . . . . . . . . . . . . . . . . . . . . . . 52.2 SDN-based Resiliency Mechanisms . . . . . . . . . . . . . . 62.3 VXLAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4 EVPN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.5 Model Driven Network Management . . . . . . . . . . . . . 8

3 Related Work 93.1 Live VM Migration . . . . . . . . . . . . . . . . . . . . . . . 9

3.1.1 Retain Network Connectivity . . . . . . . . . . . . . 93.1.2 Large Convergence Time . . . . . . . . . . . . . . . . 103.1.3 Sub-Optimal Routing . . . . . . . . . . . . . . . . . 10

3.2 EVPN Automation and Management . . . . . . . . . . . . . 113.3 EVPN Policy . . . . . . . . . . . . . . . . . . . . . . . . . . 113.4 EVPN Scalability . . . . . . . . . . . . . . . . . . . . . . . . 11

4 Research Questions 12

5 Contributions 13

6 Research Methodology 14

7 Summary of Appended Papers 16

8 Conclusions and Future Work 18

Paper I:Minimizing Live VM Migration Downtime Using Open-Flow based Resiliency Mechanisms 27

1 Introduction 29

2 Background 312.1 SDN-based Resiliency Mechanisms . . . . . . . . . . . . . . 312.2 Live VM Migration . . . . . . . . . . . . . . . . . . . . . . . 32

x

3 Flow Restoration for VM Migration 333.1 Legacy Network based Live VM Migration . . . . . . . . . . 333.2 SDN-based Live VM Migration . . . . . . . . . . . . . . . . 343.3 SDN-based Live VM Migration with FastFailover . . . . . . 343.4 SDN-based Live VM Migration with Packet bicasting . . . . . 353.5 SDN-based Live VM Migration using Stateful Forwarding . . 36

4 Experimental Evaluation 37

5 Conclusion 40

Paper II:Automating Ethernet VPNDeployment in SDN-basedDa-ta Centers 43

1 Introduction 45

2 Background 47

3 Architecture and Implementation 483.1 High-level Architecture . . . . . . . . . . . . . . . . . . . . . 483.2 Enhanced SDN Functionalities for EVPN . . . . . . . . . . . 493.3 SDN Controller Modules . . . . . . . . . . . . . . . . . . . 50

3.3.1 Neutron . . . . . . . . . . . . . . . . . . . . . . . . . 503.3.2 L2VPN Service . . . . . . . . . . . . . . . . . . . . . 503.3.3 BGP-EVPN . . . . . . . . . . . . . . . . . . . . . . . 513.3.4 PEConfigure . . . . . . . . . . . . . . . . . . . . . . 523.3.5 Existing modules . . . . . . . . . . . . . . . . . . . . 52

4 Evaluation 534.1 Evaluation Methodology . . . . . . . . . . . . . . . . . . . . 534.2 EVPN Deployment Performance . . . . . . . . . . . . . . . 534.3 Module Performance Test . . . . . . . . . . . . . . . . . . . 55


Paper III:Policy-based Routing and Load Balancing for EVPN-basedData Center Interconnections 61

1 Introduction 63

2 Background and Related Work 64

3 Use Cases 65

xi

4 Proposed SDN-based Framework 664.1 SDN Controller Modules . . . . . . . . . . . . . . . . . . . 67

4.1.1 The Neutron module . . . . . . . . . . . . . . . . . . 674.1.2 The L2VPN Service module . . . . . . . . . . . . . . 674.1.3 The Policy Manager (PM) module . . . . . . . . . . . 674.1.4 The Strategy Manager module . . . . . . . . . . . . . 68

4.2 Routing Policy Attributes . . . . . . . . . . . . . . . . . . . 694.2.1 Multi-Homing . . . . . . . . . . . . . . . . . . . . . 694.2.2 Load Balancing . . . . . . . . . . . . . . . . . . . . . 704.2.3 Bandwidth Reservation . . . . . . . . . . . . . . . . 70

4.3 Policy Enforcement . . . . . . . . . . . . . . . . . . . . . . . 704.4 Exemplary Work Flow . . . . . . . . . . . . . . . . . . . . . 71

5 Evaluation and Results 725.1 Evaluation Methodology . . . . . . . . . . . . . . . . . . . . 725.2 Policies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

5.2.1 No Multi-Homing (NO_MH) . . . . . . . . . . . . . 745.2.2 Multi-Homing . . . . . . . . . . . . . . . . . . . . . 745.2.3 Multi-Homing and Load Balancing (MHLB) . . . . . 745.2.4 Load Balancing, but Not Multi-Homing (LB_NO_MH) 755.2.5 Bandwidth Guarantee QoS (QoS) . . . . . . . . . . . 75

5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75


Paper IV:SDN Enhanced Ethernet VPN for Data Center Intercon-nect 81

1 Introduction 83

2 Background 85

3 Proposed Architecture 863.1 BUM Traffic Routing . . . . . . . . . . . . . . . . . . . . . . 873.2 Multicast Tree Inside a DC . . . . . . . . . . . . . . . . . . . 893.3 Proposed Solution . . . . . . . . . . . . . . . . . . . . . . . 893.4 Using SDN Controller for DF Selection . . . . . . . . . . . . 90

4 Evaluation 914.1 Experimental Methodology . . . . . . . . . . . . . . . . . . 914.2 DF Switch-Over . . . . . . . . . . . . . . . . . . . . . . . . 924.3 SDN Controller Triggered DF Change . . . . . . . . . . . . 94

5 Conclusions 95

xii

Paper V:EVPN/SDN Assisted Live VM Migration between Geo-Distributed Data Centers 99

1 Introduction 101

2 Design Challenges for live VM migration across the WAN 103

3 Background 1053.1 VXLAN and EVPN . . . . . . . . . . . . . . . . . . . . . . 1053.2 VM Mobility in EVPN . . . . . . . . . . . . . . . . . . . . 1063.3 Distributed Gateway using EVPN . . . . . . . . . . . . . . 107

4 Architecture and Implementation 1084.1 Controller Modules . . . . . . . . . . . . . . . . . . . . . . . 108

4.1.1 L2VPN-Service . . . . . . . . . . . . . . . . . . . . . 1084.1.2 BGP-EVPN . . . . . . . . . . . . . . . . . . . . . . . 1094.1.3 VXLAN-Manager . . . . . . . . . . . . . . . . . . . 110

4.2 Improving Network Convergence Time across DCs . . . . . 1104.3 Addressing the Hair-Pinning Problem . . . . . . . . . . . . . 112

5 Evaluation 1135.1 Intra Subnet . . . . . . . . . . . . . . . . . . . . . . . . . . . 1155.2 Inter Subnet . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

6 Conclusions 120

Introductory Summary

Towards Seamless Live Migration in SDN-Based Data Centers 3

1 IntroductionServer virtualization has significantly improved levels of efficiency, agility,availability, and flexibility of modern Data Centers (DCs). The ability toseamlesslymigrate a VirtualMachine (VM) between physical servers improvesthe flexibility, makes DCs more fault-tolerant, and helps DC providers toachieve a wide range of objectives such as dynamic load balancing.

Live VMmigration needs to be seamless which requires maintaining ongo-ing connections, providing a negligible downtime, and avoiding performancedegradation after the migration. To conduct a seamless migration, a compre-hensive support from the underlying network is as crucial as exploiting anoptimal migration scheme to transfer the system states. A network can sub-stantially improve the migration procedure in the following ways:

Preserve Network Connectivity: The ability to maintain ongoing con-nections is a prerequisite for seamless VM migration. Typically, migration ofa VM between different networks may require the IP address(es) of the VMto be changed. As a result, ongoing connections of the VM ought to be re-established which respectively violates the seamless feature of live VM migra-tion. A network can provide an opportunity for themigrating VM to preserveits network connections, for instance, by using overlay network technologies.

Fast Network Convergence: Unless a network is not informed aboutthe new location of the migrated VM, the peers of the VM continue sendingtraffic to its former location which consequently introduces further serviceinterruption. Therefore, it is important that the network reduces the inter-ruption interval by promptly re-routing the north and south traffic of the VMto its new location.

Optimal Traffic Routing: When a VM migrates, the network ought tofind a new routing path for its ingress and egress traffic in accordance with itsnew location. Using a sub-optimal path for the ingress and egress traffic ofa migrating VM may significantly degrade the application performance andintroduce problems such as congestion in the network.

Performance Improvement of Migration: A network can significantlyreduce the migration downtime by selecting an appropriate path to transferthe system states of a VM.Moreover, the network can utilize various technolo-gies and protocols to improve the performance of live migration. For instance,the network can prioritize the migration traffic using protocols such as Differ-entiated Services (DiffServ) [13]. Additionally, if the network is intrinsicallydesigned with a high degree of available path diversity, then new transmissionprotocols such as Multipath TCP (MPTCP) [23] can substantially speed upthe migration procedure [40].

Performance Improvement of Required Technologies: To conduct aseamless migration, network providers may need to employ various technolo-gies. For instance, DC providers interconnect their remote sites through over-lay networks or Layer 2 Virtual Private Networks (L2VPNs) solutions whenthey want to conduct inter-DC migration. Typically, these technologies are

4 Introductory Summary

naturally decentralized as they are developed to operate in legacy networks.Depending on the network size, reaching a convergence in decentralized sys-tems is time-consuming. Moreover, the output of the convergence can be sub-optimal as each entity makes a decision according to its local view at a giventime. The migration will improve if a network addresses the problems of theinvolved technologies (e.g., scalability problems).

However, legacy DC networks fall short to provide the aforementionedfunctionalities due to their inflexible and decentralized characteristics, as wellas reactive approaches they take to live migration. Furthermore, conductinga seamless VMmigration in legacy DC networks is difficult as it requires a no-table amount of effort to configure the underlying network, for instance, toprovide interconnection among DCs. Software Defined Networking (SDN)has revolutionized the traditional network architecture, allowing for newwaysto address the network constraints. In the context of live VMmigration, SDNcan help the procedure in the following ways:1) The holistic view of the SDN controller over the network provides oppor-tunities to i) determine optimal routing paths for migration traffic, ii)managethe ingress and egress traffic of the migrating VM efficiently, iii) prepare thenetwork for migration, for instance, by installing new forwarding rules whilehypervisor transfers the system states of the VM, and iv) propagate appropri-atemessages in the control plane to advertise themigrationwhen it is required.2) The SDN controller may benefit from a tight integration with public cloudplatforms such as OpenStack [7] to automatically deploy and flexibly managevarious technologies which may be used for VM migration.3) The SDN controller can speed up the convergence procedure, improve per-formance, and enhance the scalability of decentralized network technologieswhich may be used for VM migration.

The ultimate goal of this thesis is to improve the live VM migration pro-cedure by addressing related networking challenges using SDN-based archi-tecture. Specifically, this thesis targets two problems that legacy networksare unable to tackle as well as those of other problems that affect technologiesrequired to conduct a seamless migration. Slow network convergence and sub-optimal routing problem are two network challenges that are addressed here-in. In the context of intra-DC migration, a number of SDN-based resiliencymechanisms are proposed to decrease the network convergence time. Later,the controller re-optimizes the path to remove sub-optimal routing problems.For inter-DC migration, a novel SDN-based approach is presented that accel-erates the network convergence through message passing in the control planeand optimizes the post-migration traffic routing.

Additionally, this thesis attempts to improve the Ethernet Virtual PrivateNetwork (EVPN) technology [20]. EVPN, which is an L2 interconnectionsolution, is selected to be improved as it plays a key role in inter-DCmigrationscenarios. EVPN helps the VM to retain its ongoing connections while itmigrates between remote sites. Moreover, EVPN is designed to address therequirements of modern DCs such as VM mobility and fine-grained traffic


load balancing. In this thesis, an SDN-based framework is developed insidethe OpenDaylight (ODL) [6] controller to automate the deployment andimprove the management of EVPN-based interconnections. The aforemen-tioned framework is further extended to improve the performance and scal-ability of such interconnections by deploying routing policies and handlingthe broadcast traffic in a better way.

The remainder of this thesis is organized as follows. Section 2 provides anoverview of the technologies that are used through this thesis. Related worksare discussed in Section 3. The research questions and contributions are out-lined in Sections 4 and 5, respectively. The research methods employed bythe appended papers are discussed in Section 6. Section 7 provides a summa-ry of all the appended papers. Finally, Section 8 concludes the introductorysummary of this work.

2 BackgroundThis section provides an overview of the underlying concepts and technologiesthat are used to improve live VMmigration in this thesis. The discussion startswith an overview of live VM migration. Then the discussion proceeds witha description of SDN-based resiliency mechanism as it is the main conceptused to address the challenges of intra-DC live migration. Finally, this sectionconcisely explains Virtual Extensible LAN (VXLAN), EVPN, and model-driven network management technologies. The aforementioned technologiesare exploited in this thesis to improve the live migration procedure.

2.1 Live VM MigrationIn live VM migration, a VM transfers its states such as CPU, associated mem-ory, and storage from one physical server to another. There are mostly threedifferent schemes for live VM migration: pre-copy [18], post-copy [27], andhybrid [46]. All these migration schemes are constituted of the followingphases: i) initialization, ii) reservation, iii) iteration, iv) stop-and-copy, v)commitment, and vi) activation. Initialization and reservation are conductedbefore the VM states are transferred to the destination. In the initialization,the host is checked for compatibility of images, CPU architecture, etc. Duringthe reservation, resources on the destination host required for the newVM arereserved. In the iteration phase, the system states of the VM are transferredfrom the source to the destination node over several iterations while the VM isstill providing its services. In the stop-and-copy phase, the VM stops servicingclients at the source node and transfers its latest system states including themodified or remained memory pages to the destination node. The last twosteps are performed after the stop-and-copy phase is finished. In the commit-ment phase, the destination host acknowledges receiving the consistent copyof the VM and finally the VM starts after the activation phase [9].

Besides the transmission of VM, its corresponding traffic has also to beresumed to finish the migration procedure. To do so, once network devices


are informed about the new location of the VM, they update their routing in-formation and steer the north and south traffic of the VM to its new location.In this thesis, we propose to update routing tables during the initialization(Paper I) and stop-and-copy phase (Paper V) for intra- and inter-DC migra-tion, respectively. The rationale behind this idea is to reduce the migrationdowntime by conducting independent tasks in parallel.

2.2 SDN-based Resiliency MechanismsTypically, SDN mechanisms to cope with network failures are classified intotwo general approaches: i) recovery, and ii) protection. The recovery schemerequires communication between a switch and its controller in order to dy-namically provide backup paths. Once a link/node fails, the controller has tobe notified which then reacts by finding an alternative path. Depending on theworkload of the controller this procedure may require a significant amountof time. In contrast, in the protection schemes the network is designed inadvance to cope with failures. In OpenFlow-enabled networks, protectionschemes are typically implemented using OpenFlow group tables. A flow rule,in an OpenFlow group table, can be defined based on several action bucketsin which actions are defined based on status parameters. A predefined actionis then executed locally without involvement of the controller once param-eters change. To prepare the network for intra-DC VM migration, severalOpenFlow-based protection mechanisms have been exploited in Paper I.

2.3 VXLANTo conduct a seamless migration, the network ought to prevent the ongoingconnections of the migrating VM from being re-established. To do so, thenetwork can either provide an opportunity for the migrating VM to maintainits configuration (e.g., VLAN) or convert the old configuration to the newone by manipulating the south and north traffic of the VM. However, not allsolutions can be deployed in modern DCs as they are not designed for multi-tenant environments and have major scalability problems. VXLAN [35] isan overlay technology that provides L2 extension over a shared L3 underlayinfrastructure network by using MAC in IP/UDP tunneling encapsulation.In the VXLAN-based network, a VM can retain its network configurationwhile the multi-tenancy requirements are provided at scale and the tenant’straffic are clearly isolated. In this thesis we assume that the VXLAN overlaytechnology is deployed inside all DCs. As a result, a VM can retain its networkconfiguration while it migrates inside a DC network.

2.4 EVPNAlthough overlay technologies such as VXLAN are widely deployed in DCnetworks, they are not designed to be a DC interconnect solution. Extend-ing the overlay network across DCs expands the broadcast domain from one


DC network to another which consequently introduces scalability, efficiency,and security problems. Instead, L2VPN is a common solution that networkproviders use to stretch the layer 2 domain between their remote sites.

EVPN encompasses the next-generation Ethernet L2VPN solutions andhas been designed to provide per-flow load balancing, enhance the flexibili-ty, improve the scalability, and decrease the operational complexity of exist-ing L2VPN solutions. EVPN aligns the well-understood technical and oper-ational principles of IP VPNs to Ethernet services by utilizing MP-BGP inthe control plane as the signaling method which removes the need for tradi-tional flood-and-learn1 in the data plane. EVPN in conjunction with VXLANoverlay technology is an appropriate solution to span layer 2 domains betweenmultiple DCs [44].

EVPN comprises four types of messages: Ethernet auto-discovery, Ether-net segment, inclusive multicast, and MAC/IP advertisement route. In thefollowing, we briefly describe the MAC advertisement message and its cor-responding extended community as they have been used in this thesis. Forthe description and use cases for other routing messages, we refer the readerto [20].

MAC Advertisement: The EVPNMAC/IP advertisement message is de-signed to advertise MAC/IP reachability information of a given VM. Whenan EVPN capable node is informed about a new MAC address, it advertisesthe information to its peers through the MP-BGP protocol. All remote peersthat belong to the same EVPN instance import this route and insert the an-nounced MAC address and its reachability information (e.g., Ethernet tag2 )into their MAC VRF (Virtual Routing and Forwarding) table. This processallows the remote nodes to know where to send the traffic [20].

VM Mobility: By adding an additional extended community section tothe MAC/IP advertisement message, EVPN capable nodes can update eachother about VMmovement. Every MACmobility event for a given MAC ad-dress contains a sequence number that increases with eachMACmove. This isused by EVPN capable nodes to ensure that the MAC advertisements are pro-cessed correctly. An EVPN capable node advertises a MAC address for thefirst time with no MAC mobility extended community attribute. When an-other EVPN capable node detects a locally attached MAC address for whichit had previously received a MAC/IP advertisement route, it advertises theMAC address in a MAC/IP advertisement route. The advertisement route istagged with a MAC mobility extended community attribute with a sequencenumber one greater than the last received sequence number [20]. Fig. 1 illus-trates an EVPN operational scenario. A PE (PE-1) advertises a newly learnedMAC address provisioned on a customer network (DC-1) to its peers (PE-2and PE-3) with no additional extended community attribute (Fig.1a). Later,the VM migrates between remote sites. As it is shown in Fig. 1b, the PE ofthe DC on the right side (PE-3) re-advertises the MAC address of the migrat-

1In the context of L2VPNs, the flood-and-learn is the procedure of disseminating mac-addressinformation in the dataplane for the remote PE to learn.

2An Ethernet tag identifies a particular broadcast domain, e.g., a VLAN.


ed VM with an updated sequence number in conjunction with some otherupdated parameters in the MAC advertisement message.

(a) Initial advertisement

(b) Advertisement after the first migration

Figure 1: EVPN MAC mobility scenario.

Distributed Gateway: EVPN offers a unique and scalable solution whichallows gateways to be actively distributed across an arbitrary number of net-work elements. This is especially relevant in cloud environments where a ten-ant may exist or migrate anywhere in the network. Using the combinationof MAC/IP advertisement message and default gateway extended community,an EVPN capable node can distribute the gateway information to its peers.The remote peers treat the received MAC/IP address equivalent to their owngateway interface for the purposes of gateway processing. As a result, the gate-way is distributed around all DC networks that are part of the same EVPNinstance.

In Paper V, EVPN is the key technology that is used for inter-DC VMmigration. First, it is used as the solution that interconnects remote sites. Sec-ond, its capability in advertising themigration is improved in away to decreasethe network convergence time. Finally, the EVPN capability to distribute thegateway information is used to resolve the sub-optimal routing problem.

2.5 Model Driven Network ManagementBy increasing the size of networks and emergingDCs, it is gettingmore difficultfor infrastructure providers to configure the network devices. Large networks


are usually multi-vendor where each network element is configured in a dif-ferent way, e.g., using different command line interface. As a result, there is aclear need across the industry to simplify the configuration and managementfor both networks and devices. Model-driven network management auto-mates and accelerates the procedure of creating services through the wholenetwork. In model-driven network management, a data model is used forrepresenting services and configurations together with standard protocols totransmit the modeled data. YANG [12] has clearly positioned itself as the da-ta model language for representing network device configurations, state data,remote procedure calls, and process notifications in a standard way. Data de-fined in YANG is transmitted to a network device using a protocol such asNETCONF [22]. Over the last couple of years, YANG and NETCONFhave gained traction in the networking industry and there is a growing set ofproducts from all vendors supporting YANG as data model and NETCONFas the networkmanagement protocol. In Paper II, the SDN controller uses themodel-driven network management to automate the configuration of EVPNinstances on edge routers of a DC.

3 Related WorkThis section describes the research related to the work presented in this thesis.

3.1 Live VM MigrationIn general, there are two ways to improve the performance of VM live mi-gration: i) improve the algorithm for live migration used by the hypervisors,e.g., by compressing the memory pages during migration, and ii) improve theperformance of the migration on the network level [30]. Although researchfocusing on the former solution abound [15,17,37,51,52,55,58], the impact ofDC network in conducting a seamless migration is less investigated. Support-ing seamless live VMmigration poses several important networking challengeswhich are discussed in the following sections.

3.1.1 Retain Network Connectivity

The ability to maintain ongoing connections is a prerequisite for seamless VMmigration. This goal is easy to achieve when the VM migrates between twophysical servers inside a DC where an overlay technology covers the wholenetwork. However, different network settings (e.g., different IP address space)in remote sites make it difficult to seamlessly transfer active network con-nections. In this regard, various solutions have been proposed to help a mi-grating VM to maintain its ongoing connections. Mobile IP-based solution-s [28, 42] have been proposed in [26, 48] to address this problem. Bradford etal. [15] used a combination of dynamic DNS and IP tunneling to maintain thenetwork state during long-haul live migration. Alternative solutions such as


legacy L2VPN technologies, overlay networks, and SDN-based methods havebeen deployed in other studies [14,24,36,43,55] to address the same problem.

However, there are problems with the applicability of the proposed so-lutions. For instance, mobile IP-based solutions cause triangular routing, re-quire the VM or its corresponding hypervisor to have a modified protocolstack, or need all involved networks to support a specific protocol. Extendingan overlay network from one DC to another DC is neither scalable nor effi-cient as it extends the broadcast domain. Moreover, the DC administratorsmay need to deploy different overlay technologies in their remote sites. Lega-cy L2VPN solutions are limited in terms of redundancy, scalability, flexibility,and forwarding policies. Finally, proposed SDN-based solutions maps the oldnetwork addresses to new ones to maintain the ongoing connections whichis not a scalable solution. Furthermore, the SDN-based solutions assume thatthe controller has a holistic view over all DC networks. Nonetheless, due tothe constraint of security policies and scalability requirements, DCs usuallyhave their own controller.

3.1.2 Large Convergence Time

Besides the system state migration, the ingress and egress traffic of the VMmust also be migrated to its new location. The total time that is required fornetwork devices to update their routing tables according to the latest changesin the network (e.g., VM migration) is known as convergence time. In thelegacy networks, the procedure of network convergence is postponed to afterthe state migration is finished which introduces further service interruption.The ideal solution is to conduct flow migration in parallel with the systemstate migration as proposed in [14, 57]. The key idea in these papers is toproactively prepare the network for VM migration using an SDN-based ar-chitecture and OpenFlow-based forwarding rule rewriting. However, neitherof these solutions use resiliency mechanisms to re-route the north and southtraffic of the VM. Additionally, the proposed solutions are not applicable tointer-DC VM migration as different DCs usually have independent manage-ment infrastructures while these papers consider a single controller for allnetworks.

3.1.3 Sub-Optimal Routing

Using a sub-optimal path for egress and ingress traffic of the migrating VM isan important problem that has to be addressed. In addition to degrading theperformance of the migrated application, sub-optimal routing decreases theperformance of the whole network by increasing the congestion level of thelinks. Although the impact of sub-optimal routing in intra-DC migration isnegligible, it may have a disastrous effect on inter-DC VM migration. To thebest of our knowledge, sub-optimal routing problems related to inter-DC liveVM migration have not been investigated previously.


3.2 EVPN Automation and ManagementService deployment is one of the main concerns of network providers. Manu-al deployment of services in the network is a labor-intensive, slow, and error-prone task. Moreover, considering the size of modern DCs, manual deploy-ment of a service is not feasible. Therefore, both industry and academia puta lot of effort in developing new protocols (e.g., SNMP [16], NETCONF,etc.) and solutions (e.g., model-driven network management) to automate orpartially facilitate the deployment of services in a network. However, only afew of these efforts addressed the complexity of VPNs deployment.

Authors in [32, 50, 54] propose solutions to facilitate the deployment ofL3VPN and alleviate its corresponding complexities. Regarding the L2VPNsolutions, the number of studies are even less. Authors in [55] utilize a cen-tral VPN controller to establish the Virtual Private LAN Service (VPLS) [29]connections between remoteDCs. Likewise, authors in [31] propose an SDN-based solution to automate VPLS tunnel establishment between authorizedProvider Edges (PEs). To the best of our knowledge, a framework that auto-mates the deployment of EVPN on PE routers of a DC does not exist.

Once a service is deployed in the network, the next challenge is to flexiblymanage the service. The management complexity of a network technologymay hamper the efficiency of provisioning that technology. This fact is par-ticularly true for MPLS-based VPN solutions as a high number of protocolsare involved which make the management procedure of VPNs cumbersome.VPNService [8] is among few efforts that is developed to facilitate the man-agement of VPN services in a network. The VPNService module interactswith the OpenStack as well as other modules inside the ODL controller andimproves the management of L3VPNs that are deployed in the network. Tothe best of our knowledge, a similar framework for EVPN does not exist.

3.3 EVPN PolicyPolicies specify conditions and actions that are applied to a system in order toachieve a specific system operation goal. For instance, the network providermay desire to prioritize a specific traffic (e.g., inter-DC migration traffic) overthe others during a pre-defined time interval. Ideally, network providers shouldbe able to define the corresponding policy through high-level programmingabstractions, without having to deal with the implementation complexities.Although various intent-based solutions are developed to realize this require-ment, they are limited to a number of predefined policies [49,56]. Moreover,the network actions taken by these policies are static [21, 33, 34]. Besides,routing policies for DC interconnection solutions, such as EVPN, are notaddressed in any of the existing frameworks.

3.4 EVPN ScalabilityManaging broadcast traffic is a key requirement for L2 technologies. Broad-cast traffic not only consumes excessive resources but also introduces security


vulnerabilities. For DC interconnect technologies such as EVPN, managingbroadcast traffic is even more imperative as it can severely degrade the perfor-mance of inter-DC migration. However, management of broadcast traffic inEVPN technology confronts new challenges. One of the advantages of EVPNover preceding L2VPN technologies is that it provides an All-Active (A-A)mode of operation by which the traffic can truly be multi-homed. Althoughthe A-Amode of operation is a very beneficial feature, it may introduce severescalability problems if broadcast traffic is importedmultiple times into theDCand through different paths. The router that is responsible to handle broadcasttraffic is known as Designated Forwarder (DF). The default DF election algo-rithm defined by the EVPN standard [20] is called “service-carving” which isa distributed algorithm that each PE runs independently. However, service-carving encounters a number of fundamental problems such as inconsistentoutput, undesirable DF swap, and fairness problems. Although a number ofsolutions [25,39,45,47] are proposed to address the aforementioned problems,they all fail to fully address the problems.

4 Research QuestionsThis thesis addresses the following research questions:

RQ1: How can SDN improve live VM migration in DC networks?

To improve the live migration in DC networks, most of the research focusedon enhancing migration schemes. However, live VM migration confronts anumber of networking challenges that can severely degrade the performanceof migration. Among all networking challenges of VM migration, two chal-lenges are considered in this thesis: i) slow network convergence, and ii) sub-optimal routing problem. We aimed at identifying, using, and extending theSDN capabilities to mitigate these problems (Paper I and Paper V).

RQ2: How can SDN automate the deployment and improve the management ofDC interconnections?

To conduct inter-DC live migration, remote sites ought to be interconnect-ed. Typically, interconnecting remote sites (e.g., geo-dispersed DCs) is a verytime-consuming and error-prone task as it requires significant efforts for con-figuration. The next challenge is to efficiently manage these interconnections.The ability of the SDN controller in centrally managing the network devicesthrough various protocols motivated us to investigate an SDN-based solutionto automate the deployment and improve the management of EVPN-basedinterconnections (Paper II).

RQ3: How can SDN improve the performance and scalability of DC intercon-nections?

Improving the performance and scalability of DC interconnect solutions can


consequently enhance the total performance of migration. The performanceof DC interconnect solutions can be improved by applying routing policiesin the DC network. However, DC providers are usually unable to apply poli-cies as it requires them to have a comprehensive knowledge of network pro-tocols. On the other hand, inefficient management of broadcast traffic is acommon problem for most of the network technologies which introduces se-rious scalability problems. The efficient management of broadcast traffic is ofa paramount importance in EVPN technology as it provides the A-A modeof operation which leads to the broadcast traffic imported into the networkthrough different paths. The comprehensive network view of the SDN con-troller and its close integration with DCmanagement systems motivated us todevelop an SDN-based framework that facilitates the deployment of policies(Paper III) and improves the scalability of DC interconnections built aroundEVPN (Paper IV).

5 ContributionsThe main objective of this thesis is to improve the intra- and inter-DC live VMmigrations using SDN. Additionally, this thesis proposes and develops SDN-based solutions to automate the deployment of EVPN connections amongDCs, improve the management, enhance the performance, and increase thescalability of EVPN-based interconnections. These general contributions arereflected in various partial contributions made in Papers I-V, as follows:

1. An SDN-based framework for live VM migration.

Paper I and Paper V address research question RQ1. The proposed SDN-based solutions in these papers accelerate the network convergence and resolvesub-optimal routing problems for both intra- and inter-DC VMmigration. InPaper I, we propose to split the intra-DC live VM migration procedure intotwo parts: i) a temporary local repair, and ii) a path re-optimization. In thefirst phase, the SDN controller proactively installs backup paths for all theongoing connections of the migrating VM towards the new location usingOpenFlow-based resiliency mechanisms. Once the VM is resumed at the newlocation, the SDN controller enters the second phase and removes sub-optimalrouting by re-optimizing the paths.

In the inter-DCVMmigration (Paper V), the controller serializes a partic-ular EVPN message, the MAC Advertisement message with MAC Mobilityextended community, when the VM enters the stop-and-copy phase. Thismessage contains information about the migration which the controller prop-agates to its peers in the control plane through MP-BGP protocol. Upon re-ceiving themessage, remoteDCs update their corresponding forwarding rules.This procedure reduces the migration downtime as it conducts network con-vergence in parallel with state migration. Furthermore, by deploying the gate-way functionality in the controller the inter-subnet sub-optimal routing prob-lem is addressed. Results show that the SDN-based solutions can significantlyimprove the performance of migration in comparison to legacy methods.


2. An SDN-based framework to automate the deployment and improve the man-agement of EVPN.

In Paper II, multiple modules are designed, extended, and implemented inthe ODL controller to develop an SDN-based framework which automatesthe deployment and improves the management of EVPN-based interconnec-tions. The implemented modules use model-driven network management toautomate the deployment of EVPN instances on PE routers of the DC. Addi-tionally, the controller integrationwithOpenStack and its capability in under-standing EVPN related messages help the controller to collect a comprehen-sive information about EVPN instances deployed in the DC network. There-fore, the DC administrator can retrieve information about EVPN instancesthrough northbound APIs and manage the instances through high-level com-mands without being involved in the complexity of the underlying network.It is worth mentioning that the developed framework also mitigates existingproblems such as ARP flooding and silent host problem3 within the DC net-works. This framework contributes to address the research questionRQ2 andprovides the baseline needed to obtain other objectives in Paper III, Paper IV,and Paper V.

3. An SDN-based framework to improve the performance and scalability of EVPN.

Improving the quality of EVPN-based interconnections (RQ3) is the mainmotivation for Paper III and Paper IV. Paper III presents a policy-basedframework to flexibly manage and deploy routing policies for EVPN-basedinterconnections. The main motivation of this paper is to help DC providersto deploy various policies, e.g., traffic engineering policies, for EVPN-basedinterconnections without being involved in network complexities. Paper IVproposes an SDN-based solution to improve the management of broadcasttraffic in EVPN-based interconnections. The proposed solution in this papermitigates the problem of standard methods in dealing with multi-destinationtraffic and further increases the scalability of EVPN technology.

6 Research MethodologyThe research work described in this thesis follows the traditional scientific ap-proach in experimental computer science [19] which includes an iterative cy-cle of literature review, problem formulation, hypothesis building or descrip-tion, verification, and analysis. Typically, hypothesis verification methods in-clude analytical model, simulation, real-world measurements, and emulation.All of these methods have their respective strengths and weaknesses.

An analytical method, mathematically models a system under investiga-tion, and can provide a quick insight into the overall behavior of the system.However, the analytics results can be less accurate in comparison with other

3Silent host is a host that is in operation of which network nodes are not aware.


methods. Computer simulations, on the other hand, involve more detailedfeatures of the underlying system but it is often based on many assumptionsand artificial modeling in order to reach a certain realistic degree. As a result,if the utilized model ignores a critical behavior of the system, and improperlyhandles initial conditions it may lead to incorrect conclusions. Furthermore,a considerable effort is needed to write and debug a reasonably sized simula-tion program. Although difficult to design and expensive to deploy, real-worldmeasurement represents the lowest level of abstraction compared to analyti-cal and simulation methods. Finally, we have emulation, which is a hybridapproach between the simulation and experimentation. In emulation, somecomponents of the experimental setup are abstracted and some componentsrun within a real environment.

In this thesis, both emulation and real-world measurements are used to val-idate hypotheses. Although we are aware that emulation-based measurementsdo not allow to understand all the implications of a real-life situation such asthe variable VM downtime, the emulation-based measurements are more pre-vailed. The main reason is that conducting a real VM migration, for instance,long-haul migration, requires significant infrastructure such as a distributedOpenStack environment, EVPN-VXLAN capable routers, etc., to which wedo not have access.

An emulation approach is utilized to address the RQ1 and RQ3 using thewell-known CORE [2] and Mininet [4] network emulators. These emulatorsprovide us with an opportunity to evaluate the efficiency of our proposedsolutions in network topologies similar to real DC networks. The experimentresults are presented in Paper I, Paper III, Paper IV, and Paper V.

To tackle RQ2, we developed several modules inside the ODL controller.The performance of the implemented modules are evaluated within the fol-lowing ways: 1) a black-box test, and 2) two white-box tests. The black-boxtest and the first white-box test are real-world measurements. The secondwhite-box test, on the other hand, is an example of an emulation-based mea-surement. While the black-box test evaluates the performance of the wholecontroller, white-box tests evaluate the performance of specific modules insidethe controller.

To conduct the black-box and the first white-box tests, the Bagpipe soft-ware router [1] is extended to generate EVPN workloads in three modes in-cluding: burst, one-by-one, and single workload. For the black-box test, theperformance of the whole controller is evaluated when the Bagpipe routeroperates in one-by-one and burst modes. In the first white-box test, the per-formance of the modules we added to the controller is evaluated by measuringthe processing time of EVPN messages when the Bagpipe router operates inthe one-by-one mode.

In the second white-box test, the time consumed by each module insidethe controller to initialize and deploy EVPN instances in the DC network isevaluated while the controller interacts with an EVPN-enabled Nokia routerimported into the GNS3 [3] emulator. To conduct the same evaluation withthe real-world measurement we need a router which can support a wide range


of protocols similar to real DC edge routers. In particular, we need an EVPN-enabled router supporting NETCONF protocols and YANG data model lan-guage to which we do not have access. The performance results are presentedin Paper II.

7 Summary of Appended PapersPaper I – Minimizing Live VM Migration Downtime Using OpenFlowbased Resiliency Mechanisms

Besides the transmission of VM, its corresponding traffic has also to be re-sumed to finish the migration procedure. The time required to update thenetwork connections further increases the service downtime. Hence, fromthe networking point of view, it is very important to restore connectivityas fast as possible to provide a resilient and seamless live VM migration. Inthis paper, we proposed several novel schemes based on SDN that allow a fastrestoration of network connectivity for a VM migration within a DC. Theproposed solutions include OpenFlow resiliency method, packet bicasting,and stateful forwarding. Unlike legacy networks that defer the network con-vergence to after the VM is up-and-run at the new location, in our proposedmethod the controller proactively exploits one of the aforementioned schemesonce it is informed about the VM migration. An evaluation using SDN ex-tensions of the network emulator CORE shows that our proposals effectivelyreduce the downtime leading to a more seamless live VM migration.

Paper II – Automating Ethernet VPN Deployment in SDN-based DataCenters

By increasing the size of networks and emerging multi-vender environmentssuch as DCs, it is getting difficult for infrastructure providers to deploy aservice in their networks. Despite the efforts made to offer a faster config-uration of network devices, network providers are not able to deal with on-demand services. The introduction of a new customer or service requires aset of configuration procedures which involves administrators to go througha time-consuming and error-prone configuration process. The next challengeis to effectively manage the services that are deployed in the network. Tomanage a service in a network, the administrators ought to have extensiveknowledge about network status and protocol specifics. However, the feasi-bility of this approach is questionable due to the size of modern DCs and thewide range of services and protocols deployed in DCs. This paper proposes anSDN-based framework that automates the EVPN deployment and improvesits management inside DCs using OpenStack and OpenDaylight. First, theOpenDayligh controller is extended with several modules to receive high-levelcommands from the OpenStack and deploy EVPN instances on DC routersusing YANG data model language and NETCONF protocol. Second, theclose integration of the OpenDaylight controller with public cloud platforms


such as OpenStack helps the controller to have a comprehensive informationabout the underlying network. On the other hand, the controller knows howto communicate with network devices. As a result, the SDN controller canflexibly manage various services such as EVPNs from a centralized point. Thescalability analysis shows the feasibility of our proposed solution.

Paper III – Policy-based Routing and Load Balancing for EVPN-basedData Center Interconnections

Policy-based management attempts to simplify the management of DC andhelps DC providers to meet the service level agreements negotiated with eachend-user. However, applying policies within a DC is complex, prone to mis-configuration, and requires the administrator to have a comprehensive insightinto the network status and protocol specifics. This paper presents an SDN-based framework for policy-driven DC interconnections that are built aroundEVPN. The framework is designed to translate routing and other traffic engi-neering policies, which are defined for EVPN instances, into an appropriatelow-level network actions to meet the policy goals. The proposed frameworkavoids the need to hard-code the controller behavior and allows to modify therouting, multi-homing, and load balancing strategies within and across DCs.To illustrate the benefits of the presented approach, we have implemented fivesimple traffic engineering strategies and evaluated them in emulated intra-DCand inter-DC networks. Our evaluation results show how different trafficengineering policies lead to a different performance in terms of throughput,latency, and flow completion time.

Paper IV – SDN Enhanced Ethernet VPN for Data Center Interconnect

One of the major advantages of EVPN over legacy layer 2 VPN solutionsis providing an All-Active mode of operation so that the traffic can truly bemulti-homed on PE routers. However, when the Customer Edge (CE) routeris multi-homed to one or more PE routers, it is necessary that only one of thePE routers should forward broadcast, unknown unicast, and multicast trafficinto the DC. Importing multi-destination packets through multiple routers isdestructive and leads to scalability problems such as undesirable flooding, indata and control plane. This problem ought to be addressed as it may severelydegrade the performance of inter-DC migration. The PE router that assumesthe primary role for forwarding BUM traffic to the CE device is called thedesignated forwarder. The designated forwarder election algorithm definedby the EVPN standard encounters a number of fundamental problems suchas inconsistent output, undesirable designated forwarder swap, and fairnessproblems. In this paper, we introduce an SDN-based architecture for EVPNsupport, where the controller selects a designated forwarder in accordanceto link utilization of DC. We show how the comprehensive view over thenetwork using the SDN architecture helps to select an appropriate designatedforwarder leading to lower overhead and better performance.


Paper V –EVPN/SDNAssisted Live VMMigration betweenGeo-DistributedData Centers

In geo-distributed DCs, multiple DC sites are interconnected over the WAN,typically using MPLS networks. In contrast to intra-DC networks, wherelinks have typically less than one millisecond latency and 40 or 100 Gbps linkcapacity is common, WAN connections have significantly higher latency andlower capacity. The higher latency and lower link capacity prolongs the mi-gration downtime and seriously degrades the performance of VM migration.This paper presents a novel approach for long-haul live VMmigration betweengeo-distributed DCs that accelerates the network convergence and optimizesthe post-migration traffic routing. First, the controller reduces the networkconvergence time by pre-advertising the migration when the VM enters thestop-and-copy phase. To do so, the controller serializes and propagates an ap-propriate EVPNmessage in the control plane using MP-BGP protocol. Uponreceiving this message, the peers of the controller in remote DCs start updat-ing the flow tables inside their domain. As a result, the network starts con-vergence while the state migration is in progress. Second, the SDN controllerresolves the sub-optimal routing problem that arises as a result of migrationimplementing a distributed anycast gateway. By performing experiments inemulated scenarios, we find that our approach significantly reduces the down-time compared to alternative schemes, particularly when the latency betweenDCs is higher. Furthermore, addressing the sub-optimal routing problem re-markably increases the performance of migrating VM.

8 Conclusions and Future WorkLive VM migration is a promising solution for data center administrators toachieve a wide range of objectives, from load balancing to disaster evacuation.Although many solutions have been proposed to improve the VM migrationschemes, the networking aspects of live VMmigration are mainly overlooked.The work presented in this thesis investigates the networking challenges ofVMmigration, in particular, slow network convergence and sub-optimal rout-ing problem, and proposes SDN-based solutions to improve the intra- andinter-DC migration procedure.

To conduct inter-DC migration remote sites ought to be interconnected.The EVPN is the interconnection technology that is used in this thesis due toits outstanding features. Automating the deployment, improving the manage-ment, enhancing the performance, and increasing the scalability of EVPN-based interconnections are other objectives that are investigated in this the-sis. We developed several modules inside the ODL controller to automate theEVPN interconnection deployment onDC edge routers using Yang datamod-el language and NETCONG protocol. Further, we extended the controller toimprove the management and enhance the quality of EVPN-based intercon-nections. Table 1 summarizes the research objectives of this thesis and howthese objectives are evaluated.


To have a seamless migration addressing the system and networking chal-lenges of migration is crucial. Nonetheless, scheduling the migration is of thesame importance. Migration of a VM regardless of its system and the under-lying network states may lead to severe service interruptions. This problemis exacerbated when applications that are running on multiple VMs constructa service chain. In such a scenario, migration of the VM can significantly de-crease the performance of the whole chain. In a broader picture, any changein the current state of the network ought to be carefully arranged otherwise,it may lead to notable service disruptions. Principally, scheduling reconfigu-ration in a network helps administrators to achieve their goals, for instance toreach an energy-efficient state, while the negative impacts on the network andapplications during the reconfiguration are minimized. Knowing the cost ofmigration and its impact on the network, we intend to evaluate the reconfig-uration costs of service chains and propose solutions to minimize them in thefuture extension of this work.


Table 1: Summary of research questions, objectives, and methods.

RQ 1 RQ 2 RQ 3ResearchObjectiv

e

Reduce migration down-time. [Paper I, Paper V]

Resolve sub-optimal rout-ing problem. [Paper V]

Automate the deploymentof EVPN. [Paper II]

Improve the managementof EVPN. [Paper II]

Improve the performanceof EVPN-based intercon-nections. [Paper III]

Enhance the scalability ofEVPN-based interconnec-tions. [Paper IV]

Emulation

DC topology and back-ground traffic in COREnetwork emulator.[Paper I]

DC topology in Mininetnetwork emulator.[Paper V]

Black-box test: Interactionof ODL controller withEVPN-enabled router im-ported into the GNS3 em-ulator. [Paper II]

DC topology andbackground traffic inCORE network emulator.[Paper III]

Emulation of DF swapfunctionality. [Paper IV]

DC topology in Mininetnetwork emulator.[Paper IV]

Real-W

orld

Measurement

—

White-box and Black-boxtest: Add EVPN capabil-ity to Bagpipe router togenerate artificial EVPNworkloads. [Paper II]

—

Implem

entatio

n&

Measurement

Implementation of SDN-based resiliency mecha-nisms. [Paper I]

Extension of ODL mod-ules which are developedin Paper II. [Paper V]

Measurement of down-time and throughput.[ Paper I, Paper V]

Measurement of FCT.[Paper V]

Implementation of severalnew modules inside ODLcontroller. [Paper II]

Measurement of con-troller performance indeploying EVPNs andprocessing EVPN relatedmessages. [Paper II]

Implementation of trafficengineering routing poli-cies. [Paper III]

Measurements of portutilization of DC edgeswitches, FCT, and RTT.[Paper III]

Measurements of packetloss percentage and num-ber of received broadcasttraffic. [Paper IV]

FCT = Flow Completion Time.RTT = Round Trip Time.DF = Designated Forwarder.


References[1] BaGPipe: A lightweight implementation of BGP VPNs. https://

github.com/Orange-OpenSource/bagpipe-bgp/.

[2] Core: Common open research emulator. https://www.nrl.navy.mil/itd/ncs/products/core.

[3] Gns3: Graphical network simulator. https://www.gns3.com/.

[4] Mininet. http://mininet.org/.

[5] Open vSwitch: Multilayer Open Virtual Switch. https://openvswitch.org/.

[6] OpenDaylight: Open Source SDN Platform. https://www.opendaylight.org/.

[7] OpenStack. https://www.openstack.org//.

[8] VPNService. https://github.com/opendaylight/netvirt/tree/master/vpnmanager.

[9] S. Akoush, R. Sohan, A. Rice, A. W. Moore, and A. Hopper. Predict-ing the performance of virtual machine migration. In IEEE Symposiumon Modelling, Analysis & Simulation of Computer and TelecommunicationSystems (MASCOTS), Miami, Fl, USA, Aug. 2010.

[10] Y. Ben-Itzhak, K. Barabash, R. Cohen, A. Levin, and E. Raichstein. En-forSDN: Network policies enforcement with SDN. In IFIP/IEEE Sym-posium on Integrated Network Management (IM), Ottawa, ON, Canada,May 2015.

[11] G. Bianchi, M. Bonola, A. Capone, and C. Cascone. OpenState:Programming platform-independent stateful OpenFlow applications in-side the switch. ACM SIGCOMM Computer Communication Review,44(2):44–51, Aug. 2014.

[12] M. Bjorklund. YANG - a data modeling language for the Network Con-figuration Protocol (NETCONF). RFC 6020, Oct. 2010.

[13] D. L. Black, Z. Wang, M. A. Carlson, W. Weiss, E. B. Davies, and S. L.Blake. An architecture for differentiated services. RFC 2475, Dec. 1998.

[14] B. Boughzala, R. B. Ali, M. Lemay, Y. Lemieux, and O. Cherkaoui.OpenFlow supporting inter-domain virtual machine migration. In IEEEConference on Wireless and Optical Communications Networks (WOCN),Paris, France, May 2011.


[15] R. Bradford, E. Kotsovinos, A. Feldmann, and H. Schiöberg. Live wide-area migration of virtual machines including local persistent state. InACM Conference on Virtual Execution Environments (VEE), San Diego,CA, USA, June 2007.

[16] J. Case, M. Fedor, M. Schoffstall, and J. Davin. Simple Network Man-agement Protocol (SNMP). RFC 1157, May 1990.

[17] K. Chanchio and P. Thaenkaew. Time-bound, thread-based live migra-tion of virtual machines. In IEEE/ACM Symposium on Cluster, Cloudand Grid Computing (CCGrid), Chicago, IL, USA, May 2014.

[18] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt,and A. Warfield. Live migration of virtual machines. In USENIX Sym-posium onNetworked Systems Design & Implementation (NSDI), Berkeley,CA, USA, May 2005.

[19] P. J. Denning. What is experimental computer science? Communicationsof the ACM, 23(10):543–544, Oct. 1980.

[20] J. Drake, W. Henderickx, A. Sajassi, R. Aggarwal, D. N. N. Bitar,A. Isaac, and J. Uttaro. BGP MPLS-Based Ethernet VPN. RFC 7432,Feb. 2015.

[21] X.Du, Z. Lv, J.Wu, C.Wu, and S. Chen. PDSDN:A policy-driven SDNcontroller improving scheme for multi-tenant cloud datacenter environ-ments. In IEEE Conference on Services Computing (SCC), San Francisco,CA, USA, June 2016.

[22] R. Enns, M. Bjorklund, J. Schoenwaelder, and A. Bierman. NetworkConfiguration Protocol (NETCONF). RFC 6241, June 2011.

[23] A. Ford, C. Raiciu, M. J. Handley, and O. Bonaventure. TCP extensionsfor multipath operation with multiple addresses. RFC 6824, Jan. 2013.

[24] F. Hao, T. V. Lakshman, S. Mukherjee, and H. Song. Enhancing dynam-ic cloud-based services using network virtualization. In ACMWorkshopon Virtualized Infrastructure Systems and Architectures (VISA), Barcelona,Spain, Aug. 2009.

[25] W. Hao, L. Yong, and Q. Liang. Handshaking mechanism for DF elec-tion, May.

[26] E. Harney, S. Goasguen, J. Martin, M. Murphy, and M. Westall. Theefficacy of live virtual machine migrations over the internet. In ACMWorkshop on Virtualization Technology in Distributed Computing (VTD-C), Reno, NV, USA, 2007.

[27] M. R. Hines, U. Deshpande, and K. Gopalan. Post-copy live migrationof virtual machines. ACM SIGOPS operating systems review, 43(3):14–26,July 2009.


[28] D. B. Johnson, J. Arkko, and C. E. Perkins. Mobility support in IPv6.RFC 6275, Jul. 2011.

[29] K. Kompella and Y. Rekhter. Virtual Private Lan Service (VPLS) usingBGP for auto-discovery and signaling. RFC 4761, Jan. 2007.

[30] J. Liu, Y. Li, and D. Jin. SDN-based live VM migration across datacen-ters. In ACM SIGCOMM, Chicago, IL, USA, Aug. 2014.

[31] M. Liyanage, M. Ylianttila, and A. Gurtov. Improving the tunnel man-agement performance of secure VPLS architectures with SDN. In IEEEAnnual Consumer Communications Networking Conference (CCNC), LasVegas, NV, USA, Jan. 2016.

[32] G. Lospoto, M. Rimondini, B. G. Vignoli, and G. D. Battista. Rethink-ing virtual private networks in the software-defined era. In IFIP/IEEESymposium on Integrated Network Management (IM), Ottawa, ON, Cana-da, May 2015.

[33] C. C. Machado, L. Z. Granville, A. Schaeffer-Filho, and J. A. Wick-boldt. Towards SLApolicy refinement forQoSmanagement in software-defined networking. In IEEE Conference on Advanced Information Net-working and Applications (AINA), Victoria, BC, Canada, May 2014.

[34] C. C. Machado, J. A.Wickboldt, L. Z. Granville, and A. Schaeffer-Filho.Arkham: an advanced refinement toolkit for handling service level agree-ments in software-defined networking. Journal of Network andComputerApplications, 90:1 – 16, 2017.

[35] M. Mahalingam, D. Dutt, K. Duda, P. Agarwal, L. Kreeger, T. Sridhar,M. Bursell, and C. Wright. Virtual eXtensible Local Area Network(VXLAN): A framework for overlaying virtualized layer 2 networksover layer 3 networks. RFC 7348, Aug. 2014.

[36] V. Mann, A. Vishnoi, K. Kannan, and S. Kalyanaraman. Crossroad-s: Seamless VM mobility across data centers through software definednetworking. In IEEE Network Operations and Management Symposium(NOMS), Maui, HI, USA, Apr. 2012.

[37] A. J. Mashtizadeh, M. Cai, G. Tarasuk-Levin, R. Koller, T. Garfinkel,and S. Setty. XvMotion: Unified virtual machine migration overlong distance. In USENIX Conference on Annual Technical Conference(USENIX ATC), Philadelphia, PA, USA, June 2014.

[38] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson,J. Rexford, S. Shenker, and J. Turner. OpenFlow: Enabling innovation incampus networks. ACM SIGCOMM Computer Communication Review,38(2):69–74, Mar. 2008.

[39] S. Mohanty, K. Patel, A. Sajassi, J. Drake, and T. Przygienda. A newdesignated forwarder election for the EVPN, Apr. 2017.


[40] R. Nasim and A. J. Kassler. Network-centric performance improve-ment for live VM migration. In IEEE Conference on Cloud Computing(CLOUD), New York, NY, USA, June-July 2015.

[41] ONF. Software-DefinedNetworking: The new norm for networks.ON-F White Paper, Apr. 2012.

[42] C. E. Perkins. IP mobility support for IPv4. RFC 3344, Aug. 2002.

[43] P. Raad, S. Secci, D. C. Phung, A. Cianfrani, P. Gallard, and G. Pujolle.Achieving sub-second downtimes in large-scale virtual machine migra-tions with LISP. IEEE Transactions on Network and Service Management,11(2):133–143, June 2014.

[44] J. Rabadan, S. Sathappan, W. Henderickx, A. Sajassi, and J. Drake. Inter-connect solution for EVPN overlay networks. Internet-draft, InternetEngineering Task Force, Mar. 2018.

[45] J. Rabadan, S. Sathappan, T. Przygienda, W. Lin, J. Drake, A. Sajassi,and S. Mohanty. Preference-based EVPN DF election, Dec. 2016.

[46] S. Sahni and V. Varma. A hybrid approach to live migration of virtualmachines. In IEEE Conference on Cloud Computing in Emerging Markets(CCEM), Bangalore, India, Oct. 2012.

[47] A. Sajassi, G. Badoni, D. Rao, P. Brissette, and J. Drake. Fast recoveryfor EVPN DF election, Mar. 2017.

[48] E. Silvera, G. Sharaby, D. Lorenz, and I. Shapira. IP mobility to supportlive migration of virtual machines across subnets. In ACMConference onThe Israeli Experimental Systems, Haifa, Israel, May 2009.

[49] M. Stein, M. Scharf, and V. Hilt. SDN policy-driven service chain place-ment in OpenStack. In IFIP/IEEE Symposium on Integrated Network andService Management (IM), Lisbon, Portugal, May 2017.

[50] K. Suzuki and H. Kaneko. An openflow controller for reducing opera-tional cost of IP-VPNs. Technical report, Apr. 2014.

[51] P. Svärd, B. Hudzia, S. Walsh, J. Tordsson, and E. Elmroth. The nobleart of live vm migration-principles and performance of pre copy, postcopy and hybrid migration of demanding workloads. Technical report,2014.

[52] P. Svärd, J. Tordsson, B. Hudzia, and E. Elmroth. High performancelive migration through dynamic page transfer reordering and compres-sion. In IEEE Conference on Cloud Computing Technology and Science(CLOUDCOM), Athens, Greece, Nov. 2011.

[53] B. J. Van Asten, N. L. Van Adrichem, and F. A. Kuipers. Scalability andresilience of software-defined networking: An overview. arXiv preprintarXiv:1408.6760, 2014.


[54] R. van der Pol, B. Gijsen, P. Zuraniewski, D. F. C. Romão, and M. Kaat.Assessment of SDN technology for an easy-to-use VPN service. FutureGeneration Computer Systems, 56:295–302, 2016.

[55] T. Wood, K. K. Ramakrishnan, P. Shenoy, J. Van Der Merwe, J. Hwang,G. Liu, and L. Chaufournier. Cloudnet: Dynamic pooling of cloud re-sources by live WAN migration of virtual machines. IEEE/ACM Trans-actions on Networking, 23(5):1568–1583, Oct. 2015.

[56] K. Wrona, M. Amanowicz, S. Szwaczyk, and K. Gierłowski. SDNtestbed for validation of cross-layer data-centric security policies. InIEEE Conference on Military Communications and Information Systems(ICMCIS), Oulu, Finland, May 2017.

[57] S. Q. Zhang, P. Yasrebi, A. Tizghadam, H. Bannazadeh, and A. Leon-Garcia. Fast network flow resumption for live virtual machinemigrationon SDN. In IEEEConference onNetwork Protocols (ICNP), San Francisco,CA, USA, Nov. 2015.

[58] W. Zhang, K. T. Lam, and C. L. Wang. Adaptive live vm migration overa wan: Modeling and implementation. In IEEE Conference on CloudComputing (CLOUD), Anchorage, AK, USA, June–July 2014.



Kyoom

ars Alizadeh N

oghani | Towards Seam

less Live Migration in SD

N-B

ased Data C

enters | 2018:55


Live migration of Virtual Machines (VMs) has significantly improved the flexibility of modern Data Centers (DCs). Ideally, live migration ought to be seamless which requires a comprehensive support from the underlying network. However, legacy DC networks fall short to address the challenges of migration due to their inflexible and decentralized characteristics. In contrast, Software Defined Networking (SDN) is a new networking paradigm, which has the potential to improve the live migration thanks to its comprehensive view over the network, flexible structure, and its close integration with DC management infrastructures.

This thesis investigates networking challenges of short and long-haul live VM migration in SDN-based DCs. We propose solutions to make the intra- and inter-DC live migration procedures more seamless. Furthermore, our proposed SDN-based framework for inter-DC migration improves the management, enhances the performance, and increases the scalability of interconnections among DCs.


Faculty of Health, Science and Technology

Computer Science


ISSN 1403-8099

ISBN 978-91-7063-991-3 (pdf)

ISBN 978-91-7063-896-1 (print)

Towards Seamless Live Towards Seamless Live Migration in...

Documents

Transcript of Towards Seamless Live Towards Seamless Live Migration in...