Improvement and Evaluation of an OpenStack Extension to ... · in OpenStack, the prototype...

Improvement and Evaluation of anOpenStack Extension to Enforce

Cloud-wide Multi-resource Fairnessduring VM Runtime

Riccardo PatanèZurich, Switzerland

Student ID: 09-177-270

Supervisor: Patrick Gwydion Poullie, Dr. Thomas BocekDate of Submission: January 7, 2017

University of ZurichDepartment of Informatics (IFI)Binzmühlestrasse 14, CH-8050 Zürich, Switzerland ifi

MA

ST

ER

TH

ES

IS–

Com

mun

icat

ion

Sys

tem

sG

roup

,Pro

f.D

r.B

urkh

ard

Stil

ler

Master ThesisCommunication Systems Group (CSG)Department of Informatics (IFI)University of ZurichBinzmühlestrasse 14, CH-8050 Zürich, SwitzerlandURL: http://www.csg.uzh.ch/

Abstract

In the last years the problems of fairness in Cloud computing have frequently been ex-plored by computer scientists. The usage of job/Virtual Machine (VM) scheduling wasgenerally proposed to enforce fairness. But for long running jobs, scheduling is insufficientand other approaches had to be found to enforce fairness during runtime as well. ThisMaster Thesis contributes to the less researched area of multi-resource fairness during VMruntime.Previously to this Master Thesis, the Nova Fairness Service –a prototype embeddedin OpenStack– was developed at the University of Zurich to enforce cloud-wide multi-resource fairness during VM runtime. The subsequent experiments yielded great results.Since OpenStack has short release cycles, the prototype was already outdated when fur-ther experiments needed to be performed. Due to the constant and intensive code changesin OpenStack, the prototype architecture and the way it was implemented, the prototypewas not compatible anymore with the current OpenStack release.Besides an analysis of the architecture and the program sequence of the prototype, andthe description of the problems for porting the prototype to the current OpenStack re-lease, this Master Thesis proposes a new software architecture and describes its partialimplementation, resulting in the new Fairness Service 2.0. Unlike the embedded proto-type, the new software is independent and has therefore the advantages that it is immuneto changes in the source code of OpenStack, the code is better understandable and theprocesses are better comprehensible as well.To our best knowledge, in the OpenStack ecosystem, there is no comparable project thataddresses multi-resource fairness during runtime.

i

Zusammenfassung

In den letzten Jahren wurde oft das Problem von Fairness im Cloud Computing von Com-puterwissenschaftler erforscht. Um Fairness durchzusetzen wurde jeweils in verschiedenenVarianten die Verwendung von Job/Virtual Machine (VM) Scheduling vorgeschlagen. Furlangfristige Jobs jedoch, ist Scheduling nur unzureichend geeignet und andere Ansatzemussten gefunden werden, um Fairness auch wahrend der Laufzeit durchzusetzen. DieseMaster-Thesis tragt zu einem weniger erforschten Bereich bei: der Multi-Ressourcen-Fairness wahrend der VM-Laufzeit.Vor dieser Arbeit wurde an der Universitat Zurich der Nova Fairness Service entwick-elt. Ein in OpenStack eingebetteter Prototyp, welcher Multi-Resource-Fairness wahrendder VM-Laufzeit Cloud-weit durchgesetzt. Die anschliessenden Versuche lieferten sehrgute Ergebnisse. Als erneut Experimente durchgefuhrt werden sollten, war der Prototypbereits veraltet, weil OpenStack kurze Release-Zyklen hat. Aufgrund der konstanten undintensiven Code-Anderungen in OpenStack, war der Prototyp wegen seiner Architekturund die Weise der Implementierung jedoch nicht mehr mit der aktuellen OpenStack Ver-sion kompatibel.Neben einer Analyse der Architektur und des Programmablaufs des Prototyps sowieeiner Beschreibung der Probleme die wahrend der Portierung des Prototyps auf die ak-tuelle OpenStack-Version aufgetreten sind, schlagt diese Master-Tesis eine neue Soft-warearchitektur vor und beschreibt deren Teilimplementierung, die zum neuen FairnessService 2.0 gefuhrt hat. Im Gegensatz zum eingebetteten Prototypen ist die neue Soft-ware unabhangig und hat daher die Vorteile, dass sie gegen anderungen im Quellcodevon OpenStack bestandig ist, der Code besser verstandlich ist und die Prozesse bessernachvollziehbar sind.Gemass heutigem Wissensstand, gibt es kein vergleichbares Projekt im OpenStack Okosys-tem, bei dem es um multi-resource Fairness zur Laufzeit geht.

iii

Acknowledgments

First of all, I would like to thank Prof. Burkhard Stiller for giving me the opportunity towrite this thesis. Furthermore, I thank my supervisors Dr. Thomas Bocek for his supportwith the testbed and especially Patrick Gwydion Poullie for his advice, support and theinteresting discussions on the subject of multi-resource fairness in cloud computing.I also thank Stephan Mannhart for providing me with valuable information about theFairness Service prototype he made.Thank you goes also to Gaby Miranda for proofreading this document several times.Last but not least, a big thank you to my life partner Nalleli Garcıa Aguero for her tirelesspatience and her moral support over the last six months. To her I dedicate this thesis.

v

Contents

Abstract i

Zusammenfassung iii

Acknowledgments v

1 Introduction 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Description of Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Related Work 5

3 Fairness Service for OpenStack 7

3.1 Heaviness and Greediness . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.2 Quotas, VRs, overcommitment ratios and scales . . . . . . . . . . . . . . . 8

3.3 Main tasks of the Fairness Service . . . . . . . . . . . . . . . . . . . . . . . 8

3.3.1 Collect physical resource data . . . . . . . . . . . . . . . . . . . . . 9

3.3.2 Collect resource usage statistics . . . . . . . . . . . . . . . . . . . . 9

3.3.3 Calculate heavinesses . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.3.4 Mapping user heavinesses to VM priorities . . . . . . . . . . . . . . 9

3.3.5 Resource reallocation . . . . . . . . . . . . . . . . . . . . . . . . . . 10

vii

viii CONTENTS

4 Setting up OpenStack 11

4.1 The testbed at University of Zurich . . . . . . . . . . . . . . . . . . . . . . 12

4.2 Project, Domains, Roles, Users and Quotas . . . . . . . . . . . . . . . . . . 12

5 The Nova Fairness Service 15

5.1 Architecture and installation . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5.2 Program sequence and execution . . . . . . . . . . . . . . . . . . . . . . . 16

5.3 Strong integration into the Nova project . . . . . . . . . . . . . . . . . . . 17

5.3.1 The Nova Service Framework . . . . . . . . . . . . . . . . . . . . . 17

5.3.2 Extending the Nova Compute API . . . . . . . . . . . . . . . . . . 18

5.4 Decentralized messaging system . . . . . . . . . . . . . . . . . . . . . . . . 19

5.5 Porting to the current OpenStack version . . . . . . . . . . . . . . . . . . . 19

5.6 Solution Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

6 The Fairness Service 2.0 23

6.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

6.2 Program sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

6.2.1 Fairness Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

6.2.2 Fairness Node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

6.3 Information to be collected . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

6.4 Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

6.5 Drivers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

6.5.1 Libvirt driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

6.5.2 OpenStack driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

6.5.3 MySQL driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

6.6 ZeroMQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

6.7 State of the implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 34

6.8 Integration into Community . . . . . . . . . . . . . . . . . . . . . . . . . . 35

CONTENTS ix

7 Conclusions and future work 37

7.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Bibliography 39

Abbreviations 43

Glossary 45

List of Figures 45

List of Listings 47

A Contents of the CD 51

x CONTENTS

Chapter 1

Introduction

Cloud Computing is a paradigm with increasing popularity in recent years. It is the ab-straction of more concrete layers like IaaS, PaaS and SaaS. These Cloud-based servicesthat appear like seamless platforms, in reality, unify globally distributed resources [35].Dependent on what layer is looked at, we can find different user types: SaaS is used bya lot of people that use the internet. Examples are paid services by Google, Dropbox,LinkedIn, etc. But PaaS and IaaS are used by software developers, since the main pur-pose is to provide infrastructure and tools to develop and deploy software applications.The foundation of these mentioned XaaS models is an infrastructure consisting of baremetal computing resources that host virtual machines (VM). On only one computer, themanagement of these VMs can be simple. But since Cloud Computing paradigm entailsalso the idea of unlimited resources from the point of view of the cloud user it should beobvious that such an infrastructure can be very extensive, complex, heterogeneous andeven distributed over several continents. Therefore with the rising of Cloud Computingalso software to operate the cloud infrastructure appeared. Not surprisingly, beside com-mercial software and private solutions, there is also open source software. One of thoseopen source Cloud Computing Operating Systems is called OpenStack, and the mainpurpose is to manage the entire lifecycle of virtual machines, cloud users and other cloudelements. In a cloud it is possible that a lot of users share the same resources such as theCentral Processing Unit (CPU). Thus, such resources are subject to conflicting interests.Then when it comes to scarcity, the question is: Who gets the resource? what is a fairresource allocation?Together with the rise of Cloud Computing also academic research made lots of effortsin exploring a variety of different aspects. But especially the fairness aspect that is inour interest was not explored as deep as other aspects. The fairness problem is oftenaddressed by VM scheduling [2], [5]. But, since VM scheduling is done at VM creationor VM migration time we have to be aware that this is not the same thing that we arelooking at. We are interested in fairness during runtime of the VMs and in techniques toenforce fairness.

1

2 CHAPTER 1. INTRODUCTION

1.1 Motivation

But why is fairness so important to us and why should we bother to enforce fairnessduring runtime? Let us take again the example of the CPU: a CPU is a time sharedresource and can be allocated to only one process at a time. As long as all VMs on thehost belong to the same user, no fairness questions among users exists. But as soon asmore than one user needs to access the CPU at the same time, the question is: Whogets how much CPU time? When creating a VM we use flavors in which the numberof VCPU is defined but not what percentage of the time-multiplexed resource the VMgets during runtime. Thus, at VM creation time we do not have a mechanism to enforcefairness during runtime. So, a new mechanism is required here to close this gap. Closingthe gap can reduce the necessity to re-schedule and migrate VMs around the entire cloud.Poullie’s research in multi-resource fairness resulted in the design of a Fairness Servicefor OpenStack that aims to close the gap. It is not a service that replaces any existingfunction, but it is a mechanism that complements existing cloud infrastructures extendingits functionality with fairness during runtime. This is especially interesting for two usecases: (i) in Private Clouds that run without Service Level Agreements (SLA) and (ii) asan enabler for Cloud Computing Flat Rates in Public Clouds. In contrast to Public Cloudswhere accounting and terms of use are based on SLAs in Private Clouds we normally donot have SLAs because they are often commodities [30]. Thus, no rules on how to allocatescarce resources are in place. Concerning the second use case, from the communicationmarket we know that users prefer to pay a flat rate even if the amount to pay is higherthan a billing per actual usage.

1.2 Description of Work

The initial goals for this Master Thesis were others than those actually achieved. Basedon a software prototype implemented by Mannhart [9], the idea was to further developthe prototype and do further extensive evaluations on a real OpenStack infrastructure.To achieve this, two warm up tasks and three main tasks were defined at the beginningof this Master Thesis:

� W1 Setting up a current OpenStack environment in the testbed of the Communi-cation Systems Group of the University of Zurich.

� W2 Porting the Nova Fairness Service prototype to OpenStack Mitaka.

� M1 Finding a suited priority function1.

� M2 Finding and integrating a better metric to compare CPU time.

� M3 Evaluate the Fairness Service exhaustively.

1The priority function is used to calculate the priorities of the VMs on a host for resource allocation.

1.3. THESIS OUTLINE 3

But, as it will be described in Chapter 5, during processing the second task, unexpectedcomplications appeared and we realized that it did not make sense to port the prototypeto the newer OpenStack version and do further development on it. Instead it was de-cided to change W2 to design and develop a new software for the a Fairness Service forOpenStack. This resulted in an creation and description of a new architecture based ongained OpenStack knowledge and a partial implementation of the new Fairness Service2.0 according to this new architecture. The software implementation progress is estimatedat roughly 75%. Nonetheless the results of the present work represent a good foundationfor future work.

1.3 Thesis Outline

In the next chapter the reader will find an overview of previous work related to thisproject. Then, Chapter 3 will explain what the Fairness Service is and how it works.Chapter 4 will talk about OpenStack and how it was set up in the CSG testbed at theUniversity of Zurich. It is the necessary environment to run the Fairness Service. Chapter5 describes the first OpenStack Fairness Service prototype and its pro and cons and whywe decided to design the new Fairness Service 2.0 that is extensively outlined in Chapter6. This Master Thesis will end with the conclusions and future work in Chapter 7.

4 CHAPTER 1. INTRODUCTION

Chapter 2

Related Work

Multi-resource fairness has been approached by several researchers. In [4] for example, theauthors suggest determining bottlenecks as decision basis for fair allocation and definingBottleneck-based Fairness (BBF). Computer scientists at Berkeley University introducedthe Dominant Resource Fairness (DRF) [5] where they basically suggest allocating ac-cording to the resource that is more dominant than any other. But, the problem is thatthese approaches for multi-resource fairness apply to VM scheduling time and not to VMruntime. But since we are interested in fairness during runtime, multi-resource fairness forclouds was not defined in a satisfactory manner. This gap was closed by Poullie with animportant work that explores multi-resource fairness in Cloud Computing during runtimeand that is highly relevant for this work [27]. In Cloud Computing the physical resourcesthat have been utilized so far is the only information that is available concerning the user’sdemand [28]. This information is used to calculate the heaviness of a user. The heavinessis the amount of stress a single user imposes to the entire Cloud. Using user’s heavinessto allow multi-resource fairness in Cloud Computing is one of the main contribution ofPoullie [29]. With this information at hand, fairness in the Cloud is defined as prioritizingcloud users inversely to their heaviness. Heaviness can be defined in different ways. Theeasiest way is to just sum up all the resource utilization statistics. But, to have a moreaccurate definition for the the heaviness metric, Poullie conducted a survey about Fairnessand Heaviness with more than 600 individuals and developed the Greediness Metric (GM)based on this study [26].This Master Thesis builds on the bachelor thesis done by Mannhart [9] and supervisedby Poullie where he implemented the first prototype for the OpenStack Fairness Service.The goal was to be able to do first experiments on a real OpenStack infrastructure for theresearch work in the field of multi-resource fairness in Cloud Computing done by Poullie.Out of this bachelor work Poullie published [28] where the experiments on a real Open-Stack infrastructure are presented.Practical considerations for the resource reallocation in OpenStack can be found in Gruh-ler [6].

5

6 CHAPTER 2. RELATED WORK

Chapter 3

Fairness Service for OpenStack

The Fairness Service for OpenStack, or just Fairness Service, was introduced by PatrickGwydion Poullie at the Department of Informatics of the University of Zurich. It waspart of his dissertation to explore fairness in Cloud Computing. In the context of severalsmaller and bigger student projects, supervised by Poullie, a Fairness Service prototypefor OpenStack was developed in early 2016 that allowed to carry out first experiments ina real OpenStack environment [6], [9].The Fairness Service allows to enforce fairness among Cloud users in an OpenStack infras-tructure. This is done with the periodical reallocation of resources of all VMs in the Cloudduring runtime. In this work with resources, it is meant, exclusively, the four resourcesCentral Processing Unit (CPU), Random Access Memory (RAM), Disk I/O and NetworkI/O. The last two are splitted up in disk read, disk written and network received, networktransmitted for practical reasons. Thus, in total there are 6 individual resource values.The term user refers to an OpenStack Cloud user that has one or more Virtual Machines(VMs) assigned Cloud-wide and therefore utilizes resources on one or more hosts.The definition of Fairness is not trivial. The economic approach is to use consumer’s util-ity functions to define fairness. In other words, what is fair and what is not fair, is highlysubjective and it depends on the consumer’s demand. The VMs demand for resources isvery dynamic. It can change from one instant to another and depending on the task to beperformed the resource mix that is actually demanded can be very different. A user mayperform a calculation intensive task that requires more CPU and RAM, while anotheruser may use the Cloud mainly for data storage that requires more Disk I/O and NetworkI/O. Because of these dynamics in demands and in utility functions for Cloud resourcesthe economic approach is not a good fit.

3.1 Heaviness and Greediness

The heaviness notion is the heart of the Fairness Service. As already mentioned, it ex-presses how much stress a user or a VM imposes to the cloud and it is calculated basedon the resources that have been utilized. There are different ways how heaviness can becalculated. The most simple approach is to just sum all values. Another approach is the

7

8 CHAPTER 3. FAIRNESS SERVICE FOR OPENSTACK

Greediness Metric mentioned in Chapter 2. For the Greediness Metric first, we have tocalculate the equal share value for all resources among all VMs on the host. Then, fromthe utilization statistics, all values that exceed the equal share are summed up and allvalues that were not consumed up to the equal share are subtracted. If a VM’s resourceutilization statistic corresponds exactly to the equal shares, then the Greediness is zero. Ifit utilized more resources than the equal shares then the Greediness is positive, otherwisethe Greediness is negative.From here on, in this Master Thesis, the expressions heaviness and greediness can be usedinterchangeable.

3.2 Quotas, VRs, overcommitment ratios and scales

In most Clouds like for example OpenStack, the resources on a host are managed byquotas. Quotas are limits that describe the maximum amount of resources available fora specific user. A realistic quota could be to have at most 20 VCPUs assigned to useru1. When creating VMs the sum of all VCPU in all VMs of u1 cannot exceed 20 VCPUs.The VRs are the amount of resources actually assigned when creating VMs.Normally in Clouds, underutilization is mitigated with the concept of overcommitment.The overcommitment allows to have more VRs assigned to VMs on one host than theavailable PRs. If, for instance, the overcommitment ratio for CPU is 10 and the physicallyavailable CPUs on a host are 4, then the Cloud allows to spawn VMs with in total 40VCPUs on that specific host. This introduces the practical scale and the theoretical scale,where VRs and quotas (eg. VCPU) are on the theoretical scale and PRs (e.g. CPU) areon the practical scale.

3.3 Main tasks of the Fairness Service

The Fairness Service decides what is fair and how to reallocate the resources, based onwhat resources were used so far (demand) and how big the Cloud (supply) is. We cannow look at the main tasks of the Fairness Service that are:

1. Collect physical resource data of the nodes

2. Collect resource usage statistics of the VMs

3. Calculate heavinesses of VMs and users

4. Mapping user heavinesses to VM priorities

5. Reallocate resources based on the VM priorities

3.3. MAIN TASKS OF THE FAIRNESS SERVICE 9

3.3.1 Collect physical resource data

Since the Fairness Service takes into consideration the six resources values available forthe entire cloud, all nodes need to know the sum of them. In order to get this sum,when the Fairness Service starts its operations it retrieves the Node Resource Information(NRI) on every host. The NRI is a vector with the six values of the physical resourcesof a specific node. As shown in Equation 3.1 the NRI is then summed up to the CloudResource Supply (CRS) that is used for normalization purposes.

CRS =n∑

i=1

NRIn (3.1)

3.3.2 Collect resource usage statistics

The Fairness Service collects periodically the resource utilization statistics, also knownas Resources Utilization Information (RUI), of all VMs in all nodes. It is the amount ofresource utilized in the last period. For instance, for CPU it is measured in seconds andfor disk read it is measured in bytes per seconds. RUI is the most important informationthat is gathered and it is used as input for the calculations of the new priorities. EachVM has its own RUI since each VM has different utilization statistics.

3.3.3 Calculate heavinesses

This is the first part of the core algorithm of the Fairness Service. The inputs to calculatethe heaviness are the RUI, Endowments, and a normalization vector. The output is theheavinesses of the VMs h(vi) and the users h(uj). The user heaviness h(uj) is the sumof the VM heaviness belonging to that user. The Endowment of a VM e(vi) as shown inEquation 3.2 is the VR of vi multiplied by the ratio of the PR to the sum of all VRs onthat host. Equation 3.3 shows the example for a host with 5 CPUs and the sum of allVMs VCPUs is 15. Then e(vi) for a VM with 3 VCPUs on that host is 1.

e(vi) =V Rvi × PR∑

V R(3.2)

e(vi)cpu =3× 5

15= 1 (3.3)

3.3.4 Mapping user heavinesses to VM priorities

This is the second part of the core algorithm of the Fairness Service. The VM prioritiesare inversely proportional to the user heavinesses. The proportional priorities is a conceptthat is also known as weighted Max-min fairness [33] and Operating Systems allow toallocate most time-shared resources according to those proportional priorities [28].

10 CHAPTER 3. FAIRNESS SERVICE FOR OPENSTACK

3.3.5 Resource reallocation

In this step the resources CPU, RAM, Disk I/O, and Network I/O are going to be re-allocated according to the proportional priorities calculated in the previous step. It isimportant to note that the VRs of the VMs are not changed. Every resource has its ownterms for those proportional priorities: shares for CPU, soft-limits for RAM, weights forDisk I/O, and traffic classes for Network I/O.

In this chapter we had an overview of relevant Fairness Service concepts developed byPoullie and looked at some specific aspects to have a properly understanding needed toread the next chapters. In the next chapter, the OpenStack infrastructure and how tosetup OpenStack in the University’s testbed are described.

Chapter 4

Setting up OpenStack

For this Master Thesis a running OpenStack infrastructure was needed in order to be ableto analyse the Nova Fairness Service and to perform further developments. To this endthe instruction of the official OpenStack Website [16] to install OpenStack Newton onUbuntu 16.04 was used. Currently, OpenStack is only available for Linux and installationtutorials are available for Debian, openSUSE, Red Hat and Ubuntu operating systems.The main OpenStack components (also called services or projects) used in this MasterThesis are Nova, Keystone, Neutron and Glance. Setting up OpenStack requires a fairamount of know-how on Linux systems and Networking.Nova is the OpenStack Compute project and it is the core service in OpenStack. TheCompute project is at the heart of the whole OpenStack project and it is one of the twoinitial components when OpenStack was released for the first time in October 2010. It isthe service that manages the entire lifecycle of the virtual machines. Common tasks arefor example VM scheduling, VM creation, VM deletion, etc.Keystone is the identity service and has two responsibilities. On one hand it is respon-sible for management of users, roles, projects, and domains. On the other hand it isresponsible to authenticate API clients: before being able to access to any other Open-Stack service, a client must authenticate with the Keystone identity service.Neutron is the network service since OpenStack Mitaka and it is responsible for managingall networking aspects in the OpenStack infrastructure like for example virtual networks,subnets, DHCP, Firewall, IPv6, etc. Previous to Mitaka the nova-networking, a featureof the Nova Compute service, was used for network managing.Glance is the image storage center. It stores and retrieves virtual machine disk imagesthat are used during instance provision.Beside the OpenStack services, there are several third party services that are needed bythe OpenStack services like for example MariaDB, MongoDB, Memcached or RabbitMQ.The correct installation and configuration of those third party services is integral partof the tutorials. An example is MariaDB [11]. Among theses third party services, therelational database has an important role for the Fairness Service described in Chapter 6.A typical OpenStack infrastructure is composed by a Controller node and multiple Com-pute nodes. For high availability reasons it can also have two Controllers. But, we arenot considering high availability in this project, thus it could be a topic for future work.Almost all services are located only on the controller node. The only service that is on

11

12 CHAPTER 4. SETTING UP OPENSTACK

the Compute nodes is the Nova Compute service and has its counterpart, the ComputeService server, on the Controller node. The VMs are running exclusively on the the Com-pute nodes. Figure 4.1 shows this architecture installed in the testbed of the Departmentof Informatics at the University of Zurich. The Compute service on the compute node isresponsible for the communication to the hypervisor, while the Compute service on theController node is responsible for scheduling and for providing a REST API to any clientwith access to its network.

Figure 4.1: OpenStack infrastructure installed in the testbed at the University of Zurich.

4.1 The testbed at University of Zurich

The OpenStack infrastructure was set up in the testbed of the Communication SystemsGroup (CSG) at the Department of Informatics (IFI) at the University of Zurich (UZH).It consists of five bare metal rack servers, each with two Intel(R) Xeon(R) CPU E3113@ 3.00GHz, 4.0 GB RAM and a 64-bit Ubuntu 16.04 as host OS for the Compute nodesand one VM with 8 Intel Xeon E312xx 2.4GHz VCPUs and 10.0 GB VRAM with thesame Ubuntu OS for the OpenStack Controller. Previous to the virtual machine for theController, there was a dedicated server with 2 CPUs and 3.0 GB RAM in place. Butthis turned out to be too weak for a Controller.

4.2 Project, Domains, Roles, Users and Quotas

In OpenStack the Cloud is managed by projects, domains, users, roles and quotas tomention the most important entities for the Fairness Service. In previous OpenStackversion the term tenant is used and was replaced by the term project in more recentversions. In some documentation tenant is still present. Please refer to the Glossary atthe end for explanation of the individual terms.

This chapter described the most important aspects of the OpenStack installation in thetestbed. The next chapter is devoted to the Nova Fairness Service prototype done by a

4.2. PROJECT, DOMAINS, ROLES, USERS AND QUOTAS 13

bachelor student and points out the problems encountered and the rationales behind thedecision to create a new software.

14 CHAPTER 4. SETTING UP OPENSTACK

Chapter 5

The Nova Fairness Service

The Nova Fairness Service (NFS) is the actual prototype for the Fairness Service that isbased on the research done by Poullie at the University of Zurich and it was implementedas an extension of OpenStack Nova by Mannhart, supervised by Poullie. This was donefor the OpenStack Juno version in early 2016. The purpose of this prototype was tohave a proof-of-concept for the Fairness Service and to perform first experiments on a realOpenStack environment. In fact those experiments have been carried out very successfully.It showed that fairness can successfully be enforced in the Cloud with the Fairness Service.The Nova Fairness Service and the experiments done are described in detail in [9].For the purpose of this Master Thesis some aspects of the prototype in the followingsubsections will be highlighted. In particular those aspects that convinced us to moveaway from the original Master Thesis goals described in the introduction to the redesignand a partial implementation of a new Fairness Service software.

5.1 Architecture and installation

For the development of the proof-of-concept of the Fairness Service it was chosen to notcreate an independent software, but rather to use the existing open source Nova code andto extend it with custom code to the Nova Fairness Service. This extension was done withthe help of two mechanisms: (i) using the Nova Fairness Framework to create a customNova service and (ii) extending the Nova Compute API. Beside these two mechanismsfor extensions, a third major change was made, but on the internal messaging system:the default OpenStack centralized messaging system RabbitMQ was replaced with thedecentralized messaging system ZeroMQ. Later in the chapter, these three points will begone through.According to the installation instruction in [9] the Nova Fairness Service needs to beinstalled on each OpenStack compute node and on the controller node as well. But itneeds to run only on the compute nodes because it is a decentralized architecture. Thecontroller node only needs the custom Nova Fairness installation because of the NovaFairness API bindings. To install the Nova Fairness Service it is first necessary to havea minimal OpenStack installation in place (with Nova, Keystone, Glance, Neutron) and

15

16 CHAPTER 5. THE NOVA FAIRNESS SERVICE

1 while NRI of some node in Nf is missing:2 send own NRI to nodes of which NRI is missing3 use NRIs to calculate CRS and normalization vector4 every µ seconds:5 collect RUI of all VMs hosted by ni6 apply hv to collected RUI in order to calculate7 heaviness of all VMs hosted by ni8 send this heaviness set to all n ∈ Nf − {ni}9 wait to receive heaviness set from all n ∈ Nf − {ni}

10 apply hu to calculate the heaviness of all u ∈ U11 for every VM v hosted by ni :12 set priorities of v on according to hv(v) and hu(o(v))

Listing 5.1: The steps conducted by the fairness service running on node ni.

then to replace the official Nova code on the Compute nodes1 with the Nova Fairnesscode [10]. To have the inter-node communication decentralized as well, the RabbitMQmessaging service has to be replaced by a ZeroMQ messaging service. With this changesin place the Fairness Service works on the OpenStack Juno version.

5.2 Program sequence and execution

Listing 5.1 gives a high-level overview of the Fairness Service algorithm running on everynode. Here it is not necessary to go in more detail because it would go beyond the scopeof this Master Thesis and because more detailed information of this algorithm can befound in [28].To start the nova-fairness service after the installation, the user types into the bash shellthe command $ sudo service nova-fairness start This will look into /etc/init/nova-fairness.conf and will start the nova-fairness daemon similar to the command for manuallystart: $ sudo /usr/bin/nova-fairness -config-file=/etc/nova/nova.conf Then thenova.cmd.fairness starter script will be called. This is the entry point for the Nova Fair-ness Service. Until now, the files and code mentioned are part of the custom Fairnesscode. But now, the fairness starter script calls the nova.service.Service class that belongsto the official Nova code. From the description of the nova.service.py file:

”A service takes a manager and enables rpc by listening to queues based ontopic. It also periodically runs tasks on the manager and reports its state tothe database services table.”

The Service class calls at some point the fairness manager that belongs to the customFairness code and incorporates the main logic of the Nova Fairness Service. The Ser-vice class schedules periodically the collection of the Node Resource Information (NRI)and the collection of the Resource Utilization Information (RUI). Beside the just de-scribed behaviour, the Service class does a lot of other things which here are not explored

1on Ubuntu at /usr/lib/python2.7/dist-packages available

5.3. STRONG INTEGRATION INTO THE NOVA PROJECT 17

thoroughly. The nova.service.py module has around 440 lines of code and calls severalother Nova modules, like for instance conductor.py, context.py, rpc.py, servicegroup.py,etc. Those are all functionalities that are needed and customized for OpenStack NovaCompute. Thus, only a small fraction of those functionalities can be used for the FairnessService if any.

5.3 Strong integration into the Nova project

As already mentioned, the Nova Fairness code is embedded into the OpenStack Novaproject, that is the compute service of OpenStack that manages the VMs. With embeddedis meant that it is integrated into the code of the official Nova project. Basically, thestudent who did the prototype forked the free available OpenStack Nova source code fromGithub.com [17] and started to add his own code and made usage of lots of Nova modulesand its classes, methods and functions.OpenStack has a new release every 6 months and the changes from one release to thenext are considerable. The official OpenStack code [17] evolves on a daily basis, while theFairness code [10] has not been further developed since the bachelor thesis was submittedin March 2016. This means that at least every 6 months the Nova Fairness Service isbroken because no maintenance is given. The Nova Fairness Service today does not workwith the current stable OpenStack version. Since OpenStack is an open source software,there are obviously ways to integrate own developed patches or new features into theofficial code. Especially for the integration of new features like the Fairness Service is,OpenStack has a well defined process to integrate the features through Blueprints [15].But since this integration process was never done, the Nova Fairness Service is also notmaintained by the OpenStack Community. In other words, no maintenance is givenoutside from the Communication Systems Group at Departement of Informatics of theUniversity of Zurich. The integration to the OpenStack Community was not consideredfrom the beginning because the Nova Fairness Service represented a very first proof-of-concept project in an OpenStack environment for fair resource allocation during runtime.For the time being no other OpenStack project in this direction is known.

5.3.1 The Nova Service Framework

This section describes the first extension mechanism used for the prototype.The nova.service.Service class that was already mentioned in Section 5.2, is used to createa new Nova service: the nova-fairness service. Among other things, the Service class is re-sponsible for remote procedure calls (RPCs) and messaging between nodes running Nova.The RPCs and the messaging service are close related because RPCs use the messagingservice to call objects and methods on other nodes. A further example is that the NovaFairness Service makes usage of methods available in the Libvirt drivers of the Nova codelike, for instance, the _lookup_by_name() method from the nova.virt.driver.py module.Thus, using the Nova Service framework clearly has the advantage of avoiding implement-ing all those functionalities separately. Listing 5.2 illustrates the strong incorporation ofthe Fairness code into the Nova project with an incomplete list of Nova modules and


classes that are used by the main module manager.py of the Nova Fairness Service. Theother modules show a similar behaviour.

1 from nova import context2 from nova import exception3 from nova import manager4 from nova import rpc5 from nova import servicegroup6 from nova.compute import api as compute api7 from nova.compute import rpcapi as compute rpcapi8 from nova.fairness import api as fairness api9 from nova.fairness import cloud supply

10 from nova.fairness import metrics11 from nova.fairness import resource allocation12 from nova.fairness import rui stats13 from nova.fairness import timing stats14 from nova.objects import instance as instance objects15 from nova.openstack.common import importutils16 from nova.openstack.common import log as logging17 from nova.openstack.common import periodic task18 from nova.openstack.common import timeutils19 from nova.virt import driver20 from nova.virt import virtapi

Listing 5.2: An excerpt of Nova modules and classes that are used by the Nova Fairnesscode.

Then, in this way, he did not have to take care of the configuration file, the servicedaemonization and the communication between nodes. A further advantage of using theNova Service Framework, is that Nova adds an entry to the Nova service list. This listcan be called with the Nova command line client: $ nova service-list. Together withthe nova-compute services and other nova services it also lists the nova-fairness servicesof all nodes. The Service class has further internal calls to other Nova classes. Therefore,integrating this Service class implies the integration of an entire network of Nova modules.Because of this tightly integration, even small changes in the Service class code or thecode in other classes can easily break the Fairness Service. According to the author of theNova Fairness Service the main reason behind using the Nova Service class was to reducethe amount of code to be written for the Nova Fairness Service.

5.3.2 Extending the Nova Compute API

The Nova Fairness Service uses also the API Extension mechanism to achieve its goals.Basically, this is extending the official Compute API with new endpoints to access to newfunctionalities. This was supported until API version 2.0. But nowadays according to theOpenStack Foundation [14] API Extensions are a deprecated concept and consequentlyshould not be used anymore. The current API 2.1 does not support API Extensionsanymore.

5.4. DECENTRALIZED MESSAGING SYSTEM 19

5.4 Decentralized messaging system

The Nova Fairness Service needs to exchange messages between nodes. Since the NovaFairness Service is embedded into the Nova code it uses the Nova internal messagingsystem. A typical OpenStack infrastructure uses the RabbitMQ messaging system. Ifrequired, the Cloud Engineer can use other messaging systems, like for example ZeroMQ.Since RabbitMQ has a central message broker that resides on the OpenStack controller,for the prototype the developer decided to switch to ZeroMQ in order to avoid that everymessage had to traverse the controller node. In this way the system should become morescalable and avoids a single point of failure for inter-node messaging. Using ZeroMQ themessage volume was of O(|N | × |V |) sent by |N | × (|N | − 1) messages every µ seconds.

Figure 5.1: Central message broker (RabbitMQ) vs. decentralized brokerless (ZeroMQ)communication.

5.5 Porting to the current OpenStack version

When starting this Master Thesis the current stable OpenStack version was Mitaka. Forbeing able to run the Fairness Service and do further Fairness experiments, it is essentialto have a stable and running OpenStack infrastructure. Since no infrastructure was ready,the current Mitaka version was set up, as described in Chapter 4. But to improve theNova Fairness Service and to run further experiments on current infrastructure, it wasalso necessary to port the Nova Fairness Service from Juno to the current OpenStackversion. To do that on one hand, there was the custom Fairness code, that was writtenby a bachelor student and no test cases exist. And on the other hand, there was theofficial Nova code, that is a very mature codebase, written by OpenStack engineers withmany years of Python coding experience and it was assumed that many advanced codingpractices and design patterns are used. Also a sound code review process is in place andto provide test cases is a common practise.With this preconditions, it was started the porting/debugging task mentioned in Section1.2. Obviously when fixing a bug, the bugfix has to be tested. But since no test caseswere available, every bigger and smaller fix in the running University testbed had to betested. This resulted in the following workflow: First, it was changed the code locallyon my laptop. Then, it was made a git commit and finally it was pulled the code onthe corresponding testbed server. After the git pull it was possible to manually restart


the nova-fairness daemon and check for errors. To have debug information, the PyCharmremote debugger was used that allowed to have an overview of objects and variable val-ues, to use variable watchers, and to evaluate expressions at program break points. Itcannot be firmly stated if the problems and bugs encountered were mostly because ofthe evolution from release to release of the Nova project or because of the Nova Fairnesscode quality. But since the Nova Fairness worked for a previous OpenStack version it wasassumed that the main cause are the code changes from release to release.As an example for the changes in Nova it can be mentioned the get_all(self, group_id)

method from the servicegroup.API() class. This is used to get a list of all members ofthe given group. For example the nova-fairness group. But this method was simply notavailable anymore. The OpenStack Engineers just deleted this method because it was notused by any other client. In this case the problem was fixed taking the code from theold OpenStack Juno verion, adapting it, and appending the missing code to the callingcloud supply.py file. Or another similar case was that the lookupByName(instance_name)method int the nova/virt/libvirt/driver.py module to get instance information from Lib-virt was also not available anymore. The solution was similar to the former one: It wascopied back from the OpenStack Juno version. Another example is the configuration op-tions to import in the nova/fairness/cloud supply.py module: the cpu_allocation_ratiothat defines the default CPU overcommitment ratio was not anymore available in thenova.scheduler.filters.core_filter file. It was found that it was now available inthe nova.compute.resource_tracker file. But then it seemed that the import from thislast mentioned file did not work properly. Therefore, even though not desirable at all, theresult was just hardcode the overcommitment ratios for CPU, RAM and Disk.Conversations in the official OpenStack IRC channels with individuals of the OpenStackcommunity also confirmed that among other codes, especially the resource allocation part,which was not even started to fix by that time, it had changed a lot between Juno andMitaka, and will change again for Newton/Octava.

5.6 Solution Approach

All mentioned facts and problems were the reason to consider alternative approaches toget a running Fairness Service that allows to perform the tasks for this Master Thesis. Inthe first few weeks the debugging process just continued. But then doubts raised if thisreally would be the right way. Since the original Nova Fairness code was not written bythe author of this thesis, he had not a clear idea on how long this task would take. Atthis point, it should be remembered that this task was meant as a short warm up andpreparation step to start working on the main tasks of this Master Thesis mentioned inSection 1.2.After dealing for some time with OpenStack it could be conceived how to design a FairnessService and a talk was carried out with the supervisor of this Thesis about it. Then, itwas created a new Fairness Service architecture that was proposed to the supervisor.The new Fairness Service should have a modular architecture, decoupled from all otherOpenStack Projects and therefore, it should be able to be developed independently. Theusage of the Nova Service class, or any other OpenStack code, should be completelyavoided to achieve the loose coupling principle that is well known from the software

5.6. SOLUTION APPROACH 21

engineering best practices. There are plenty of official and stable RESTful APIs in allimportant OpenStack projects that can easily be called with a simple http request if someinformation from OpenStack is needed.Based on the architecture proposal the supervisor and the author of this Thesis decidedto start with a new independent software for the Fairness Service that satisfies theseprinciples and at the same time incorporates the lessons learned from the first prototypesoftware.

In the next chapter, the new Fairness Service 2.0 that resulted from the reflexions in thischapter will be introduced.

Chapter 6

The Fairness Service 2.0

The new Fairness Service 2.0 is entirely redesigned from the scratch taking into accountthe experience and knowledge learned from the the Nova Fairness Service (NFS). Somecodes of the old Nova Fairness Service could be reused in the new Fairness Service 2.0but most of the code had to be re-written. For instance, the Fairness Controller is a newcomponent that did not exist in the previous Nova Fairness Service architecture.As a development basis for the new Fairness Service 2.0, Ubuntu 16.04 and the OpenStackNewton distribution is used because it is the current, available and stable OpenStackversion, released on October 6, 2016 [13]. This allows to have a well functioning OpenStackinfrastructure to test the code while developing the Fairness Service 2.0. However, theFairness Service 2.0 should be almost immune to a new release because, as we will seelater, in its new architecture it is loose coupled from the rest of the infrastructure.It is crucial to have the new Fairness Service decoupled from the Nova code. Otherwiseeven small changes to the Nova code can break the Fairness Service and as we have alreadyseen in Chapter 4 and 5, changes in the whole OpenStack project are very frequent. Theway to access to OpenStack is through the different RESTful APIs like the Compute APIor the Identity API. Another way to get information from OpenStack is to access theMySQL database directly that resides on the controller. In fact, not all information thatthe Fairness Service needs is available through the APIs, therefore database queries arenecessary.

6.1 Architecture

In an OpenStack Compute architecture we have two main components: the OpenStackcontroller and the OpenStack nodes. In a similar way also the Fairness Service architecturehas one Fairness Controller (FC) and one or more Fairness Nodes (FN). For the inter-nodecommunication it uses the ZeroMQ library. It is written in C and has a Python bindingthat supports all functions needed. To use it, it is as easy as to import a module inPython.The Fairness Controller needs information from OpenStack. For example a user list. Toget the information, there is the official OpenStack API. But not every piece of information

23

24 CHAPTER 6. THE FAIRNESS SERVICE 2.0

that is needed is also available from the API. Thus the Fairness Controller has also a directaccess to the OpenStack relational DB that runs on the OpenStack controller as well.The Fairness Node needs some infrastructure information as well and the most convenientway is a direct call to the API that is accessible on the OpenStack controller. To gatherVM utilization information and to reallocate the resource the Fairness Node uses Libvirt.Libvirt is another C library, also used by OpenStack itself to communicate with thehypervisors situated on the Compute nodes. Libvirt gives a unified access interface to thedifferent hypervisors. Using Libvirt a developer can avoid to write different code for everysingle hypervisor. This architecture is depicted in Figure 6.1: It shows the OpenStackController and one OpenStack Compute Node. In a real environment it can have morethan one Compute Node.

Figure 6.1: Architecture of the new Fairness Service 2.0.

The responsibilities of the two Fairness Service components become clearer now: TheFairness Controller is responsible for the registration of new Fairness Node and to controlthe inter-node communication. It is mainly responsible for the start and the intervaltime of the communication. But it also establishes for example the communication path.Whereas the Fairness Node is responsible for the main tasks described in Section 3.3.

6.2 Program sequence

Since the role of the Fairness Controller is a different one as the role of the Fairness Node,this section is divided in two subsections: the Fairness Controller and the Fairness Node.

6.2.1 Fairness Controller

To have a complete overview of the entire cloud, the controller first gathers the IP ad-dresses of all Nova nodes directly from the local relational database which is also usedby the other OpenStack components like Nova or Keystone. The controller needs to havethe overview of all existent hosts in the cloud because it is the entity that is responsiblefor building the communication ring that will be explained in section 6.4. It needs the

6.2. PROGRAM SEQUENCE 25

IP addresses of all hosts to establish the communication path through all of them. Atthis point one of the lessons learned of the prototype mentioned previously was enforced:using a ring topology to keep the communication overhead low for the inter-node mes-saging. The controller assigns itself as the successor node to the first Fairness Node thatregisters to it. Then, the second Fairness Node gets as successor the first node, the thirdnode gets the second, and so forth, until all nodes are registered. Another way would beto just take all IP addresses in a sorted list to establish the ordering of the hosts in thering. But with the former and also implemented way, no node has to be updated withits new successors when a new node registers and the ring is always closed and readyto communicate even not all existent Nova nodes registered as Fairness Nodes. A third,more complicated and maybe more efficient way for a geographically distributed cloud,would be a reconfiguration of the topology based on geological data every time a newnode connects to the Fairness Controller. But, this could be considered for future work.For the time being the controller is waiting until all nodes are connected to it assumingthat all Nova nodes will participate as Fairness Nodes as well. In other words, as soon asa node connects to the controller, the controller communicates to the requesting node itssuccessor node. When the controller realizes that all hosts with a running Nova serviceeventually connected its Fairness Service as well, the controller starts the ring communi-cation process. The controller is part of the ring and supervises the whole communication.The last Node nn sends its message to the controller, then the controller measures thetime passed since the start and then starts the next cycle according to the desired cycleduration defined in the configuration file.The hosts IP addresses are the only piece of information that the Fairness Controller needsthat is not available from the OpenStack API. Any other information that the FairnessController needs and can be obtained from the OpenStack API, for example a user list andthe user’s quota q(ui). This is needed to initialize the User Greediness Vector (UGV). An-other metric that is also needed to initialize the UGV is the Cloud Resource Supply CRS.The CRS is calculated every time when the Fairness Controller node gets a registrationrequest from a Fairness Node. It receives the Fairness Node’s Node Resource Information(NRI). The NRI is a vector with the values of the actual physical resources (PR) of therequesting host. The controller updates its local Cloud Resource Supply (CRS) vectorsumming the received NRI to the CRS. It is an element wise summing of the physicalresource values. The CRS was implemented as a dictionary as shown in Listing 6.1.

1 {2 ’cpu’: ’23996’,3 ’memory’: ’8081816’,4 ’disk read bytes’: ’318903333’,5 ’disk write bytes’: ’121266666’,6 ’network receive’: ’250000000’,7 ’network transmit’: ’250000000’8 }

Listing 6.1: An example of the CRS.

The CRS is always attached to the User Greediness Vector (UGV) when sending themessage in the communication ring. This ensures that the Fairness Nodes have always thenewest CRS vector for their calculations, for the case that a new Fairness Node registeredor unregistered since the last message cycle. With all required information available the


Fairness Controller initializes the UGV with the negative quota values. To calculate thisvalues it first calculates the values of all resources. This calculation is shown in Equation6.1.

resource value array =CRS × q(ui)

q(uall)(6.1)

The resource value array calculates all six CRS resources simultaneously with elementwise operations on NumPy arrays. Then all six values are summed up to one scalar andthis scalar is then used as the initial user greediness value. The initial user values of allusers are then put into a dictionary that eventually forms the User Greediness Vector(UGV). Thus, the User Greediness Vector (UGV) is a Python dictionary that has as keythe username and as value the user’s greediness. The user’s greediness is a scalar thatis calculated on every Fairness Node. An example for the initial User Greediness Vectorsent by the controller is shown in Listing 6.2.

1 {2 ’user 1’: −254052908.0,3 ’user 2’: −254052908.04 }

Listing 6.2: The UGV sent from the controller.

In the example in Listing 6.2 the scalars of the two users are the same. This happenswhen the Quotas are for both users exactly the same. OpenStack usually gives all usersof the same project the same quotas. The administrator is able to manually change thequotas for a single user or the whole project with the $ nova quota-update command[12].Since the User Greediness Vector is initialized with the negative sum of quota values, thismeans that the user that does not use all of their quotas will get a heaviness premium.If for example there are two users and u1 has only one VM running and u2 has 10 VMsrunning with ten times more VRs bound to them, and both users impose the same stress(resource utilization) to the Cloud, then u1 will be less heavy because she did not use allof her available quotas.There are several quotas metrics available from OpenStack. But for the Fairness Serviceonly VCPU and VRAM are usable. Since the resources important for the Fairness Serviceare six, namely CPU, RAM, disk read, disk write, network received, network transmitted,for the four restant resources it is sufficient to have just a dummy value of 1.While controlling the CRS+UGV message ring, the Fairness Controller keeps all the timelistening for new nodes that want to register. Listing 6.3: shows the algorithm in acompact way.

The controller runs with 3 threads. One is the main thread that sends the CRS+UGVmessage. The second one is listening to new Fairness Nodes that want to register. Andthe last thread is for the server that listens to the CRS+UGV message that comes fromthe ring.

6.2. PROGRAM SEQUENCE 27

1 get all node IP addresses from DB2 While TRUE:3 listen to incoming node requests4 for every node that connects:5 send the IP address of its successor node6 sum NRI to CRS7 If all nodes registered to the controller:8 every µ seconds:9 start CRS+UGV message cycle

10 Measure elapsed time

Listing 6.3: The Fairness Controller algorithm.

6.2.2 Fairness Node

When a Fairness Node ni starts running, its first task is to gather information about thephysical resources (PR) of the host where it is actually running. This information is alsocalled Node Resource Information (NRI) and it is needed by the Fairness Controller. Thenit connects to the Fairness Controller to communicate its NRI, to learn about its successornode ni+1, and to inform about its readiness. Since the NRI, and therefore also the CRS,represents the actual physical resources (PR), and the PRs of a node are static, it issufficient to communicate the NRI only once, at the time when a Fairness Node registers.Next, in order to have an own understanding of the cloud the host is running, and the VMsthat run on the host, it starts by constructing the internal representation of the cloud.Basically, it creates an object for the node and for every VM. The cloud is representedwith the global normalization factor. At this point, the Fairness Node is ready for theCRS+UGV communication procedure. That means the node is waiting in an idle state forthe CRS+UGV message from its predecessor node ni−1. The node ni does not know if itspredecessor is another Fairness Node or the Fairness Controller. Thus, in this subsectionthe predecessor node ni−1 can be either a node or the controller. As already mentioned,the CRS+UGV message is triggered by the Fairness Controller as soon as all nodes haveregistered. The controller sends to the first node a message containing the current CRSand the User Greediness Vector (UGV). In the very first cycle the Fairness Node cannotuse the information because it does not have the greediness values of all its successornodes. So before the User Greediness Vector can be used for further calculations it needsto cycle at least once. When a node ni gets the CRS+UGV message, it updates the vectorwith the greediness of the users that have VMs running on the node ni. Updating meansto first subtract the own user greediness from the last cycle and then sum the new ownvalue to the User Greediness Vector that was just received. After the User GreedinessVector is updated it will be sent to the successor node ni+1. The Greediness calculation isdescribed in Section 3.1. In the meantime, during a node waits for the next user greedinessmessage (representing the next cycle) it uses the updated UGV to calculate the prioritiesand applies the reallocation parameters for CPU, RAM, disk and network on every VMpresent on that host. Since the CRS changes every time a host comes up or shuts down,the first thing a compute node does when a User Greediness message arrives is check theCRS hash value to quickly determine if the CRS has changed compared to the last cycle.This avoids the necessity to compare all values from the received CRS with all values


of the CRS saved on the compute node. If the received CRS really changed, then thenormalization vector has to be updated with the new CRS values.

This User Greediness communication ring only stops when a Node shuts down. In otherwords, the same User Greediness Vector cycles through all nodes as long as the FairnessService runs. Listing 6.4: shows the algorithm in a compact way.

1 collect NRI2 send NRI to Controller and wait for successor IP address3 while TRUE:4 listen to CRS+UGV message5 for all VMs hosted by ni :6 collect RUI7 calculate VM v heaviness h(v)8 for every user u that has VMs v hosted by ni :9 calculate the user’s heavinesses h(u)

10 send the updated CRS+UGV message to the successor node ni+1

11 for every VM v hosted by ni :12 Map heaviness h(u) to priorities13 calculate resource shares, soft−limits, weights, traffic classes14 reallocate resources

Listing 6.4: The Fairness Node algorithm running on node ni.

Figure 6.2 shows a sequence diagram that sums up the most important steps from the startof the service until after the first CRS+UGV message cycle. For the sake of simplicity thevery first CRS+UGV message cycle, that is not usable for the nodes, is not illustrated.

6.3 Information to be collected

There are several pieces of information that have to be collected to be able to calculatethe priorities and heavinesses. In particular the NRI and the RUI. In this section we willgo through these data.The information that the Fairness Nodes collect in order to send their NRI to the controllerare the actual physical resources (PR):

� Number of CPUs:This value is retrieved by calling cpu_count() from the multiprocessing package.The number of CPUs is then multiplied by the BogoMIPS value to have a more orless comparable value across different host. Although it is not a scientific benchmark,BogoMIPS values are a sufficient reference point for our calculations because in thetestbed there are five identical physical hosts. In the future work section alternativeways to get a more reliable benchmark value for CPUs are described.

� Random Access Memory in bytes:This is the total installed RAM and it is retrieved by calling in the Linux tool$ free -k

6.3. INFORMATION TO BE COLLECTED 29

Figure 6.2: Sequence diagram of the new Fairness Service 2.0.


� Disk read speed in bytes per seconds:This value is retrieved by calling $ lsblk -io KNAME,TYPE and using the resultsof TYPE disk to call $ sudo hdparm -t /dev/<disk_name>. This has to be per-formed three times to take an average out of the retrieved value.

� Disk write speed in bytes per seconds:Similar to the read performance measures we have to call a Linux tool three timesand take the average out of it:$ dd if=/dev/<disk_name> of=∼/disk_benchmark_file bs=8k count=200k+

� Max bandwidth in bits per seconds:This value is manually set at 1000 Mbit/s for both network received and networktransmitted.

The retrieved Disk I/O speeds are used in the NRI and CRS metrics and are valuesthat describe hardware characteristics. They differ considerably for the read and writeoperations. The old Nova Fairness Service did not distinguish between disk read speedand disk write speed and did not measure three times. It made only a one-time disk readtest with the hdparm Linux tool to determine the Disk I/O speed and the result was thenused for both the disk read and disk write values. Since the results from this test can varyaccording to current disk utilization, the new Fairness Service 2.0 takes the same test threetimes and uses the average value. In addition to the hdparm disk read tests it performsalso disk write tests. This is done with the dd tool as shown in the example above andlike the read test it is performed three times to average disk utilization variations. Bothtools return the disk speeds in MB/s.The dd tool, especially in the case of the write test, does not simulate a real environmentclose enough. According to [1] the problems are:

� The tests are single-threaded, sequential-writing. But the opposite is more commonon real world stress.

� Caching because of the small amount of data written.

� The test period is to short for consistent results.

To overcome these shortenings instead of measuring in MB/s, Input/Output Operationsper Second (IOPS) could be implemented in a future work.

Periodically on every host the VMs RUI have to be retrieved The statistics needed arethe following:

� CPU time in seconds

� Memory usage in bytes

� Disk read in bytes

� Disk written in bytes

6.4. COMMUNICATION 31

� Network interface received in bytes

� Network interface transmitted in bytes

All of these values are retrieved with the Libvirt driver implementation described in Sec-tion 6.5.1.The virtual resources (VR) of a VM are defined by the flavour at VM creation time. Forthe Fairness Service we need the VCPU and the VRAM values. These values are retrievedwith libvirt in number of VCPUs and Bytes of VRAM.The VRs are needed to calculatethe endowment of the VMs resources and also to reward the cloud user for correct VMconfiguration.Some information cannot be retrieved from the system and has to be configured beforethe Fairness Service starts. For this purpose the Fairness Service has a fairness.ini config-uration file, where all important information can be configured. For example the FairnessController’s IP address or the message cycle time are such information pieces that theconfiguration file allows to define.

6.4 Communication

As it was seen in Section 5.4, with O(n2) the messages sent for the Nova Fairness Servicewas still too high. For this reason a given specification for the new Fairness Service 2.0 wasto implement a ring topology for inter-node communication that can reduce the numberof messages sent to O(n), with n for the number of active Fairness Nodes. Figure 6.3,shows the message ring.

Figure 6.3: Ring communication topology for the Fairness Service 2.0.

The concatenation of the Cloud Resource Supply and the User Greediness Vector, thatforms the CRS+UGV message, is the most important message for running the FairnessService. It is sent around and is then updated and used by all nodes in the ring. First, itis updated on every node with the user’s heaviness values of the current node and then,it is also used as input for the calculations of the new heaviness values and therefore also


for the new priorities needed for the reallocation of the resources. As it will be seen in thesubsequent sections, the Fairness Node’s role from the point of view of the communicationring is to forward the message, whereas the Fairness Controller’s role is to manage thecommunication with starting and timing functions.

6.5 Drivers

The function of drivers is to provide a means to talk to other software components like Lib-virt, MySQL, etc. Some drivers are already extensively implemented by OpenStack. Forexample the Libvirt driver in OpenStack Newton has 7740 lines of code [23]. This exten-sive usage is due to the fact that Libvirt is handling everything between OpenStack Novaand the hypervisors. The prototype from Chapter 5 depends heavily on the OpenStackimplementations of these drivers. Unfortunately, these drivers are frequently changed andadapted by the OpenStack engineers to Nova’s needs. Since the functions that the Fair-ness Service needs are only a very small subset of the implemented by Nova and somefunctions are even not implemented at all, it is useful to have the own implementation ofthese drivers.

6.5.1 Libvirt driver

Libvirt is the standard virtualization API [7] to manage virtual machines and other vir-tualization functionality and it provides a way to talk to various available virtualizationprovider/hypervisor technologies, like KVM, Hyper-V, Xen, VMWare ESX, etc. Thus,Libvirt allows to implement only one code to manage all hypervisors that are supportedby Libvirt. The Fairness Service 2.0 calls Libvirt whenever it needs a piece of informationabout the Resource Utilization Information (RUI) of a specific VM on a specific host orwhenever it has to reallocate those resources. The VRs of the VMs are also retrievedvia Libvirt. The Libvirt driver’s main function is to communicate to the Libvirt API [8].The RUI and VR information is retrieved from the running VMs by the Libvirt driver.There are two main classes to use. The virConnect class for managing connections tothe hypervisors and the virDomain class for managing virtual machines and retrieve theirinformation and statistics. The statistics retrieved are returned accumulated since VMboot time. Therefore, when querying statistics for the RUI, it is necessary to subtract theprevious values every time the new values are retrieved.Libvirt can access a hypervisor in two ways: local or remote. This means that from thepoint of view of Libvirt, we could design the Fairness Service to run on every node sepa-rately with direct access to the hypervisors, or on one node centralized with remote accessto the hypervisors of all Fairness Nodes. A hybrid version is theoretically also possiblebut it would not make sense because the retrieved RUI and VR values are never sharedwith other nodes or the controller. It is very likely that the hypervisor could provide moredetailed information. It is also expected that the hypervisor is able to perform additionalvirtualization operations than Libvirt does. This is due to the fact that the differenthypervisors have specialized functions, but Libvirt does not cover specialized functions.

6.6. ZEROMQ 33

Since Libvirt is a very mature software, it can be assumed that Libvirt provides all func-tionalities needed for the new Fairness Service 2.0 because the Nova Fairness Service alsoused the Libvirt API and not the individual hypervisor APIs. In the unlikely case thatthe hypervisor API should be needed in the future, we could limit ourselves to the KVMhypervisor since we are working with a pure Ubuntu environment.

6.5.2 OpenStack driver

The OpenStack driver’s task is to get information from the different OpenStack servicesthat expose a RESTful API for http requests. However, the Fairness Service 2.0 needsto communicate only with the Nova Compute API v2.1 and the Keystone Identity APIv3 [18]. The first step when talking to any OpenStack API is to authenticate with theIdentity API. This can be done with an http request to the /v3/auth/tokens endpointsending username and password provided by the Fairness Service configuration file. Theidentity service then checks the user rights and issues a security token. The answer to thecalling client will include this security token that is needed for any other API call in allother OpenStack services. If the token expires the Fairness Service will issue a new token.There are two types of security tokens. Scoped and unscoped. A scoped token is boundto a specific project. It is important to issue a scoped security token. The unscoped tokengives no access to any service.Starting from Compute API version 2.1 the Nova project has introduced Microversions[22]. Microversions allow old clients to talk to newer clouds because it maintains backwardcompatibility. An old client can check for the oldest API Microversion available to verifythe compatibility of the cloud. These mechanisms make the API quite stable over a longperiod of time because it does not break older clients.

6.5.3 MySQL driver

Currently, a way to get the IP addresses of the Nova nodes with the OpenStack API cannotbe found. The answer to a similar question in Stackoverflow [34] supports this statement.This could be the case because OpenStack abstracts the underlying infrastructure fromthe cloud user. As suggested by a Stackoverflow user, one way is to query the Novadatabase [34]. Since the IP addresses are needed by the Fairness Controller only, and theFairness Controller runs on the same host like the relational database MariaDB does, it isa convenient way to implement a MySQL driver to perform queries directly to MariaDB.Note at this point that the Fairness Service does not have an own table or database. Itqueries the data from the Nova database. This is not optimal from the point of view ofthe loose coupling principle, but at the moment no other way is known.

6.6 ZeroMQ

The communication ring is implemented on the nodes with the extended request-replypattern in ZeroMQ [36], [37], [24]. The Fairness Nodes have a Dealer/Router ZeroMQ


socket combination that allows to receive a message, manipulate the message and forwardit to the successor node. The Fairness Controller acts as a client and as a server at thesame time. It has a ZeroMQ server that listens to incoming messages that were origi-nated from the ZeroMQ client on same controller and traversed all Fairness Nodes. Whenthe message arrives at the ZeroMQ server, the ZeroMQ server, sends an acknowledgmentmessage back that traverses all nodes again until eventually reaching the ZeroMQ client.Basically, it is the same as a common client-server communication but with intermediatestops.Implementing the own ZeroMQ messaging queue allows us to be flexible with the experi-mentation of different communication models since on this way we do not use the Open-Stack messaging system anymore. This flexibility is needed because the first experimentswith the Fairness Service prototype showed us the presence of too much communicationoverhead. The independence from the OpenStack messaging system stands also in align-ment with the design requirement of independence of other OpenStack components.ZeroMQ supports the desired flexibility with many different communication patterns de-scribed in the official ZeroMQ guide. The four built-in core ZeroMQ patterns are:

� Request - reply is the most similar pattern to the common Client - server paradigm.It connects a set of clients to a set of servers.

� Publisher - subscriber allows clients to subscribe on servers for different topics.

� The Pipeline pattern can have multiple steps and loops with the connected nodes.

� The Exclusive pair pattern is a special case. It connects two threads in a processover to sockets exclusively.

6.7 State of the implementation

As already stated in the introduction, the software implementation progress is estimatedat roughly 75%. To conclude the implementation that would allow for the evaluations ofthe Fairness Service several tasks have still to be done.The Fairness Controller is almost completed. Only a minor part is missing: the measure-ment of the message cycle time that allows to delay the start of the next message cycleaccording to the value configured in the configuration file.On the Fairness Node, the main part that is not implemented is task Number 5 fromSection 3.3: the resource reallocation part. This encompasses as well the subtask to mapthe heaviness to priorities. To be able to do the evaluation, a dedicated logging function isneeded. This should be implemented beside the regular logging function for troubleshoot-ing in order to have the data in a separated file.All these mentioned and other minor open issues are tracked on freedcamp.com1. The justmentioned issues are marked with high priority. The tasks marked with medium priorityare not necessary to be able to perform the evaluations but would help to produce betterand faster results. And the tasks marked as low priority are considered improvements

1https://freedcamp.com/Riccardos Projects pTq/Fairness Servic AYfU/todos

6.8. INTEGRATION INTO COMMUNITY 35

or tasks for future work. To have a relatively mature software for integrating it into theOpenStack community, it is necessary to take care of the low priority tasks as well.Moreover, when starting the evaluations, the system has to be finely tuned and furtherchanges are not excluded.

6.8 Integration into Community

An important aspect of the new software is to integrate it to the OpenStack community.Since the Fairness Service 2.0 is now a separate OpenStack project and not anymore partof the Nova Compute project the integration process into the OpenStack Community isnow a different and easier one than the integration of patches and new features into anexisting OpenStack project. It is necessary to follow the official guidelines [19] to create anew project and it is possible to apply with the technical committee to become an officialOpenStack project. In [20] the Requirements for new OpenStack Projects applicationsare described. According to these guidelines for creating an OpenStack project the firststeps towards to become an official project have already been done. It was registered theFairness Service 2.0 as a subproject in Launchpad [3]. Launchpad is a website that allowsusers to develop and maintain open source software. It was also uploaded the code on thePython Package Index under the Apache 2.0 licence [32].Another feature that is available for both, official OpenStack projects and also for relatedOpenStack projects is the OpenStack continuous integration (CI) infrastructure for acontinuous integration with Gerrit, Zuul and Jenkins. The workflow of this CI systemis shown in Figure 6.4. Gerrit is the code review system that is used for OpenStack.It is closely integrated with the Git version control system and since it is web-based, itallows users to accept or reject code changes using a Web browser. Zuul is the gatekeeperbetween Gerrit and Jenkins. When a change in Gerrit occurs it triggers the correct jobsto run in Jenkins. Jenkins is an open source continuous integration server. It allows thedeveloper to have frequent builds and run any required tests in an automated way afterevery code commit.

In this chapter the architecture, the program sequence, and different important aspectsof the new Fairness Service 2.0 was outlined. More detailed information is available ascomments in the codefiles on Github.com [25]. It also described the way to make a softwareproject part of the official OpenStack projects and a short overview of the OpenStack CIflow is given. The next chapter is devoted to the conclusions and describes possible futurework.


Figure 6.4: OpenStack continuous integration (CI) flow-chart. (Source: openstack.org)

Chapter 7

Conclusions and future work

While porting the first software prototype to the newest OpenStack version we came tothe conclusions that it did not make sense to port it because of several reasons. One reasonwas that since the prototype is basically an old Nova version extended by the FairnessService the system was too complex to port it to the new OpenStack version and to furtherextend its functionalities. Thus, it was not known how long this would take. Anotherreason was that the software lifetime would then still be limited by the OpenStack versionand the next OpenStack version was about to be released in the coming couple of weeks.

During the creation of this Master Thesis we attended two OpenStack meetups in Zurich,Switzerland and various individuals took interest in the Fairness Service. This fact in-creased the desire to work towards a Fairness Service that also other individuals can useand even improve according to their specific needs. Thus, if beside the already mentionedreasons, it is also added the ability to make it one of the official OpenStack projectsor to publish it as a free OpenStack related project then there is no better way thanto deprecate the old prototype and to develop a new software. The new architectureand the implementations of the Fairness Service 2.0 are a sound foundation for furtherdevelopment and research in the area of multi-resource fairness during VM runtime.

7.1 Future Work

The result of this Master Thesis is an excellent basis for further student projects. In thissection It will be outlined a few ideas for future work.

Since OpenStack can be installed and configured for high availability [21], a possible futurework could be to make also the Fairness Controller ready for high availability.

Currently when sending the CRS+UGV message in the ring, the server sends only theACK message back. To make use of the communication ring more efficiently, this wayback to the client could also be used to send the CRS+UGV message the other wayaround. This would increase the messaging efficiency from 2n to n, with n for the numberof nodes.

37

38 CHAPTER 7. CONCLUSIONS AND FUTURE WORK

In this work, it is assumed that the physical host are geographically close to each other.But as we know from public clouds like Google Cloud Platform or Amazon Web Servicesa cloud can even be distributed over countries or continents. For this case, it wouldmake sense to take care of the communication topology. For example, it could be basedon geological data. This means that when a node registers to the controller, it alsocommunicates its position on earth. This could be with degree of longitude and latitudeor maybe with the help of system locales. This geological data can be used to reconfigurethe communication topology, if necessary.

Hard disk measurements for NRI are performed with dd and hdparm and the resultsare in MB/s. According to [1] it would be more precise to measure in Input/OutputOperations per Second (IOPS). First tries with the recommended benchmark tool in [1]lasted roughly 30 minutes per trial. So, replacing the current MB/s measurements withthe IOPS measurements would result in a start-up time of 30 minutes for every host.Since this behaviour would cause a poor user experience, one option could be to extendthe software with the new IOPS functionality and when the Fairness Service starts, it firstretrieves the MB/s values in order to have a fast start of the service, and then in a secondtime to retrieve the IOPS values and update the CRS.

The NRI values for network received and network transmitted is currently manually set at1000 MBit/s. Here the idea is to measure the actual network speeds or at least to retrievethe theoretical NIC speeds.

The number of CPUs in the NRI is multiplied with the BogoMIPS value to have a morecomparable metric across different computers. This is not optimal but, since in the testbedof the University of Zurich there are five identical servers it does not matter. Nevertheless,the Fairness Service is meant to run on a heterogeneous infrastructure. That is why abetter metric has to be used. Possible alternatives are Sysbench or the SPEC benchmarkdatabase.

Bibliography

[1] BinaryLane: How to benchmark disk I/O, https://www.binarylane.com.

au/support/solutions/articles/1000055889-how-to-benchmark-disk-i-o,Online; accessed December 25th, 2016

[2] T. Bonald, J. Roberts: Enhanced Cluster Computing Performance through Propor-tional Fairness; Performance Evaluation 79, April 2014, pp 134-145.

[3] Canonical Ltd.: Launchpad, fairness-service. https://launchpad.net/

fairness-service, Online; accessed December 31st, 2016

[4] Y. Etsion, T. Ben-Nun, D. G. Feitelson: A Global Scheduling Framework for Vir-tualization Environments; IEEE International Symposium on Parallel DistributedProcessing, IPDPS’09, Rome, Italy, May 2009, pp 1-8.

[5] A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, I. Stoica. Domi-nant Resource Fairness: Fair Allocation of Heterogeneous Resources in Datacenters;Technical Report UCB/EECS-2010-55, EECS Department, University of California,Berkeley, CA, USA, May 2010.

[6] A. Gruhler: ”Investigation of Resource Reallocation Capabilities of KVM and Open-Stack”, Bachelor Thesis, University of Zurich, 2015

[7] Libvirt Project: libvirt virtualization API https://libvirt.org/, Online; accessedDecember 29th, 2016

[8] Libvirt Project: libvirt Application Development Guides, https://libvirt.org/

devguide.html, Online; accessed December 29th, 2016

[9] S. Mannhart: ”Development and Evaluation of an OpenStack Extension to EnforceCloud-wide, Multi-resource Fairness during VM Runtime”, Bachelor Thesis, Univer-sity of Zurich, 2016

[10] S. Mannhart: novafairnes code repository https://github.com/savf/

novafairness, Online; accessed December 28th, 2016

[11] OpenStack Foundation: OpenStack Installation Tutorial for Ubuntu: SQLdatabase. http://docs.openstack.org/newton/install-guide-ubuntu/

environment-sql-database.html, Online; accessed December 23rd, 2016

39

40 BIBLIOGRAPHY

[12] OpenStack Foundation: Compute service (nova) command-line client: novaquota-update. http://docs.openstack.org/cli-reference/nova.html#

nova-quota-update, Online; accessed December 26th, 2016

[13] OpenStack Foundation: OpenStack Releases. https://releases.openstack.org,Online; accessed December 28th, 2016

[14] OpenStack Foundation: Compute API: Extensions. http://developer.openstack.org/api-guide/compute/extensions.html, Online; accessed December 28th, 2016

[15] OpenStack Foundation: Blueprints. https://wiki.openstack.org/wiki/

Blueprints, Online; accessed December 28th, 2016

[16] OpenStack Foundation: OpenStack Installation Tutorial for Ubuntu. http:

//docs.openstack.org/newton/install-guide-ubuntu/index.html, Online; ac-cessed December 28th, 2016

[17] OpenStack Foundation: Official code repository. https://github.com/openstack,Online; accessed December 29th, 2016

[18] OpenStack Foundation: OpenStack API Documentation. http://developer.

openstack.org/api-guide/quick-start/, Online; accessed December 30th, 2016

[19] OpenStack Foundation: Project Creator’s Guide. http://docs.openstack.org/

infra/manual/creators.html, Online; accessed December 31st, 2016

[20] OpenStack Foundation: Requirements for new OpenStack Projectsapplications. https://governance.openstack.org/tc/reference/

new-projects-requirements.html, Online; accessed December 31st, 2016

[21] OpenStack Foundation: OpenStack High Availability Guide. http://docs.

openstack.org/ha-guide/, Online; accessed December 31st, 2016

[22] OpenStack Foundation: Microversions. http://developer.openstack.org/

api-guide/compute/microversions.html, Online; accessed January 1st, 2017

[23] OpenStack Foundation: Github repository for /nova/virt/libvirt/driver.pyhttps://github.com/openstack/nova/blob/stable/newton/nova/virt/

libvirt/driver.py, Online; accessed January 1st, 2017

[24] R. Patane: ZeroMQ for ring topology - how to config-ure sockets? http://stackoverflow.com/questions/40947877/

zeromq-for-ring-topology-how-to-configure-sockets, Online; accessedDecember 29th, 2016

[25] R. Patane: Fairness Service 2.0 source code. https://github.com/patanric/

fairness-service, Online; accessed January 4th, 2017

[26] P. Poullie, B. Stiller: Cloud Flat Rates Enabled via Fair Multi-resource Consumption;Technical Report IFI-2015.03, https://files.ifi.uzh.ch/CSG/staff/poullie/

extern/publications/IFI-2015.03.pdf, Universitat Zurich, Zurich, Switzerland,October 2015

BIBLIOGRAPHY 41

[27] P. Poullie, B. Stiller: Cloud flat rates enabled via fair multi-resource consumption;IFIP International Conference on Autonomous Infrastructure, Management and Se-curity. Springer International Publishing, 2016.

[28] P. Poullie, S. Mannhart, B. Stiller: Defining and Enforcing Fairness Among CloudUsers by Adapting Virtual Machine Priorities During Runtime;, 2016.

[29] P. Poullie: Virtual Machine Priority Adaption to Enforce Fairness Among CloudUsers;, 12th International Conference on Network and Service Management. 2016.(Conference Presentation)

[30] P. Poullie, B. Stiller: The Design and Evaluation of a Heaviness Metric for CloudFairness and Correct Virtual Machine Configurations; 13th International Conferenceon Economics of Grids, Cloud, Systems and Services (GECON 2016), ”Lecture Notesin Computer Science (LNCS)”, Athens, Greece, September 2016, pp 1-14.

[31] Python Software Foundation: The Python Debugger. https://docs.python.org/2/library/pdb.html, Online; accessed December 29th, 2016

[32] Python Software Foundation: Package Index > fairness-service. https://pypi.

python.org/pypi/fairness-service, Online; accessed December 31st, 2016

[33] B. Radunovic, J.Y.L. Boudec: A Unified Framework for Max-Min and Min-MaxFairness With Applications; IEEE/ACM Transactions on Networking 15(5), October2007, pp 1073-1083.

[34] Stack Exchange Inc.: Get Ip address of computes nodes in anopenstack network http://stackoverflow.com/questions/18010889/

get-ip-address-of-computes-nodes-in-an-openstack-network, Online;accessed January 2nd, 2017

[35] G. Wei, A. Vasilakos, Y. Zheng, N. Xiong. A Game-theoretic Method of Fair ResourceAllocation for Cloud Computing Services; The Journal of Supercomputing 54(2),November 2010, pp 252-269.

[36] ZeroMQ: Shared Queue (DEALER and ROUTER sockets) http://zguide.zeromq.org/py:all#Shared-Queue-DEALER-and-ROUTER-sockets, Online; accessed De-cember 29th, 2016

[37] ZeroMQ: The Extended Reply Envelope http://zguide.zeromq.org/py:all#

The-Extended-Reply-Envelope, Online; accessed December 29th, 2016

42 BIBLIOGRAPHY

Abbreviations

CPU Central Processing unitCRS Cloud Resource SupplyCSG Communication Systems GroupDB databaseFC Fairness ControllerFN Fairness NodeFS Fairness ServiceGM Greediness MetricIFI Computer Science Departement, ger: Institut fur InformatikNFS Nova Fairness ServiceNRI Node Resource InformationRAM Random Access MemoryRUI Resource Utilization InforationSLA Service Level AgreementUGV User Greediness VectorVM Virtual Machine

43

44 ABBREVIATONS

Glossary

Most descriptions especially those concernig OpenStack are taken from the official Open-Stack Glossary1.

Blueprint Launchpad blueprints are used to track the implementation of significant fea-tures in OpenStack. In addition to tracking implementation status, blueprints canalso be used to propose a design and get it approved.

domain An Identity API v3 entity. Represents a collection of projects, groups and usersthat defines administrative boundaries for managing OpenStack Identity entities.Also, a domain is an entity or container of all DNS-related information containingone or more records.

host A physical computer. Host and node are used interchangeable in this work.

Libvirt Libvirt is a collection of software that provides a convenient way to managevirtual machines and other virtualization functionality, such as storage and networkinterface management. These software pieces include an API library, a daemon(libvirtd), and a command line utility (virsh).

node see host.

project Projects represent the base unit of ”ownership” in OpenStack, in that all re-sources in OpenStack should be owned by a specific project. In OpenStack Identity,a project must be owned by a specific domain.

quota In Compute and Block Storage, the ability to set resource limits on a per-projectbasis.

role A personality that a user assumes to perform a specific set of operations. A roleincludes a set of rights and privileges. A user assuming that role inherits those rightsand privileges.

user In OpenStack Identity, entities represent individual API consumers and are ownedby a specific domain. In OpenStack Compute, a user can be associated with roles,projects, or both.

vitrual machine An operating system instance that runs on top of a hypervisor. Multi-ple VMs can run at the same time on the same physical host. OpenStack calls VMssometimes servers or instances and libvirt calls VMs domains.

1http://docs.openstack.org/user-guide/common/glossary.html

45

46 GLOSSARY

List of Figures

4.1 OpenStack infrastructure installed in the testbed at the University of Zurich. 12

5.1 Central message broker (RabbitMQ) vs. decentralized brokerless (ZeroMQ)communication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

6.1 Architecture of the new Fairness Service 2.0. . . . . . . . . . . . . . . . . . 24

6.2 Sequence diagram of the new Fairness Service 2.0. . . . . . . . . . . . . . . 29

6.3 Ring communication topology for the Fairness Service 2.0. . . . . . . . . . 31

6.4 OpenStack continuous integration (CI) flow-chart. (Source: openstack.org) 36

47

48 LIST OF FIGURES

Listings

5.1 The steps conducted by the fairness service running on node ni. . . . . . . 165.2 An excerpt of Nova modules and classes that are used by the Nova Fairness

code. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.1 An example of the CRS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256.2 The UGV sent from the controller. . . . . . . . . . . . . . . . . . . . . . . 266.3 The Fairness Controller algorithm. . . . . . . . . . . . . . . . . . . . . . . 276.4 The Fairness Node algorithm running on node ni. . . . . . . . . . . . . . . 28

49

50 LISTINGS

Appendix A

Contents of the CD

The attached CD contains the following files and directories:

� abstract.txt: A plain text of the English abstract.

� zusammenfassung.txt: A plain text of the German abstract.

� thesis.pdf: PDF version of the thesis.

� fairness-service: Directory containing the source code.

� documents/presentations: Directory containing the mid-term presentation andfinal presentation.

� documents/thesis: Directory containing the LaTeX source of this thesis.

51

Improvement and Evaluation of an OpenStack Extension to ... · in OpenStack, the prototype...

Documents

Transcript of Improvement and Evaluation of an OpenStack Extension to ... · in OpenStack, the prototype...