PowerScaling: An Elastic PowerManagement Solution inVirtualized Data Center

Selome Kostentinos Tesfatsion

May 24, 2013Master’s Thesis in Computing Science, 30 credits

Supervisor at CS-UmU: Johan TordssonExaminer:

Umea UniversityDepartment of Computing Science

SE-901 87 UMEaSWEDEN

Abstract

At present, large-scale data centers are typically over-provisioned in order to handle peakload requirements. The resulting low utilization of resources contribute to a huge amountsof power consumption in data centers. The effects of high power consumption manifest in ahigh operational cost in data centers and carbon footprints to the environment. Therefore,the management solutions for large-scale data centers must be designed to effectively takepower consumption into account. In this work, we present the data center managementdimensions that can be used to control servers in an energy-efficient manner. The identifieddimensions integrate software solution that make use of virtualization capabilities to dynam-ically configure resources of a service and hardware solution that changes the configurationof physical resources to achieve high power efficiency. The proposed system consists of acontroller that combines feedback and feedforward information to determine a configurationthat minimize power consumption while meeting the performance target. The controller canalso be configured to accomplish power minimization in a stable manner, without causinglarge oscillations in the resource allocations. Our experimental evaluation based on the Sys-bench benchmark combined with workload traces from production systems shows that ourapproach achieves the lowest energy consumption among the compared three approacheswhile meeting the performance target.

ii

Contents

1 Introduction 1

2 Introduction to Cloud Computing and Virtualized Data Center 3

2.1 Cloud Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.1 Characteristics of Cloud Computing . . . . . . . . . . . . . . . . . . . 3

2.1.2 Service Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.1.3 Deployment Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 Virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.3 Data Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Related Work 7

4 The System Model 9

4.1 Power Management Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.2 System Modeling Experimentation . . . . . . . . . . . . . . . . . . . . . . . . 11

4.2.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4.2.2 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5 System Architecture 15

6 Design of Optimal Controller 17

7 Experimental Evaluation & Analysis 21

7.1 Experiment Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

7.2 Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

8 Conclusion and Possible Improvements 25

9 Acknowledgements 27

References 29

iii

iv CONTENTS

List of Figures

2.1 Virtual Machine Abstraction. . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4.1 Relative power impact of different resources on dynamic power consumption. 10

4.2 An input-output model for a system. . . . . . . . . . . . . . . . . . . . . . . . 11

4.3 The impact of each management dimension on power usage and performance. 13

4.4 Effect of tuned Vs non-tuned system on performance and power usage. . . . . 13

5.1 Overall system architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

6.1 Stepwise penality function. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

7.1 Comparison of power saving approaches. . . . . . . . . . . . . . . . . . . . . . 23

7.2 Impact of performance-power trade-off. . . . . . . . . . . . . . . . . . . . . . . 23

7.3 Impact of configuration cost on performance and cores. . . . . . . . . . . . . . 24

7.4 Impact of VM booting time on VM selection. . . . . . . . . . . . . . . . . . . 24

v

vi LIST OF FIGURES

List of Tables

7.1 VM booting time in secondsfor different number of cores and CPU frequen-

cies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

7.2 Cost of SLA violation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

vii

viii LIST OF TABLES

Chapter 1

Introduction

Cloud computing is emerging as one of the key technologies in the IT industry. It providesa model for delivering configurable computing resources on-demand to customers over anetwork in a self-service fashion, independent of device and location [35]. The resourcesrequired to provide the necessary Quality-of-Service(QoS) levels are virtualized, shared,rapidly provisioned and released with minimal service provider interaction.

With the rise of cloud computing, the number of data centers is growing fast. Accordingto a report in [43] there are about 500000 data centers worldwide. These data centers arenot only growing in numbers but also in size. For instance, as reported by [45] google runsaround 900000 servers in its data centers.

Unlike traditional data centers that run services on a dedicated hardware infrastructure,modern data centers use virtualization to facilitate dynamic sharing of physical resourceswhile retaining application isolation and provide application portability. The virtualizationtechnologies enable rapid provisioning of Virtual Machines (VMs) and thus allow cloudservices to scale up and down on-demand. This elasticity can be achieved using horizontalelasticity where VMs are changed during a service’s operation and vertical elasticity wherecapabilities of a running VM, typically in terms of CPU and RAM, are changed dynamically.

At present, large-scale data centers are typically over-provisioned in order to handlepeak load requirements. This low utilization of resources contribute to a huge amounts ofpower consumption. A recent survey [47] reports that worldwide, the digital warehouses useabout 30 billion watts of electricity. Data centers that have re-engineered their systems todecrease wasted power, still consume huge amount of electricity. Among them are Google’sdata centers that consume nearly 300 million watts and Facebooks about 60 million watts.As discussed in [48], in a typical Google cluster it is common for a server with only 10%load to draw 50-60% of its peak power consumption. The effects of high power consumptionmanifest in a high operational cost in data centers. A study in [44] presents that the costof energy has steadily risen to capture 25% of total operating costs, and it now sits close tothe top of the list. The costs to operate servers are expected to exceed the costs to purchaseserver hardware by 2015 [42]. What is more, the power consumption in large-scale datacenters also raises many other serious issues including carbon dioxide and system reliability.As reported in [50], running a single high performance 300-watt server during a year canemit as much as 1300kg CO2, without mentioning the cooling equipment. These facts meanthat management solutions for large-scale data centers must be designed to effectively takepower consumption into account.

While power consumption must be minimized, one of the important requirements for

1

2 Chapter 1. Introduction

a data center, is to guarantee reliable QoS required by customers. QoS can be definedin terms of Service Level Agreements (SLAs) that are part of customer commitments anddescribe the key performance metrics as minimal throughput and maximal response time[46]. Minimizing power consumption may inherently result in performance loss. It is thus,important to guarantee the application QoS while minimizing the power consumption ofdata centers. To achieve this goal, a well designed trade-off between power saving andsystem performance is necessary.

The main objective of this work is to discuss the data center management dimensionsthat can be used to control servers in an energy-efficient manner, present a model that repre-sent the relationship between power consumption and performance for the different possibleconfigurations of a service and an architectural framework and algorithm for energy-efficientmanagement of cloud computing environments. The proposed system consists of an optimalcontroller that combines feedback and feedforward to solve resource configuration problem.The controller determines the optimal configuration to minimize power consumption basedon the discrepancy between the measured performance of the system and the target perfor-mance. It can also be configured to accomplish the minimization in a stable manner, withoutcausing large oscillations in the resource allocations and a higher reconfiguration delay. Ourexperimental evaluation based on the Sysbench benchmark combined with workload tracesfrom production systems shows that our configuration approach achieves the lowest energyconsumption among the compared three approaches.

The rest of the work is organized as follows. In Chapter 2, we present a brief descriptionof the current cloud computing and virtualization technologies. Chapter 3 presents survey ofrelated work. In Chapter 4, we present the design of the system model. Chapter 5 describesthe architecture of our proposed system. This is followed by a detailed description of thedesign of the optimal controller in Chapter 6. In Chapter 7, we present implementationdesign with some preliminary evaluation results. Possible improvements and conclusion ofthe work is presented in Chapter 8.

Chapter 2

Introduction to CloudComputing and VirtualizedData Center

In this chapter we give a brief description of the current cloud computing environment, itsessential characteristics, the different service and deployment types. Then virtualizationtechnology and its advantages is presented. Finally, we introduce Virtualized data centers.

2.1 Cloud Computing

Various definitions of cloud computing have been proposed [35], [36], [37]. According tothe US National Institute of Standards and Technology (NIST) [35] Cloud computing is amodel for enabling ubiquitous, convenient, on-demand network access to a shared pool ofconfigurable computing resources (e.g., networks, servers, storage, applications, and services)that can be rapidly provisioned and released with minimal management effort or serviceprovider interaction. This cloud model is composed of five essential characteristics, threeservice models, and four deployment models:

2.1.1 Characteristics of Cloud Computing

• On-demand self-service. A consumer can unilaterally provision computing capabilities,such as server time and network storage, as needed automatically without requiringhuman interaction with each service provider.

• Broad network access. Capabilities are available over the network and accessed throughstandard mechanisms that promote use by heterogeneous thin or thick client platforms(e.g., mobile phones, tablets, laptops, and workstations).

• Resource pooling. The providers computing resources are pooled to serve multipleconsumers using a multi-tenant model, with different physical and virtual resourcesdynamically assigned and reassigned according to consumer demand.

• Rapid elasticity. Capabilities can be elastically provisioned and released, in somecases automatically, to scale rapidly outward and inward based on demand. To the

3

4 Chapter 2. Introduction to Cloud Computing and Virtualized Data Center

consumer, the capabilities available for provisioning often appear to be unlimited andcan be appropriated in any quantity at any time.

• Measured service. Cloud systems automatically control and optimize resource use byleveraging a metering capability at some level of abstraction appropriate to the typeof service (e.g., storage, processing, bandwidth, and active user accounts). Resourceusage can be monitored, controlled, and reported, providing transparency for both theprovider and consumer of the utilized service

2.1.2 Service Models

The cloud computing stack is typically depicted as consisting of three layers, namely Soft-ware as a Service, Platform as a Service and Infrastructure as a Service.

• Software as a Service. The consumer is able to use the provider’s applications runningon a cloud infrastructure or platform. The consumer has no control over the infras-tructure the application runs, but just uses its functionality, usually through a thinclient. This type of Cloud is also described as Service or Application Cloud. Examplesinclude Salesforce Customer Relationships Management (CRM) system, SAP Businessby Design and Google Docs.

• Platform as a Service. Resources are provided via a platform upon which servicesand applications developed or owned by the consumer are deployed. The providerexposes a specific API that the consumer can use to develop or extend an application.The consumer has no control over the infrastructure upon which the application isdeployed. Examples include Google App Engine and Windows Azure.

• Infrastructure as a Service. This type of Cloud provides virtualized resources to theuser. The resources can be processing, storage or communication networks which theconsumer can use to deploy arbitrary software. The consumer does not have controlover the physical resources, but has control over operating systems, storage, deployedapplications, and possibly limited control of select networking components (e.g., hostfirewalls).

2.1.3 Deployment Models

The deployment types of clouds are:

• Private cloud. The cloud infrastructure is operated solely for an organization. It maybe managed by the organization or a third party and may exist on premise or offpremise.

• Community cloud. The cloud infrastructure is shared by several organizations andsupports a specific community that has shared concerns (e.g. security requirements,policy and compliance considerations). It may be managed by the organizations or athird party and may exist on premise or off premise.

• Public cloud. The cloud infrastructure is made available to the general public or alarge industry group and is owned by an organization selling cloud services.

• Hybrid cloud. The cloud infrastructure is a composition of two or more clouds (pri-vate, community, or public) that remain unique entities but are bound together bystandardized or proprietary technology that enables data and application portability(e.g., cloud bursting for load-balancing between clouds).

2.2. Virtualization 5

Figure 2.1: Virtual Machine Abstraction.

2.2 Virtualization

Virtualization is often used in cloud computing platforms for its several advantages in ef-ficiently managing resources. The term virtualization broadly describes the separation ofa resource or request for a service from the underlying physical delivery of that service.Virtualization provides a layer of abstraction between physical hardware resources and theapplications running on it as shown in Figure 2.1. It allows sharing of resources amongdifferent applications while at the same time providing isolated environment called VirtualMachine(VM) for each application. VM is a virtual environment created by VMM(describedbelow), which simulates all the hardware resources needed by an operating system. The OSrunning in such environment is called a guest OS. Guest OS has a virtual view of the un-derlying hardware. Virtual Machine Monitor (VMM/hypervisor) is the interface betweenthe guest OS and the underlying hardware. All the administrative tasks like adding a newguest OS, allocation of resources to each of guest OS is done through VMM. Some popularexamples of VMM are VMware [38], Xen [39], KVM [40] . In our study, we use KVM. Thenative OS running on the given hardware is called the Host OS. The VMM is installed onHost OS. This OS has all the privileges on the given hardware.

In general, cloud computing can benefit from virtualization in the following aspects:

• Multiple services can share the same platform. Virtualization can be used to maintainmultiple software environments on the same host.

• Management flexibility. Virtual machines can be started, shutdown and migrateddynamically. It is therefore easier to manage virtual machines in a computing center.

• Isolation of services. Services running in a shared hardware resources are isolatedfrom each other and run in their own sandbox. This isolation of services also facilitatesystem security.

6 Chapter 2. Introduction to Cloud Computing and Virtualized Data Center

2.3 Data Center

Data centers are buildings where multiple servers and communication gear are colocatedbecause of their common environmental requirements and physical security needs, and forease of maintenance [41]. Traditional datacenters, typically host a large number of relativelysmall- or medium-sized applications, each running on a dedicated hardware infrastructurethat is de-coupled and protected from other systems in the same facility. Modern datacenter, on the other hand, use virtualization to provide application portability and facil-itate dynamic sharing of physical resources. The hardware for a data center may consistof thousands of individual computing nodes with their corresponding networking and stor-age subsystems, backup power supplies, Power Distribution Unit(PDU) and conditioningequipment, and extensive cooling systems.

Chapter 3

Related Work

There are previous works in the area of elasticity on Cloud infrastructures. Ali-Eldin et al.[4] considered an adaptive horizontal elasticity controller for Cloud infrastructures, where aCloud service is modeled by queue theory and service load is estimated to build proactivecontrollers. Lim et al. [5] design an integral elasticity controller with proportional threshold-ing for elastic storage systems. Urgaonkar et al. [6] propose a hybrid control mechanism thatincorporates both a proactive controller and a reactive controller to determine when to provi-sion resources. These works focus on horizontal elasticities and address resource provisioningby allocating and releasing virtual server instances of predetermined sizes. Concerning ver-tical elasticity, the work by Kalyvianaki et al. [7] integrates the Kalman filter into feedbackcontrollers to dynamically allocate CPU resources to VMs. The CPU utilization is trackedand the allocations are updated accordingly. Dawoud et al. [8] compares the benefits anddrawbacks of vertical elasticity with respect to horizontal elasticity and focus on verticalelasticity. They propose an Elastic VM architecture which scales number of cores and CPUcapacity. The aforementioned works do not address the combined vertical and horizontalelasticity. In addition, power consumption is not taken into account in these works. Thiswork on the other hand combines vertical and horizontal elasticity to dynamically adapt theunderlying change in demand by minimizing power consumption.

Significant research efforts have been expended on applying dynamic voltage scaling(DVS) to computing systems in order to save power consumption while meeting performanceconstraints. [10], [15], [20], [23], [24]. For instance, Elnozahy et al. [15] studied voltage scal-ing and request batching for reducing server energy consumption in web servers. Elnozahy[20] investigated using cluster-level reconfiguration in conjunction with DVFS Laszewski etal. [10] propose a scheduling algorithm to allocate virtual machines in a DVFS-enabledcluster by dynamically scaling the supplied voltages.

Chase et al. [16] considered how to reduce power consumption in data centers by turningservers on and off based on resource demand. Niyato et al. [17] proposed an optimal powermanagement based on a constraint Markov model to adjust the number of active servers formaximum energy savings. While a considerable amount of power can be saved by turningservers on and off, the main goal of this work is to achieve autonomous resource configurationby online adjusting service’s capacity according to power usage and users’ SLA requirements.

There exists some studies on autonomous resource provisioning in the cloud. Rao et al.[26] applied reinforcement learning to adaptively adjust resource allocations based on SLAconformance feedback. Padala et al. [27] present a self-adaptive automated control policyin virtualized computing system, in which a auto-regression moving average model utilize

7

8 Chapter 3. Related Work

previous configuration and to determine to select future configurations in terms of optimizingperformance. However, they do not consider power usage in decision making. Stoess et al.[28] propose a power management framework in multi-layer virtualized system architecturebut it ignores the system-wide performance. Jung et al. [29] propose a framework calledMistral to dynamically coordinate power efficiency and performance, which is based on aheuristic method. Beloglazov et al. [34] proposed energy-aware allocation heuristics forthe provision of data center resources to client applications in a way that improves energyefficiency of the data center, while delivering the negotiated Quality of Service. Generally,heuristic methods are efficient but usually tend to trap the configuration into a local optimalsolutions. Zhang et al. [30] proposed an adaptive power management framework usingreinforcement learning to achieve autonomic resource allocation and optimal power efficiencymeanwhile satisfying SLA. Reinforcement learning techniques generally do not require amodel but they need some time to converge to the optimal(near-optimal) decisions [31].

Control-theoretic techniques have recently shown a lot of promise on power managementdue to their better control performance such as accuracy, stability, short settling time, andsmall overshoot and a better quantitative control analysis when the system is suffering un-predictable workload variations. Works in [32], [33], [52], [53], [54], [55] have successfullyapplied control theory to power management on servers. Padala et al. [32] applied a propor-tional controller to dynamically adjust CPU shares to VM-based multi-tier web applications.Gao et al. [33] designed an integrated multi-input, multi-output (MIMO) performance con-troller and an energy optimizer management solution which takes advantages of both VMresizing and server consolidation to achieve energy efficiency and QoS in virtualized datacenters. Although all these works use control theory to manage power consumption, theydo not consider both ’hard’ and ’soft’ scaling in the power savings decision.

Chapter 4

The System Model

In control theory, an object to be controlled is typically represented as an input-outputsystem, where the inputs are the control knobs and the outputs are the metrics beingcontrolled. Control theory mandates that a system model that characterizes the relationshipbetween the inputs and the outputs be specified and analyzed before a controller is designed.For traditional control systems, such models are often based on first-principles, for whichlittle empirical validation is done. For computer systems, although there is queueing theorythat allows for analysis of aggregate statistical measures of quantities such as utilizationand latency, it may not be fine-grained enough for run-time control over short time scales,and its assumption about the arrival process may not be met by certain applications andsystems [32]. Therefore, most prior work on applying control theory to computer systemsemploys an empirical and ”black box” approach to system modeling by varying the inputs inthe operating region and observing the corresponding outputs. We adopted this approach inour work. A feedback control loop requires an actuator to implement the changes indicatedby the control knobs and a sensor to measure the value of the output variables. In the nextsection, we provide description of dimensions or knobs that can be used to manage serversin an energy efficient manner. Then we describe a set of system modeling experiments weperformed to analyze the impact of each management dimension on performance and powerconsumption and design a system model for the dynamic behaviour of the service undervarious configurations.

4.1 Power Management Dimensions

Two basic power management mechanisms are Dynamic Voltage and Frequency Scaling(DVFS) and Vary-On/Vary-Off (VOVO). DVFS changes the operating frequency and volt-age of a given computing resource. Vary-On/Vary-Off (VOVO) turns servers on and off toadjust the number of active servers according to the workload. Studies in [15], [16], [17],[18], [19], [20], [21] show that these techniques can save power consumption considerably.However, there is limited work on the use of virtualization capabilities to dynamically scaleresources of a service to meet the performance requirement while minimizing power us-age. In this work we combine ’hard’ and ’soft’ scaling management techniques. The ’hard’scaling leverages underlying hardware support for changing the configuration of physicalresources. The ’soft’ scaling combines horizontal and vertical scaling of virtual resources.The techniques are described as follows:

9

10 Chapter 4. The System Model

Figure 4.1: Relative power impact of different resources on dynamic power consumption.

• ’Soft’ horizontal scaling. This technique exploits a hypervisor’s ability to change thenumber of VMs for the running service. System-wide power consumption in twoaspects. The first one is the static power which reflect the idle power consumptionof a consumption of a system. The second is the dynamic power consumption causedby running additional intensive workloads.The energy impact of theVM can thus beconsidered in terms of the dynamic power used. VMs use dynamic power consumingdevices like CPU, memory and disks. The power consumption of a VM greatly dependson the extent to which these resources are utilized and differs from one application toanother.

• ’Soft’ vertical scaling. This technique exploits a hypervisor’s ability to scale the al-location of physical resources to guest VM. We select core or CPU as a resource formanagement as it dominates the overall power usage profile of a machine. Figure4.1 provides some insight into the relative non-idle dynamic power usage by a datacenter server categorized by main component group [25]. CPU intensive applications,in general, benefit more from this type of scaling. Some works in [1], [32] show savingon power consumption by modifying the hypervisor’s scheduling attributes to changeguest’s maximum time slice on a core. In contrast,in this work we address ’soft’ ver-tical scaling by changing the number of cores of a VM and without modifying thehypervisor scheduler.

• ’Hard’ power scaling. Dynamic Voltage and Frequency Scaling (DVFS) has beenapplied within clusters and supercomputers to reduce power consumption and achievehigh reliability and availability [11], [12] [13], [14]. By using DVFS one can change theoperating frequency and voltage, which results in power consumption change of a givencomputing resource considerably. This technique was originally used in portable andlaptop systems to conserve battery power, and has since migrated to the latest serverchipsets. Current technologies exist within the CPU market such as Intel’s SpeedStep[56] and AMD’s PowerNow [57] technologies. These dynamically raise and lower bothfrequency and CPU voltage using ACPI P-states [27]. Since processors are often thedominant consumers of power in servers, we change the frequencies of the cores basedon demand. We also consider turning on/off cores in the power scaling decision.

4.2. System Modeling Experimentation 11

Figure 4.2: An input-output model for a system.

4.2 System Modeling Experimentation

Once the above management dimensions are identified, a set of experimentations have beenperformed to understand the relationship between them and their impact on application’sperformance and power usage. Figure 4.2 illustrates an input-output representation of thesystem we are controlling. The inputs to the system are server CPU frequency, cores andVMs. The The outputs include power usage and performance of the system. To understandhow the outputs in Figure 4.2 change as we vary the inputs, we performed a set of systemmodeling experiments to determine a model for the dynamic behavior of the system underdifferent configurations.

4.2.1 Experimental Setup

The experiments are performed on a ProLiant DL165 G7 machine, with AMD Opteron(TM)Processor containing 32 CPUs and 56GB of physical memory. The server runs Ubuntu 4.6.3with Linux kernel 3.2.0. The frequency of CPU can be scaled to 1.4GHz, 1.5GHz,1.7GHz,1.9GHz and 2.1GHz. In order to adjust the CPU frequency, we utilize the cpufrequtils [2]package. The package contains two utilities for inspecting and setting the CPU frequencythrough CPUFreq kernel interfaces provided by the kernel on hardware. We changed thedefault ondemand CPU frequency scaling policy to usersspace policy to have a control onfrequency changes. A special power meter attached to the Power Distribution Unit (PDU) isused for measuring power usages of servers. We used Simple Network Management Protocol(SNMP) for extracting power consumption of the servers.

The virtual machines run on Debian 2.6.32 with kernel version 2.6.32. Version of kvmhypervisor is KVM QEMU 1.0.0. A maximum of 10 VM instances are made to run at thesame time. The maximum of virtual core allocation for each VM is 32 and and exact numberof cores depend on specific experiments. The amount of memory allocated to a vm is setto 5GB memory size. A base image is created and used as template disk image for creatingother clone VM images. Guest VMs are created by modifying the UUID, name, number ofCPUs and IP address of a template XML definition.

We chose the CPU intensive benchmark from the SysBench [11] package which consistsof configurable number of events that compute prime numbers from 1 to N (user-specified)for different number of requests. The benchmark reports the average event handling timewhich we used it to obtain the throughput metric.

Scaling cores of a VM dynamically is not currently supported by KVM supervisor. Weaddress this limitation by stopping and starting VMs with different number of cores. Inaddition, there is no support to change the frequency of virtual cores from the guest ma-chine and changing it from the underlying hardware requires virtual cores to be pinned to

12 Chapter 4. The System Model

physical cores. However, locking VMs to particular physical cores negates the notion of vir-tualized hardware resources that is running guests on any available physical processing cores.Moreover, pinning may threaten desired performance isolation properties, as evidenced onthermally sensitive multicore chips where a VM increasing its frequency may heat up itscore in a manner that compromises another core’s ability to run at a speed sufficient formeetings its VMs desired QoS [1]. To address this issue, we change the frequency of all thecores of a server by running one service per node. This simplification avoids the impact ofphysical cores being shared by multiple services running on a server. Besides, since it is notpossible to connect physical power meters to VMs, a more accurate power measurement canbe obtained by running a service on a node .

4.2.2 Experimental Results

For this experiment a single server is used to host CPU intensive application of SysBench.At each setting, the application was loaded with 144000 requests while the average averageevent handling time for each VM was measured. We used this metric to calculate the averagethroughput. The experiments were repeated at 4 different times to get more accurate results.The number of threads in the client were changed depending on the number of cores a VMused.

We run different experiments by combining VMs that range from 1 to 10, cores thatrange from 1 to 32 and server CPU frequencies running at 1.4GHz, 1.5GHz,1.7GHz, 1.9GHzand 2.1GHz. We got a total of 6585 results for all possible combinations of configurations.Figures 4.3(a)-(d) plot some results that show the impact of each management dimensionon performance and power usage. Figures 4.3 (a),(b),(c),(d) show the throughput andpower consumption as a function of core, VM, CPU frequency and turning on/off coresrespectively. Increasing cores, VMs and frequency results in the increase of performanceand power consumption as shown in Figures 4.3 (a)-(c). The figures reflect the importanceof these three dimensions in the power management solutions. On the contrary, Figure4.3(d) shows small difference in power consumption (less than 1 watt) between a VM using1 core and 31 of the remaining cores being idle and a VM using 1 core when the 31 idlecores are turned off. This test confirms that the power-save mode of modern CPUs is good.

Understanding the underlying architecture of the system can also help tune the perfor-mance of a service. In particular, for systems with a Non-Uniform Memory Access (NUMA)architecture VCPUs can be pinned to physical processing cores on the same NUMA node.This avoids cross-node memory transports by allocating memory to the VM that is alwayslocal to the NUMA node it is running on. Figure 4.3 shows the improvement in perfor-mance in the tuned system as compared to one which is not. Locking a VM to a particularNUMA node offers the benefit if that node has sufficient free memory for that VM. Besides,tuning requires pinning and proper understanding of the underlying system architecture.To keep our work as general as possible, We have thus considered the default non-pinningvirtualization policy to perform the experimentation.

The detailed information about power consumption and performance of a service forthe different combinations of cores, VMs and CPU frequencies yields a system model thatcontains the overall power-performance information for determining future adjustment insystem configuration in terms of balancing power saving and performance.

4.2. System Modeling Experimentation 13

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

22000

1 2 4 8 16 32 180

190

200

210

220

230

240

250

260

Thro

ughput (R

equests

/sec)

Watt

Number of cores

ThroughputWatt

(a) Impact on power and performance by varyingthe number of cores of a VM running at 2.1 CPUfrequency.

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

1 2 4 8 10 175

180

185

190

195

200

205

Thro

ughput (R

equests

/sec)

Watt

Number of vms

ThroughputWatt

(b) Impact on power and performance by changingthe number of VMs, each running at 1.4 CPU fre-quency and with 2 cores.

850

900

950

1000

1050

1100

1150

1200

1250

1300

1350

1.4 1.5 1.7 1.9 2.1 177

178

179

180

181

182

183

184

185

186

187

Thro

ughput (R

equests

/sec)

Watt

CPU frequency

ThroughputWatt

(c) Impact on power and performance by runninga 2-core VM at different CPU frequency.

(d) Impact on power and performance by turningcores on and off.

Figure 4.3: The impact of each management dimension on power usage and performance.

(a) Impact of architecture on per-formance.

(b) Impact of architecture onpower usage.

Figure 4.4: Effect of tuned Vs non-tuned system on performance and power usage.

14 Chapter 4. The System Model

Chapter 5

System Architecture

In this section, we provide a high-level description of the system architecture including thesystem model and the optimal controller.

The system in general can be viewed as a feedforward and feedback control system. In afeedforward system the control action does not depend on the system output and no systemmonitoring and feedback are required. Conversely, in a feedback system the action dependson the system output and on monitoring some system parameters [3]. The advantage offeedforward is that it can compensate for load changes before they are detected by thefeedback control system as an error. Feedback systems can only react to correct for a loadchanges after an error is detected. We combined these approaches to improve the response ofthe system under drastic variations in workloads in one control interval by using feedfowardinformation and increase the accuracy of the system by relying on feedback information.Figure 5.1 shows the overall system architecture. The high level function of each componentcan be described as follows:

• The system model models the relationship between performance and power usage ofa service. The various combination of cores, VMs and CPU frequencies. It containsall possible configurations of a service with performance and power consumptions asdescribed in Chapter 4.

• The system monitor periodically polls the measured performance of an application aswell as measures the power usage. We refer to this period as the control interval. Themonitor can collect a number of performance data, such as the average response timeand throughput. We currently measure performance in terms of the throughput butit can be extended to control other performance metrics too. The performance valuesof the service is then sent to the optimal controller as feedback.

• The workload monitor gathers performance information of workloads. The frequencyof workload monitoring depends on the characteristic of the workload. If dramaticload changes are expected within one control interval, the workload monitor can bemade to monitor the load more frequently than system monitoring. The performancedemand is sent to the optimal controller as target performance for making decisionsof auto-configuration.

• The optimal controller compares the measured performance with the application per-formance target. Based on the discrepancy, it determines the optimal configuration forthe next control interval to minimize power consumption by achieving the SLA. The

15

16 Chapter 5. System Architecture

Figure 5.1: Overall system architecture.

new configuration may result in CPU frequency change, change in number of coresor/and change in the number of VMs.

Chapter 6

Design of Optimal Controller

This section details the design of optimal controller. We first introduce some notations. LetN denote the total number of cores on the server, xi denote the number of VMs with 2i cores,i = 0,1,..., log2 (N). Let f denote the CPU frequency of the server. Let t denote the controltime. Let W(t) denote power consumption at time t. Let P(t) denote performance at time t.Assume that at each point in time, we are given a minimum performance target Pref (t). Thetask of minimizing the power consumption at each point of time while attaining at least thepre-specified performance target can be formulated as the constrained optimization problem

min(x,f)∈A

W (t)(x, f) subject to P (t) ≥ Pref (t) (6.1)

where the set of possible server configurations is given by

A = {(x, f) | x ∈ Axandf ∈ Af} (6.2)

Here Af is the set of clock frequencies available on the server and the set of feasible VMcombinations Ax is given by

Ax = {x | xi ∈ N, i = 0, 1, ..., log2(N)and

log2(N)∑n=0

2ixi ≤ N}. (6.3)

Ax represent all possible combinations of VMs with different number of cores such thatthe total number of cores is less than or equal to the number of cores in the server. Theoptimization formulation in Equation(6.1) always achieves the minimum performance target.But this formulation is inflexible. In some cases, it might be acceptable to miss performancetarget by some fraction and assign a penalty for the deviation of the application’s measuredperformance from its target. In this way, performance smaller than the target is feasiblebut become increasingly expensive. To achieve this we refine Equation (6.1) into:

min(x,f)∈A

aW (t)(x, f) + bmax{0, Pref (t)− P (t)(x, f)}. (6.4)

We define Pf = Pref (t) − P (t)(x, f). It is easy to see that Pf = 0 when the application ismeeting its performance target. Otherwise, Pf serves as a penalty for the deviation of theSLA. We refer to Pf as the performance cost. Parameters a and b represent priorities givento power minimization and meeting performance target. In cases where power minimization

17

18 Chapter 6. Design of Optimal Controller

can not be achieved without violating some SLA, a and b parameters control the trade-off between power efficiency and performance. The control formulation in Equation (6.4)minimizes power consumption by responding to performance changes, but it is prone tooscillations if performance target changes frequently.

An additional goal can be set to accomplish the minimization in a stable manner, withoutcausing large oscillations in the resource allocations and a higher delay in reconfiguration.Assume that we are at time t and that the machine status is x running at frequency f andwith c total number of cores and that we have an updated performance P (t) and powerconsumption W (t).

We are interested in finding the best reconfiguration so that given a performance goalPref (t) is met at time t = t + 4t. Our control strategy for the server is to set the newconfiguration (x(t),f(t)) to the configuration that minimizes problem

min(x,f)∈A

aW (t)(x, f) + bmax{0, Pref (t)− P (t)(x, f)}+ sR. (6.5)

The function R = r(∆f,∆c,∆x), referred to as the control cost, is the control penalty, whichpenalizes big changes in the resource allocation in a single interval and contributes towardshigh system stability. The first term in the control cost function ∆f represents the timeit takes to change CPU frequency. The next term ∆c represents the total number of corechanges, denoted as |c− c|. The last term ∆x denotes the number of booted VMs (xi− xi).The change may lead to an increase in the number of active VMs. Here VM boot time mayinfluence performance greatly. The delay in starting machines is calculated as the averagetime it takes to start new VMs as shown in Equation (6.6)

1

n

n∑i=0

xibt

ip(6.6)

where xibt represent the time it takes to boot xi machine. The term ip denotes the intervalperiod and n represent the total number of VM types booted. It is assumed that VMs ofsame size boot in parallel. Incorporating VM booting time into stability decision can alsohelp to differentiate configurations having the same total number of cores but with differentVM type configurations. For instance, the ability to differentiate between starting a 2-coreVM and starting two 1-core VMs.

As it is shown in Equation(6.5), the parameter s controls the stability of the system. Itprovides an intuitive way to control the trade-off between the controller’s stability and itsability to respond to changes in the workloads and performance, and hence is referred toas the stability factor. For a larger s value, the controller stabilize the system by reactingslowly to the performance changes. As the value of s decreases the controller responds morerapidly to changes in performance resulting in a higher oscillation in the new configuration.

We translate the power consumption, performance and configuration penalties in theoptimization formulation into costs to make the overall minimization computation possible.Power usage in KWatt hour is converted to dollar. Performance penalty is mapped to mon-etary penalties using SLA definition. The exact relation between the metric of performanceconformance and the monetary penalties due to violation is an important parameter of theSLA. There are multiple ways in relating the two and thus a variety of penalty functionscan be chosen [9]]. In this work, we have used step-wise penalty function in relating thetwo. The Stepwise penalty function enables categorization of a range of service parametervalues into groups or classes having the same utility value. Figure 6.1 shows the stepwisefunction (for a case of n = 5 steps).The function is defined via n pairs (Pi, Xi), where Pi is

19

Figure 6.1: Stepwise penality function.

the penalty charged when the conforming performance percentage is Xi. The configurationpenalty in terms of number of core changes and average booting time are also convertedinto costs using linear cost function.

20 Chapter 6. Design of Optimal Controller

Chapter 7

Experimental Evaluation &Analysis

In this chapter, we present the implementation tails of the controller described in Chapter6. Then, experimental results that validate the effectiveness of our controller design in avariety of scenarios are presented.

7.1 Experiment Design

In order to evaluate the optimal controller, we define a number of parameters, including theworkload to use, control interval and cost definitions.

To implement the controller we used simulation using traces from the FIFA 1998 worldcup [51] as input to the system. We used the same experimental testbed as used in Section4.2.1. We calculated the aggregate number of requests per 15 seconds from these traces.The Java programming language is used for implementing the proposed system. We adoptedan optimization control loop of t=15 seconds interval. This value is found to be applicableto our case, but it typically depends on a system and application characteristics. Thefrequency of workload monitoring depends on the characteristic of the workload. Currently,we have used the control interval as a default workload monitoring period. But, in general,it can be made to monitor the load more frequently to handle dramatic load changes thatoccur within a single control interval. We instrumented the VM boot process by using thebootchart tool [49] to measure the time taken during boot and visualize the results. Table7.1 shows the average time it takes to boot VMs with different number of cores and CPUfrequencies. Power consumption is set to cost 1.52$ per KWatt hour over control intervalt. The cost for SLA penalty used in the system is shown in Table 7.2. Since the timeto make CPU frequency changes is small (8 milliseconds) we omit the cost from furtherconsideration. Costs of 0.01$ and 0.01*averagebootingtime$ are used per core change andVMs booting time respectively. Currently there are 6585 possible configurations and goingthrough each of them to find the most optimal solution takes around 26 milliseconds usingexhaustive search. With more configurations, this method tends to be slower and otherefficient optimization techniques such as branch and bound need to be adapted.

21

22 Chapter 7. Experimental Evaluation & Analysis

Table 7.1: VM booting time in secondsfor different number of cores and CPU frequencies.

Core 1.4GHz 1.5GHz 1.7GHz 1.9GHz 2.1GHz1 10 9 8 7 72 6 6 5 5 54 6 5 5 4 48 5 5 4 4 416 5 5 5 4 432 6 6 5 5 5

Table 7.2: Cost of SLA violation.

Perfromance con-formance( %)

cost($)

[>=100] 0[99:100] 0.1[98:99] 0.4[97:98] 0.8[96:97] 1.6[95:96] 3.2[<95] 100000

7.2 Evaluation Results

Figure 7.1 shows the results of different power saving policies. Four different policies arecompared: CPU frequency policy, VM policy, Core policy and our approach that combinesCPU frequency, VM and core policies.

• VM policy adds/removes VMs of same sizes. Allocating fixed-size VMs may give moreperformance than what it is required. This may result in higher power consumption.

• CPU frequency policy changes the CPU frequency of the server. This approach mayalso result in higher performance and power usage because of a coarse-grained fre-quency changes.

• Core policy changes the number of cores of a VM. This policy provides more powersavings than the first two policies as it provides a fine-grained control in load changes.Whether this savings holds for other system depends on the number of cores availablein system and the extent to which a VM is allowed to scale vertically.

• Combined policy combines the VM, core and CPU frequency policies. This approachesshows a greater power saving than when each of the policies are applied in isolation.This approach provides a very fine-grained scaling option and selects a configurationthat meets the performance target while minimizing power usage. It achieves 9% powersaving as shown in Figure 6.1(b), while meeting application performance target, shownin Figure 6.1(a).

7.2. Evaluation Results 23

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

0 500 1000 1500 2000 2500 3000

Thr

ough

put (

Req

uest

s/se

c)

Time (t=15 sec)

CPU FrequencyVM

CoreCombined

Target

(a) performance.

185

190

195

200

205

210

215

220

225

0 500 1000 1500 2000 2500 3000

Wat

t

Time (t=15 sec)

FrequencyVM

CoreCombined

(b) power usage.

Figure 7.1: Comparison of power saving approaches.

12000

13000

14000

15000

16000

17000

18000

19000

20000

21000

0 100 200 300 400 500 600 700 800

Thro

ughput

time (t=15 sec)

Targetα-0.5

α-0.933

(a) Impact of performance-power trade-off onthroughput.

210

215

220

225

230

235

240

245

250

255

0 100 200 300 400 500 600 700 800

Watt

time (t=15 sec)

α-0.5 α-0.933

(b) Impact of performance-power trade-off on watt.

Figure 7.2: Impact of performance-power trade-off.

The extent to which the controller can save power depends on how much SLAs areviolated. By varying the values of constants a and b in Equation (6.4) we can get differentpower consumption and performance values as shown in Figure 7.2. In this figure, the trade-off between a and b are represented by α and (1−α) respectively. Figure 7.2(a) shows with ahigher α (α= 0.933) value the controller reacts aggressively to minimize power consumption.With a lower α(α=0.5) value, the controller responds more quickly to performance deviationas shown in Figure 7.2(b) .

Minimizing power consumption may causing large oscillations in the system as evidencedin Figure 7.2. To deal with larger changes the controller can be tuned to accomplish theoptimization in a stable manner. Figure 7.3 illustrate the tradeoff between controller sta-bility and responsiveness by adjusting the stability factor s in Equation (6.5). Figure 7.3(a)shows the achieved throughput for s values of 0 and 10. The result for s = 0 the controllerreacts to the workload change more quickly and aggressively, resulting in large oscillationsin the number of cores allocated and thus also in performance. For s=10, the controllerdoes not adjust resource allocations fast enough to always track the performance targets

24 Chapter 7. Experimental Evaluation & Analysis

0

5000

10000

15000

20000

25000

0 1000 2000 3000 4000 5000 6000 7000

Thro

ughput

Time (t=15 sec)

Targetconf-0

conf-10

(a) Impact of configuration cost on performance.

0

5

10

15

20

25

30

35

0 1000 2000 3000 4000 5000 6000 7000

Core

Time (t=15 sec)

conf-0conf-10

(b) Impact of configuration cost on number of cores.

Figure 7.3: Impact of configuration cost on performance and cores.

Figure 7.4: Impact of VM booting time on VM selection.

but achieves a much more stable operation. Figure 7.3(b) shows the corresponding changeson the number of cores used for the same run. For s = 0 the controller makes 137 additionsand 133 removal of cores for 145 control intervals. For s=10 only 31 additions and 3 removalof cores for the same interval periods.

In addition to considering number of core changes, the controller takes VM bootingtime into account when making a control decision. Considering delay in starting new VMscan help differentiate configurations having the same total number of cores but differentVM types. Figure 6.4 shows the difference between VM Booting-aware(B) controller andNon Booting-aware (NB) controller. B controller selects a configuration that starts 2-coreVM which takes only 6 seconds where as NB controller starts two 1-core VMs that take 9seconds.

Chapter 8

Conclusion and PossibleImprovements

To conclude, in this work we present a system that integrate ’hard’ and ’soft’ scaling man-agement solution to achieve high power efficiency. The ’soft’ scaling combines horizontal andvertical scaling to provide richer choice for applications with different requirements. Thesystem consists of a system model that captures the relationship between performance andpower consumption and a controller that combines feedforward and feedback informationto determine a configuration that minimize power consumption. Combining these controlinformation enable the controller to improve the response of the system under drastic work-load variations and increase the accuracy of the system. Our experimental results showedthat the controller saves more power while still achieving the SLA when a combinationsof CPU frequency, core and VM approaches are used. The controller can also be made tomake a trade-off between power minimization and achieving the SLA . Finally, we showedthat the controller can be configured to accomplish the minimization in a stable manner,without causing large oscillations in the resource allocations.

There are a number of limitations with our current approach. Changing CPU frequencyof virtual cores of the running VM is not possible in the KVM hypervisor. KVM is onlycapable of controlling ”shares” or changing the number of cores. One method that can beused to deal with this limitation is to pin virtual cores to physical cores and change theCPU frequency of physical cores. Pinning may decrease performance of the overall systemwhen physical cores are idle because they are pinned to inactive VMs. In addition, piningvirtual cores that run different tasks to the same physical cores may result in the loss ofperformance. This is due to the lost benefit of sharing the same cache when one of thevirtual cores finds no data in the cache as other virtual cores may have filled the cache.To make the work as general as possible, we use the default virtualization technique to runvirtual cores to on any of the available physical cores. To make CPU frequency changesstill possible, we used an approach that changes the CPU frequencies of all the cores ofthe server and run one service per node to avoid the impact of physical cores being sharedby multiple services. Since VMs are started with fixed number of virtual cores and idlephysical cores consume almost no power, the impact of changing the frequencies of all coreson power consumption is minimal compared to only changing the CPU frequencies of theaffected physical cores.

One possible extension to the work to address the limitation of hardware power scaling isto map the required CPU frequency change to change that control ”shares” of CPU [32]. This

25

26 Chapter 8. Conclusion and Possible Improvements

involves emulating the hardware scaling action to change CPU frequency and mapping to asoftware scaling action to get the the VM’s desired performance. Changing ”shares” of CPUentails modifying the hypervisor’s scheduling attributes to change the guest’s maximum timeslice. For instance, if a VM has requested changes to a lower frequency then the hypervisorscheduler may reduce the guest’s maximum core time slice.

The work can also be extended to include other resources types like memory and disk .By analyzing the behavior of these resources and their impact on performance and powerconsumption a higher power savings can be achieved.Another improvement that can be madeis to make the system model learn automatically and make updates online to the systemmodel based on real-time measurements of performance and power usage under differentoperating regimes and workload conditions.

Acknowledgements

I would like to express my gratitude to my supervisor Johan Tordsson for the useful com-ments, remarks and engagement through the learning process of this master thesis. I amalso grateful to Erik Elmroth and Eddie Wadbro for providing useful suggestions about thisthesis. Finally, I would like to thank my loved ones, who have supported me throughoutentire process.

27

28 Chapter 8. Conclusion and Possible Improvements

References

[1] Nathuji, Ripal, and Karsten Schwan. VirtualPower: coordinated power management invirtualized enterprise systems. ACM SIGOPS Operating Systems Review 41.6 (2007):265–278.

[2] ”Cpufrequtils.” [Online]. Available:http://packages.debian.org/sid/cpufrequtils

[3] Astrom, Karl Johan, and Richard M. Murray. Feedback systems: an introduction forscientists and engineers. Princeton university press, 2010.

[4] Ali-Eldin, Ahmed, Johan Tordsson, and Erik Elmroth. ”An adaptive hybrid elasticitycontroller for cloud infrastructures.” Network Operations and Management Symposium(NOMS), 2012 IEEE. IEEE, 2012.

[5] Lim, Harold C., Shivnath Babu, and Jeffrey S. Chase. ”Automated control for elas-tic storage.” Proceedings of the 7th international conference on Autonomic computing.ACM, 2010.

[6] Urgaonkar, Bhuvan, et al. ”Agile dynamic provisioning of multi-tier internet applica-tions.” ACM Transactions on Autonomous and Adaptive Systems (TAAS) 3.1 (2008):1.

[7] Kalyvianaki, Evangelia, Themistoklis Charalambous, and Steven Hand. ”Self-adaptiveand self-configured cpu resource provisioning for virtualized servers using kalman filters.”Proceedings of the 6th international conference on Autonomic computing. ACM, 2009.

[8] Dawoud, Wesam, Ibrahim Takouna, and Christoph Meinel. ”Elastic virtual machine forfine-grained cloud resource provisioning.” Global Trends in Computing and Communi-cation Systems. Springer Berlin Heidelberg, 2012. 11–25.

[9] Y. C. Lee, C. Wang, A. Y. Zomaya, and B. B. Zhou, OProfit-driven service requestscheduling in clouds,O in Cluster, Cloud and Grid Computing (CCGrid), 2010 10thIEEE/ACM International Conference on, May 2010, pp. 15–24.

[10] von Laszewski, Gregor, et al. ”Power-aware scheduling of virtual machines in dvfs-enabled clusters.” Cluster Computing and Workshops, 2009. CLUSTER’09. IEEE Inter-national Conference on. IEEE, 2009.

[11] Gorton, Ian, et al. ”Data-intensive computing in the 21st century.” Computer 41.4(2008): 30–32.

[12] Feng, W-C., Avery Ching, and C-H. Hsu. ”Green supercomputing in a desktop box.”Parallel and Distributed Processing Symposium, 2007. IPDPS 2007. IEEE International.IEEE, 2007.

29

30 REFERENCES

[13] Feng, Wu-chun, and Kirk W. Cameron. ”The green500 list: Encouraging sustainablesupercomputing.” Computer 40.12 (2007): 50-55.

[14] Feng, Wu-chun, Xizhou Feng, and Rong Ce. ”Green supercomputing comes of age.” ITprofessional 10.1 (2008): 17–23.

[15] Elnozahy, Mootaz, Michael Kistler, and Ramakrishnan Rajamony. ”Energy conserva-tion policies for web servers.” Proceedings of the 4th conference on USENIX Symposiumon Internet Technologies and Systems-Volume 4. USENIX Association, 2003.

[16] Chase, Jeffrey S., et al. ”Managing energy and server resources in hosting centers.”ACM SIGOPS Operating Systems Review. Vol. 35. No. 5. ACM, 2001.

[17] Niyato, Dusit, Sivadon Chaisiri, and Lee Bu Sung. ”Optimal power management forserver farm to support green computing.” Proceedings of the 2009 9th IEEE/ACMInternational Symposium on Cluster Computing and the Grid. IEEE Computer Society,2009.

[18] Rajamani, Karthick, and Charles Lefurgy. ”On evaluating request-distribution schemesfor saving energy in server clusters.” Performance Analysis of Systems and Software,2003. ISPASS. 2003 IEEE International Symposium on. IEEE, 2003.

[19] Yuan, Lin, et al. ”PowerTracer: Tracing requests in multi-tier services to save clusterpower consumption.” arXiv preprint arXiv:1007.4890 (2010).

[20] Elnozahy, EN Mootaz, Michael Kistler, and Ramakrishnan Rajamony. ”Energy-efficientserver clusters.” Power-Aware Computer Systems. Springer Berlin Heidelberg, 2003. 179–197.

[21] Zheng, Xinying, and Yu Cai. ”Optimal server provisioning and frequency adjustmentin server clusters.” Parallel Processing Workshops (ICPPW), 2010 39th InternationalConference on. IEEE, 2010.

[22] Bianchini, Ricardo, and Ram Rajamony. ”Power and energy management for serversystems.” Computer 37.11 (2004): 68–76.

[23] Sharma, Vivek, et al. ”Power-aware QoS management in web servers.” Real-Time Sys-tems Symposium, 2003. RTSS 2003. 24th IEEE. IEEE, 2003.

[24] Pillai, Padmanabhan, and Kang G. Shin. ”Real-time dynamic voltage scaling for low-power embedded operating systems.” ACM SIGOPS Operating Systems Review. Vol.35. No. 5. ACM, 2001.

[25] Kansal, Aman, et al. ”Virtual machine power metering and provisioning.” Proceedingsof the 1st ACM symposium on Cloud computing. ACM, 2010.

[26] Rao, Jia, et al. ”VCONF: a reinforcement learning approach to virtual machines auto-configuration.” Proceedings of the 6th international conference on Autonomic comput-ing. ACM, 2009.

[27] Padala, Pradeep, et al. ”Automated control of multiple virtualized resources.” Proceed-ings of the 4th ACM European conference on Computer systems. ACM, 2009.

REFERENCES 31

[28] Stoess, Jan, Christian Lang, and Frank Bellosa. ”Energy management for hypervisor-based virtual machines.” Proceedings of the USENIX Annual Technical Conference.2007.

[29] Jung, Gueyoung, et al. ”Mistral: Dynamically managing power, performance, andadaptation cost in cloud infrastructures.” Distributed Computing Systems (ICDCS),2010 IEEE 30th International Conference on. IEEE, 2010.

[30] Zhang, Ziming, Qiang Guan, and Song Fu. ”An adaptive power management frameworkfor autonomic resource configuration in cloud computing infrastructures.” PerformanceComputing and Communications Conference (IPCCC), 2012 IEEE 31st International.IEEE, 2012.

[31] Maggio, Martina, et al. ”Decision making in autonomic computing systems: comparisonof approaches and techniques.” Proceedings of the 8th ACM International Conferenceon Autonomic Computing (ICAC11). 2011.

[32] Padala, Pradeep, et al. ”Adaptive control of virtualized resources in utility computingenvironments.” ACM SIGOPS Operating Systems Review 41.3 (2007): 289-302.

[33] Gao, Yongqiang, et al. ”Quality of Service Aware Power Management for VirtualizedData Centers.” Journal of Systems Architecture (2013).

[34] Beloglazov, Anton, Jemal Abawajy, and Rajkumar Buyya. ”Energy-aware resource al-location heuristics for efficient management of data centers for cloud computing.” FutureGeneration Computer Systems 28.5 (2012): 755–768.

[35] Mell, Peter, and Timothy Granc NIST Definition of cloud computing v15, NIST, Editor.2009, National Institute of Standards and Technology: Gaithersburg, MD (2009).

[36] Van Renesse, Robbert, Kenneth P. Birman, and Werner Vogels. ”Astrolabe: A ro-bust and scalable technology for distributed system monitoring, management, and datamining.” ACM Transactions on Computer Systems (TOCS) 21.2 (2003): 164–206.

[37] Zhang, Shuai, et al. ”Cloud computing research and development trend.” Future Net-works, 2010. ICFN’10. Second International Conference on. IEEE, 2010.

[38] VMware site. http://www.vmware.com/

[39] Barham, Paul, et al. ”Xen and the art of virtualization.” ACM SIGOPS OperatingSystems Review 37.5 (2003): 164–177.

[40] http://www.linux-kvm.org/

[41] Barroso, Luiz Andre, and Urs Hoelzle. ”The datacenter as a computer: An introductionto the design of warehouse-scale machines.” Synthesis Lectures on Computer Architec-ture 4.1 (2009).

[42] Microsoft Microsoft environment - the green grid consortium http://www. mi-crosoft.com/environment/ourcommitment/articles/greengrid.aspx, 2009

[43] http://www.datacenterknowledge.com/archives/2011/12/14/how-many-data-centers-emerson-says-500000/

32 REFERENCES

[44] http://www.informationweek.com/hardware/data-centers/5-data-center-trends-for-2013/240145349

[45] http://www.datacenterknowledge.com/archives/2011/08/01/report-google-uses-about-900000-servers/

[46] Gao, Yongqiang, et al. ”Quality of Service Aware Power Management for VirtualizedData Centers.” Journal of Systems Architecture (2013).

[47] http://www.nytimes.com/2012/09/23/technology/data-centers-waste-vast-amounts-of-energy-belying-industry-image.html

[48] Luiz Andre Barroso and Urs Hoelzle The case for energy-proportional computing Com-puter, 40(12):33–37,2007

[49] http://www.bootchart.org/

[50] R. Bianchini, R. Rajamony Power and energy management for server systems Com-puter, 37 (2004), pp. 68–76

[51] Arlitt, Martin, and Tai Jin. ”A workload characterization study of the 1998 world cupweb site.” Network, IEEE 14.3 (2000): 30-37.

[52] Minerick, Robert J., Vincent W. Freeh, and Peter M. Kogge. ”Dynamic power man-agement using feedback.” (2002).

[53] Lefurgy, Charles, Xiaorui Wang, and Malcolm Ware. ”Server-level power control.” Au-tonomic Computing, 2007. ICAC’07. Fourth International Conference on. IEEE, 2007.

[54] Skadron, Kevin, Tarek Abdelzaher, and Mircea R. Stan. ”Control-theoretic tech-niques and thermal-RC modeling for accurate and localized dynamic thermal manage-ment.” High-Performance Computer Architecture, 2002. Proceedings. Eighth Interna-tional Symposium on. IEEE, 2002.

[55] Wu, Qzang, et al. ”Formal control techniques for power-performance management.”Micro, IEEE 25.5 (2005): 52-62.

[56] ”SpeedStep” [Online]. Available:http://en.wikipedia.org/wiki/SpeedStep

[57] http://en.wikipedia.org/wiki/PowerNow!