Contents lists available at ScienceDirect

Optical Switching and Networking

Optical Switching and Networking 23 (2017) 225–240

http://d1573-42

n Corr305-701

E-mdkkangchyoun

journal homepage: www.elsevier.com/locate/osn

Energy and QoS aware resource allocation for heterogeneoussustainable cloud datacenters

Yuyang Peng, Dong-Ki Kang, Fawaz Al-Hazemi, Chan-Hyun Youn n

Department of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea

a r t i c l e i n f o

Article history:Received 31 August 2015Received in revised form1 February 2016Accepted 19 February 2016Available online 27 February 2016

Keywords:Sustainable cloud datacentersRenewable energyVirtual machine allocationHeterogeneity

x.doi.org/10.1016/j.osn.2016.02.00177/& 2016 Elsevier B.V. All rights reserved.

esponding author at: 373-1 Guseong-dong,, Korea. Tel.: +82 42 350 3495; fax: +82 42ail addresses: yuyangpeng@kaist.ac.kr (Y. Pen@kaist.ac.kr (D.-K. Kang), fawaz@kaist.ac.kr (@kaist.ac.kr (C.-H. Youn).

a b s t r a c t

As the demand on Internet services such as cloud and mobile cloud services drasticallyincreases recently, the energy consumption consumed by the cloud datacenters hasbecome a burning topic. The deployment of renewable energy generators such as Pho-toVoltaic (PV) and wind farms is an attractive candidate to reduce the carbon footprintand, achieve the sustainable cloud datacenters. However, current studies have focused ongeographical load balancing of Virtual Machine (VM) requests to reduce the cost of brownenergy usage, while most of them have ignored the heterogeneity of power consumptionof each cloud datacenter and the incurred performance degradation by VM co-location. Inthis paper, we propose Evolutionary Energy Efficient Virtual Machine Allocation (EEE-VMA), a Genetic Algorithm (GA) based metaheuristic which supports a power hetero-geneity aware VM request allocation of multiple sustainable cloud datacenters. Thisapproach provides a novel metric called powerMark which diagnoses the power efficiencyof each cloud datacenter in order to reduce the energy consumption of cloud datacentersmore efficiently. Furthermore, performance degradation caused by VM co-location andbandwidth cost between cloud service users and cloud datacenters are considered toavoid the deterioration of Quality-of-Service (QoS) required by cloud service users byusing our proposed cost model. Extensive experiments including real-world traces basedsimulation and the implementation of cloud testbed with a power measuring device areconducted to demonstrate the energy efficiency and performance assurance of the pro-posed EEE-VMA approach compared to the existing VM request allocation strategies.

& 2016 Elsevier B.V. All rights reserved.

1. Introduction

The electric energy consumption of datacenters isaccounted to be 1.5% of the worldwide electricity usage in2010, and the energy cost is a primary fraction of a data-center's maintenance expenditure [1,2]. Therefore, there isa growing push to improve the energy efficiency of the

Yuseong-gu, Daejeon350 7260.g),F. Al-Hazemi),

data centers behind cloud computing [3,4]. Traditionally,datacenters get their power supply from the utility gridwhich is generated by dirty energy generators, such ascoal, or nuclear plants [15]. These conventional energygenerators not only produce much carbon but alsoincrease the operation cost for datacenters. Towardsaddressing this inefficiency, a promising solution receivingspotlights is the incorporation of renewable energy gen-erators such as PhotoVoltaic (PV) and wind turbines intothe design of datacenters (i.e., achieving “sustainable”datacenters which reduce not only the electricity cost butalso the carbon footprint). Renewable energy generator isbecoming drastically an attractive candidate for designing

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240226

green datacenters in academia. Recently, researchers haveproposed several studies to integrate renewable energysources into cloud datacenters. The cost optimizationmodel considering both of renewable energy source andcooling infrastructure is proposed to realize the potentialof sustainable cloud datacenters [9]. They propose thedemand shifting which schedules non-interactive work-load to maximize the utilization of renewable powersource. The energy storage management of sustainablecloud datacenters has been proposed to minimize thecloud service provider's electricity cost [10,11]. The sche-duling scheme for parallel batch jobs has been proposed inorder to maximize the utilization of green energy con-sumption while ensuring the Service Level Agreements(SLAs) of requests [16]. However, there are still remainingchallenges to achieve the energy efficient sustainablecloud datacenters.

First, each cloud datacenter have heterogeneous serverarchitecture, i.e., they require different power consump-tion even for serving of the same amount of workload. Theserver heterogeneity is caused by hardware upgrades,capacity extension, and the replacement of peripheraldevices [6–8]. However, traditional cloud datacentermanagement schemes assume that all the cloud data-centers have homogeneous server architecture with samepower efficiency although this assumption is unrealisticfor most cloud resource providers. Second, two issues ofgreening cloud datacenters and Quality-of-Service (QoS)assurance are conflicting goals in the resource manage-

Fig. 1. Cloud environment consists of multiple cloud datacenters and Cloud

ment. Especially, the performance degradation might beinduced by VM co-location interference when multiple VMinstances are running on common physical server in clouddatacenters [12]. As more VM instances are packed intocommon servers, the required number of active servers isdecreased, while the resource contention is deteriorated.This means that the energy consumption is reduced withsacrificing the QoS assurance of the processing for VMrequests. It is important to find a desirable tradeoffbetween above two goals corresponding to the dynamicworkload level.

To solve these challenges, we propose an EvolutionaryEnergy Efficient Virtual Machine Allocation (EEE-VMA)approach which depends on an energy optimizationmodel for sustainable cloud datacenters having hetero-geneous power efficiency with renewable energy gen-erators. This paper proposes four contributions as belows.

First, our approach tries to find a near optimized solu-tion of VM request allocation by applying Genetic Algo-rithm (GA) with consideration for both of renewableenergy cost and traditional utility grid cost. The funda-mental strategy adopted in EEE-VMA as an energy savingscheme is Dynamic Right Sizing (DRS) which is for makingcloud datacenters be power-proportional (i.e., consumespower only in proportion to the workload level) byadjusting the number of active servers in response toactual workload (i.e., to adaptively “right-size” the data-center) [3,5]. In DRS, the energy saving can be achievedthrough allowing idle servers that do not have any running

Request Brokers (CRBs) with Cloud Request Broker Manager (CRBM).

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 227

VM instances to be low-power mode (e.g., sleep or hiber-nation). Note that our proposed energy consumptionmodel for the EEE-VMA approach includes a switchingcost of DRS which is incurred by toggling a server fromlow-power mode into active mode (i.e., awaken transi-tion). This makes our proposed approach more practicalfor energy efficient cloud datacenter in real world.

Second, in our proposed EEE-VMA approach, in order toadopt the heterogeneous power efficiency of each clouddatacenters, we propose a novel metric called powerMarkto quantize the power efficiency of servers by measuringtheir power consumption at each utilization level ofresources such as CPU, memory, and I/O bandwidth.Especially, we compute powerMark for serveral types ofserver by measuring their power consumption for pro-cessing CPU-intensive applications.Through powerMark,we are able to determine the allocation priority of eachcloud datacenter based on their power efficiency so as toimprove the performance of energy saving.

Third, we achieve the significant energy saving of clouddatacenters while minimizing the performance degrada-tion caused by VM co-location interference through ourEEE-VMA approach. The workload model including both ofthe number of co-located VM instances and the resourceutilization which are key factors reflecting VM co-locationinterference is applied to the cost model of the EEE-VMAapproach. Moreover, we consider the bandwidth costbetween cloud service users and cloud datacenters as anadditional part contributing the QoS deterioration of VMrequest processing [31,32]. The desirable cloud datacenterselection for each VM request assignment are conductedwith consideration for both of energy saving and QoSassurance corresponding to the dynamic workload level.

Finally, we conduct extensive experiments throughsimulations at various workload levels based on real-worldtraces such as dynamic capacity of renewable energy andelectricity prices of traditional grid power [9,18–20], andthe implementation of testbed with a power measuring

Table 1Set of key notations.

Notation Description

DC The set of cloud datacentersCRB The set of CRBsF The set of flavor types of VM request supported by cloud resourRC The set of resource components such as CPU and memoryΛi tð Þ The set of VM requests arrived at whole CRBs at time tX tð Þ Resource allocation plan of VM requests from CRBs to cloud datac

each VM requestM tð Þ DRS plan of cloud datacenters at time t, which determines the nS tð Þ A solution including resource allocation plan X tð Þ and DRS planDj tð Þ Performance degradation of cloud datacenter DCj by CPU resourUR The set of predetermined resource utilization levels

pwMjrch i An average power consumption per an unit level of utilization o

pivotS The predetermined pivot server used as a criterion of resource cej tð Þ The energy consumption of cloud datacenter DCj at time t

ctotal tð Þ The total cost of whole cloud datacenters at time t

f EEE�VMA Uð Þ The objective function to get ctotal tð Þ in the EEE-VMA solver

device called Yocto-Watt to measure a real power con-sumption of several cloud server types [21].

The rest of the paper is organized as follows. Section 2gives an overview of the proposed system architecture ofmultiple cloud datacenters and cloud request brokers. InSection 3, the objective cost model including workload andenergy consumption model with powerMark are for-mulated. Our EEE-VMA approach based on Genetic Algo-rithm is proposed to obtain the approximated optimalsolution minimizing the total cost of cloud datacenters inSection 4. Section 5 shows the various experimentalresults that demonstrate the effectiveness of our proposedapproach based on real-world traces. The conclusion isgiven in Section 5.

2. System architecture and design

Our considered cloud environment including multipleCloud Request Brokers (CRBs) which support mesh net-working with distributed multiple cloud datacenters isdepicted in Fig. 1. There are h CRBs and m cloud data-centers with h�m communication links. In each clouddatacenter, the information of resource utilization, theavailable renewable energy, and the power consumptionof each server are collected through monitoring modules,power measuring devices, and reported to the CloudRequest Broker Manager (CRBM) which is responsible forsolving the allocation of VM requests submitted to CRBs.The CRBM has two modules: the powerMark analyzer andthe EEE-VMA solver. The powerMark analyzer is respon-sible for capturing the power efficiency of each clouddatacenter through our proposed novel metric calledpowerMark. We describe this metric in detail in Section 3.The EEE-VMA solver is responsible for finding a nearoptimal solution of VM request allocation from CRB to thecloud datacenter. The solution derived by the EEE-VMAsolver based on the amount of submitted VM requests in

ce provider

enters at time t, which determines the destined cloud datacenters for

umber of active servers of cloud datacentersM tð Þce contention at time t

f resource component rcARC of servers in the cloud datacenter DCj

apacity

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240228

each CRB and the reported information from each clouddatacenter is delivered to whole CRBs and all thesubmitted VM requests are allocated to their destinedcloud datacenters.

The owner of cloud datacenters has to minimize costsfor resource operation while boosting benefits whichcan be realized since cloud service users have a goodreputation on observed QoS by cloud services. In thispaper, our EEE-VMA solver tries to find a solution tominimize the total cost of resource operation includingthree sub cost models: energy consumption cost, band-width cost, and performance degradation cost. In theperspective of energy consumption cost, the EEE-VMAsolver tries to maximize the utilization of renewableenergy with consideration on the dynamic capacity ofeach renewable energy generator since the price ofrenewable energy is much cheaper than the one of gridenergy.

VM requests from CRBs are preferably allocated tocloud datacenters which have the higher capacity ofrenewable energy and the higher power efficiency (i.e., thelower powerMark value). In the perspective of bandwidthcost, the EEE-VMA tends to route VM requests to clouddatacenters having the cheaper bandwidth cost. Obviously,different pairs of CRB and cloud datacenter have differentbandwidth cost according to the hop distance and theamount of transferred data of routed VM requests.Therefore, it is clear that VM requests need to be allocatedto the closest cloud datacenter to their source CRB in orderto minimize the bandwidth cost. To simplify our model,we assume that the transferred data size of each VMrequest is known to the EEE-VMA solver in CRBMbeforehand.

In the perspective of performance degradation cost, theEEE-VMA solver tries to spread whole VM requests overmultiple cloud datacenters in order to avoid QoS dete-rioration of VM request processing. In cloud datacenters,the VM co-location interference is the key factor thatmakes servers undergo severe performance degradation[12,22]. The VM co-location interference is caused byresource contention which can be reflected mainly by thenumber of co-located VM instances and resource utiliza-tion of them. In brief, the VM co-location interferencegrows bigger as more VM instances are co-located on thecommon server and the higher resource utilization isoccurred. Therefore, VM requests have to be scattered inorder to try its hardest to avoid performance degradationby VM co-location interference. Because of the complexityof optimization for aggregated cost model, the EEE-VMAsolver adopts metaheuristic based on GA to obtain nearoptimal solution of VM requests allocation within theacceptable computation time. In next section, we proposea mathematical model to describe the cost of cloud data-center and describe the metric powerMark in detail. Theset of involved key notations are shown in Table 1.

3. Problem formulation

3.1. Workload model

There are many different kinds of workloads in clouddatacenters which can be classified into two categories:interactive or transactional (delay-sensitive) and non-interactive or batch (delay-tolerant) workload [9]. The inter-active workloads such as Internet web services and multi-media streaming services have to be processed within a cer-tain response time defined by service users. They are oftennetwork I/O intensive jobs which have less impact to thepower consumption of servers. In contrast, the batch work-loads such as scientific applications and big data analysis canbe scheduled to process anytime as long as the whole tasksare finished before the predetermined deadline. They areusually computation intensive jobs that require a lot of CPUutilization causing a significant power consumption of ser-vers. In this paper, we are interested in the computationintensive batch workloads since they have a greater influenceto server power consumption than interactive workloads. Weassume that all the VM requests have computation intensiveworkloads, and the resource contention is always occurred inCPU resource. A workload λki tð ÞAΛi tð Þ denotes the number ofarrived VM requests with a required flavor type (e.g., instancetype such as m3.medium or c4.large in Amazon EC2) FkAF atthe CRBiACRB at time t [29]. We use rkrch i to denote therequired amount of resource component rcARC by a VMrequest with flavor type Fk where RC¼ rcCPU ; rcMEMf g. Forexample, rkrch isuchth at Fk ¼m3:medium and rc¼ rcCPUrepresents the required number of CPU cores by a VM requestof which the flavor type is m3:medium. When multiple VMrequests are arrived at the CRB, then the CRB would decide inwhich cloud datacenters each VM request should be routedfor processing. We assume no data buffering at the CRB sothat whenever a VM request arrives at the CRB, it would berouted to a cloud datacenter for processing immediately [11].We denote the number of VM requests with the flavor type Fkrouted from the CRBi to DCj at time t as xi;kj tð Þ, which isderived by a resource allocation plan for cloud datacenters,X tð Þ. Then we have the following constraints:X

8DCj ADCxi; kj tð Þ ¼ λki tð Þ; 8CRBiACℛℬ; 8FkAℱ; 8 t ð1Þ

0rxi;kj tð Þrλki tð Þ; 8CRBiACRB; 8FkAF ; 8DCjADC; 8t ð2Þ

Above Eq. (1) means that the total number of VMrequests arrived at CRBs must agree with the one of wholeVM requests allocated to cloud datacenters. Another con-straint we should consider is a resource capacity of thecloud datacenter. Each cloud datacenter only can accom-modate VM requests within their resource capacity (e.g.,the total number of CPU cores). Then, we have the fol-lowing constraintsX

8CRBi ACℛℬ

X8Fk Aℱ

rk⟨rc ¼ rcCPU ⟩ Uxi; kj tð Þrscpj⟨rc ¼ rcCPU ⟩

Umj tð Þ; 8DCjADC; 8t ð3ÞX

8CRBi ACℛℬ

X8Fk Aℱ

rk⟨rc ¼ rcmem⟩Uxi; kj tð Þrscpj⟨rc ¼ rcmem⟩

Umj tð Þ; 8DCjADC; 8t ð4Þ

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 229

0rmj tð ÞrN DCj� �

; 8DCjADC; 8t ð5Þ

where rkrc ¼ rcCPUh i and rkrc ¼ rcmemh i are required CPU coresand memory size of VM request with flavor type FkAF .scp j

rch i is the physical capacity of resource componentrcARC of an arbitrary server in the cloud datacenter DCj.Constraints (3) and (4) represent that allocated VMrequests can not always exceed the capacity of resourceprovided by cloud datacenter DCj. We use mj tð Þ to denotethe number of active servers in cloud datacenter DCj attime t and it is determined by a DRS plan M tð Þ, and itsupper bound is N DCj

� �which is the number of total phy-

sical servers in the cloud datacenter DCj. The constraint (5)represents that mj tð Þ can be determined in the range of0 to N DCj

� �through the DRS plan. mj tð Þ ¼ 0 means that

whole servers in the cloud datacenter DCj are in the sleepstate, while mj tð Þ ¼N DCj

� �means that they are in the

active state at time t.Next, we consider a VM co-location interference to

build a performance degradation model of resource allo-cation in cloud datacenter [12]. The VM co-location inter-ference implies that the virtualization of cloud supportsresource isolation explicitly when multiple VM requestsare running simultaneously on common PM, but it doesnot mean the assurance of performance isolation betweenVM requests internally. In the perspective of CPU resource,physical CPU cores of the server are not pinned to eachrunning VM request, but assigned dynamically. Theswitching overhead by the dynamical CPU assignmentpolicy might cause the undesirable performance degra-dation of allocated VM requests. Moreover, the CPUresource contention aggravates the performance degra-dation since it is very difficult to isolate the cache space ofCPU. There is a strong relationship between VM co-location interference and the number of co-located VMsin PM [12]. The more co-located VM instances, the moresevere VM co-location interference is occurred. Based on[12], we estimate the performance degradation Dj tð Þ of thecloud datacenter DCjADC by the CPU resource contentionat time t as follows,

Dj tð Þ ¼P

8CRBi ACℛℬ

P8 Fk Aℱx

i; kj tð ÞUrk⟨rc ¼ rcCPU ⟩ U vu⟨rc ¼ CPU⟩ tð Þþtsj tð Þ

� �scp j

⟨rc ¼ rcCPU ⟩Umj tð Þ

;

8DCjADC; 8t ð6Þ

where tsj tð Þ is an average allocated time slice deter-mined by Hypervisor [25,28] for VM requests allocated tothe cloud datacenter DCj at time t. We use vu rch i tð Þ todenote an average utilization of assigned virtual resourcesof whole VM requests allocated to cloud datacenters attime t. Note that in Eq. (6), tsj tð Þ and rkrc ¼ CPUh i can beknown in advance, while vu rch i tð Þ can not be recognizedbeforehand, until the utilization of CPU resource is mea-sured through the internal monitoring module of eachserver in cloud datacenters at time t [24]. Therefore it isrequired to use the historical information of CPU resourceutilization of VM requests to find optimal solution ofresource allocation for the current workload. As shown in

Fig. 1, the data repository module is responsible for col-lecting and storing the monitoring information of resourceutilization of each VM request to estimate the futuredemand. Our EEE-VMA solver uses the historical data ofthe resource utilization from the data repository module ineach cloud datacenter to estimate the expected perfor-mance degradation of solution candidates.

3.2. Energy consumption model

3.2.1. The renewable energy modelThe renewable energy such as the PV and wind energy

is more sustainable than the traditional grid power, andits price is low and the less carbon is emitted [9]. Thereare two models to achieve the sustainable cloud data-centers by deploying the renewable energy generation.One is on-site deployment of renewable energy genera-tion at the datacenter facility itself. For example, Applehas built its own local biogas fuel cells and two 20-MWsolar arrays in Maiden, NC and they have been poweredby 100% renewable energy sources [26,27]. Such on-siterenewable energy generator can alleviate energy lossesdue to the transmission and distribution of generatedenergy, but its energy potential depends greatly on thelocation of the cloud datacenter. Another model isbuilding the renewable energy generator at off-sitefacilities. It has the flexibility to locate the generator ina location with good weather (e.g., strong wind speed orbright sunshine), but the significant transmission lossesof energy can be occurred. In this paper, we use the firstmodel which has been adopted by most major datacenterowners.

We denote rwej tð Þ and rpej tð Þ as dynamic capacity ofrenewable wind energy and renewable photovoltaicenergy of the cloud datacenter DCjADC at time t, respec-tively. Obviously, it is required to forecast the futurecapacity of renewable energy to achieve energy efficientresource management of cloud datacenters since they areusually intermittent and irregular. Therefore, we estimatethe future capacity of renewable energy generation byusing the historical data from the data repository modulein the cloud datacenter through calculating an Exponen-tially Weighted Moving Average (EWMA) values. Thedetailed descriptions of the EWMA based forecastingscheme for estimated capacity of renewable energy isomitted in this paper.

3.2.2. Heterogeneous power consumption modelWe propose a novel power efficient metric called

powerMark to evaluate the heterogeneous power con-sumption of cloud datacenters. Servers consist of eachcloud datacenter have heterogeneous architecture, whichimplies that the specification of their resources are dif-ferent, consequently, even though they process the sameapplication, for which each required power consumptionmight be different [8,13]. To describe powerMark in detail,we propose Definition 1 and 2 as belows,

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240230

Definition 1. (powerMark):

The powerMark pwMjrch i is an average power con-

sumption per an unit level of utilization ofresource component rcARC of servers in the clouddatacenter DCj.

Definition 2. (pivot server):

The predetermined pivot server pivotS used as a cri-terion of resource capacity for normalizingpowerMark of each cloud datacenter.

Moreover, we propose the novel concept pivot serverpivotS in Definition 2 to normalize the powerMark of eachcloud datacenter. For simplicity, we assume that servers inthe same cloud datacenter has power-Homogeneity to eachother. To obtain the powerMark value, we predetermined theset of resource utilization levels UR¼ ur1;ur2;…;urk

� �.

The powerMark pwMjrch i represents the power efficiency of

servers in the cloud datacenter DCj with respect to thecertain resource component rcAR by calculating an arith-metic mean of power consumption measured at eachresource utilization level urkAUR; 8urk40. For example,we set UR¼ ur1 ¼ 0:1;ur2 ¼ 0:2;…;ur9 ¼ 0:9f g and rc¼rcCPU , then the power consumption of server is measured ateach CPU utilization level 0:1;0:2;…;0:9 respectively. Basedon the data of measured power consumption, the power-Mark pwMj

rch i is given by

pwMj⟨rc⟩ ¼

1Uℛj j

X8urk AUℛ

pw j⟨rc⟩; urk

urk: ð7Þ

npwMj⟨rc⟩ ¼

1Uℛj j

X8urk AUℛ

scppivot⟨rc⟩

scpj⟨rc⟩

Upw j⟨rc⟩;urk

urk: ð8Þ

where pw jrch i;urk is the power consumption of servers in

the cloud datacenter DCj at the utilization level urk of the

resource component rc. npwMjrch i is the normalized value

of pwMjrch i based on the capacity of resource component

rc of the pivot server pivotS wherescppivotrch iscpjrch i

Upw jrch i;urk is the

normalized value of pwjrch i;urk . The lower powerMark

represents the higher power efficiency of the clouddatacenter and, with larger URjj , powerMark can accu-rately describe the power efficiency of the cloud data-center. In order to investigate the availability of power-Mark, we conduct the preliminary experiment to obtain

Table 2Three server types for an experiment to investigate powerMark.

Server types CPU architecture CPU cores CPU

Server-1 Intel i5-760 4 2.8Server-2 Intel i5-4590 4 3.3Server-3 Intel i7-3770 8 3.4

power consumption of heterogeneous servers with run-ning VM requests processing computation-intensive jobson a real test bed. There are 3 server types to investigatethe heterogeneity of power consumption. The hardwarespecifications of each server type are shown in Table 2.Each server has two Ethernet interface cards with 1 Gbpsand uses Ubuntu 14.04. The test application for theexperiment is mProject module (m108 with range 1.7) ofMontage Project to make astronomy image files of spacegalaxy, which is the computation-intensive application[14]. Fig. 2 shows curves of power consumption of eachserver type as the resource utilization of CPU is increasedby mProject running when Server-1 type is pivotS. Asmentioned earlier, each server requires different powerconsumption even at same resource utilization level. TheServer-1 has the largest amount of power consumptionthan others, which means that this server type has theworst performance in terms of energy consumption.Fig. 3 shows the calculated normalized powerMark npwMvalues with rc¼ rcCPU of Server-1, 2, and 3 based on Eq.(8). Note that the difference of normalized powerMarknpwM values among servers of Fig. 3 is bigger than theone of powerMark pwM values of Fig. 2. The Server-3 hasthe smallest value of npwM, which means that this serverhas the best power efficiency among three servers, andthis is in concordance with the results in Fig. 2. Based onresult curves in Figs. 2 and 3, we conclude that our pro-posed metric powerMark is simple and useful to representthe relative power efficiency of heterogeneous clouddatacenters in practice.

3.2.3. Dynamic right sizing modelTo achieve power-proportional cloud datacenter

which consumes power only in proportion to the work-load, we consider DRS approach which adjusts thenumber of active servers by turning them on or offdynamically [3]. Obviously, there is no need to turn allthe servers in cloud datacenter on when the totalworkload is low. In DRS approach, the state of serverswhich have no running applications can be transit to thepower saving mode (e.g., sleep or hibernation) in orderto avoid wasting energy as shown in Fig. 4. In order tosuccessfully deploy DRS approach onto our system, weshould consider the switching overhead for adjustingthe number of active servers (i.e., for turning sleep ser-vers on again).

The switching overhead includes: (1) additional energyconsumption by transition from sleep to active state (i.e.,

clocks (GHz) Cache size (kB) Memory size (GB)

8192 36144 88192 16

Fig. 2. Normalized power consumption results of Server-1, 2, 3 underexecution of Montage applications as an example.

Fig. 3. Results of normalized powerMark of Server-1, 2, 3 based on Eq. (7).

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 231

awaken transition); (2) wear-and-tear cost of server;(3) fault occurrence by turning sleep servers on whentoggled is high [3]. We only consider the energy con-sumption as the overhead by DRS execution. Therefore, wedefine a constant αaWaken to denote the amount of energyconsumption for awaken transition of servers. Then thetotal energy consumption ej tð Þ of cloud datacenter DCj attime t is defined as follows,

ej tð Þ ¼X

8 rcAℛpwMj

⟨rc⟩ U

P8CRBi ACℛℬ

P8 Fk Aℱ rk⟨rc⟩ Ux

i; kj tð ÞUvurc tð Þ

scpjrc Umj tð Þ

!

þαaWaken � mj tð Þ �mj t�1ð Þ� �þ; 8DCjADC; 8t ð9Þ

where xð Þþ ¼max 0; xð Þ. The first term of the right handin (9) represents an energy consumption for using serversto serve VM requests allocated to the cloud datacenterDCj at time t and the second term represents an energyconsumption for awaken transition of sleeping servers.Especially, the second term implies that a frequentchanges in the number of active servers might increasethe undesirable waste of energy. Note that the overheadby transition from active to sleep state (i.e., asleep tran-sition) is ignored in our model since a time required forasleep transition is relatively short compared to the onefor awaken transition.

3.3. The cloud datacenter cost minimization problem

We build a cost model based on workload model andenergy consumption model proposed in Sections 3.1 and3.2. We focus on minimizing the total cost including threesub costs: (1) energy cost; (2) performance degradationcost; (3) bandwidth cost. In our energy cost model, tosimplify it, we assume that the price for renewable energyusage is zero in this paper (strictly, the real price is notzero since the investment expense and the maintenanceexpenditure for renewable energy generation equipments

are required to deploy the renewable energy generatoronto the cloud datacenter). Generally, the price of powergrid and the capacity of the generated renewable energyare time-varying according to the electricity market andthe location of the cloud datacenter [17,19]. We use cej tð Þ todenote the energy cost of cloud datacenter DCj at time t asfollows,

cej tð Þ ¼ ρgrid tð ÞU ej tð Þ�rwej tð Þ�rpej tð Þ� �þ

; 8DCjADC; 8tð10Þ

where ρgrid tð Þ denotes the time-varying price ofpower grid at time t. Next, the performance degradationcost can be determined by the total performancedegradation of the cloud datacenter based on Eq. (6).When we use ρperf to denote the constant of penaltyprice for performance degradation, then the perfor-mance degradation cost of the cloud datacenter DCj attime t, cperfj tð Þ is given by,

cperfj tð Þ ¼ ρperf � Dj tð Þ; 8DCjADC; 8 t ð11Þ

Note that ρperf is a constant in contrast with ρgrid tð Þwhich is dynamically changed according to time. Third,the bandwidth cost is the one for the data transferbetween the cloud service users closed to CRBs and VMrequests allocated on servers in cloud datacenters.Obviously, different links between CRB and cloud data-center require the different bandwidth cost. The band-width cost is determined by the network distance (e.g.,hop distance) and the transferred data size. We use cbwj tð Þto denote the bandwidth cost of the cloud datacenter DCj

at time t as given by,

cbwj tð Þ ¼X

8CRBi ACℛℬ

X8DCj ADC ρbw

i; j UX

8Fk Aℱxi; kj tð ÞUdsk

� �;

8DCjADC; 8t ð12Þ

where ρbwi;j denotes is the bandwidth cost coefficient of

the communication link between the cloud request brokerCRBi and the cloud datacenter DCj, and dsk denotes thetransferred data size of VM request with flavor type Fk.

Fig. 4. Illustration of Dynamic Right Sizing procedure.

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240232

Obviously, as the hop distance between the CRBi and DCj

grows longer, ρbwi;j is also increased. (12) implies that the

allocation of more VM requests to the cloud datacenterwhich is far away (i.e., has long hop distance) from thesource CRB increases the bandwidth cost cbwj tð Þ. It isadvantageous for bandwidth cost saving to allocate VMrequests to the nearest cloud datacenter to theirsource CRB.

Consequently, we focus on minimizing the total cost ofwhole cloud datacenters through our proposed approachof the EEE-VMA solver. We use ctotal tð Þ to denote the totalcost of whole cloud datacenters at time t, which includesthe energy costs, the performance degradation cost andthe bandwidth cost. Then we define the objective functionf EEE�VMA S tð Þð Þ to calculate the total cost determined by thesolution S tð Þ ¼ X tð Þ ¼ x1; 11 tð Þ;

nnx1;1 2 tð Þ; …; x Cℛℬj j; ℱj j

DCj j tð Þ g;M tð Þ ¼ m1 tð Þ; m2 tð Þ; …; m DCj j tð Þ

� �g at time t as belows,

f EEE�VMA S tð Þð Þ : ctotal tð Þ ¼X

8DCj ADC cej tð Þþcperfj tð Þþcbwj tð Þ� �

s:t 1ð Þ � ð5Þ

ð13ÞTo solve this function, we propose the EEE-VMA

approach based on GA in order to find an approximatedoptimal solution for VM request allocation. In next section,we describe our algorithm in detail.

3.4. Evolutionary energy efficient virtual machine allocation

In this section, we propose EEE-VMA approach basedon GA which is one of efficient metaheuristics to solve acomplex optimization problem. In order to successfullydeploy GA onto the EEE-VMA, we should define accu-rate strategies for GA and set their appropriate para-meters. To do this, we consider five basic steps of GA asfollows.

3.4.1. Encoding schemeA chromosome (i.e., individuals in the population)

features the solution S tð Þ of our datacenter managementscheme in cloud datacenters. The format of genes in thechromosome is described as an integer value. The chro-mosome includes multiple genes which are divided intotwo parts: the first part is for the VM request allocationplan X tð ÞAS tð Þ; and the second part is for the DRS planM tð ÞAS tð Þ. This detailed structure of the chromosome isshown in the Fig. 5.

In the first part of the chromosome, gene values repre-sent the number of VM requests allocated to the clouddatacenter at time t. For example, as shown in Fig. 5, thegene 1;210ð Þ in the chromosome represents x1;11 tð Þ ¼ 210which means that the number of allocated VM requestswith flavor type F1 from the CRB1 to DC1 is 210 at time t. Inthe second part of the chromosome, gene values representthe number of active servers in the cloud datacenter at timet. In Fig. 5, the gene CRB � F � DC þ1;3200j Þ

����������� in thechromosome means that the number of active servers inthe cloud datacenter DC1 is 3200 at time t.

3.4.2. InitializationIn the first generation g¼ 1, GA in the EEE-VMA

approach begins with randomly generated populationsaccording to submitted VM requests at each CRB. Toreduce the computation time for GA execution, the rangeof value for each gene can be predetermined based on (2),and (5).

3.4.3. EvaluationIn EEE-VMA approach, we use (13) to evaluate the

performance of each chromosome (i.e., solution) in thepopulation. The fitness value of a chromosome is inversely

Fig. 5. Encoding example of VM allocation with chromosome.

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 233

related to the cost value. The higher fitness function valueimplies the higher performance of the chromosome. Notethat if a certain chromosome violates any of constraints(1)–(5), then its cost value is counted as “positive infinity”.Otherwise, the chromosome which has the smallest costvalue among all the chromosomes in the generatedpopulation at g¼ gMax (a max step of generation) is cho-sen as an optimal solution S� tð Þ finally.

3.4.4. SelectionThere are several candidate schemes for selection of

appropriate solutions in GA. Especially, we adopt theroulette-wheel selection which determines the probabilityof each chromosome to be chosen according to their fit-ness function value. This scheme tends to preservesuperior solutions and evolve them in the next generation[30].

3.4.5. CrossoverThe role of crossover is to generate offspring from

two parents by cutting certain genes of parents andconducting recombination of each gene fragment. Theoffspring inherits characteristics of each parent. OurEEE-VMA approach adopts the simple crossover schemeby which the first half of the first parent and the secondhalf of the second parent are aggregated to genes of theiroffspring. Note that crossover has to be conductedseparately on each part of the chromosome since it hastwo parts of the VM request allocation plan and theDRS plan.

3.4.6. MutationIt is necessary to ensure the diversity of the generated

population at all generation steps in order to avoid localminima problem in GA. At each generation step g, genevalues constituting chromosome can be modified ran-domly according to the predetermined probability pbmt .If pbmt is too large, the superior genes inherited fromparents can be loss, otherwise, the diversity of popula-tion might be lower when pbmt is too small. It is

important to determine the appropriate pbmt in order tomaintain the diverse and superior population. However,we do not consider this issue since it is out of scope inthis paper.

The proposed GA for EEE-VMA approach is described inAlgorithm 1. In order to get the near optimal solution x�t ofdatacenter management for VM requests arrived at CRBsat time t, the state information of servers in all the data-centers at time t�1 is required. If the current time t ¼ 0,then we assume that the previous state of all the servers isactive (i.e., all the servers are switched on). In line 02, weinitialize the candidate population cand_popg tð Þ, g ¼ 1ð Þwith the population size ps (represents the limit numberof chromosomes in the population) randomly. The popu-lation cand_popg tð Þ evolves until g¼ gMax to generate thefinal population popg ¼ gMax tð Þ to search the near optimalsolution x�t as shown from line 03 to 29. Two parentchromosomes Sg;i tð Þ and Sg;j tð Þ are released fromtemp_popg tð Þ to produce an offspring Sg;k tð Þ from line 06 to10. In line 11, each offspring in the set offspringg tð Þ ismutated by modifying each gene according to the prob-ability prmt to maintain the diversity of the population. Wecheck constraints (i.e., Eqs. (1)–(5)) of each solution Sg;i tð Þin cand_popg tð Þ are whether violated or not in line 14. Ifthey are violated, the corresponding solution has the costvalue counted as “positive_infinity”. Otherwise, theobjective function value of the solution is calculatedthrough f EEE�VMA Uð Þ in line 17. If we find the solution Sg;i tð Þhaving the objective function value cg;i tð Þ which is smallerthan the predetermined threshold value cthr , then wecount Sg;i tð Þ as the near optimal solution S� tð Þ and thealgorithm 1 is finished. Otherwise, chromosomes to bepreserved until the next generation are chosen from thecurrent population through the Iterative Roulette-wheelSelection (Algorithm 2) procedure as shown in line 23.When reaching the max step of generation gMax, the nearoptimal solution S� tð Þ having minimum cost value off EEE�VMA Uð Þ in popg ¼ gMax tð Þ is found and returned to theEEE-VMA solver in our system in line 30.

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240234

The IterativeRwSelection for Algorithm 1 is describedin Algorithm 2. From line 02 to 07, the cost values whichhave “positive_infinity” are released from cand_Cg tð Þ andadded to illeg_Cg tð Þ since solutions which do not violateconstraints (i.e., (1)–(5)) are preferentially considered ascandidates to be preserved until the next generation. Themaximum (worst) and minimum (best) objective function

Algorithm 1.

values are found from cand_Cg tð Þ in line 09 and 10. Fitnessvalues of each solution are calculated as shown in line 13.Through this equation, the fitness value of the best solu-tion comes out as α times of one of the worst solution. Theselection pressure which represents the differencebetween fitness values of superior solutions and inferiorsis increased as α is increased. The sum of fitness values ofeach solution, SF is updated in line 15.

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 235

Algorithm 2.

This value represents the total size of roulette-wheel, andeach solution is assigned to spaces on the roulette-wheel.That means that the selection probability of each solution isproportional to the size of their assigned spaces. The selec-tion procedure of the roulette-wheel is described from line19 to 26. At every step, the cumulative summation QS is

updated according to the fitness value fvg;i tð Þ in FVg tð Þ. If theselection point SP is smaller than QS with the latest update

by fvg;i tð Þ (i.e., Pi�1

k ¼ 1fvg;k tð ÞrSPr Pi

k ¼ 1fvg;k tð Þ), then the index

i of Sg;i tð Þ in cand_popg tð Þ is added to chIdxSetg tð Þ. If the total

Fig. 6. Wind speed (m=s) and its corresponding amount of generated wind energy (kW) at Oak Ridge National Lab (a and b), Univ of Arizona (c and d), andUniv of Nevada (e and f) at EST 05:20–17:54 on September 9, 2015 [33].

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240236

number of chosen chromosomes by the roulette-wheelprocedure is not sufficient (i.e., the cardinality ofchIdxSetg tð Þ is smaller than the predetermined size of thepopulation ps), then we supplement chIdxSetg tð Þ by ran-domly putting out the indices of chromosomes fromilleg_Cg tð Þ. After all the procedures are finished, thenchIdxSetg tð Þ is finally returned to the Algorithm 1 in line 32.

4. Performance evaluation

In this section, we evaluate the performance of ourproposed EEE-VMA approach based on both of simulationanalysis and experiments on real testbeds. To highlight the

benefits of our design for renewable and QoS awareworkload management, we perform a numerical simula-tion based on real-world traces of renewable energycapacity.

4.1. Dynamic capacity of renewable energy

We consider three locations to employ the raw data inorder to build a capacity trace of renewable energyincluding wind energy: Oak Ridge National Lab (EasternTennessee); University of Arizona (Tucson, Arizona); Uni-versity of Nevada, Las Vegas (Paradise, Nevada) [11,33]. Weobtain the capacity traces of wind energy at those threelocations baced on [33] that collects data of wind speedevery day. The capacity traces of each location at EST

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 237

05:20–17:54 on September 9, 2015 are shown in Fig. 6.Fig. 6(a), (c), and (e) shows the wind speed of each locationand we can find that it is fluctuated a lot even during shortperiod. We assume that each generator has 30 wind tur-bines and the amount of generated wind energy is esti-

Fig. 7. Real time price of power grid at three regions.

Fig. 8. CPU utilization of the running VM request including pbzip2,iozone3, and netperf.

Fig. 9. Total cost of datacenters including servers with heter

mated based on the wind power prediction scheme from[34]. Then the curves of the amount of available windenergy are shown in Fig. 6(d), (e), and (f).

4.2. Energy price description

As mentioned earlier, only the grid power price isconsidered since we assume that the renewable energyprice is free. The grid power price is dynamically changedaccording to the electricity consuming time. We use theelectricity price information in our simulation based onthe real time pricing during 24 h in the electricity marketwhich is shown in Fig. 7 [23,35]. Note that the electricityprice is high from 6 a.m. to 14 p.m., and from 19 p.m. to21 p.m. The electricity usage is usually increased duringthese periods due to the needs of industrial and householdappliances. In our simulation, each cloud datacenter ran-domly has the electricity pricing curve among datacenter1, 2, and 3 in Fig. 7.

4.3. Cloud resource description

The total number of multiple cloud datacenters is nine,and each datacenter owns 2 � 103 homogeneous servers inthis paper. In perspective of VM instance specifications, weadopt the policy of Amazon EC2 Web Services (AWS), ourcloud datacenters support the set of flavor typesF ¼ F1 ¼ CPU ¼ 2cores;mem¼ 4 GBð Þ; F2 ¼ 4;8ð Þ; F3 ¼ 8;16ð Þ;�F4 ¼ 16;32ð Þg and each VM request has an arbitrary flavortype FkAF randomly [29]. As mentioned in Section 3, eachcloud datacenter has heterogeneous server architecture,they have different powerMark value in the range of 200–500 based on results in Fig. 3.

4.4. Workload scenario

Our considered workload includes two parts: thenumber of VM requests tð Þ , and their required resourceutilization vu rch i tð Þ at time t. The number of VM requestsΛ tð Þ can be defined as from 3 � 103 to 100 � 103 in thispaper. Obviously, as Λ tð Þ is increased, both of energy con-sumption and performance degradation are also increased.

ogeneous (a) and homogeneous power efficiency (b).

Fig. 10. Active server ratio of cloud datacenters including servers with heterogeneous (a) and homogeneous power efficiency (b).

Fig. 11. Performance degradation of CPU contention by co-located VM requests in Server-1 (a), 2 (b), and 3 (c).

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240238

In perspective of resource utilization, we only consider theresource component rc¼ rcCPU and ignore the resourcecomponent rcmem since the energy consumption and per-formance degradation caused by rcmem is negligible com-pared to ones by rcCPU . We use the real traces of CPU

resource utilization measured by the monitoring modulewith serveral benchmark applications on the physicalmachines. Fig. 8 shows the CPU resource utilization ofrunning benchmarks including a mixture of pbzip2,iozone3, netperf on VM instances.

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240 239

4.5. GA Parameters for EEE-VMA approach

We consider a population size ps with a range from 102

to 104, the max step of generation gMax in the range of100 to 1000, and the mutation probability 0.001, 0.005 and0.01 in the Algorithm 1 and 2. As the parameters such psand gMax are increased, the performance of the derivedsolution is increased, but its computation need is alsodeteriorated.

4.6. Traditional resource management schemes

To demonstrate that our proposed approach outper-forms existing resource management schemes, we com-pare the EEE-VMA approach to both of VM consolidationand VM balancing based allocation approaches. The VMconsolidation approach tries to pack multiple VM requestsas many as possible in the common physical server. Thisscheme tends to reduce the number of active servers.Therefore, the energy saving performance is increased,while the performance degradation is deteriorated. Incontrast, the VM balancing approach splits VM requestsover multiple cloud datacenters. This scheme avoids theperformance degradation of resource contention by VMrequest co-location, but causes the large energy con-sumption due to a lot of active servers.

Figs. 9, 10, and 11 show the performance of our pro-posed EEE-VMA approach and existing VM balancing andconsolidation approaches at ps¼ 102, gMax¼ 500, andmutation probability is 0.001. Fig. 9 shows the total cost inEq. (13) of the VM balancing, VM consolidation and ourproposed EEE-VMA approach at different workload offeredload level. Fig. 9(a) shows the curves of total cost of all theapproaches assuming that each cloud datacenter has het-erogeneous power efficiency. Our proposed approachachieves the improvements of the cost saving performanceabout 8% and 53% compared to VM consolidation and VMbalancing approaches, respectively. However, the differ-ence of the cost saving performance between traditionalapproaches and our EEE-VMA approach in Fig. 9(b) assuming that each cloud datacenter has homogeneouspower efficiency is relatively small compared to the one inFig. 9(a). The EEE-VMA approach achieves the improve-ments of the cost saving performance about 10% and 15%compared to VM consolidation and VM balancingapproaches, respectively. Note that our EEE-VMA approachfurther improves the performance of energy saving in theheterogeneous cloud datacenters since it uses powerMarkvalue which can rank the power efficiency of each clouddatacenter to maximize the energy efficiency of resourceallocation. However, our proposed approach still has thebetter performance than ones of existing approaches evenunder the assumption of homogeneous power efficiency ofeach cloud datacenter. Fig. 10 shows the active server ratioof cloud datacenters by our EEE-VMA approach andexisting resource management approaches. In Fig. 10(a),the average active server ratio of the EEE-VMA approach isunder 30%, while the ones of VM balancing is closed to60%. Our EEE-VMA approach considers both of energyconsumption and performance degradation of VMrequests, while the VM balancing only focuses on the

performance degradation. Note that the energy savingperformance of VM consolidation is worse than the one ofthe EEE-VMA approach even though the VM consolidationfocuses on the energy consumption of cloud datacenters.This is because our EEE-VMA approach allocates VMrequests to power efficient cloud datacenters pre-ferentially based on their powerMark values, while the VMconsolidation randomly assigns VM requests to clouddatacenters. In Fig. 10(b), the active server ratio of VMconsolidation is lower than the one of our EEE-VMAapproach, this is because the VM consolidation approachonly focuses on the energy consumption of cloud data-center, but the EEE-VMA approach avoids unacceptableperformance degradation of running VM requests throughEq. (11).

Fig. 11 shows the performance degradation of allocatedVM requests in each cloud datacenter by the EEE-VMA, VMconsolidation, and VM balancing approaches based on theserver types. The performance degradation is calculated byEq. (6). In the perspective of performance degradation, theVM balancing approach outperforms the others includingour proposed EEE-VMA approach. The VM balancing triesto spread submitted VM requests over whole cloud data-centers as fair as possible, therefore the CPU resourcecontention of co-located VM requests can be minimized. InServer-1 type, the performance degradation of VM balan-cing is lower than the ones of the EEE-VMA approach andthe VM consolidation by 40% and 60%, respectively. InServer-2 type, the performance degradation of VM balan-cing is lower than ones of the EEE-VMA approach and theVM consolidation by 55% and 62%, respectively. Finally, inServer-3 type, the VM balancing approach can improve theperformance degradation about 39% and 55% compared tothe EEE-VMA approach and the VM consolidation,respectively.

5. Conclusions

In this paper, we introduced the EEE-VMA approach forgreening cloud datacenters with renewable energy gen-erators. We proposed a novel energy efficient metricpowerMark to classify the power efficiency of hetero-geneous servers in cloud datacenters and built a con-siderate cost model considering switching overheads inorder to reduce efficiently the energy consumption ofservers without a significant performance degradation byco-located VM requests and DRS execution. We deployedthe iterative roulette-wheel algorithm for GA of the EEE-VMA approach in order to solve the complex objectivefunction of our cost model. Through various experimentalresults based on simulation and Openstack platform justifythat our proposed algorithms are supposed to be deployedfor prevalent cloud data centers. In the perspective of totalcost, our EEE-VMA approach can improve the average costby 28% compared to existing resource managementschemes at all the workload level. With the increase of thecomputation investment for GA in EEE-VMA approach, ourproposed approach can get arbitrarily close to theoptimal value.

Y. Peng et al. / Optical Switching and Networking 23 (2017) 225–240240

Acknowledgments

This work was supported by 'The Cross-Ministry GigaKOREA Project' of the Ministry of Science, ICT and FuturePlanning, Korea [GK13P0100, Development of Tele-Experience Service SW Platform based on Giga Media].

References

[1] J. Hamilton, Cost of Power in Large-Scale Data Centers, Nov. 2009.Available Online: ⟨http://perspectives.mvdirona.com/⟩.

[2] J. Koomey, Growth in Data Center Electricity Use 2005–2010, Ana-lytics Press, Burlingame, CA, USA, 2011.

[3] M. Lin, A. Wierman, L. Lachlan, H. Andrew, E. Thereska, Dynamicright-sizing for power-proportional data centers, IEEE/ACM Trans.Netw. 21 (5) (2013) 1378–1391.

[4] L.A. Barroso, U. Holzle, The case for energy-proportional computing,Computer 40 (12) (2007) 33–37.

[5] T. Lu, M. Chen, L. Lachlan, H. Andrew, Simple and effective dynamicprovisioning for power-proportional data centers, IEEE Trans. Par-allel Distrib. Syst. 24 (6) (2013) 1161–1171.

[6] Z. Ou, H. Zhuang, J. K. Nurminen, A. Yla-Jaaski, and P. Hui, Exploitinghardware heterogeneity within the same instance type of AmazonEC2, In: Proceedings of the 4th USENIX Workshop on HotCloud,2012.

[7] Z. Ou, H. Zhuang, A. Lukyanenko, J.K. Nurminen, P. Hui, V. Mazalov,A. Yla-Jaaski, Is the same instance type created equal? Exploitingheterogeneity of public clouds, IEEE Trans. Cloud Comput. 1 (2)(2013) 201–214.

[8] A. Beloglazov, R. Buyya, Optimal online deterministic algorithms andadaptive heuristics for energy and performance efficient dynamicconsolidation of virtual machines in cloud data centers, Concurr.Comput.: Pract. Exp. 24 (2012) 1397–1420, http://dx.doi.org/10.1002/cpe.1867.

[9] Z. Liu, Y. Chen, C. Bash, A. Wierman, D. Gmach, Z. Wang, M. Marwah,C. Hyser, Renewable and cooling aware workload management forsustainable data centers, In: Proceedings of the ACM SIGMETRICS,London, UK, 2012.

[10] Y. Guo, Y. Fang, Electricity cost saving strategy in data centers byusing energy storage, IEEE Trans. Parallel Distrib. Syst. 24 (6) (2013)1149–1160.

[11] Y. Guo, Y. Gong, Y. Fang, P.P. Khargonekar, X. Geng, Energy andnetwork aware workload management for sustainable data centerswith thermal storage, IEEE Trans. Parallel Distrib. Syst. 25 (8) (2014)2030–2042.

[12] F. Xu, F. Liu, L. Liu, H. Jin, B. Li, B. Li, iAware: making live migration ofvirtual machines interference-aware in the cloud, IEEE Trans. Com-put. 63 (12) (2014) 3012–3025.

[13] X. Wang, B. Li, B. Liang, Dominant Resource Fainess in Cloud Com-puting Systems with Heterogeneous Servers, In: Proceedings of theIEEE INFOCOM, Toronto, Canada, 2014.

[14] Montage, ⟨http://montage.ipack.caltech.edu/⟩.[15] S.K. Garg, R. Buyya, Green cloud computing and environmental

sustainability, in: S.M. A. G. G. (Ed.), Harnessing Green IT: Principlesand Practices, Wiley Press, UK, 2012, pp. 315–340.

[16] I. Goiri, M.E. Haquc, K. Le, R. Beauchea, T.D. Nguyen, J. Guitart,J. Torres, R. Bianchini, Matching renewable energy supply anddemand in green datacenters, Elsevier Ad Hoc Netw. 25 (2015)520–534.

[17] C. Wu, H. Mohsenian-Rad, J. Huang, A. Y. Wang, Demand sidemanagement for wind power integration in microgrid usingdynamic potential game theory, In: Proceedings of the IEEE CLO-BECOM Workshop on Smart Grid Communications and Networking,Houston, Tx, Dec. 2001.

[18] Solar Anywhere, Solar anywhere overview, Clean Power Research,Web. 17 Apr. 2012. ⟨http://www.solaranywhere.com/Public/Overview.aspx⟩.

[19] R. Huang, T. Huang, R. Gadh, Solar Generation Prediction using theARMA Model in a Laboratory-level Micro-grid, In: Proceedings of theIEEE SmartGridComm Symposium, Tainan, Taiwan, Nov. 2012.

[20] Openstack, ⟨http://www.openstack.org/⟩.[21] YOCTO-WATT, ⟨http://www.yoctopuce.com/EN/products/usb-elec

trical-sensors/yocto-watt⟩.[22] D. Gupta, L. Cherkasove, R. Gardner, A. Vahdata, Enforcing perfor-

mance isolation across virtual machines in Xen, In: Proceedings ofthe ACM/IFIP/USENIX 2006 International Conference on Middle-ware, Nov. 2006.

[23] A. Qureshi, R. Weber, H. Balakrishnan, J. Guttag, B. Maggs, Cuttingthe electric bill for internet-scale systems, In: Proceedings of theACM SIGCOMM Computer Communications Review, vol. 39(4), Aug.2009, pp. 123–134.

[24] T. Wood, P. Shenoy, A. Venkataramani, M. Yousif, Sandpiper: black-box and gray-box resource management for virtual machines,Elsevier Comput. Netw. 53 (17) (2009) 2923–2938.

[25] Credit Scheduler, ⟨http://wiki.xen.org/wiki/Credit_scheduler⟩.[26] C. Ren, D. Wang, B. Urgaonkar, A. Sivasubramaniam, Carbon-aware

energy capacity planning for datacenters, In: Proceedings of the IEEEMASCOTS, 2012, pp. 391–400.

[27] Apple Environmental Responsibility, ⟨http://www.apple.com/environment.renewable-resources/⟩.

[28] H. Fawaz, Y. Peng, C. Youn, A MISO mode for power consumption invirtualized servers, Clust. Comput. 18 (2) (2015) 847–863.

[29] Amazon Web Services, ⟨https://aws.amazon.com⟩.[30] E.D. Dasgupta, Z. Michalewicz, Evolutionary Algorithms in Engi-

neering Applications, Springer, Berlin, Germany, 1997.[31] M. Chen, Y. Wen, H. Jin, V. Leuna, Enaling technologies for future

data center networking: a primer, IEEE Netw. 27 (4) (2013) 8–15.[32] M. Chen, Y. Zhana, L. Hu, T. Taleb, Z. Shena, Cloud-based wireless

network: virtualized, reconfigurable, smart wireless network toenable 5G technologies, ACM/Springer Mob. Netw. Appl. 20 (6)(2015) 704–712.

[33] Measurement and Instrumentation Data Center (MIDC), ⟨http://www.nrel.gov/midc/⟩.

[34] C. Wu, H. Mohsenian-Rad, J. Huang, A.Y. Wang, Demand side man-agement for wind power integration in microgrid using dynamicpotential game theory, In: Proceedings of the IEEE GLOBECOMWorkshop on Smart Grid Communications and Networking, Hous-ton, Tx, Dec. 2001.

[35] H. Mohsenian-Rad, A. Leon-Garcia, Optimal residential load controlwith price prediction in real-time electricity pricing environments,IEEE Trans. Smart Grid 1 (2) (2010) 120–133.