Future Generation Computer Systems ( ) –

Contents lists available at ScienceDirect

Future Generation Computer Systems

journal homepage: www.elsevier.com/locate/fgcs

Energy accounting for shared virtualized environments under DVFS usingPMC-based power modelsRamon Bertran a,∗, Yolanda Becerra a,b, David Carrera a,b, Vicenç Beltran a, Marc Gonzàlez a,b,Xavier Martorell a, Nacho Navarro a,b, Jordi Torres a, Eduard Ayguadé a

a Barcelona Supercomputing Center, C. Jordi Girona 1-3, 08034 Barcelona, Spainb Universitat Politècnica de Catalunya (UPC), Campus Nord UPC D6 Building, C.Jordi Girona 1-3, 08034, Spain

a r t i c l e i n f o

Article history:Received 28 October 2010Received in revised form8 February 2011Accepted 8 March 2011Available online xxxx

Keywords:Modeling techniquesEnergy accountingPerformance countersPower modelingVirtualizationDVFS

a b s t r a c t

Virtualized infrastructure providers demand new methods to increase the accuracy of the accountingmodels used to charge their customers. Future data centers will be composed of many-core systems thatwill host a large number of virtual machines (VMs) each. While resource utilization accounting can beachieved with existing system tools, energy accounting is a complex task when per-VM granularity is thegoal.

In this paper, we propose a methodology that brings new opportunities to energy accounting byadding an unprecedented degree of accuracy on the per-VM measurements. We present a system —which leverages CPU andmemory power models based in performance monitoring counters (PMCs) — toperform energy accounting in virtualized systems. The contribution of this paper is threefold. First, weshow that PMC-based power modeling methods are still valid on virtualized environments. Second, weshow that the Dynamic Voltage and Frequency Scaling (DVFS) mechanism, which commonly is used byinfrastructure providers to avoid power and thermal emergencies, does not affect the accuracy of themodels. And third,we introduce a novelmethodology for accounting of energy consumption in virtualizedsystems. Accounting is done on a per-VM basis, even in the case where multiple VMs are deployed ontop of the same physical hardware, bypassing the limitations of per-server aggregated power metering.Overall, the results for an Intel r⃝ CoreTM 2 Duo show errors in energy estimations <5%. Such an approachbrings flexibility to the chargeback models used by service and infrastructure providers. For instance, weare able to detect cases where VMs executed during the same amount of time, present more than 20%differences in energy consumption even only taking into account the consumption of the CPU and thememory.

© 2011 Elsevier B.V. All rights reserved.

1. Introduction

Advanced service data centers leverage virtualization tech-nologies to run applications from multiple customers in sharedenvironments. Initiatives such as the Amazon EC2 platformhave brought to reality the creation of large pools of resourceswhere end users deploy applications and services. Infrastructureproviders manage user applications transparently thanks to virtu-alization operations, and due to the benefits of server and work-load consolidation, they offer affordable and virtually unlimited

∗ Corresponding address: Despatx 102 K2M-Planta 1, Campus Nord, BarcelonaSupercomputing Center, C. Jordi Girona 1-3, 08034 Barcelona, Spain. Tel.: +34 934137245; fax: +34 93 4017055.

E-mail addresses: rbertran@ac.upc.edu, ramon.bertran@bsc.es (R. Bertran),yolandab@ac.upc.edu (Y. Becerra), dcarrera@ac.upc.edu (D. Carrera),vincenc.beltran@bsc.es (V. Beltran), marc@ac.upc.edu (M. Gonzàlez),xavim@ac.upc.edu (X. Martorell), nacho@ac.upc.edu (N. Navarro),torres@ac.upc.edu (J. Torres), eduard@ac.upc.edu (E. Ayguadé).

computing resources to bulk customers. Moreover, the customersbenefit from avoiding the direct maintenance costs of a large ITinfrastructure.

Power and energy consumption plays an important role in themaintenance cost of a large data center. The energy and power con-sumption includes the consumption of computational resourcessuch as processors, memories, storage and networking resources,and also the consumption of other required infrastructures, suchas AC facilities. As a result, this power and energy consumptionaffects the pricing policies that infrastructure providers charge totheir costumers.

Infrastructure providers usually offer different pricing plans totheir customers. For instance, there exists plans based on resourceutilization — also named pay as you go — and plans based onresource reservation, in which the customers pays a fixed amountindependently of the resource usage they do. Table 1 summarizesthe methods used by different providers to account for CPU andmemory resources. As can be seen, different service providers

0167-739X/$ – see front matter© 2011 Elsevier B.V. All rights reserved.doi:10.1016/j.future.2011.03.007

2 R. Bertran et al. / Future Generation Computer Systems ( ) –

Table 1Accounting methods and granularity used by infrastructure providers to chargetheir customers.

Provider Accounting method

Amazon Time VM deployed in hoursRackspace Time VM deployed in hoursGoogle app engine CPU cycles usedWindows azure CPU cycles usedJoyent Time VM deployed in daysGogrid Time VM deployed in hoursTerremark Time VM deployed in hoursVerizon Time VM deployed in hours

use different granularity and accounting methods for charging thecustomers but all of them use time as the end billing unit to thecustomer since is a metric in which end users can estimate theirrequirements. Exposing other metrics such as power or energy asthe billing unit to the customers would make difficult to themestimate their requirements, or impossible given the fact that theirdo not know the underlying infrastructure.

Nevertheless, there is a need for more granular and detailedtracking of resource costs, even this granularity is not passedthrough to the customer as a billing unit. It is necessary for theservice provider to understand in detail their own cost structurebecause they decide their pricing policies based on such costs. Atthe end, they charge customers based on fixed costs ratios as wellas following the usage that they do of the computing resources.

Therefore, detailed measurement of resource consumption isa critical point for shared data centers. Accounting of resourceutilization must be done in a per-virtual machine (VM) basis, evenwhen multiple VMs are deployed on top of the same physicalhardware. While measuring CPU, storage and network utilizationis feasible with existing system tools, per-VM energy consumptioncan be hardly estimated in resource sharing environments. Powermetering devices measure aggregate power consumption of thewhole system, but detailed per-VM power-related metrics canonly be measured with advanced techniques. The new many-core systems will take this limitation even further, envisioningenvironments in which large number of VMs will be deployed ontop of tenths of cores in a single server.

As stated, there aremethods to gather information about powerconsumption. However, the need to estimate and predict powerconsumption in the case where measurement devices are notapplicable, has justified the research onpowermodeling strategies.One strategy is to use the performancemonitoring counters (PMCs)present in all current processors as proxies to estimate power.

Methods based PMCs have been shown to be a good solutionto estimate power consumption. Their applicability has beendemonstrated on several fields such as power management andapplication profiling. For instance, they can be used to performlive predictions of power behavior in order to guide power awarepolicies and mechanisms such as DVFS (Dynamic Voltage andFrequency Scaling) [1–4]. In the end, they have been crucial in theprocess of addressing power and energy management issues.

In this paper we join the virtualization technology with PMC-based power modeling techniques to derive energy consumptionestimates of the CPU and memory at per-VM level. Furthermore,we also analyze the effects of DVFS on such technologies. Themaincontributions of this work are three-fold:

• We apply already proposed PMC-based power modelingtechniques [5] in virtualized environments for different DVFSlevels. We produce single core and dual core models for anIntel r⃝ CoreTM 2 Duo platform under 4 different DVFS. Themodels are validated using the entire SPECcpu2006 suiteexecuted within virtualized environment for the differentfrequencies. We obtain average errors below 3%, which letsus conclude that the virtualization layer does not introduce

noticeable interferences in power predictions and that the PMC-based power models are still valid on such environments.

• We propose a simple methodology to perform per-VM energyaccounting on current multicore processors which leveragesthe PMC-based power models. The results show that theapproach is able to estimate, with less than 5% error, the energyconsumption of each VM in the system, even in the case wherethe system is stressed andmultiple VMs aremultiplexed on topof the same physical core.

• Finally, we present a case of applicability: the quantification ofthe biases in energy consumption of different VMs. We are ableto detect cases where VMs executing the same amount of CPUtime, present 20% variations in energy consumption.

In overall, the novel methodology introduced provides accurateper-VM energy accounting. This approach increments the granu-larity at which service and infrastructure providers can evaluatetheir operation costs. At the same time, it enables the explorationof chargeback models with a more fair mapping of the operationalcosts to customers because with this technique it is possible toestimate per customer CPU and memory energy consumption.

The rest of this paper is organized as follows: Section 2 des-cribes the power modeling and energy accounting methodology.Section 3 validates the methodology in virtualized systems anddemonstrates how this technology can be used to derive energyconsumption estimates. Section 4 describes the related work.Finally, Section 5 concludes the paper.

2. Methodology

This section describes the details of the methodology usedto perform the experiments. First, we present the experimentalenvironment,which in our case is an Intel r⃝ CoreTM 2Duoplatform.Then, we explain the details of the DVFS mechanism available inour platform. Next, we justify the methodology used for gatheringthe power measurements and the PMC traces of the workloads.Finally, we present the details of the virtualization environment.

Once the experimental setup is explained, an overview of thetechnique used to produce the power models is presented. Andfinally, this section ends with the explanation of the methodologyto perform the per-VM energy accounting and a summary of theexperiments performed.

It should be noted that even though the study presented isfor a particular architecture, the same approach can be applicableto other architectures in which PMC-models and virtualizationtechniques are applicable.

2.1. Experimental environment

Our experimental framework is similar to the ones used inprevious works [5,6]. We ran all the experiments on a workstationthat features an Intel r⃝ CoreTM 2 Duo T9300 processor [7] andtwo modules of 1 GB of memory. The embedded controllerfirmware of the platform provides information about the powersource, which fulfills the SMAPI specification [8]. This specificationdefines an interface to obtain power consumption measurementswith a guaranteed granularity and accuracy. We gather powermeasurements in a granularity ofmilliwattswith amaximumerrorof 2%. For correctness, the readings were also validated against themeasurements of a Watts Up.Net [9] power meter.

The platform ran a Linux kernel 2.6.28 with virtualizationsupport enabled (see Section 2.4) and the required patches to allowPMC readings. The tp_smapi [10] module was loaded in order tobe able to gather the power measurements. This module createsa set of entries in the sysfs filesystem that provide informationsuch as the current power consumption.

R. Bertran et al. / Future Generation Computer Systems ( ) – 3

Table 2Frequencies of the Intel r⃝ CoreTM 2 Duo T9300 processor evaluated.

Frequency (MHz) Bus freq. (MHz) Bus multiplier

2500 200 12.52000 200 101600 200 8800 100 8

The system ran the minimum required system processesto minimize possible interferences in power consumption andPMC readings. Moreover, we switched off all sources of powerconsumption — i.e. the display — that we did not want to interfereto the power model generation. For platform components whereit was not possible to switch off, we configured them at constantoperation mode — i.e. the CPU fan or the disk.

In order to validate that we nullified eventual power interfer-ences, we tracked the power consumption during 2min before thestart of each experiment. The fact that we found a fairly constantidle power consumption confirmed that we minimized the powerinterferences.

For the purpose of the model formation only, we assumed thatidle power consumption, 18.2 watts, to be the power consumptionof the platform except the processor and the memory. Thismethodology was used and validated previously in other PMC-based power modeling works [2,5].

2.2. Dynamic voltage frequency scaling

The experimental platform implements DVFS1. DVFS is a mech-anism that allows to modify the processor operational frequencyand voltage, so that it is possible to dynamically adapt the pro-cessor operating point to a given performance, power or energyrequirements. The flexibility of this mechanism is used by infras-tructure providers to avoid power or thermal emergencies, or toreduce operational costs while maintaining the quality of service.

Concretely, the frequency and voltage ranges are [0.800,2.500]GHz and [0.75,1.25]2 volts respectively. The core frequency is amultiple of the external (bus) clock speed, which in our system is200Mhz. Frequencies below1.600GHz are achieved by halving theexternal (bus) clock speed to 100MHz, affecting the access tomainmemory.

Table 2 summarizes the frequencies studied in this work.Although the platform supports 18DVFS states in steps of 100MHz,we only studied the maximum (2.5 GHz) and minimum (0.8 GHz)levels plus two intermediate ones (2 GHz and 1.6 GHz). Selectingsuch frequencies allowedus to cover the entireDVFS spacewithoutenlarging excessively the experimentation time.

We used the CPUfreq [12] interface to configure the currentDVFS in our platform.We configuredCPUfreq to use the userspacegovernor. This governor allows the user, or any userspace programrunning with enough privileges, to set the processor to a specificfrequency by making a file named scaling_setspeed available inthe sysfs filesystem. In each experiment, we set up the requiredfrequency by writing it to the scaling_setspeed file.

2.3. Power metering and data gathering

We used the pfmon [13] utility to access the performancecounters. This tool allows users to perform system-wide or per-process PMC readings. Moreover, in pfmon an interval time can

1 Also known as Enhanced Intel r⃝ SpeedStep Technology.2 Each processor is programmed with a maximum valid voltage identification

value (VID), which is set at manufacturing and cannot be altered. IndividualmaximumVID values are calibrated duringmanufacturing such that two processorsat the same frequency may have different settings within the VID range [11].

be configured to print intermediate deltas of the PMCs counts,resulting in a PMCs trace. However, we needed to correlate thePMC reading traces with trace of actual power measurements forcorrectness.

For that purpose, we modified pfmon to access to the PMCsand current power consumption simultaneously by reading fromthe interface created by the tp_smapi module explained inSection 2.1. By using this version of pfmon, we were able togenerate correlated PMCs and power traces.

For validating the models generated, we configured pfmonin system-wide mode, retrieving PMCs counters and currentpower consumption every 2 s. Whereas for performing per-VMaccounting, we used the per-process mode (specifying the PID3

to follow), retrieving PMCs counters and last minute averagepower consumption information (which is also provided by thetp_smapi module) every 60 s.

In both cases, at the end of the experiments we obtainedtwo traces of information: one containing the PMC readings andthe other one containing the real power consumption. Then,these traces were processed offline to generate the power models(see Section 2.5) and to perform the energy accounting (seeSection 2.6). All these procedures are validated using the realpower consumption traces in the validation and evaluation section(see Section 3).

2.4. Virtualization environment

Since one of the purposes of this work is to quantify theeffects of the virtualization layer on the power estimations, weran the experiments on a non-virtualized environment and into avirtualized one. This allows the direct comparison and evaluationpresented in Section 3.1.

The two environments are depicted in Fig. 1. On the left,Fig. 1(a) shows the non-virtualized environment. In this case, theapplications run on the native operating system (OS) and pfmongathers information about their activity in the processor. This isthe common scenario in which PMC modeling techniques havebeen validated and used extensively, which is not the case of thescenario shown in Fig. 1(b).

Fig. 1(b) shows the virtualized environment. This environmentintroduces the extra layer of virtualization represented by theVMM (Virtual Machine Monitor). Concretely, in Fig. 1(b) scenariotwo VMs coexist, and within every VM two applications arerunning on top of virtual hardware. In this case, pfmon (executedon the VMM) gathers information about the activity in the entireVMs. One of the main objectives of this paper is to confirmthat PMC-based power models are still valid in this scenario. Wevalidate this hypothesis in Section 3.1, before using them as amechanism to account per-VM energy consumption.

We used QEMU 0.10.0 [14] in conjunction with KVM [15]as the virtualization infrastructure. QEMU is a generic and opensource machine emulator and virtualizer. QEMU, when used as avirtualizer, achieves near native performances by executing theguest code directly on the host CPU. However, for that purposeit needs KVM. KVM (Kernel Virtual Machine) is a Linux kernelmodule that allows a user-space program to utilize the hardwarevirtualization features of the processor.

We used libvirt 0.6.1 [16] to interact with the virtualizationcapabilities of our host system and to manage the VMs. We set upa set of VMs which feature a Linux kernel 2.6.28 and 512 MB ofmemory. The number of total virtual machines deployed and thenumber virtual cores per VM varies depending on the experimentperformed. At the end of this section, Table 4 summarizes all theconfigurations used for all the experiments.

3 PID stands for ‘Process IDentifier’, which is a number used by the operatingsystem to uniquely identify a process.

4 R. Bertran et al. / Future Generation Computer Systems ( ) –

a b

Fig. 1. Non-Virtualized and Virtualized system. On the left, the non-virtualized environment in which pfmon is attached directly to the applications. On the right, the samesystem with a virtualization layer (VMM) in which pfmon is attached to the process that represent the entire VM.

Table 3Coefficients of the power models generated for our experimental architecture.

Frequency(MHz) FEcoef INTcoef FPcoef BPUcoef L1coef L2coef MEMcoef Const1core Const2cores

2500 1092.40 333.10 1348.42 4526.15 1078.70 36108.81 23139.77 8339 59192000 581.14 233.84 818.44 1851.14 706.15 19679.74 18358.84 7024 52621600 344.77 119.47 240.17 1041.39 401.59 12639.42 16665.68 5986 4343800 92.25 127.14 88.31 446.00 100.48 3725.81 14258.04 3328 2139

2.5. Power modeling

Our methodology for producing the power model follows thewell-known modeling steps performed in previous works [17,5].Concretely, we followed the same methodology already validatedin [5]. The methodology ensures the generation of responsive,accurate, decomposable and benchmark-independent PMC-basedpowermodels. The outcome of that methodology is a formula like:

Power = f (PMC0, PMC1, . . . , PMCn−1, PMCn)

that predicts power consumption of the CPU and thememory froma set of performancemonitoring counters. For the sake of simplicityand because the objective of this work is not to present PMC-basedmodeling techniques, we only summarize the main methodologysteps performed:

• Define power model inputs from microarchitectural compo-nents. They are PMC-based formulas that represent ratios ofactivity of microarchitectural components.

• Design a specific set of microbenchmarks that stress each of theinputs defined for a wide set of situations.

• Execute the microbenchmarks in order to gather the requireddata to train the model.

• Produce the model by applying incrementally linear regressiontechniques on the training data for deriving theweight in powerconsumption of each power component defined.

First we did the first two steps and then we repeated the lasttwo ones for each of the frequencies studied. At the end of theprocess, we obtained a model for each of the frequencies. Forinstance, the model obtained for our architecture when runningat maximum frequency, 2.5 GHz, is the following:

Power = 1092 × ARFe + 333 × ARInt + 4526 × ARBpu

+ 1348 × ARFp + 1078 × ARL1 + 36108 × ARL2

+ 23139 × ARMem +

8339 if 1 core enabled5919 if 2 cores enabled

where each ARi is a PMC-based formula accounting for the activityratio of the i component. For instance, the ARL1 is the activity ratio

of the first level cache, and its translation to a PMC-based formula isthe number of L1 accesses per cycle4. All the models generated areshown in Table 3. Please refer to the related work [5] for a moredetailed description of the power components, the performancecounters selection and the model generation.

2.6. Per virtual machine energy accounting

In order to perform per-VM energy accounting, we profitedfrom the abstraction provided by the OS in our experimentalenvironment. From the host OS, each VM is seen as an independentprocess. In the end, this abstraction in conjunction with thecapability of pfmon to gather per process PMCs allowed us to trackthe activity (generating PMC traces) of eachVMrunning on thehostOS.

From each of the activity traces of each VM, we estimated theirpower consumption using the power models produced. Then, wederived the energy by multiplying the predicted power by theexecution time, which is present in the PMC traces as cpu cycles.

However, when performing per-VM accounting, it is notpossible to validate the estimations against empirical data becausethe platform only provides aggregated power readings. Hence, forvalidation purposes, we assume that the sum of the predictedenergy consumption of each virtual machine running on thesystem should be the same as the overall platform’s energyconsumption. In Section 3.2, we will see this assumption to bevalid.

2.7. Experiments

Before moving to the validation and evaluation section, werecap the steps and experiments performed. Table 4 summarizesthem. The results of experiments 1–4, in a non-virtualizedenvironment, are required for generating the PMC-based power

4 The rest of activity ratios represent the following:ARFe: Front-End,ARInt : Integerunits,ARBpu: Branch predictor unit, ARFp: Floating point units, ARL2: level 2 cache andARMem: main memory.

R. Bertran et al. / Future Generation Computer Systems ( ) – 5

Table 4Experiments executed for each of the frequencies studied. The columns denote the environment, the benchmarks, the number of CPUs enabled, the number of VMs, thenumber of VCPUs per VM and the tracing configuration of pfmon in conjunction with the purpose of the experiment.

# Environment Benchmarks #CPU #VM #VCPU Tracing Purpose

1 Non virtualized micro-benchmark 1 – – system-wide/2 s Power model generation2 Non virtualized SPECcpu2006 1 – – system-wide/2 s Power model validation3 Non virtualized micro-benchmark 2 – – system-wide/2 s Power model generation4 Non virtualized SPECcpu2006 subset 2 – – system-wide/2 s Power model validation5 Virtualized SPECcpu2006 1 1 1 system-wide/2 s Power model validation6 Virtualized SPECcpu2006 subset 2 1 2 system-wide/2 s Power model validation7 Virtualized SPECcpu2006 subset 2 1 2 process/60 s VM energy accounting8 Virtualized SPECcpu2006 subset 2 2 2 process/60 s VM energy accounting9 Virtualized SPECcpu2006 subset 2 3 2 process/60 s VM energy accounting

10 Virtualized SPECcpu2006 subset 2 4 2 process/60 s VM energy accounting11 Virtualized SPECcpu2006 subset 2 5 2 process/60 s VM energy accounting

Fig. 2. Comparison of average percentage error of the estimations for the single-core models in virtualized and non-virtualized environments.

models and validate them. Note that the number of physicalcores enabled (column 4 in the table) varies depending on theexperiment executed.

Then, the results of experiments 5 and 6 are used to validatethat PMC-based power models are still valid on virtualizedenvironments. The rest of experiments stress our proposed energyaccounting methodology by deploying up to 5 VMs with 2 virtualCPUs (VCPUs) each one on our dual-core platform. The same setof experiments were repeated for each of the four frequenciesstudied.

The applications executed for gathering data to generatethe power models are a set of microbenchmarks defined inthe second step of the power modeling methodology (seeSection 2.5). Conversely, for validating and performing the energyaccounting experiments, we used the SPECcpu2006 benchmarksuite [18]. This suite is designed to stress processor and memoryperformance, which is what our models are designed to accountfor.

3. Validation and evaluation

The model and methodology evaluation is organized in threemain sections. The first one covers the power model validation invirtualized systems for a single-core and dual-core configurationsfor the four frequencies studied. In both cases, the average percenterror of the power predictions of the power models when appliedon a virtualized environment remains low (below 3% in single-coremodels and dual-coremodels) for all the frequencies. This factcorroborates that the power model generated in a non-virtualizedenvironments is still valid in a virtualized one.

Then, we validate the energy accounting methodology pre-sented in Section 2.6 showing that the aggregated energy ac-counted for each VM present on the system matches with theoverall platform energy consumption. From that, we conclude thatPMC-based power models — which allow for energy accounting at

process level — can be applied to perform energy accounting at VMlevel.

The last part of this section discusses an applicability case. Weshow that VMs that run during the same amount of CPU cycles donot consume the same amount of energy. Hence, this methodologycan be used to introduce new chargeback models that take intoconsideration fine grained energy consumption measurements.

3.1. Power model validation

This section addresses the validation of the power model invirtualized environments. For this purpose, we will compare theaverage error of the model in both virtualized and non-virtualizedsystems. In the non-virtualized system, applications run on topof the native OS and hardware whereas in virtualized tests,applications run within a VM.

3.1.1. One core validationTable 5 summarizes the results on a single-core configuration

for each tested application in both non-virtualized and virtualizedsystems for all the frequencies analyzed. Columns labeled ‘%error ’and ‘std’ correspond to the average percentage error and standarddeviation that the model incurs respectively. The average errornever exceeds 6% for all the configurations studied, and the averageerror of virtualized experiments never exceeds 3%.

Non-virtualized experiments exhibit higher average (2–3 more% points) errors than the virtualized ones. The reason is two-fold. On one side, the interference of the VMM layer introducesa constant activity across benchmarks reducing the variabilityamong them. This can be shown by analyzing the standarddeviation. And on the other side, this activity removes specialpathological cases, such as the 470.lbm.

Graphically, this can be observed in Fig. 2. In this figure wecompare the average error for every application in virtualized and

6 R. Bertran et al. / Future Generation Computer Systems ( ) –

Table 5Average error and standard deviation of the predictions generated by the single-core power models for the entire SPECcpu2006 suite. Results are given by frequency andper type of system: non-virtualized and virtualized.

Benchmark Frequency0.8 GHz 1.6 GHz 2 GHz 2.5 GHzNo Virtualized Virtualized No Virtualized Virtualized No Virtualized Virtualized No Virtualized Virtualized%err std %err std %err std %err std %err std %err std %err std %err std

400.perlbench 2.49 1.84 1.65 4.58 2.75 0.59 1.14 1.80 4.50 0.71 0.81 1.34 2.12 1.42 2.79 0.92401.bzip2 3.04 1.38 2.66 3.86 0.59 1.36 1.19 2.55 4.31 5.26 2.11 1.60 3.49 1.62 3.53 1.45403.gcc 2.43 1.52 1.71 2.12 1.20 1.92 0.56 0.84 1.61 1.56 1.82 1.06 1.51 1.07 3.96 1.14410.bwaves 2.48 1.34 1.88 3.37 1.63 0.95 1.16 2.37 1.16 0.85 1.20 3.38 0.26 0.38 2.46 5.09416.gamess 5.67 3.02 1.71 2.19 0.52 0.48 1.48 0.87 2.63 0.58 0.64 0.55 3.86 0.34 2.76 0.63429.mcf 10.61 2.76 1.19 2.83 3.77 1.31 0.87 2.45 6.68 4.38 2.07 3.00 6.95 2.54 2.71 4.10433.milc 10.34 2.25 1.53 2.16 3.37 1.38 0.54 0.84 2.41 1.05 0.53 1.05 1.62 0.83 2.01 1.08434.zeusmp 1.15 1.23 1.32 2.44 0.54 0.75 0.71 1.34 0.42 0.55 0.72 0.83 0.45 0.49 2.88 1.37435.gromacs 7.93 2.98 1.72 1.95 6.60 0.61 2.08 1.21 5.19 0.82 1.55 0.89 0.33 0.32 1.22 1.02436.cactusADM 3.39 3.63 1.71 2.78 3.17 1.15 1.05 1.20 3.00 1.31 1.09 0.99 0.81 0.51 1.66 0.87437.leslie3d 7.51 1.62 1.31 1.41 1.41 0.93 0.40 0.86 2.73 0.57 0.99 0.60 1.31 0.51 3.66 0.70444.namd 4.57 1.76 1.41 1.36 2.79 0.84 0.57 0.87 5.82 0.78 0.56 1.32 6.26 0.33 2.62 0.80445.gobmk 0.85 1.68 1.83 4.52 3.38 0.94 1.51 1.54 5.50 0.68 1.04 0.88 0.64 0.59 3.12 0.76450.soplex 8.69 2.04 2.27 4.94 1.65 1.03 1.32 3.04 2.34 0.76 1.10 2.20 0.35 0.32 2.83 1.16453.povray 2.54 0.71 1.20 1.86 0.18 0.90 0.90 0.74 4.06 0.62 0.42 0.90 3.01 0.26 2.42 0.73454.calculix 2.13 1.26 1.75 1.78 2.74 0.71 1.01 1.30 4.88 0.71 0.70 1.12 2.68 0.53 2.56 1.25456.hmmer 1.03 2.32 2.48 2.08 1.64 0.63 2.47 1.22 5.35 0.59 2.11 0.88 3.07 1.88 4.36 0.89458.sjeng 0.77 2.40 1.88 1.85 4.42 0.88 1.28 0.70 7.38 0.74 1.07 0.51 4.72 0.35 3.71 0.72459.GemsFDTD 8.67 1.73 2.19 3.64 1.80 0.87 1.55 2.82 1.69 0.90 1.81 2.68 0.65 0.93 2.71 3.50462.libquantum 5.03 1.55 1.06 2.00 3.97 0.98 0.53 0.82 4.32 0.95 1.50 0.61 8.65 0.93 3.83 0.76464.h264ref 7.59 3.06 1.62 1.88 2.15 0.58 1.11 0.92 0.49 0.57 2.00 0.87 0.96 0.50 3.84 0.91465.tonto 2.16 1.77 1.60 1.93 0.44 0.56 0.65 0.70 2.96 0.80 0.60 0.68 2.94 0.61 3.51 0.63470.lbm 20.81 0.75 2.15 1.64 15.51 0.52 2.63 1.45 19.13 2.51 2.82 1.27 3.41 0.35 0.34 1.11471.omnetpp 6.79 1.35 1.81 1.91 5.45 0.81 1.37 0.72 5.34 0.58 1.35 0.92 3.09 0.33 3.01 1.04473.astar 3.52 4.29 1.85 1.70 4.60 1.80 1.40 0.66 7.66 2.41 1.75 0.78 4.63 2.23 3.79 0.65482.sphinx3 13.17 1.77 1.08 3.42 0.60 0.58 1.16 1.67 3.34 0.98 1.03 1.00 2.89 0.63 1.60 1.17Average 5.59 2.00 1.71 2.55 2.96 0.93 1.18 1.37 4.42 1.24 1.28 1.23 2.71 0.79 2.84 1.32

non-virtualized systems for all the frequencies. The differencesbetween virtualized and non-virtualized results are always below5 points except for the alreadymentioned special cases such as the470.lbm.

Moreover, if we analyze the results in terms of frequency, weconclude that on average the operation frequency does not affectthe validity of the power models. In fact, the average percent errortends to increase slightlywhen lowering the frequency butwithoutcompromising the accuracy because it remains below a reasonable6% error on average.

In the virtualized experiments, we have also measured theresponse of the power model during the VM boot and VMshutdown phases. These results were discarded for computingthe power model average since they include for I/O disk activitythat the power model does not track. Nevertheless, we presentthe prediction of the PMC-based model for an example caseapplication, 454.calculix.

Fig. 3 shows the power evolution for the 454.calculixapplication. The three phases of execution are clear: the VMboot, execution of the application, and the VM shutdown. Asexpected, during the application’s execution the model succeedsin accurately predicting the power consumption. But this is notthe case for the boot and shutdown phases. These two phases areintensive in IO operations that are not captured by the model. Themodel just accounts for activity in the processor andmainmemory.Themodeling of IO operations and other platform elements are outof the scope of this paper and it is one subject of future study.

In conclusion, the power models generated for one core arevalidated in both virtualized and non-virtualized systems, and wecan affirm that the layer corresponding to theVMthat encapsulatesthe test applications does not introduce any significant distortionover the power predictions obtained through themodel.Moreover,we also confirmed that the running DVFS state does not affect thevalidity of the predictions.

0 50 100 150 200 250 300

Sample

Pow

er(m

W)

12000100008000600040002000

0

454.calculix Power Consumption

Real PowerPredicted Power

Fig. 3. Complete real power and predicted power trace for the 454.calculixapplication executed at 2.5 GHz.

3.1.2. Two core validationThis section addresses the validation of the power model for

the entire chip — when both cores are enabled — in virtualizedenvironments. Similarly as in the previous section, we willcompare the average error of the model in both virtualized andnon-virtualized systems.

We have selected a subset of 8 (4 integer and 4 floating point)applications of the SPECcpu2006 benchmarks to evaluate thepower model. The applications have been chosen according to theclassification and characterization of the SPECcpu2006 for multi-programming evaluation [19,20]. We paired them in order to beexecuted concurrently in both scenarios — virtualized and non-virtualized. At the end, we end up with 28 different pairs.

Table 6 describes the data obtained in both scenarios, similarlyas in the case with just one core being active. Both environmentsexpose similar average errors in every case. The [0.91–3.28]%average range of error demonstrates the reliability of the powermodel for both environments—single-core and dual-core.

Again, like in the single-core validation experiments, the non-virtualized ones exhibit higher average errors, but never morethan 2 percentage points. Nevertheless, all the errors remain inreasonable ranges. Actually, the trend is that dual-core modelpredictions are more accurate (in percentage) that single-coreones. There are two reasons for that behavior. First, the absolute

R. Bertran et al. / Future Generation Computer Systems ( ) – 7

Table 6Average error and standard deviation of the predictions generated by the dual-core power models for SPECcpu2006 suite pairs. Results are given by frequency and per typeof system: non-virtualized and virtualized.

Benchmark Frequency0.8 GHz 1.6 GHz 2 GHz 2.5 GHzNoVirtualized

Virtualized NoVirtualized

Virtualized NoVirtualized

Virtualized NoVirtualized

Virtualized

%err std %err std %err std %err std %err std %err std %err std %err std

400.perlbench-403.gcc 0.68 0.68 0.67 1.04 1.69 0.92 0.51 0.82 0.81 0.72 1.88 0.68 6.71 2.12 1.28 0.64400.perlbench-435.gromacs 1.50 0.86 2.47 1.46 0.47 0.21 1.72 1.14 0.13 0.14 0.51 0.52 4.53 2.68 0.70 0.42400.perlbench-436.cactusADM 0.67 0.66 1.95 1.51 0.55 0.84 0.95 1.28 0.89 0.33 0.22 0.51 1.36 0.40 0.33 0.38400.perlbench-450.soplex 6.73 0.86 1.31 2.11 3.79 0.32 0.66 2.00 1.90 0.31 0.82 0.48 0.25 0.28 0.85 0.42400.perlbench-453.povray 0.71 0.50 0.65 1.43 3.88 0.28 0.43 1.67 2.15 0.25 0.66 0.42 1.10 0.23 0.43 0.34400.perlbench-462.libquantum 4.45 0.70 1.09 1.33 0.40 0.31 0.47 1.19 0.66 0.30 1.57 0.51 0.54 0.55 1.51 0.48400.perlbench-473.astar 4.51 3.29 0.87 1.35 4.45 0.43 0.72 1.46 3.15 0.77 1.46 0.41 2.23 1.26 0.70 0.36403.gcc-435.gromacs 1.31 0.79 0.72 1.19 2.83 2.06 1.78 0.89 2.67 1.03 0.34 0.38 0.66 0.80 0.42 0.50403.gcc-436.cactusADM 0.63 0.65 0.67 1.30 1.20 1.18 1.28 0.87 1.15 0.84 0.74 0.47 0.62 0.68 0.37 0.38403.gcc-450.soplex 6.39 1.05 1.11 1.47 1.66 0.72 1.09 2.19 0.60 0.72 1.92 0.63 0.71 0.79 1.55 0.65403.gcc-453.povray 0.63 1.06 1.76 0.97 0.82 1.01 1.03 0.87 0.51 0.64 1.64 0.54 0.69 0.73 0.90 0.69403.gcc-462.libquantum 4.31 0.79 0.70 0.99 2.67 1.14 0.51 1.07 3.30 0.92 2.15 0.61 2.36 1.14 1.82 0.76403.gcc-473.astar 3.92 3.16 1.26 1.01 1.01 0.91 0.34 1.16 0.66 0.77 2.55 0.59 0.93 1.22 1.49 0.68435.gromacs-436.cactusADM 4.09 0.99 3.94 1.71 3.10 0.38 2.90 0.66 2.93 0.36 2.13 0.51 1.81 0.50 1.93 0.45435.gromacs-450.soplex 2.25 1.02 1.83 1.38 0.22 0.22 1.91 0.65 1.64 0.34 1.04 0.51 0.71 0.53 0.76 0.47435.gromacs-453.povray 5.81 0.52 2.64 1.09 2.19 0.77 2.50 0.55 0.93 0.21 1.58 0.57 1.93 0.24 1.71 0.47436.cactusADM-450.soplex 2.99 1.06 3.67 3.67 0.35 0.30 1.81 1.12 1.49 0.38 0.20 0.36 0.24 0.26 0.78 0.35436.cactusADM-453.povray 4.17 1.03 2.60 1.59 1.21 0.32 2.12 0.60 0.15 0.17 0.83 0.58 1.72 0.31 1.52 0.42450.soplex-453.povray 5.20 0.92 1.75 5.67 3.43 0.30 1.85 1.02 1.60 0.30 0.29 0.28 0.95 0.34 0.60 0.37462.libquantum-435.gromacs 0.49 0.57 2.58 1.52 3.34 0.49 2.22 1.00 4.65 0.37 0.51 0.38 1.21 0.49 0.21 0.29462.libquantum-436.cactusADM 2.29 0.97 2.15 1.60 2.45 0.55 1.27 1.43 3.88 0.42 0.94 0.48 1.45 0.45 0.21 0.29462.libquantum-450.soplex 5.32 0.88 1.04 1.49 1.97 0.40 1.25 0.74 4.89 0.34 1.91 0.46 3.44 0.36 1.39 0.46462.libquantum-453.povray 3.40 0.82 1.71 1.25 0.41 0.32 0.49 0.66 0.94 0.33 1.14 0.39 0.21 0.27 0.70 0.29462.libquantum-473.astar 5.54 2.69 0.51 0.92 0.27 0.46 0.41 0.86 1.37 0.75 2.51 0.48 1.42 0.56 1.78 0.44473.astar-435.gromacs 2.37 2.55 1.46 1.39 0.98 0.60 1.83 0.61 0.52 0.40 0.21 0.38 2.86 1.46 0.40 0.38473.astar-436.cactusADM 1.74 3.48 1.30 1.77 1.45 0.76 1.06 0.83 0.87 0.74 0.57 0.27 2.07 1.18 0.16 0.27473.astar-450.soplex 8.86 2.96 0.52 1.26 3.28 0.43 0.34 1.17 1.16 0.84 1.40 0.45 0.77 0.98 0.68 0.34473.astar-453.povray 0.87 3.61 0.48 0.82 4.22 0.46 0.22 0.40 3.13 0.22 0.90 0.28 3.51 1.35 0.24 0.26Average 3.28 1.40 1.55 1.58 1.94 2.07 1.20 1.03 1.74 0.50 1.17 0.47 1.68 0.79 0.91 0.44

Fig. 4. Comparison of average percentage error of the estimations for the dual-core models in virtualized and non-virtualized environments.

power increases, so that the same absolute error is seen aslower percentage errors. And second, the percentage of staticpower consumption over the total power consumption increaseswhen two cores are active, and since this static (constant) powerconsumption is easier tomodel, themodels increase their accuracy.

Fig. 4 summarizes graphically the average errors of Table 6.As can be seen, all average errors are below 10%, and onlyin the lower frequency (0.8 GHz) we obtained higher errors.Again, this is because that low values are more sensitive to thesame absolute interferences. The rest of frequencies do not showany noticeable trend, which lets us to conclude the PMC-basedmodeling methodology is orthogonal to DVFS.

In conclusion, the power models generated for the entire chipare validated in both the virtualized and the non-virtualizedsystems for wide set of frequencies. The layer corresponding tothe VM that encapsulates the test applications does not introduceany significant distortion over the power predictions obtainedthrough the model. This validates one of the contributions of this

paper: PMC-based power models can be applicable on virtualizedenvironments for any given frequency. The next section validatesthe applicability of these power models to perform per-VM energyaccounting.

3.2. Per virtual machine energy accounting validation

In this section, we validate the methodology used to performthe energy accounting. We confirm that pfmon is able to track theactivity of the VMs and also that the power model estimations areaccurate. Although we validated the 4 DVFS frequencies, in thissectionweonly present results for the 2.5GHz frequency. It sufficesto say the rest of frequencies were also validated and presentedsimilar results.

For the experiment performed to validate the energy account-ingmethodology, we chose three SPECcpu2006 applicationswithdifferent power behavior. Table 7 summarizes them in conjunction

8 R. Bertran et al. / Future Generation Computer Systems ( ) –

Table 7SPECcpu2006 benchmarks used to perform the energy accounting experimentsin conjunction with is average power consumption, when executed in a non-virtualized environment at 2.5 GHz, and their assigned mnemonic.

Benchmark Power consumption Avg. power(mW) Mnemonic

454.calculix High 12432.47 +

482.sphinx Medium 10976.9 =

435.gromacs Low 9974.62 −

Table 8Workloads executed for each frequency configuration in order to evaluate theenergy accounting methodology.

Test 1 VM 2 VMs 3 VMs 4 VMs 5 VMs

1 -- =-,-- ==,=-,-- +-,==,=-,-- +=,+-,==,=-,--2 =- ==,-- +-,=-,-- +=,==,=-,-- ++,+-,==,=-,--3 == ==,=- +-,==,-- +=,+-,=-,-- ++,+=,==,=-,--4 +- +-,-- +=,=-,-- +=,+-,==,-- ++,+=,+-,=-,--5 += +-,=- +-,==,=- ++,==,=-,-- ++,+=,+-,==,--6 ++ +=,-- +=,==,-- +=,+-,==,=- ++,+=,+-,==,=-7 +-,== +=,==,=- ++,+-,=-,--8 +=,=- ++,=-,-- ++,+-,==,--9 +=,== +=,+-,-- ++,+=,=-,--

10 ++,-- ++,==,-- ++,+=,==,--11 ++,=- +=,+-,=- ++,+-,==,=-12 +=,+- ++,==,=- ++,+=,==,=-13 ++,== +=,+-,== ++,+=,+-,--14 ++,+- ++,+-,-- ++,+=,+-,=-15 ++,+= ++,+-,=- ++,+=,+-,==16 ++,+=,--17 ++,+-,==18 ++,+=,=-19 ++,+=,==20 ++,+=,+-

with its power behavior, average power consumption and the as-sociated mnemonic5.

By combining such applications within different numbers ofVMs, we were able to gather results for a wide space of situations.The combinations of the experiments performed to validate themethodology are summarized in Table 8. Two applications runinside each dual-core VM, and depending on the number ofVMs explored (each column in the table), we generated a set ofworkloads. They are shown in Table 8, sorted by the number of‘-’, ‘=’ and ‘+’ respectively (from low to high power consumption).For instance, the test 10 with 2 VMs runs two instances of454.calculix in one VM and two instances of 435.gromacsin the other one. We executed each of the 62 different workloadsduring fiveminutes in order to gather their PMC activity and powertrace.

3.2.1. Activity tracking validationAs we explained in Section 2.3, pfmon is used to track the

activity of each VM. Ideally, the activity tracking could be doneby the host OS. Then, inaccuracies that might be introduced byhaving another user-level application running on the system couldbe avoided. However, we see that pfmon is able to track most ofthe VM activity. Fig. 5(a) shows the % of time tracked by pfmondepending on the number of VMs. The time tracked is derivedfrom the sum of the CPU cycles that pfmon accounted for eachVM. Therefore, during the 300 s duration of the experiments, andassuming no other processes running, we need to account for atotal of about 600 s6 of CPU time. We see that unless more than 4VMs are configured on the system,pfmon is able to track about 99%of VM activity. Notice that we stress our mechanism by deploying

5 The names:‘+’,‘=,‘-’ denote high, medium and low power consumptionrespectively.6 We have 2 CPUs. Hence, during 300 s, we have 600 s of CPU time.

a

b

Fig. 5. Percentage of CPU time tracked by pfmon and CPU time consumed per-VMfor the 2.5 GHz frequency.

fairly more than the usual number of VM on the same system.7Even so, our mechanism is able to track fairly well the system’sactivity.

Moreover, we want to validate that the scheduler performscorrectly and assigns the same CPU time to each VM. In particular,our 600 s of CPU time should be split equally among the VMs. Forexample, when 3 VMs are present on the system, their executiontime should be around 200 s (600 s/3 VMs). We confirm thatthe host scheduler assigns the same amount of CPU time to eachVMin Fig. 5(b). Notice that all the points are concentrated ontheir expected value, without any significant variation. From thatobservation, one can think that the chargeback that a providermust perform for each VM machine should be the same. We seelater on, in Section 3.3, that in terms of energy this is not true.

3.2.2. Energy accounting validationOnce we have validated that our methodology is able to

track the VM activity, we apply the power model generated inSection 2.5. Fig. 6 shows the results for the workloads studiedwith 4 VMs enabled. The x axis denotes the experiment workload.At the top, the real energy consumption and a stacked bar ofenergy contribution of each VM is shown. The energy is derivedfrom the average power consumption predicted by the modelmultiplied by the VM execution time. We see that we are able toaccurately predict overall energy consumption from the sum ofper-VM energy consumption. This is summarized at the bottom ofFig. 6, where the % of accuracy and CPU time tracked are shown.First, we see again that we are able to track almost 100% of theCPU time. And second, the error in energy prediction is alwaysbelow 5%.

We do not show absolute energy workload results for theother number of VMs to not overload the reader. However, wesummarize the error in energy prediction for all configurationstudied in Fig. 7(a). The main observation is that the error inprediction is not affected by the number of VMs present on thesystem — keeping the same magnitude. From that observationwe can expect this methodology to be valid on future many-coreprocessors.

We also show the relation between energy prediction and timetracked in Fig. 7(b). We see that if we are able to track most ofthe VM activity, the error in prediction is low. Once we lose someaccuracy in tracing system— the case of 5 VMs—, we obtain higher

7 Usually, a 1 to 1 mapping is deployed between physical CPUs and virtual CPUs.Whereas we perform a maximum of 5 to 2 mapping.

R. Bertran et al. / Future Generation Computer Systems ( ) – 9

Fig. 6. Energy accounting results and error of predictions of the workloads using 4VMs running at 2.5 GHz frequency.

a

b

Fig. 7. Error in energy prediction and relation between energy prediction error andtime tracked by pfmon for all workload running at 2.5 GHz frequency.

errors in energy prediction. The reason is that in that configurationthe system starts to be overloaded — including some thrashing —and as a result, our mechanism based on the user-space processpfmon does not accounts all the VMactivity. This supports the ideaof that a future implementation of an activity tracker mechanismshould be done into the host OS in order to avoid losses in activityaccounting.

In this section, we have validated and shown that we are able toperformper-VMenergy accountingwhile keeping acceptable errorlevels. The following section explains a case of applicability.

3.3. Applicability: quantification of biases in energy consumption

It is obvious that it is possible that VMs that spend the samenumber of CPU cycles, do not always consume the same amountof energy due to the differences of workloads. In this section, wequantify these differences, showing the fact that our approach canbe used to increase the flexibility and the fairness of the chargebackmodels that service and infrastructure providers use.

Fig. 8 shows the maximum difference in joules between VMsexecuted concurrently. As we showed in Fig. 5(b), the OS schedulerdoes a good job assigning a similar amount of CPU cycles to

Fig. 8. Absolutemaximumdifference in joules between VM executed concurrentlyat 2.5 GHz frequency.

each VM. However, as we see in Fig. 8, in most cases there existdifferences in energy consumption among the VMs. We also seethat the absolute difference between VMs decreases when thenumber of concurrent VMs increases. However, that does notmeanthat the percentage of difference between VMs decreases. Forinstance, having 500 joules difference in the 2 VMs workloads, isthe same percentage of difference of having 200 joules differencewhen executing 5 concurrent VMs on the system. As a result,we have evidenced that energy differences between VMs do notdepend on the number of VMs running on the system. Hence, theyonly depend on the workload characteristics.

In order to quantify themagnitude of the differenceswepresentfour cases in detail, Fig. 9 shows them. Each subfigure showsthe time and energy breakdown pies in conjunction with theinformation of theworkload executed in each VM and the absolutenumbers of execution time and energy consumption. We alsoshow the percentage difference in execution time and energyconsumption with respect to the VM1. In all the cases, we seethat VMs were executed during a fairly similar amount of time.For instance, in Fig. 9(a) the VM1 only was running a 0.37% moretime than VM2. The bigger difference in execution time is shownin Fig. 9(d). In that experiment, the VM1 was running 9.91% moretime thanVM4. In any case, a chargebackmodel basedon executiontime will charge all the VMs similarly. However, if we take intoaccount the energy consumption we see that the differences arequite higher. For instance, in Fig. 9(b) and (c), the VM1 consumes>20% more energy than VM2 and VM3, even with execution timedifferences below 5%. The smaller difference is seen in Fig. 9(d),where there is 9.89% energy difference between VM1 and VM2,even there only exists a 0.52% difference in execution time. If weanalyze the workloads being executed and the differences, weclearly see the expected results, which are that the ‘++’ workloadsconsume more energy than the rest.

In conclusion,wequantified thedifferences in energy consump-tion of VMs, showing that chargeback models based only on CPUexecution time may fail assigning the same charge to VMs thatactually consumed fairly different amount of energy (>20%). Thereason is that such chargeback models assume the same averagepower consumption for allworkloads,whichwill result in the samedifferences in execution time than in energy. However, as seen inFig. 9, that assumption introduces huge errors because the averagepower consumed by each workload can differ substantially. As aresult, this methodology can be used to obtain more accurate andfair chargeback models.

4. Related work

In general, previousworks onPMC-basedpowermodeling focuson two main aspects: power model generation and its usage toguide power aware policies. For instance, Tao Li et al. [21] study thepower consumption at the OS level using IPC as a predictor. In [22],Bellosa implements an OS power aware policy based on datacollected at runtime through the PMCs. In this work, overall powerconsumption is being modeled using a reduced set of PMCs. Theworks of Joseph, Isci, Contreras and Martonosi [23,17,24] describePMC-based power models of other architectures, but following a

10 R. Bertran et al. / Future Generation Computer Systems ( ) –

a

b

c

d

Fig. 9. Time and energy breakdowns of four workloads studied at 2.5 GHzfrequency.

similar methodology as the one used to derive the Intel r⃝ CoreTM2 Duo power model. In conclusion there has been sufficient priorwork that proves power models based on PMCs are functional andoffer more than satisfactory levels of accuracy. However, as far aswe know, they have not been validated on virtualized systems.

Regarding the consideration of energy-costs in virtualized data-center management, some research has been conducted in orderto manage energy. In [25,26] the authors propose a placementoptimization process that considers both performance and energy-costs. The proposed placement algorithms aim to satisfy theperformance goal of each application and, at the same time, tosave energy throughworkload consolidation, whichminimizes thenumber of physical machines required to support the execution.

Finally, there exist some works that perform per-VM energyaccounting. In [27], the authors propose an experimental OSprototype to perform per-VM energy accounting. For that purpose,they propose modifications on host and virtualized OS to achieveper-VM energy accounting and assume the PMC-based models arestill valid on such environments. In a recent proposal [28], theauthors perform per-VM energy accounting. However, they usesimple modeling methods — not PMC-based — for the CPU andmemory and as a result, they face the problem evidenced in thispaper: the same CPU time usage does not imply the same energyconsumption. They overcome the problem proposing adaptivetechniques. In the end, our main difference over previous worksis that we do not assume that PMC-based models are valid onvirtualized systems. Instead, we firstly validate that assumption,and we propose a mechanism — that does not require any codemodification — to track per-VM power consumption. Moreover,the mechanism is also validated in terms of accuracy for a wideset of configurations, workloads and DVFS settings.

5. Conclusion

This paper joins the virtualization technology with specificpower modeling techniques. We validate the PMC-based powermodeling methodology in virtualized systems. The validation pro-cess has been performed on an Intel r⃝ CoreTM 2 Duo platform. Theresultingmodels are able to account for the power consumption forCPU and memory at process level, corroborating that the virtual-ization layer does not introduce inaccuracies on predictions. More-over, we showed that DVFS configuration does not affect the accu-racy of model predictions. In overall, all PMC-based power modelsshow errors below 3% on average for all the configurations studied.

Then, we have applied the power model to account for theenergy consumption of each VM in a virtualized system. Theaccounting is based on collecting the number of cycles that aVM runs, plus the PMCs values that describe its activity. Withthat we derive energy consumption estimates at VM level. Atthe end, we are able to obtain energy consumption estimationswith an error below 5%. This estimations can be used to generatemore accurate chargeback models, since, as we have shown, VMsmachines executed during the same amount of time, presentedmore than 20% variations in energy consumption.

Although this is a promising technology for energy accounting,there are several open issues. Mainly, the current model is restric-ted to the processor and memory, with no support for modelingthe IO operations. This corresponds to future work to come.

Acknowledgements

This work was supported by the Ministry of Science andInnovation of Spain (CICYT) [TIN-2007-60625], the Generalitatde Catalunya [2009-SGR-980] and the European Commissionin the context of the SARC Project #27648 (FP6). The authorsacknowledge the support of the Barcelona Supercomputing Center(BSC) and the Universitat Politècnica de Catalunya (UPC).

References

[1] A.Merkel, F. Bellosa, Balancing power consumption inmultiprocessor systems,SIGOPS Oper. Syst. Rev. 40 (4) (2006) 403–414doi:http://doi.acm.org/10.1145/1218063.1217974.

[2] K. Singh, M. Bhadauria, S.A. McKee, Real time power estimation and threadscheduling via performance counters, SIGARCH Comput. Archit. News 37 (2)(2008) 46–55. doi:http://doi.acm.org/10.1145/1577129.1577137.

[3] A. Bhattacharjee, M. Martonosi, Thread criticality predictors for dynamicperformance, power, and resource management in chip multiprocessors,in: ISCA ’09: Proceedings of the 36th Annual International Symposiumon Computer Architecture, New York, NY, USA, ACM, 2009, pp. 290–301.doi:http://doi.acm.org/10.1145/1555754.1555792.

R. Bertran et al. / Future Generation Computer Systems ( ) – 11

[4] J. Donald, M. Martonosi, Techniques for multicore thermal management:Classification and new exploration, in: ISCA ’06: Proceedings of the 33rdAnnual International Symposium on Computer Architecture, Washington, DC,USA, IEEE Computer Society, 2006, pp. 78–88.doi:http://dx.doi.org/10.1109/ISCA.2006.39.

[5] R. Bertran, M. Gonzàlez, X. Martorell, N. Navarro, E. Ayguadé, Decomposableand responsive power models for multicore processors using performancecounters, in: ICS ’10: Proceedings of the 24th ACM International Conferenceon Supercomputing, ACM, Tsukuba, Ibaraki, Japan, 2010, pp. 147–158.doi:http://doi.acm.org/10.1145/1810085.1810108.

[6] R. Bertran, Y. Becerra, D. Carrera, V. Beltran, M. Gonzalez, X. Martorell,J. Torres, E. Ayguade, Accurate energy accounting for shared virtualizedenvironments using PMC-based power modeling techniques, in: The 11thIEEE/ACM International Conference on Grid Computing, GRID-2010, Brussels,Belgium, 25–29 October, 2010, pp. 1–8.

[7] V. George, S. Jahagirdar, C. Tong, K. Smits, S. Damaraju, S. Siers, V. Naydenov, T.Khondker, S. Sarkar, P. Singh, Penryn: 45-nm next generation Intel r⃝ CoreTM2 processor, in: ASSCC’07 IEEE Asian Solid-State Circuits Conference, 2007,pp. 14–17.

[8] SBSIF, Smart specifiaction rev.1.1, December 1998. URL http://sbs-forum.org.[9] Electronic educational devices. watts up pro., October 2010, URL http://www.

wattsupmeters.com/.[10] Thinkpad smapi kernelmodule version 0.40. URL http://tpctl.sourceforge.net/.[11] Intel, Intel Core2 DuoMobile Processor, Intel Core2 Solo Mobile Processor and

Intel Core2 Extreme Mobile Processor on 45-nm Process. For platforms basedon Mobile Intel 4 Series Express Chipset Family. Datasheet, March 2009. URLhttp://download.intel.com/design/mobile/datashts/32012001.pdf.

[12] D. Brodowski, Linux kernel cpufreq subsystem. URL http://www.kernel.org/pub/linux/utils/kernel/cpufreq/cpufreq.html.

[13] Perfmon2. URL http://perfmon2.sourceforge.net/.[14] Qemu: Open source processor emulator. URL http://www.qemu.org.[15] Kernel based virtual machine. URL http://www.linux-kvm.org.[16] Libvirt virtualization api. URL http://www.libvirt.org.[17] C. Isci, M. Martonosi, Runtime power monitoring in high-end processors:

Methodology and empirical data, in: MICRO 36: Proceedings of the 36thAnnual IEEE/ACM International Symposium on Microarchitecture, IEEEComputer Society, Washington, DC, USA, 2003, p. 93.

[18] J.L. Henning, Spec cpu2006 benchmark descriptions, SIGARCH Comput. Archit.News 34 (4) (2006) 1–17. doi:http://doi.acm.org/10.1145/1186736.1186737.

[19] A. Phansalkar, A. Joshi, L.K. John, Subsetting the spec cpu2006benchmark suite,SIGARCH Comput. Archit. News 35 (1) (2007) 69–76doi:http://doi.acm.org/10.1145/1241601.1241616.

[20] A. Phansalkar, A. Joshi, L.K. John, Analysis of redundancy and applicationbalance in the spec cpu2006 benchmark suite, SIGARCH Comput. Archit. News35 (2) (2007) 412–423. doi:http://doi.acm.org/10.1145/1273440.1250713.

[21] T. Li, L.K. John, Run-time modeling and estimation of operating systempower consumption, SIGMETRICS Perform. Eval. Rev. 31 (1) (2003) 160–171.doi:http://doi.acm.org/10.1145/885651.781048.

[22] F. Bellosa, The benefits of event: driven energy accounting in power-sensitivesystems, in: EW 9: Proceedings of the 9thworkshop on ACM SIGOPS Europeanworkshop, ACM, New York, NY, USA, 2000, pp. 37–42.doi:10.1145/566726.566736. URL http://dx.doi.org/10.1145/566726.566736.

[23] R. Joseph, M. Martonosi, Run-time power estimation in high performancemicroprocessors, in: ISLPED ’01: Proceedings of the 2001 InternationalSymposium on Low power electronics and design, ACM, New York, NY, USA,2001, pp. 135–140. doi:http://doi.acm.org/10.1145/383082.383119.

[24] G. Contreras, M. Martonosi, Power prediction for Intel XScale processors usingperformance monitoring unit events, in: ISLPED ’05: Proceedings of the 2005International Symposium on Low Power Electronics and Design, ACM, NewYork, NY, USA, 2005, pp. 221–226.doi:http://doi.acm.org/10.1145/1077603.1077657.

[25] M. Steinder, I. Whalley, J.E. Hanson, J.O. Kephart, Coordinated management ofpower usage and runtime performance, in: Proceedings of the 11th NetworkOperations and Management Symposium, IEEE, 2008, pp. 387–394.

[26] J.S. Chase, D.C. Anderson, P.N. Thakar, A.M. Vahdat, R.P. Doyle, Managingenergy and server resources in hosting centers, SIGOPS Oper. Syst. Rev. 35 (5)(2001) 103–116. doi:http://doi.acm.org/10.1145/502059.502045.

[27] J. Stoess, C. Lang, F. Bellosa, Energy management for hypervisor-basedvirtual machines, in: ATC’07: 2007 USENIX Annual Technical Conference onProceedings of the USENIX Annual Technical Conference, USENIX Association,Berkeley, CA, USA, 2007, pp. 1–14.

[28] A. Kansal, F. Zhao, J. Liu, N. Kothari, A.A. Bhattacharya, Virtual machinepower metering and provisioning, in: SoCC ’10: Proceedings of the 1st ACMsymposium on Cloud computing, ACM, New York, NY, USA, 2010, pp. 39–50.doi:http://doi.acm.org/10.1145/1807128.1807136.

Ramon Bertran received a bachelor’s degree in ComputerScience and a master’s degree in Computer Architecturefrom Universitat Politècnica de Catalunya (UPC). Since2005 he is a Ph.D. student in the Computer ArchitectureDepartment at UPC. Currently, he is a resident studentat Barcelona Supercomputing Center, and his researchis focused on power and performance modelling ofmulticore architectures and energy aware computing.

Yolanda Becerra holds a Ph.D. in Computer Science since2006 from the Technical University of Catalonia (UPC).Sheis an assistant professor at the Computer Architecture De-partment of the UPC and an associate researcher in theBSC. Since 1998, she lectures courses on Operating Sys-tems in several schools of the UPC and in the Masters pro-gram of the Computer Architecture department of UPC. In2007 she joined to the ‘‘Autonomic Systems and eBussinesplatforms’’ research line. In this research group she isconducting research about application profiling and per-formance estimation, energy-efficient computing, paral-

lel and distributed systems, and about non-centralized performance managementfor heterogeneous execution environments. Since 2008 she is involved in a re-search collaboration project with IBM Research, focused on the development ofnon-centralized resource management strategies and mechanisms.

David Carrera received the M.S. degree at the TechnicalUniversity of Catalonia (UPC) in 2002 and his Ph.D. fromthe same university in 2008. Since 2003 he is an assistantprofessor at the Computer Architecture Department of theUPC. He is also an associate researcher in the BSC wherehe is the coordinator of the Performance managementfor next-generation data centers area in the AutonomicSystems and e-Business Platforms research line. He iscurrently advisor for 3 Ph.D. students.

He was a member of the European Center forParallelism of Barcelona (CEPBA) (2001–2004) and a

member of the CEPBA-IBM Research Institute (CIRI) (2000–2004). He has workedin a number of EU and industrial research and development projects at CEPBAand CIRI. His current research focuses are autonomic computing, energy-efficientcomputing, parallel and distributed systems, virtualization management, fault-tolerant systems, web and gridmiddleware, systems architecture, and performancemodeling.

In 2006 he was an intern at the IBM TJWatson research lab in NY, and currentlyleads a collaboration project between the team he coordinates in BSC and theMiddleware and Virtualization Management group in IBM Watson. He has beeninvolved in several EU and industrial research projects. He has authored a numberof papers in international journals and conferences, and has served as a reviewerfor a number of international conferences. He is an IEEE member.

He lectures courses on Computer Networks, Operating Systems, ComputerArchitecture and on Performance Evaluation of Computer Systems in the ComputerScience School (FIB) of UPC and in the Masters program of the ComputerArchitecture department of UPC.

Vicenç Beltran is a Senior Researcher at the Computer Sci-ence department of the Barcelona Supercomputing Center(BSC). He received his Ph.D. on Computer Science from theUPC in 2008. His research interests are heterogeneous andhybrid systems, domain specific languages and operatingsystems.

Marc Gonzàlez received the Engineering degree in Com-puter Science in 1996 and the Computer Science Ph.D.degree on December 2003. His research has been deve-loped around parallel programming models and highperformance computing, in particular, the OpenMP pro-gramming model. His current research is centered on thisprogramming model and its use on heterogeneous archi-tectures, based on accelerators and local memories. Themain research activities are the design and implementa-tion of software cache techniques for the IBM Cell BE pro-cessor as well as the implementation of runtime libraries

and compiler support for the OpenMP programming model on this architecture.

Xavier Martorell received the M.S. and Ph.D. degrees incomputer science from the Technical University of Cat-alonia (UPC) in 1991 and 1999, respectively. Since 1992he has lectured on operating systems, parallel runtimesystems and OS administration. He has been an associateprofessor in the Computer Architecture Department atUPC since 2001. His research interests cover the areas ofoperating systems, runtime systems, compilers and appli-cations for high-performance multiprocessor systems. Dr.Martorell has participated in several long-term researchprojects with other universities and industries, primarily

in the framework of the European Union ESPRIT, IST and FET programs. He spendone year working with the BG/L team in the IBM Watson Research Center. He iscurrently the Manager of the Parallel Programming Models team at the BarcelonaSupercomputing Center. He is participating in the Hipeac Network of Excellenceand in the SARC and ACOTES European projects.

12 R. Bertran et al. / Future Generation Computer Systems ( ) –

Nacho Navarro is Associate Professor at the UniversitatPolitècnica de Catalunya (UPC), Barcelona, Spain, since1994, and Senior Researcher at the Barcelona Supercom-puting Center (BSC). He holds a Ph.D. degree in Com-puter Science fromUPC (1991), Spain. His current interestsinclude: tools for evaluation of multi-core microproces-sors, application-specific computer architectures, dyna-mic reconfigurable logic and resource management inheterogeneous environments and sensor networks. He isalso doing research on the programmability and supportof hardware accelerators like GPUs at the University of

Illinois (IMPACT Research Group). Prof. Navarro is a member of the IEEE, the IEEEComputer Society, the ACM and USENIX.

Jordi Torres has a Masters degree in Computer Sciencefrom the Technical University of Catalonia (UPC, 1988) andalso holds a Ph.D. from the same institution (Best UPCComputer Science Thesis Award, 1993). Currently he is afull professor in the Computer Architecture Departmentat UPC. He has more than twenty years of experiencein research and development of advanced distributedand parallel systems in the High Performance ComputingGroup at UPC. He has been a visiting researcher at the‘‘Center for Supercomputing Research & Development’’ atUrbana-Champaign (Illinois, USA, 1992).

His current principal interest as a researcher is making IT resources moreefficient, and focuses on the resource management needs of modern distributedand parallel cloud computing environments. Currently he is actively working tocombine the research from different research areas such as autonomic computing,parallel and distributed systems, performance modelling, virtualization, machinelearning, amongst others, to reasonably stem the difficulties to obtain moresustainable IT by itself (Green Computing). In general my interest is Science &Technology for prosperity.

He has more than a hundred publications in journals, conferences and bookchapters. He is member of IEEE, ACM and ISOC and was involved in several

conferences organized by these associations. Currently he is also a member ofthe Management Committee of the EU COST action in Green Computing ‘‘Energyefficiency in large scale distributed systems’’ and a member of the SteeringCommittee IEEE-TCSC Technical Area of Green Computing.

He was a member of the European Center for Parallelism of Barcelona (CEPBA)(1994–2004) and a member of the board of managers of CEPBA-IBM ResearchInstitute (CIRI) (2000–2004). In 2005 the Barcelona Supercomputing Center —Centro Nacional de Supercomputación (BSC)was founded and hewas nominated asa Manager for Autonomic Systems and eBusiness Platforms research line in BSC. Hehas worked and works in a number of EU and industrial research and developmentprojects.

He lectures on Computer Networks, Operating Systems, Computer Architectureand on Performance Evaluation in the UPC. He has been Vice-dean of InstitutionalRelations at the Computer Science School (1998–2001), and a member of theCatedra Telefonica-UPC where he worked in teaching innovation (2003–2005).He has also participated in numerous academic management activities andinstitutional representation from 1990 onwards. Currently he is a member of theUniversity Senate and member of the Board of Governors.

Eduard Ayguadé received the Engineering degree inTelecommunications in 1986 and the Ph.D. degree inComputer Science in 1989, both from the UniversitatPolitècnica de Catalunya (UPC), Spain. Since 1987 he hasbeen lecturing on computer organization and architectureand parallel programming models. Currently, and since1997, he is full professor of the Computer ArchitectureDepartment at UPC. His research interests cover the areasof processor microarchitecture, multicore architecturesand programming models and their architectural support.He has publishedmore than 100 papers in these topics and

participated in several research projects in the framework of the European Unionand research collaborations with companies. He is associated director for researchon computer sciences at the Barcelona Supercomputing Center (BSC-CNS).