« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 1»

Cluster ComputDOI 10.1007/s10586-008-0054-y

1 55

2 56

3 57

4 58

5 59

6 60

7 61

8 62

9 63

10 64

11 65

12 66

13 67

14 68

15 69

16 70

17 71

18 72

19 73

20 74

21 75

22 76

23 77

24 78

25 79

26 80

27 81

28 82

29 83

30 84

31 85

32 86

33 87

34 88

35 89

36 90

37 91

38 92

39 93

40 94

41 95

42 96

43 97

44 98

45 99

46 100

47 101

48 102

49 103

50 104

51 105

52 106

53 107

54 108

Providing platform heterogeneity-awareness for data centerpower management

Ripal Nathuji · Canturk Isci · Eugene Gorbatov ·Karsten Schwan

Received: 29 February 2008 / Accepted: 20 March 2008© Springer Science+Business Media, LLC 2008

Abstract Power management is becoming an increasinglycritical component of modern enterprise computing environ-ments. The traditional drive for higher performance has in-fluenced trends towards consolidation and higher densities,artifacts enabled by virtualization and new small form fac-tor server blades. The resulting effect has been increasedpower and cooling requirements in data centers which el-evate ownership costs and put more pressure on rack andenclosure densities. To address these issues, we exploit afundamental characteristic of data centers: “platform het-erogeneity”. This heterogeneity stems from the architec-tural and management-capability variations of the underly-ing platforms. We define an intelligent heterogeneity-awareload management (HALM) system that leverages hetero-geneity characteristics to provide two data center level ben-efits: (i) power efficient allocations of workloads to the bestfitting platforms and (ii) improved overall performance ina power constrained environment. Our infrastructure reliesupon platform and workload descriptors as well as a novelanalytical prediction layer that accurately predicts workloadpower/performance across different platform architecturesand power management capabilities. Our allocation scheme

R. Nathuji (�) · K. SchwanGeorgia Institute of Technology, Atlanta, GA 30032, USAe-mail: rnathuji@ece.gatech.edu

K. Schwane-mail: schwan@cc.gatech.edu

C. IsciVMware, Palo Alto, CA 93404, USAe-mail: canturk@vmware.com

E. GorbatovIntel Corporation, Hillsboro, OR 97124, USAe-mail: eugene.gorbatov@intel.com

achieves on average 20% improvements in power efficiencyfor representative heterogeneous data center configurations,and up to 18% improvements in performance degradationwhen power budgeting must be performed. These resultshighlight the significant potential of heterogeneity-awaremanagement.

Keywords Power management · Distributed resourcemanagement · Heterogeneous systems

1 Introduction

Power management has become a critical component ofmodern computing systems, pervading both mobile and en-terprise environments. Power consumption is a particularlysignificant issue in data centers, stimulating a variety of re-search for server systems [2]. Increased performance re-quirements in data centers have resulted in elevated densitiesenabled via consolidation and reduced server form factors.This has in turn created challenges in provisioning the nec-essary power and cooling capacities. For example, currentdata centers allocate nearly 60 Amps per rack, a limit thatis likely to become prohibitive for future high density rackconfigurations such as blade servers, even if the accompany-ing cooling issues can be solved [24]. In addition, a 30,000square feet data center with a power consumption of 10 MWrequires a cooling system which costs $2–$5 million [21]. Insuch a system, the cost of running the air conditioning equip-ment alone can reach $4–$8 million a year [24]. Coupledwith the elevated electricity costs from high performanceservers, these effects can substantially affect the operatingcosts of a data center. Overall, these trends in power/coolingdelivery and cost highlight the need for power and thermalmanagement support in data centers.

« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 2»

Cluster Comput

109 163

110 164

111 165

112 166

113 167

114 168

115 169

116 170

117 171

118 172

119 173

120 174

121 175

122 176

123 177

124 178

125 179

126 180

127 181

128 182

129 183

130 184

131 185

132 186

133 187

134 188

135 189

136 190

137 191

138 192

139 193

140 194

141 195

142 196

143 197

144 198

145 199

146 200

147 201

148 202

149 203

150 204

151 205

152 206

153 207

154 208

155 209

156 210

157 211

158 212

159 213

160 214

161 215

162 216

Previous work on server management has focused onmanaging heat during thermal events [21] or utilizing plat-form power management support, such as processor fre-quency scaling, for power budgeting [9, 18, 24]. In this pa-per, we approach the problem of managing data centers froma different perspective by considering how to intelligentlyallocate workloads amongst heterogeneous platforms in amanner that (i) improves data center power-efficiency whilepreserving/satisfying workload performance requirements,and (ii) meets data-center-level power budgets with minimalimpact on workload performance. Typically, data centersstatically allocate platform resources to applications basedupon peak load characteristics in order to maintain isola-tion and provide performance guarantees. With the continu-ing growth in capabilities of virtualization solutions (e.g.,Xen [1] and VMware [25]), the necessity of such offlineprovisioning is removed. Indeed, by allowing for flexibleand dynamic migration of workloads across physical re-sources [6], the use of virtualization in future data centersenables a new avenue of management and optimization. Ourapproach begins to leverage some of these capabilities toenhance power efficiency by taking advantage of the abilityto assign virtualized applications to varying sets of underly-ing hardware platforms based upon performance needs andpower consumption characteristics.

Throughout their lifetimes, data centers continually up-grade servers due to failures, capacity increases, and mi-grations to new form factors [12]. Over time, this leads todata centers comprised of a range of heterogeneous plat-forms with differences in component technologies; power,performance and thermal characteristics; and power man-agement capabilities. When provisioning resources to work-loads in these heterogeneous environments, power efficiencycan vary significantly based on the particular allocation. Forexample, by assigning a memory bound workload to a plat-form that can perform dynamic voltage and frequency scal-ing (DVFS), run-time power consumption can be reducedwith minimal impact to performance [19]. We propose anovel heterogeneity-aware load management (HALM) ar-chitecture to achieve this power-friendly behavior in datacenters.

Allocating power and cooling resources is another sig-nificant challenge in the modern data center. Though clearlybeneficial for transient power delivery and cooling issues,power budgeting solutions can also be useful in the provi-sioning of these resources. Traditionally, power and coolinghave been allocated based on the nameplate rating of the sys-tem power supply or its maximum output power. However, afully utilized server with a typical configuration will see itselectrical load between 60%–75% of the name plate ratingwith most enterprise workloads. Therefore, providing powerand cooling capacity based on these worst case assumptions

results in either over allocation of power and cooling ca-pacity or underutilization of server rack space leading to in-creased capital costs and underutilized data centers. Allocat-ing power and cooling capacity based on the average work-load behavior within a server and across a data center allowssignificantly increased densities but requires dynamic pro-tection mechanisms that can limit server power consump-tion when demand temporarily exceeds available capacity.These mechanisms have been recently proposed in the liter-ature and explored in the industry [8]. While very effectivein limiting power and protecting the infrastructure, they mayresult in nontrivial degradation of peak performance, espe-cially when the power constraint is too prohibitive. In thispaper we illustrate how HALM can lessen the performanceimpact of data center power budgeting strategies.

Intelligent mapping of applications to underlying plat-forms is dependent upon the availability of relevant infor-mation about workloads and hardware resources. As part ofHALM, we extend the use of workload and platform de-scriptors for this purpose, which are then used by a pre-dictor component that estimates the achievable performanceand power savings across the different platforms in the datacenter. These predictions are finally used by an allocationlayer that map workloads to a specific type of platform. Thisoverall infrastructure is evaluated using data center con-figurations consisting of variations upon four distinct plat-forms. In summary, the main contributions of our HALMsystem are: (i) a platform heterogeneity-aware power man-agement infrastructure that improves data center power effi-ciency under workload performance constraints and limiteddata center power budgets; (ii) an allocation infrastructurethat uses workload and platform descriptors to perform map-pings of hardware to virtualized workloads; and (iii) an in-telligent load shedding policy to dynamically meet transientchanges in power consumption limits. Evaluations of oursystem performed on state-of-the art platforms, includingIntel® Core™ microarchitecture based hardware, demon-strate the benefits of exploiting platform heterogeneity forpower management.

2 Motivation

2.1 Data center composition and exploiting heterogeneity

Data center deployments are inherently heterogeneous. Up-grade cycles and replacement of failed components andsystems contribute to this heterogeneity. In addition, newprocessor and memory architectures appear every few years,and reliability requirements are becoming ever more strin-gent. The effect of these trends is reflected by a recent sur-vey of data center managers that found that 90% of the facil-ities are expected to upgrade their compute and storage in-frastructure in the next two years. Figure 1(a) shows a distri-bution of different systems in a representative enterprise data

« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 3»

Cluster Comput

217 271

218 272

219 273

220 274

221 275

222 276

223 277

224 278

225 279

226 280

227 281

228 282

229 283

230 284

231 285

232 286

233 287

234 288

235 289

236 290

237 291

238 292

239 293

240 294

241 295

242 296

243 297

244 298

245 299

246 300

247 301

248 302

249 303

250 304

251 305

252 306

253 307

254 308

255 309

256 310

257 311

258 312

259 313

260 314

261 315

262 316

263 317

264 318

265 319

266 320

267 321

268 322

269 323

270 324

Fig. 1 Data center heterogeneity and management benefits

center. As the figure shows, the data center contains nine dif-ferent generations of systems that have either (1) differentprocessor architectures, cores and frequencies; (2) varyingmemory capacity and interconnect speeds; or (3) differentI/O capabilities. While all systems support the same soft-ware stack, they have very different and often asymmetricperformance and power characteristics.

Traditionally, the non-uniformity of systems in a datacenter has been characterized by different levels of perfor-mance and power consumption. However, recently, anotherdimension has been added to this heterogeneity becauseserver platforms are beginning to offer rich thermal andpower management capabilities. Processors support DVFSand aggressive sleep states to conserve CPU power. Newmemory power management implementations allow differ-ent DRAM devices to go to lower power states when in-active, and enable bandwidth throttling for thermal protec-tion. Server power supplies exhibit different conversion ef-ficiencies under different loads, directly impacting the over-all power efficiency of the system. Since power efficiencyhas become an important thrust in enterprise systems, weexpect component and platform vendors to continue intro-ducing new power and thermal management capabilities intotheir products, including I/O and system buses, chipsets, andnetwork and disk interfaces, making future platforms evenmore heterogeneous.

Previous work has proposed different approaches forenergy-efficient workload allocation in clusters, but nonehave accounted for system level power management andthermal characteristics. Therefore, the workload allocationsproposed by previous approaches will yield less than idealresults since they are completely unaware of power and ther-mal management effects on system performance and powerconsumption. To illustrate this phenomenon, we experimen-tally compare two dual processor systems, A and B , run-ning two different workloads, as shown in Fig. 1(b). Thedifferences between the two systems are in the power sup-ply unit (PSU) and processor power management capabili-ties. System A has a less efficient power supply at light loadand has processors with limited power management support.System B, on the other hand, has a high efficiency powersupply across all loads and processors that support a rich set

of power management capabilities. We measure power con-sumption on these platforms using two different syntheticworkloads: one with full utilization (W1) and one with avery low level of utilization (W2). W1 consumes about thesame amount of power on both platforms. However, allocat-ing the low-utilization W2 to system A leads to very powerinefficient execution. Since A does not support power man-agement and has low PSU efficiency at light load, its totalsystem power is more than 50 W higher than that of sys-tem B . Thus, while both systems meet the performance de-mand of both workloads, heterogeneity-aware resource allo-cation can decrease total power by more than 10%, translat-ing into millions of dollars in savings for large data centers.As this example shows, a full knowledge of system powerand supported power management features is required to ef-ficiently allocate workloads. Our HALM system is designedto provide such functionality.

2.2 Benefits of heterogeneity-aware management

To further motivate the need and benefits of heterogeneity-aware management in data centers, we perform two oppor-tunity studies. The first study considers the possible benefitsof allocating workloads by matching system capabilities andworkload execution characteristics to reduce a data center’spower profile while also meeting workload performance de-mands. We analyze an example of running a set of work-loads in a data center configuration with four unique typesof platforms described later in the paper, each with differ-ent power/performance characteristics. The set of workloadsincludes ten computational benchmarks (swim, bzip2,mesa, gcc, mcf, art, applu, vortex, sixtrack, andlucas from SPEC CPU2000) and one transaction-orientedworkload (SPECjbb2005). We generate all subsets of fourfrom these eleven benchmarks and compare three alloca-tion policies for each of the subsets in Fig. 2(a). The ‘worstcase’ allocation distributes the benchmarks across platformsto maximize power consumption, ‘random’ allocates work-loads to platforms randomly, and ‘optimal’ distributes theworkloads to minimize power consumption. For each work-load, we allocate as many systems of a given type as nec-essary to meet workload throughput requirements. Subsets

« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 4»

Cluster Comput

325 379

326 380

327 381

328 382

329 383

330 384

331 385

332 386

333 387

334 388

335 389

336 390

337 391

338 392

339 393

340 394

341 395

342 396

343 397

344 398

345 399

346 400

347 401

348 402

349 403

350 404

351 405

352 406

353 407

354 408

355 409

356 410

357 411

358 412

359 413

360 414

361 415

362 416

363 417

364 418

365 419

366 420

367 421

368 422

369 423

370 424

371 425

372 426

373 427

374 428

375 429

376 430

377 431

378 432

Fig. 2 Opportunity analysis of heterogeneity-aware management

that have benchmarks with more homogeneous behavior, i.e.similar processor and memory usage behavior, appear on theleft side of the graph, while subsets with more heteroge-neous benchmarks appear on the right. As can be seen fromthe figure, subsets of workloads with more heterogeneousbehavior can substantially benefit from heterogeneity-awareresource allocation. Averaging across all subsets, the opti-mal policy can reduce total power by 18% when comparedto random allocation and by 34% over worst-case allocation,without compromising workload performance.

The second opportunity study considers how the aggre-gate throughput of a set of workloads varies within a givenpower budget based upon allocations. In particular, we as-sume that we have one of each of our four unique platformsand again generate subsets of four workloads from a set ofSPEC CPU2000 benchmarks. For each subset, we calculatethe minimum, average, and best case throughput across allpermutations of possible allocations of the four workloadsonto the four platforms. Figure 2(b) provides the results,where each scenario is normalized by the minimum through-put value to provide fair comparisons. We find that on aver-age, the best case allocation provides a 23% improvementin performance over the random allocation, and a 48% im-provement compared to the worst-case. These results high-light the relationship between allocation decisions and per-formance when a power budget must be imposed.

Summarizing, HALM addresses the power benefits ofheterogeneity-aware allocation for two cases: (1) when thereis no power budget and (2) when such a budget must beimposed temporarily due to power delivery or cooling con-straints or as part of a power provisioning strategy [8].

3 Scalable enterprise and data center management

Our previous discussions have motivated the need to aug-ment the behavior of data centers to improve manageabil-ity by leveraging the heterogeneity in platform capabilities.

HALM extends this support with its heterogeneity-awareworkload allocation infrastructure that utilizes the flexibil-ity of rapidly developing virtualization technologies. Virtu-alization attempts to provide capabilities and abstractionsthat significantly impact the landscape of enterprise man-agement. For example, there is active work to ensure per-formance isolation benefits, where it will be possible torun multiple virtual machines (VMs) within a given phys-ical platform without interference among applications [15].Currently, VMs can coexist on a platform with negligibleperformance interference as long as resources are not over-committed. Approaches that allow for resource pools andreservations as well as dynamic resource sharing and recla-mation can aid in providing isolation even when systemsare over-provisioned. Secondly, by encapsulating applica-tion state within well defined virtual machines, migrationof workloads among resources can be performed easily andefficiently. A more powerful contribution of virtualization,however, is the ability to combine multiple resources acrossphysical boundaries to create virtual platforms for applica-tions, providing a scalable enterprise environment. HALMassumes the existence of this flexible and powerful virtual-ization support.

The usage pattern of data centers is becoming increas-ingly service-oriented, where applications and workloadsmay be submitted dynamically by subscribers/clients. Whenmanaging these types of applications certain managementactions, such as allocation decisions, happen at a coarsegranularity with finer adjustments being made at runtimeto address transient issues such as reduced power budgets.One can imagine how such a data center might be managedwith the typically used assignment approaches. At some in-frequent interval the pool of applications and service levelagreements (SLAs) that specify their required performance,in metrics such as throughput or response time, are com-piled. Applications are then assigned to platforms usinga simple load balancing scheme based upon utilization orqueue lengths, possibly even accounting for differences inthe performance of the systems [26], so that SLAs are met.

« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 5»

Cluster Comput

433 487

434 488

435 489

436 490

437 491

438 492

439 493

440 494

441 495

442 496

443 497

444 498

445 499

446 500

447 501

448 502

449 503

450 504

451 505

452 506

453 507

454 508

455 509

456 510

457 511

458 512

459 513

460 514

461 515

462 516

463 517

464 518

465 519

466 520

467 521

468 522

469 523

470 524

471 525

472 526

473 527

474 528

475 529

476 530

477 531

478 532

479 533

480 534

481 535

482 536

483 537

484 538

485 539

486 540

Fig. 3 HALM architecture

When load must be reduced to address power budgeting re-quirements, load might be shed from workloads in a simi-larly random or round robin fashion. This approach clearlyleaves room for improvement, since it does not considerpower or platform differences in any way. HALM addressesthis weakness by performing heterogeneity aware alloca-tions as well as intelligent load shedding.

The HALM architecture can be organized into three ma-jor components: (1) platform/workload descriptors, (2) apower/performance predictor, and (3) an allocator, as shownin Fig. 3(a). We use platform and workload descriptors toprovide our workload allocator with the differences amongstworkloads and platforms. These descriptor inputs are uti-lized by the predictor to determine: (1) the relative perfor-mance of workloads on different types of platforms, and(2) the power savings achievable from platform power man-agement mechanisms. Coupled with coarse platform powerconsumption information (obtained via online power moni-toring) (3) the allocator, performs the assignments of work-loads to the available resources.

The purpose of platform descriptors is to convey informa-tion regarding the hardware and power management capabil-ities of a machine. A platform descriptor is made up of mul-tiple modules, representing different system components, asshown in Fig. 3(b). Each module specifies the type of com-ponent to which it refers, such as processor, memory subsys-tem, or power supply. Within each of these modules, variouscomponent parameters are defined. For example, a moduledescribing the processor component may have attributes likeits microarchitecture family, frequency, and available man-agement support. Workload descriptors are also structuredin modules, headed with attribute declarations. Within eachmodule, a list of values for that attribute is provided. Asworkload attributes often vary with the platforms on whichit executes, our descriptor design allows multiple attributedefinitions, where each definition is predicated with com-ponent parameter values that correlate back to platform de-scriptors. Figure 3(b) illustrates the structure of the resultingworkload descriptor. We further explain the meaning of the

MPI (memory accesses per instruction) and CPICORE (corecycles per instruction) attributes in subsequent sections.

Platform descriptor information can be provided in a vari-ety of ways. It can be made readily available using platformsupport such as ACPI [13], and possibly also with someadministrative input. To provide the required workload de-scriptors, we profile workloads on a minimal set of orthogo-nal platforms, with mutually exclusive component types. Wethen use an analytical prediction approach to project work-load characteristics on all available platforms. As we discussin Sect. 5, this approach provides accurate predictions thatscale with increased amounts of heterogeneity.

4 Methodology

4.1 Platform hardware

Our hardware setup consists of four types of rack mountedserver platforms summarized in Fig. 4(a), where LLC de-notes last-level cache size. All four types of platforms con-tain standard components and typical configurations that en-tered production cycles. In our experiments Linux was in-stalled on all systems for measurement of various attributes(e.g. CPI, MPI, etc.) as well as performance. We validatedthat the performance results matched those with Xen usinga subset of workloads and platforms, but performed the ma-jority of our experiments in a non-virtualized environment tohave better access to performance counters used to measureother workload attributes.

The platform names are based on their processor codename in this paper. All four platforms are dual-processorsystems. Woodcrest, Sossaman, and Dempsey are CMPdual-core processors, and Irwindale is a 2-way SMT proces-sor supporting Hyper-Threading Technology. All platformshave 8 GB of memory. Woodcrest and Dempsey supportFully Buffered DIMM (FBD) memory with a 533 MHzDDR2 bus, while Sossaman and Irwindale support unreg-istered DDR2 400 MHz memory. Woodcrest and Dempsey

« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 6»

Cluster Comput

541 595

542 596

543 597

544 598

545 599

546 600

547 601

548 602

549 603

550 604

551 605

552 606

553 607

554 608

555 609

556 610

557 611

558 612

559 613

560 614

561 615

562 616

563 617

564 618

565 619

566 620

567 621

568 622

569 623

570 624

571 625

572 626

573 627

574 628

575 629

576 630

577 631

578 632

579 633

580 634

581 635

582 636

583 637

584 638

585 639

586 640

587 641

588 642

589 643

590 644

591 645

592 646

593 647

594 648

Fig. 4 Experimental platforms

Table 1 Levels ofheterogeneity in ourexperimental platforms

have dual FSB architectures with two branches to memoryand two channels per branch.

All four types of systems are heterogeneous in a sensethat each has a unique combination of processor architec-ture and memory subsystem. If we assume that Intel Coremicroarchitecture/Pentium® M and NetBurst constitute twotypes of processors and LLC-4 MB/FSB-1066/FBD-533and LLC-2 MB/FSB-800/DDR2-400 constitute two types ofmemory, all four platforms can be mapped as having uniqueprocessor/memory architecture combinations. Note that allfour platforms also have vastly different power and perfor-mance characteristics. For example, the Intel Core microar-

chitecture is superior to NetBurst both in terms of perfor-mance and power efficiency. FBD based memory, on theother hand, provides higher throughput in our systems atthe expense of elevated power consumption due to increasedDDR2 bus speed and the power requirements of the Ad-vanced Memory Buffer (AMB) on the buffered DIMMs.The four platforms occupy separate quadrants of a hetero-geneity space with dimensions of microarchitecture hetero-geneity and memory subsystem heterogeneity, as shown inFig. 4(b). We refer to this initial level of heterogeneity as“across-platform heterogeneity”. However, in addition tothis, all these server platforms also support chip-level DVFS.

« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 7»

Cluster Comput

649 703

650 704

651 705

652 706

653 707

654 708

655 709

656 710

657 711

658 712

659 713

660 714

661 715

662 716

663 717

664 718

665 719

666 720

667 721

668 722

669 723

670 724

671 725

672 726

673 727

674 728

675 729

676 730

677 731

678 732

679 733

680 734

681 735

682 736

683 737

684 738

685 739

686 740

687 741

688 742

689 743

690 744

691 745

692 746

693 747

694 748

695 749

696 750

697 751

698 752

699 753

700 754

701 755

702 756

This leads to a second degree of heterogeneity, where onetype of platform can have instances in a data center thatare configured to operate at different frequencies. We re-fer to this as “within-platform heterogeneity”. As processvariations increasingly result in the binning of producedchips into different operating points, this within-platformheterogeneity becomes an inherent property of the gen-eral data center landscape. Finally, many of these platformsmay incorporate some processor dynamic power manage-ment (DPM) techniques that adaptively alter platform be-havior at runtime. This creates a third source of heterogene-ity, “DPM-capability heterogeneity”, where platforms withbuilt-in DPM hooks exhibit different power/performancecharacteristics from the ones with no DPM capabilities. InTable 1, we show how these three levels of heterogeneityquickly escalate the number of distinct platform configura-tions in a data center scenario.

All experimental power measurements are performed us-ing the Extech 380801 power analyzer. The power is mea-sured at the wall and represents total AC power consumptionof the entire system. The power numbers presented in thispaper are obtained by averaging the instantaneous systempower consumption over the entire run of each workload.Our assumption is that infrastructure support for monitor-ing power consumption will be utilized to obtain this typeof workload specific power characteristics online, insteadof parameterized models. For example, all power supplies,which adhere to the latest power supply monitoring interface(PSMI) specification, support out-of-band current/voltagesampling allowing for per platform A/C power monitoringreflected by our actual power measurements.

4.2 Application model

When power managing computing environments, improve-ments can be attained with a variety of approaches. In thiswork, we consider two scenarios. The first assumes a lackof budgeting constraints, concentrating on a workload allo-cation that reduces power consumption while maintainingbaseline application performance. In other words, we max-imize the performance per watt, while holding performanceconstant. The second addresses power budgeting by per-forming load shedding to reduce power consumption whileminimizing performance impact to workloads. We considerapplication performance in terms of throughput, or the rateat which transaction operations are performed. Therefore,it is not the execution time of each transaction that definesperformance, but the rate at which multiple transactions canbe sustained. This type of model is representative of appli-cations such as transaction based web services or payrollsystems.

The goal of HALM is to evaluate the power-efficiencytradeoffs of assigning a workload to a variety of platforms.

Since the performance capabilities of each platform are dif-ferent, the execution time to perform a single operable unit,or atomic transaction, varies across them. As previouslymentioned, virtualization technologies can help to extendthe physical resources dedicated to applications when nec-essary to maintain performance by increasing the number ofplatforms used to execute transactions. In particular, trans-actions can be distributed amongst nodes until the desiredthroughput is reached.

For our analysis, we consider applications that mimicthe high performance computational applications commonto data center environments and also heavily exercise thepower hungry components of server platforms, the proces-sor and memory. Two aspects of these workloads are cap-tured in our experimental analysis. First, these workloadsare inherently transactional, such as the previous financialpayroll example or the processing of risk analysis modelsacross different inputs common to investment banking. Sec-ond, with the ability to incorporate large amounts of memoryinto platforms at relatively low costs, these applications of-ten execute mostly from memory, with little or no I/O beingperformed. Though I/O such as network use can play a sig-nificant role in multi-tier enterprise applications, we leaveconsideration of such characteristics to future work. To real-ize our application model, while also providing determin-istic and repeatable behavior for our experimentation, weutilize benchmarks from the SPEC CPU2000 suite as rep-resentative examples of transaction instances. SPEC bench-marks allow for the isolation of processor and memory com-ponents, while also generating different memory loads. In-deed, many SPEC benchmarks exhibit significant measuredmemory bandwidth of 5–8 GB/sec on our systems. In orderto provide an unbiased workload set, we include all SPECbenchmarks in our experiments. For each application, wespecify an SLA in terms of required transaction processingrate, equal to the throughput achievable on the Woodcrestplatform.

5 Workload behavior estimation

The power/performance predictor component of our HALMframework can be implemented in multiple ways. For ex-ample, one can profile a set of microbenchmarks on all plat-form configurations and develop statistical mapping func-tions across these configurations. However, as the platformtypes and heterogeneity increase, the overhead of such ap-proaches can be prohibitive. Instead, we develop a predic-tor that relies on the architectural platform properties andadjusts its predictions based on the heterogeneity specifica-tions. We refer to this model as the “Blocking Factor (BF)Model”. The BF model simply decomposes execution cyclesinto CPU cycles and memory cycles. CPU cycles represent

« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 8»

Cluster Comput

757 811

758 812

759 813

760 814

761 815

762 816

763 817

764 818

765 819

766 820

767 821

768 822

769 823

770 824

771 825

772 826

773 827

774 828

775 829

776 830

777 831

778 832

779 833

780 834

781 835

782 836

783 837

784 838

785 839

786 840

787 841

788 842

789 843

790 844

791 845

792 846

793 847

794 848

795 849

796 850

797 851

798 852

799 853

800 854

801 855

802 856

803 857

804 858

805 859

806 860

807 861

808 862

809 863

810 864

Fig. 5 Performance predictionresults

the execution with a perfect last-level cache (LLC), whilememory cycles capture the finite cache effects. This model issimilar to the “overlap model” described by Chou et al. [5].With the BF model, the CPI (cycles per instruction) of aworkload can be represented as in (1). Here CPICORE repre-sents the CPI with a perfect LLC. This term is independentfrom the underlying memory subsystem. CPIMEM accountsfor the additional cycles spent for memory accesses with afinite-sized cache:

CPI = CPICORE + CPIMEM. (1)

The CPIMEM term can be expanded into architecture andworkload specific characteristics. Based on this, the CPI ofa platform at a specific frequency f1 can be expressed asin (2). Here, MPI is the memory accesses per instruction,which is dependent on the workload and the LLC size, L

is the average memory latency, which varies based upon thememory subsystem specifications, and BF is the blockingfactor that accounts for the overlapping concurrent execu-tion during memory accesses, which is a characteristic ofthe workload:

CPI(f1) = CPICORE(f1) + MPI · L(f1) · BF(f1). (2)

Using variants of (2), performance prediction can beperformed relatively easily for within-platform heterogene-ity, as well as across-platform heterogeneity. For within-platform heterogeneity, the frequency-dependent compo-nents of (2) are scaled with frequency to predict workloadperformance on a different frequency setting. The top chartin Fig. 5 provides results for an example of this type of pre-diction with an orthogonal platform (Sossaman). The figurecontains the actual measured performance for our workloadstogether with the predicted performance.

In the latter case of across-platform heterogeneity, thenatural decoupling of the microarchitectural and memory

subsystem differences in the BF model enables us to esti-mate application performance on a platform lying on a dif-ferent corner of the memory and microarchitecture hetero-geneity space. Among our four experimental platforms, two“orthogonal platforms”, which span two opposite cornersof the platform heterogeneity quadrants in Fig. 4(b), can beused to predict performance on a third “derived platform”.The lower chart in Fig. 5 shows the prediction results for theWoodcrest platform, whose performance is “derived” usingthe CPICORE and CPIMEM characteristics of the orthogonalplatforms (Sossaman and Dempsey respectively). Overall,for the orthogonal platforms, the BF model can very accu-rately predict performance with an average prediction errorof 2%. For the derived platforms, our predictor can trackactual execution times very well, though with an increasedaverage prediction error of 20%. In the following sections,we show that this performance prediction methodology pro-vides sufficient accuracy to represent workload behavior andallows HALM to achieve close to optimal allocations. Fur-ther details of this prediction methodology can be found inour previous work [22].

The final heterogeneity type supported by our predictoris the DPM-capability heterogeneity. For this, we considera platform that enables DVFS during memory bound exe-cution regions of an application. We implement this func-tionality as part of OS power management, based on priorwork [14]. To incorporate DPM awareness, we extend thepredictor component to estimate the potential power savingsthat can be attained when executing a workload on a DPMenabled platform. Experimental results show that there isa strong correlation between the MPI of a workload andits power saving potential. Therefore, we utilize the MPIattribute in the workload descriptors to predict the powersaving potentials of workloads on DPM enabled platforms.Figure 6 shows that our MPI based prediction approach ef-fectively captures the power saving potentials of differentworkloads and successfully differentiates applications that

« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 9»

Cluster Comput

865 919

866 920

867 921

868 922

869 923

870 924

871 925

872 926

873 927

874 928

875 929

876 930

877 931

878 932

879 933

880 934

881 935

882 936

883 937

884 938

885 939

886 940

887 941

888 942

889 943

890 944

891 945

892 946

893 947

894 948

895 949

896 950

897 951

898 952

899 953

900 954

901 955

902 956

903 957

904 958

905 959

906 960

907 961

908 962

909 963

910 964

911 965

912 966

913 967

914 968

915 969

916 970

917 971

918 972

Fig. 6 Power savingpredictions for DPM enabledplatforms

can benefit significantly from being allocated to a DPM en-abled machine. As we describe in Sect. 6.1, we use this pre-dictor to choose workloads that should be assigned to theDPM enabled platforms.

6 Management policies

6.1 HALM allocation policy

After processing the workload and platform descriptors,and utilizing our BF model for performance prediction, thenext step is to perform allocation of resources to a set ofapplications in a data center. Evaluations are based on agreedy policy for allocating workloads. In particular, witheach application i, we associate a cost metric for execut-ing on each platform type k. Workloads are then orderedinto a scheduling queue based on their maximum cost met-ric across all platform types. The allocator then performsapplication assignment based on this queue, where applica-tions with higher worst-case costs have priority. The plat-form type chosen for an application is a function of this costmetric across the available platforms as well as the estimatedDPM benefits. As a cost metric for our policy, we defineNi,k , the number of platforms of type k required to executea workload i, Ni,k . This value is clearly a function of boththe performance capabilities of the platform and the SLA re-quirement of the workload. Ni,k can be analytically defined,given the transaction based application model utilized in ourwork. For each application i, the service level agreement(SLA) specifies that Xi transactions should be performedevery Yi time units. If ti,k is the execution time of a trans-action of application i on platform k, the resulting numberof platforms required to achieve the SLA can be expressedwith (3)

Ni,k =⌈

Xi

�Yi/ti,k�⌉. (3)

The ti,k values are provided by the performance predictor.It should be noted that there is a discretization in Ni,k , whichis due to the fact that individual atomic transactions cannotbe parallelized across multiple platforms. Ni,k is thereforebetter able to handle errors due to the inherent discretiza-tion performed, making it a strong choice as a cost metric

(other possible metrics are discussed and defined in our pre-vious work [22]). Given the use of Ni,k as our cost metric,our allocation approach first determines the platform typesof which (1) there are enough available systems to allocatethe workload and (2) the cost metric is minimized. We thenuse DPM savings to determine whether a more power ef-ficient platform alternative should be used between thosewith the same cost value. In other words, if there are mul-tiple platform types for which an application has the sameNi,k value, we utilize a DPM specific threshold to decidewhether or not it should be scheduled to a DPM enabledplatform type. As we demonstrate in the following section,this threshold based approach can be effective in identifyingworkloads that can take advantage of DPM capabilities.

6.2 HALM power budgeting policy

In order to address transient power delivery or cooling is-sues, it is sometimes necessary to temporarily reduce powerconsumption in a data center. To provide this mechanism,we develop a load shedding policy based upon an existingworkload allocation scheme. The goal of the policy is to re-duce the amount of resources provided to applications in or-der to meet a power budget while still allowing all workloadsto make some progress. In other words, application perfor-mance may be degraded compared to prescribed SLAs, butall applications achieve some fraction of their SLA.

Our power budgeting policy is, again, a greedy approach.For all applications with resources that can be shed, i.e.applications that utilize more than one platform, we de-fine a power-efficiency metric as the throughput per Wattthat is being provided by each resource. Afterwards, the re-sources with minimal power efficiency are shed until thepower budget is met. As our experimental results demon-strate, this simple metric allows for improved performancewhen power budgeting must be performed, as well as betterfairness across workloads in terms of performance degrada-tion experienced.

7 Experimental evaluation

7.1 Increasing power efficiency

In order to evaluate our heterogeneity-aware allocation ap-proach, we perform power and performance measurements

« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 10»

Cluster Comput

973 1027

974 1028

975 1029

976 1030

977 1031

978 1032

979 1033

980 1034

981 1035

982 1036

983 1037

984 1038

985 1039

986 1040

987 1041

988 1042

989 1043

990 1044

991 1045

992 1046

993 1047

994 1048

995 1049

996 1050

997 1051

998 1052

999 1053

1000 1054

1001 1055

1002 1056

1003 1057

1004 1058

1005 1059

1006 1060

1007 1061

1008 1062

1009 1063

1010 1064

1011 1065

1012 1066

1013 1067

1014 1068

1015 1069

1016 1070

1017 1071

1018 1072

1019 1073

1020 1074

1021 1075

1022 1076

1023 1077

1024 1078

1025 1079

1026 1080

of our SPEC based representative transactional workloadsacross each type of platform. To scale these results to thenumber of platforms present in data centers, this measureddata is extrapolated analytically using a data center alloca-tion simulator which combines real power and performancedata, prediction output, and allocation policy definitions tocalculate power efficiency in various data center configura-tions. In the simulator, we provide the output of the predictoras input to the allocation policy. We always assume that theplatforms which are profiled are the 2 GHz Sossaman plat-form and the 3.7 GHz Dempsey system. Since we assumethe workload attributes are profiled accurately on these sys-tems, for fairness we also assume that for these two plat-forms performance data is obtained via profiling as well andis therefore known perfectly. We then consider three differ-ent scenarios: (1) all other platform performance informa-tion is known perfectly (oracle) (2) our BF model is used topredict performance for the remainder of platforms as de-scribed in Sect. 5 (BF model) (3) incorporating a simple sta-tistical regression approach (Stat. Est.). For this regressionmethod, we profile a subset of applications across all plat-forms to obtain linear performance prediction models para-meterized by variables that can be obtained by profiling aworkload on the 2 GHz Sossaman and 3.7 GHz Dempseysystems (CPI, MPI, etc.). The regression models can thenbe used to predict performance of any application. The base-line allocation scheme we compare against is a random one,since it closely estimates the common round-robin or uti-lization based approach.

The efficiency improvements achievable in a data centerare also dependent upon the system’s current mix of appli-cations. To obtain our results, we randomly pick applica-tions and allocate them using the random approach until nomore workloads can be scheduled. Using the resulting setof workloads, we then evaluate power consumption whenusing our prediction and allocation policies, and comparethem against the random allocation result. This is repeated ahundred times for each of our data points.

We first look at the benefits achieved with HALM indata center configurations with across-platform and within-

platform heterogeneity but no DPM support. In particular,we include the four base platforms, Woodcrest, Sossaman,Dempsey, and Irwindale, as well as the frequency variationsof the platforms. We create data center configurations withequal numbers of each type of system. Trends are consistentacross various data center sizes, so for brevity, we includehere only results with 1000 platforms of each type. The re-sulting data center has 13 types of platforms, and power con-sumptions vary with allocation as shown in Fig. 7(a). Thefirst interesting observation is that platform heterogeneity al-lows us to achieve improved benefits over a simple randomapproach. Indeed, we see improvements of 22% with perfectknowledge and 21% using our BF based prediction com-pared to a random allocation policy. We also observe a sig-nificant difference between the statistical and analytical pre-diction schemes. The regression approach is unable to scalein terms of accuracy with increased heterogeneity, whereasthe BF approach achieves close to optimal power savings.

In order to evaluate how well our allocator can exploitDPM support, we extend the thirteen platform type configu-ration with an additional Woodcrest 3 GHz platform whichprovides DPM support. We again find that our BF predictionmethod can provide improved aggregate savings across allmachines over the statistical approach as shown in Fig. 7(b).To more closely determine our ability to exploit DPM mech-anisms, we also evaluate the power consumption of the thou-sand DPM-enabled platforms (all of which are active). Wefind that our BF model based allocation is able to improvethe power efficiency of these platforms by 3.3%. This illus-trates the potential of HALM to provide additional benefitswhen platforms vary in the power management they support.

7.2 Maximizing performance under power budgets

As a second benefit of HALM, we evaluate the ability toperform load shedding when power budgeting must be per-formed. In particular, our goal is to maximize performanceobtained in the data center while observing power budgets.For the purposes of this paper, we consider transient power

Fig. 7 HALM power improvements

« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 11»

Cluster Comput

1081 1135

1082 1136

1083 1137

1084 1138

1085 1139

1086 1140

1087 1141

1088 1142

1089 1143

1090 1144

1091 1145

1092 1146

1093 1147

1094 1148

1095 1149

1096 1150

1097 1151

1098 1152

1099 1153

1100 1154

1101 1155

1102 1156

1103 1157

1104 1158

1105 1159

1106 1160

1107 1161

1108 1162

1109 1163

1110 1164

1111 1165

1112 1166

1113 1167

1114 1168

1115 1169

1116 1170

1117 1171

1118 1172

1119 1173

1120 1174

1121 1175

1122 1176

1123 1177

1124 1178

1125 1179

1126 1180

1127 1181

1128 1182

1129 1183

1130 1184

1131 1185

1132 1186

1133 1187

1134 1188

Fig. 8 Power budgeting results

budgeting where workloads are not (re-)migrated, but in-stead, based upon an existing allocation, resources are tem-porarily withheld from applications and placed into lowpower states. For comparison, we consider three such ini-tial allocations from the previous section, one based upona random allocation, one based upon perfect oracle perfor-mance information, and finally an allocation based upon pre-diction with our BF approach. For each allocation, we con-sider two load shedding policies, one which randomly se-lects an application and sheds resources (i.e. a single com-pute node) if possible, and our greedy “smart” policy de-scribed in Sect. 6.2.

Figure 8(a) provides the aggregate performance acrossall workloads in the data center for different power budgets.The performance results are normalized with respect to themaximum achievable performance at the maximum uncon-strained power budget (100%). The figure shows how theperformance of different allocation policies decrease withdecreasing power budgets. The range of applied power bud-gets is limited to 50% as there is no possible allocation thatmeets the budget below this point. The six scenarios con-sidered are random shedding based upon a random allo-cation (rand-rand), our load shedding policy based upon arandom allocation (smart-rand), similarly the two sheddingapproaches based upon oracle based allocation (rand-oracleand smart-oracle), and finally, shedding based upon a BFprediction based allocation (rand-BF and smart-BF). We seemultiple interesting trends. First, the intrinsic benefits of theheterogeneity-aware allocations towards budgeting are ap-parent in the figure by the fact that performance does notbegin to reduce until lower power budgets compared to arandom allocation scheme. We also see that given a particu-lar allocation, our shedding policy provides benefits in per-formance across the set of power budgets. Moreover, again,we find that our BF prediction model behaves very close toan oracle based allocation when our load shedding policy isused. Overall, we see benefits of up to 18% in performancedegradation compared to a random load shedding policybased upon a random allocation. Figure 8(b) evaluates theperformance degradations for the six approaches by also

taking the fairness of load shedding into account. Here, weshow the aggregate data center performance as the harmonicmean of the individual workload throughputs, which is acommonly used metric for evaluating fairness [20]. We seefrom the figure that a random allocation always exhibits poorperformance regardless of the load shedding policy used.A heterogeneity-aware allocation, on the other hand, pro-vides improved fairness, particularly when combined withour load shedding policy.

8 Related work

HALM builds upon existing work and extends the state ofthe art in power management research. A variety of mecha-nisms exist to provide power and thermal management sup-port within a single platform. Brooks and Martonosi pro-posed mechanisms for the enforcement of thermal thresh-olds on the processor [3], focusing on single platform char-acteristics as opposed to the data center level managementachieved with HALM. Processor frequency and voltagescaling based upon memory access behavior has been shownto successfully provide power savings with minimal impactto applications. Resulting solutions include hardware basedapproaches [19] and OS-level techniques, that set processormodes based on predicted application behavior [14]. HALMis designed to manage load while being aware of such un-derlying power management occurring in server platforms.Power budgeting of SMP systems with a performance lossminimization objective has also been implemented via CPUthrottling [16]. Other budgeting solutions extend platformsupport for fine grain server power limiting [18]. The powerbudgeting achieved with HALM is based upon the use of re-source allocation to reduce power consumption across mul-tiple systems, as opposed to throttling performance of indi-vidual components.

At the data center level, incorporating temperature-awareness into workload placement has been proposed byMoore et al. [21], along with emulation environments forstudies of thermal implications of power management [11].HALM can use these thermal aware strategies to perform

« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 12»

Cluster Comput

1189 1243

1190 1244

1191 1245

1192 1246

1193 1247

1194 1248

1195 1249

1196 1250

1197 1251

1198 1252

1199 1253

1200 1254

1201 1255

1202 1256

1203 1257

1204 1258

1205 1259

1206 1260

1207 1261

1208 1262

1209 1263

1210 1264

1211 1265

1212 1266

1213 1267

1214 1268

1215 1269

1216 1270

1217 1271

1218 1272

1219 1273

1220 1274

1221 1275

1222 1276

1223 1277

1224 1278

1225 1279

1226 1280

1227 1281

1228 1282

1229 1283

1230 1284

1231 1285

1232 1286

1233 1287

1234 1288

1235 1289

1236 1290

1237 1291

1238 1292

1239 1293

1240 1294

1241 1295

1242 1296

power budgeting based upon data center temperature charac-teristics. Chase et al. discuss how to reduce power consump-tion in data centers by turning servers on and off based ondemand [4]. Utilizing this type of cluster reconfiguration inconjunction with DVFS [7] and the use of spare servers [23]has been investigated as well. As opposed to these ap-proaches, HALM attempts to reduce power consumptionby intelligently managing workloads across heterogeneousservers. Enforcing power budgets within data centers by al-locating power in a non-uniform manner across nodes hasbeen shown to be an effective management technique [9].Techniques for enforcing power budgets at blade enclosuregranularities have also been discussed [24]. HALM bud-gets aggregate power consumption via resource allocationwithout assigning per server power budgets as with theseprevious approaches.

Heterogeneity has been considered to some degree inprior work, including the evaluation of heterogeneous multi-core architectures with different core complexities [17]. Incomparison, HALM considers platform level heterogene-ity as opposed to processor asymmetry. In cluster environ-ments, a scheduling approach for power control has beenproposed for processors with varying fixed frequencies andvoltages [10]. HALM supports heterogeneity across addi-tional dimensions, such as power management capabilitiesand memory. A power efficient web server with intelligentrequest distribution in heterogeneous clusters is another ex-ample which considers leveraging heterogeneity in enter-prise systems [12]. HALM goes beyond these methods, byconsidering not just the differences in platforms’ perfor-mance capabilities, but also in their power management ca-pabilities.

9 Conclusions and future work

Power management in data center environments has becomean important area of research, in part because power deliveryand cooling limitations are quickly becoming a bottleneckin the provisioning of performance required by increasinglydemanding applications. This paper makes use of the man-agement flexibility afforded by virtualization solutions todevelop a heterogeneity-aware load management (HALM)system. HALM improves power management capabilitiesby exploiting the natural heterogeneity of platforms in datacenters, including differences in dynamic power manage-ment support that may be available. We introduce a threephase approach to mapping workloads to underlying re-sources to improve power efficiency consisting of struc-tured platform and workload descriptors, a prediction com-ponent to estimate the performance and power characteris-tics of various workload to platform mappings, and finallyan allocator which utilizes policies and prediction results to

perform decisions. We also evaluate a load shedding pol-icy based upon resulting allocations to improve performancewhen power budgeting must be performed.

Our results underscore two major conclusions. First, weshow that by intelligently considering the varying powermanagement capabilities of platforms, the ability for thesesystems to obtain power savings using their managementmechanisms can be vastly improved when compared toother assignment models. Using representative data centerconfigurations consisting of older P4 based platforms up toIntel Core microarchitecture based systems, we find that ourallocation architecture can improve power efficiency by 20%on average. In addition, our results show that by performingintelligent load shedding when power budgets must be ob-served, 18% improvements in performance degradation canbe obtained when using HALM.

In this paper, we present the beginning of our investi-gation into exploiting platform heterogeneity and emerg-ing virtualization support to improve the power characteris-tics of enterprise computing environments. As future work,we plan to integrate the management tradeoffs and lessonslearned from this work into virtualization layer managementapplications. This includes the consideration of distributedvirtualized workloads such as tiered web services where dif-ferent components may be appropriate for each layer, in-cluding heterogeneous I/O devices. The results presented inthis paper support the potential of this area of research forpower managing heterogeneous computing systems.

References

1. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A.,Neugebauer, R., Pratt, I., Warfield, A.: Xen and the art of virtu-alization. In: Proceedings of the ACM Symposium on OperatingSystems Principles (SOSP), 2003

2. Bianchini, R., Rajamony, R.: Power and energy management forserver systems. IEEE Comput. 37(11), 68–76 (2004)

3. Brooks, D., Martonosi, M.: Dynamic thermal management forhigh-performance microprocessors. In: Proceedings of the 7th In-ternational Symposium on High-Performance Computer Architec-ture (HPCA), January 2001

4. Chase, J., Anderson, D., Thakar, P., Vahdat, A., Doyle, R.: Manag-ing energy and server resources in hosting centers. In: Proceedingsof the 18th Symposium on Operating Systems Principles (SOSP),October 2001

5. Chou, Y., Fahs, B., Abraham, S.: Microarchitecture optimizationsfor exploiting memory-level parallelism. In: Proceedings of theInternational Symposium on Computer Architecture (ISCA), June2004

6. Clark, C., Fraser, K., Hand, S., Hansen, J.G., Jul, E., Limpach,C., Pratt, I., Warfield, A.: Live migration of virtual machines. In:Proceedings of the 2nd ACM/USENIX Symposium on NetworkedSystems Design and Implementation (NSDI), May 2005

7. Elnozahy, E.N., Kistler, M., Rajamony, R.: Energy-efficient serverclusters. In: Proceedings of the Workshop on Power-Aware Com-puting Systems, February 2002

8. Fan, X., Weber, W.-D., Barroso, L.: Power provisioning for awarehouse-sized computer. In: Proceedings of the InternationalSymposium on Computer Architecture (ISCA), June 2007

« CLUS 10586 layout: Large v.1.3 reference style: mathphys file: clus54.tex (petras) aid: 54 doctopic: OriginalPaper class: spr-twocol-v1 v.2008/04/14 Prn:15/04/2008; 16:06 p. 13»

Cluster Comput

1297 1351

1298 1352

1299 1353

1300 1354

1301 1355

1302 1356

1303 1357

1304 1358

1305 1359

1306 1360

1307 1361

1308 1362

1309 1363

1310 1364

1311 1365

1312 1366

1313 1367

1314 1368

1315 1369

1316 1370

1317 1371

1318 1372

1319 1373

1320 1374

1321 1375

1322 1376

1323 1377

1324 1378

1325 1379

1326 1380

1327 1381

1328 1382

1329 1383

1330 1384

1331 1385

1332 1386

1333 1387

1334 1388

1335 1389

1336 1390

1337 1391

1338 1392

1339 1393

1340 1394

1341 1395

1342 1396

1343 1397

1344 1398

1345 1399

1346 1400

1347 1401

1348 1402

1349 1403

1350 1404

9. Femal, M., Freeh, V.: Boosting data center performance throughnon-uniform power allocation. In: Proceedings of the Second In-ternational Conference on Autonomic Computing (ICAC), 2005

10. Ghiasi, S., Keller, T., Rawson, F.: Scheduling for heterogeneousprocessors in server systems. In: Proceedings of the InternationalConference on Computing Frontiers, 2005

11. Heath, T., Centeno, A.P., George, P., Ramos, L., Jaluria, Y., Bian-chini, R.: Mercury and freon: Temperature emulation and manage-ment in server systems. In: Proceedings of the International Con-ference on Architectural Support for Programming Languages andOperating Systems (ASPLOS), October 2006

12. Heath, T., Diniz, B., Carrera, E.V., Meira, W. Jr., Bianchini, R.:Energy conservation in heterogeneous server clusters. In: Proceed-ings of the 10th Symposium on Principles and Practice of ParallelProgramming (PPoPP), 2005

13. Hewlett-Packard, Intel, Microsoft, Phoenix, and Toshiba: Ad-vanced configuration and power interface specification. http://www.acpi.info (2004)

14. Isci, C., Contreras, G., Martonosi, M.: Live, runtime phase mon-itoring and prediction on real systems with application to dy-namic power management. In: Proceedings of the 39th Interna-tional Symposium on Microarchitecture (MICRO-39), December2006

15. Koh, Y., Knauerhase, R., Brett, P., Bowman, M., Wen, Z., Pu, C.:An analysis of performance interference effects in virtual environ-ments. In: Proceedings of the IEEE International Symposium onPerformance Analysis of Systems and Software (ISPASS), 2007

16. Kotla, R., Ghiasi, S., Keller, T., Rawson, F.: Scheduling processorvoltage and frequency in server and cluster systems. In: Proceed-ings of the Workshop on High-Performance, Power-Aware Com-puting (HP-PAC), 2005

17. Kumar, R., Tullsen, D., Ranganathan, P., Jouppi, N., Farkas,K.: Single-Isa heterogeneous multi-core architectures for multi-threaded workload performance. In: Proceedings of the Interna-tional Symposium on Computer Architecture (ISCA), June 2004

18. Lefurgy, C., Wang, X., Ware, M.: Server-level power control. In:Proceedings of the IEEE International Conference on AutonomicComputing (ICAC), June 2007

19. Li, H., Cher, C., Vijaykumar, T., Roy, K.: Vsv: L2-miss-drivenvariable supply-voltage scaling for low power. In: Proceed-ings of the IEEE International Symposium on Microarchitecture(MICRO-36), 2003

20. Luo, K., Gummaraju, J., Franklin, M.: Balancing throughput andfairness in SMT processors. In: Proceedings of the IEEE Interna-tional Symposium on Performance Analysis of Systems and Soft-ware (ISPASS), November 2001

21. Moore, J., Chase, J., Ranganathan, P., Sharma, R.: Makingscheduling cool: Temperature-aware workload placement in datacenters. In: Proceedings of USENIX ’05, June 2005

22. Nathuji, R., Isci, C., Gorbatov, E.: Exploiting platform heterogene-ity for power efficient data centers. In: Proceedings of the IEEEInternational Conference on Autonomic Computing (ICAC), June2007

23. Rajamani, K., Lefurgy, C.: On evaluating request-distributionschemes for saving energy in server clusters. In: Proceedings ofthe IEEE International Symposium on Performance Analysis ofSystems and Software (ISPASS), March 2003

24. Ranganathan, P., Leech, P., Irwin, D., Chase, J.: Ensemble-levelpower management for dense blade servers. In: Proceedings ofthe International Symposium on Computer Architecture (ISCA),2006

25. Sugerman, J., Venkitachalam, G., Lim, B.-H.: Virtualizing i/o de-vices on VMware workstation’s hosted virtual machine monitor.In: Proceedings of the USENIX Annual Technical Conference,2001

26. Zhang, W.: Linux virtual server for scalable network services. In:Ottawa Linux Symposium, 2000

Ripal Nathuji is a Ph.D. candidatein the School of Electrical and Com-puter Engineering at the Georgia In-stitute of Technology. He previouslyreceived his M.S. in Computer En-gineering from Texas A&M Univer-sity and his B.S. in Electrical Engi-neering and Computer Science fromthe Massachusetts Institute of Tech-nology. His current research focuseson system-level power managementof computing systems, with applica-tions to virtualized enterprise serverenvironments.

Canturk Isci is a senior memberof technical staff at VMware. Hisresearch interests include power-aware computing systems and work-load-adaptive dynamic management.He has a Ph.D. and an M.A. in elec-trical engineering from PrincetonUniversity, an M.S. in VLSI systemdesign from University of Westmin-ster, London, UK, and a B.S. in elec-trical engineering from Bilkent Uni-versity, Ankara, Turkey.

Eugene Gorbatov is a researcher atIntel’s Energy Efficient Systems lab.His current research focuses on plat-form power management, particu-larly the interaction of different plat-form components and system soft-ware.

Karsten Schwan is a professor inthe College of Computing at theGeorgia Institute of Technology, andis also the Director of the Centerfor Experimental Research in Com-puter Systems (CERCS). He ob-tained his M.S. and Ph.D. degreesfrom Carnegie-Mellon University inPittsburgh, Pennsylvania, where hebegan his research in high perfor-mance computing, addressing op-erating and programming systemssupport for the Cm* multiprocessor.His current work ranges from topics

in operating and communication systems, to middleware, to paralleland distributed applications, focusing on information-intensive distrib-uted applications in the enterprise domain and in the high performancedomain.