PAC485Managing Datacenter Resources

Using the VirtualCenter Distributed Resource Scheduler

Carl WaldspurgerPrincipal Engineer, R&D

This presentation may contain VMware confidential information.

Copyright © 2005 VMware, Inc. All rights reserved. All other marks and names mentioned herein may

be trademarks of their respective companies.

Talk Overview

Context and featuresManaging resourcesVirtual machine placementSystem architectureSummary

What Is DRS?DRS = Distributed Resource SchedulerAutomatic virtual machine placement

Optimize load balance across hostsDecide if, when, and where to migrateReact to dynamic load changes

Cluster-wide resource managementScalable resource controlsConfigurable automation levelsIntegrated UI for all controls

DRS Can Help You…

Manage variable loadsWorkloads often dynamic, time-dependentQuickly shift loads in response to demand

Administer many virtual machinesHierarchical organizationDelegated administration

Move towards utility computingThink more about aggregate resource poolsThink less about individual hosts

Where Does DRS Fit In?

New productRequires VirtualCenter 2 and ESX Server 3Modular plug-in for VirtualCenter

DRS moduleImplements algorithms, enforces policiesManaged using VirtualCenter UI

Leverages core technologiesVMotion for migrating live VMs across hostsSophisticated resource management

Key FeaturesVirtual machine placement

Choose initial host when VM powers onDynamic rebalancing using VMotion

Configurable automation levelsManual – recommend initial host and migrationsPartial – automatic initial host, recommend migrationsFull – automatic initial host and migrations

Resource poolsFlexible grouping, sharing, and isolationHierarchical organization and delegation

Talk Overview

Context and featuresManaging resourcesVirtual machine placementSystem architectureSummary

Managing ResourcesBasic controls

Same as in current products Shares – specify relative importanceMin – guaranteed resource availabilityMax – limit resource consumption

Resource poolsNew feature leveraged by DRS Hierarchical management

Basic Control: SharesImportance

Entitlement directly proportional to sharesAnalogy: shares of stock in corporation

Relative unitsAbstract number, only ratios matterEntitlement depends on total shares issued

Named valuesPredefine high, normal, low with 4 : 2 : 1 ratioDefaults to normal

Shares Examples

Change shares forvirtual machineDynamic reallocation

Add virtual machine, overcommit resourcesGraceful degradation

Remove virtual machineExploit extra resources

Basic Control: MinGuaranteed resources

Minimum service level reservationEven when system overcommitted

Absolute unitsMHz for cpu, MB for memoryDefaults to zero for cpu, memory

Virtual machine admission controlReserve resources for minsSum of all VM mins ≤ capacityPrevent power-on if check fails

Min ExampleTotal capacity

600 MHz reserved400 MHz available

Admission control2 VMs try to power-on300 MHz min eachUnable to admit both

VM1 powers onVM2 not admitted

VM1VM1 VM2VM2

Basic Control: MaxResource limit

Upper bound on consumptionEven when system undercommitted

Absolute unitsMHz for CPU, MB for memoryDefaults to “unlimited” for cpuDefaults to guest RAM size for memory

Max ExampleCurrent utilization

600 MHz active400 MHz idle

Start CPU-bound VM200 MHz maxExecution throttled

New utilization800 MHz active200 MHz idleVM prevented fromusing idle resources

VMVM

Resource EntitlementsResources that each VM “deserves”

Combining shares, min, and maxAllocation primarily based on sharesConstrained by min and max

What if VM idles?Don’t give VM more than it demandsResources redistributed to active VMsUnused mins not wasted

Resource PoolsMotivation

Allocate aggregate resources for sets of VMsIsolation between pools, sharing within poolsFlexible hierarchical organizationAccess control and delegation

What is a resource pool?Named object in VirtualCenter inventoryAccess control permissionsMin, max, and shares for both CPU and memoryParent pool, child pools and VMs

Resource Pools ExampleAdmin manages usersPolicy: Alice’s share50% more than Bob’sUsers manage own virtual machines Not shown: min, maxVM allocations:

Bob

200 Admin

VM3

400 Bob

Alice300 Admin

75 Alice 75 Alice

Admin

30%

30%

40%

VM2VM1

Example: Bob Adds Virtual MachineSame policyPools isolate usersAlice still gets 50%more than BobVM allocations:

Bob

200 Admin

400 Bob

Alice300 Admin

75 Alice 75 Alice

Admin

800 Bob

30%

30% 13%

27%VM3VM2VM1 VM4

Resource Pool Admission ControlPool admission control

Same check as before, at pool levelSum of mins for pool children ≤ pool capacityWhen create pool, power-on VM, change settings

Growable Min optionDynamically request more capacity from parentSimplifies policies where hard partitions too rigid

Resource Pools UI

Delegated AdministrationCluster administrator

Default pool contains all cluster resourcesAggregate cpu and memory capacity of all hostsCarves up cluster resources into poolsProvides bulk allocations to pool administrators

Pool administratorPool may reflect department, project, client, etc.Carves up pool resources into smaller pools for users

End userAllocates resources from personal pool to virtual machinesView restricted to personal pool hierarchy

Best PracticesUse Mins and Shares appropriately

Shares generally more flexible policy toolUse shares to isolate without hard partitioningUse mins to guarantee acceptable service

Maintain some spare capacityDon’t use mins that commit entire clusterSlack for maintenance, rebalancingNeeded to tolerate host failures

Talk Overview

Context and featuresManaging resourcesVirtual machine placementSystem architectureSummary

Virtual Machine PlacementGoals

Balance virtual machine load across hosts in clusterEnforce resource policies accuratelyRespect placement constraints

Dynamic balancingMonitor key virtual machine, pool, and host metricsDeliver entitled resources to pools and VMsRecommend migrations (prioritized list)

Initial placementPower on virtual machine in resource poolRecommend host (prioritized list)

Placement ConstraintsVMotion compatibility

Processor typeSAN and LAN connectivity

Anti-affinity rulesRun virtual machines on different hostsMotivation: high-availability, clustering

Affinity rulesRun virtual machines on same hostMotivation: locality, performance benefits

Dynamic BalancingWhat to balance?

Load, adjusted for resource entitlementLoad = utilization, if all VMs equally important

When to balance?Re-evaluate every few minutesChanges to pool or VM settingsAdd or remove host

AggressivenessMigration rate, recommendation strengthDepends on severity of imbalance

Balancing DetailsCompute virtual machine entitlements

Based on pool and virtual machine resource allocationsDon’t give virtual machine more than it demandsReallocate extra resources fairly

Compute host loadsSum entitlements for virtual machines on hostNormalize by host capacity

Consider possible VMotionsEvaluate effect on cluster balanceEvaluate migration cost for involved hosts

Recommend best moves (if any)

Dynamic Balancing UI

Initial Placement UI

Best PracticesFollow strong recommendations

Otherwise balance and fairness may deteriorateSome VMotion is necessary

Enable automationChoose default based on environment, comfort levelUse per-VM automation level overridesLet DRS autonomously manage most VMsCan keep human in loop for critical VMs

Talk Overview

Context and featuresManaging resourcesVirtual machine placementSystem architectureSummary

System Architecture Overview

VirtualCenter

clients

DB

cluster n

•••cluster 1

••• •••

stats + actions

SDKUI

DRS

DRS1

n

•••

System Architecture ConstraintsCluster size

LAN, not WANUp to 32 hosts per clusterHost capacities may differ significantly

Time scaleMinutes, not millisecondsVMotion VM downtime ≈ milliseconds,but end-to-end latency ≈ tens of seconds Migrate VM infrequently ≈ minutes to hours

Algorithm performanceMilliseconds, not minutesOperations occur at human time scale

SummaryAutomatic virtual machine placement

Recommendations or full automationInitial placement at virtual machine power-onDynamic load balancing

Powerful resource controlsFlexible cluster-wide policiesHierarchical resource poolsVirtual machine affinity rules

Future directionsIntegrated I/O bandwidth managementDetect longer-term trends, proactive migration