Debbie Sheetz Sr. Staff Consultant BMC Software, Customer Support/Waltham, MA Top Ten Topics in...

Debbie SheetzSr. Staff ConsultantBMC Software, Customer Support/Waltham, MA

Top Ten Topics in Performance Reporting/Modeling for Virtualization using BMC Performance Assurance

© Copyright 04/10/23 BMC Software, Inc.2

Presentation Overview

› Overview of Customer-Selected Virtualization Topics– AIX Partitions

• SMT effect on Performance reporting• Performance reporting• Server consolidation to AIX partitions

– VMware ESX Server• Server consolidation to VMware• Performance reporting

– HP Partitions • Performance reporting

– Solaris Container (Zone)• Performance reporting• Modeling techniques and results reporting


Why virtualization?

› IT needs to improve the ratio between service delivery and cost – More cost-effective hardware can be purchased (lower power)– Higher concentrations of hardware resources = lower cost

• Hardware cost/transaction• Manpower required for management• Software license cost• Square foot/HVAC cost

› A popular choice is mainframe-like Distributed Systems hardware with Partitioning technology

– Allows dissimilar workloads to coexist, but to be managed separately– Allows for higher resource utilization, e.g. same hardware can be shared by

work which has non-overlapping peaks


Different Virtualization Strategies from each Vendor

› Several virtualization strategies are popular – IBM partitions– VMware virtual machines– HP partitions– Solaris containers

› Currently the most popular choice among BPA customers is IBM partitioning

– Many companies have production AIX 5.3 partitioned environments, and the number of POWER5 CPUs managed this way is growing

› This presentation assumes some familiarity with IBM’s AIX Partitioning, HP partitioning, VMware, and Solaris Containers

› This presentation assumes basic familiarity with the BMC Performance Assurance components: Analyzer, Predictor, Visualizer, Manager


Tools shown in this presentation

› This presentation shows the combination of BMC Performance Assurance for Virtual Servers (Release 7.4 of Analyzer/Predictor and Visualizer 4.2.02) and extensions distributed by BMC Customer Support as a result of field experience with these products

– Visualizer extensions available at ftp://ftp.bmc.com/pub/visualizer/vgd/ • Most of the custom metrics can be obtained from “out-of-the-box” graphics, with

different metric labels, metric groupings, and/or metric units– Presentation surveying different Virtualization Strategies available from

Customer Support (Do CPUs Count? written for UKCMG 2007)– Specific Virtualization techniques white papers available from BMC Customer

Support (FAQs for AIX Partitions, FAQs for VMware)– Performance Assurance Suite for Virtual Servers, product documentation (

http://documents.bmc.com/supportu/documents/88/98/68898/68898.pdf)› Not shown in this presentation (but part of BPA Virtual Servers)

– Perceiver 7.2.20 – Investigate 7.4

ftp://ftp.bmc.com/pub/visualizer/vgd/



http://documents.bmc.com/supportu/documents/88/98/68898/68898.pdf


Topic: How SMT is reported

› What is SMT?– IBM’s Simultaneous Multi-Threading provides resources to dispatch two

threads simultaneously to one physical processor, i.e. core (pretty useful doc at http://www-03.ibm.com/servers/eserver/iseries/perfmgmt/pdf/SMT.pdf) • This is NOT dual-core which provides two independent physical processors

– Available first with AIX 5.2 on POWER5 processors• Partitioning coexists with SMT, but does not depend on it

Diagram from IBM redbook Advanced POWER Virtualization on IBM System p5

http://www-03.ibm.com/servers/eserver/iseries/perfmgmt/pdf/SMT.pdf


How SMT is reported

› What is SMT (continued)?– How does it look when it’s reported?

• Each physical processor is reported as two logical processors by AIX• AIX CPU accounting (i.e. PURR) ensures that CPU usage metrics are correct

– The sum of the two logical processors’ CPU utilization = utilization of one physical processor

– This is NOT like Windows Hyper-Threading which doesn’t have equivalent CPU metrics› Approaches for reporting SMT

– SMT-aware reporting• Logical/physical processor relationship is exploited

– Recommended BPA solution is at least 7.3 Performance Assurance console and AIX 7.3 collector for AIX 5.2/5.3

– BMC Performance Manager (formerly PATROL) support first available with AIX 7.2.10 collector

– Non-SMT aware reporting• CPU utilization (i.e. % of capacity in use) is reported as ½ of what it really is

– Performance Assurance 7.2


How SMT is reported

› How is SMT presented?– Logical processors are aggregated to physical processors for

• Analyzer reports• Predictor• Visualizer (and Perceiver using Visualizer datasource)

– Logical processor view for• Native UNIX tools (e.g. lcpu, lbusy, etc. means Logical processor)• Raw measurement data per processor

– Investigate (and Perceiver using UDR datasource)– Analyzer

» Interval (“browse data”) view» Computer CPU Utilization report (Perceiver CPU Utilization – All Processors chart)

– Details “by metric and metric group” available in FAQs for AIX Partitions white paper › The aggregated (physical processor) view ensures consistency for CPU reporting, regardless of whether or not SMT is enabled

– Can compare across AIX systems, other UNIX systems


Topic: LPAR performance reporting challenges

› What is unique about performance reporting for LPARs (partitions)?– This is not “business as usual”, i.e. CPU capacity only dependent on a fixed

number of available CPUs. Instead, CPU capacity now depends on three levels of processor allocation• (1) Total physical processors• (2) Processors assigned to DLPARs vs. Shared Pool AND• (3) Each DLPAR requires adequate processors• (3) Each SPLPAR requires adequate

– Virtual processors AND– Entitlement processors AND– Shared POOL processors

• And all of this while workloads are changing– Traditional CPU capacity approaches cannot capture these simultaneous

complexities


Topic: How to report CPU for a Physical Frame

› How is CPU reported for the physical system from the CDB?Simultaneously shows %

of capacity used in 3 categories, per physical system:

1) total physical system use

2) average SPLPAR use of the entitlement processor allocation, and

3) average SPLPAR use of the virtual processor allocation

Note: This is a “custom” Visualizer graphic; also in Perceiver 7.2.20 AIX Physical System Overview


How to report CPU for a Physical Frame

› How is CPU reported for the physical system from the CDB?• Supplement previous

view for the physical system with graphics for individual partitions

• Top 10 Virtual Used and

• Top 10 Entitlement Used (next slide)

• Gives total picture for long-term trending and looks for individual trouble spots that need immediate attention

Note: This is a “custom” Visualizer graphic; also in Perceiver 7.2.20 AIX Physical System Overview



› How is CPU reported for the physical system from the CDB?• Supplement previous

view for the physical system with graphics for individual partitions

• Top 10 Virtual Used (previous slide) and

• Top 10 Entitlement Used

• Gives total picture for long-term trending and looks for individual trouble spots that need immediate attention

Note: This is a “custom” Visualizer graphic also in Perceiver 7.2.20 AIX Physical System Overview



› How is CPU reported for the physical system from the CDB?Physical System CPU usage

can be shown for all DLPARs and all SPLPARs

Can select either Processors_Used (shown) or % Physical_Used (next slide)

Can also show per DLPAR usage or per SPLPAR usage (shown next slide)

First available with 7.4/4.2.02

Note: This is a “custom” Visualizer graphic



› How is CPU reported for the physical system from the CDB?Physical System CPU usage

reflects total Shared Pool usage, even when all SPLPARs have not been instrumented (first available with 7.4/4.2.02)

ROP = Rest of Pool, represents CPU usage of SPLPARs without collectors

Can select either Processors_Used (shown previous slide) or % Physical_Used

Note: This is a “custom” Visualizer graphic; also in Perceiver 7.2.20 AIX Partition Overview


Topic: How to report CPU for LPARs

› How is CPU reported for LPARs from the CDB?

Can also drill down on one SPLPAR, all three metrics simultaneously, or can view metrics one at a time …

Note: This is a “custom” Visualizer graphic; also in Perceiver 7.2.20 AIX Partition Capacity View


How to report CPU for LPARs


% Proc SMT label indicates that this is per physical processor (not per logical processor)

Shows individual SPLPARs use of the physical system, e.g. total never exceeding 25% of capacity




% LProc label indicates that this is per logical (i.e. virtual processor)

Shows individual SPLPARs use of the virtual processor allocation, e.g. never exceeding 40% of capacity




Shows individual SPLPAR’s use of the entitlement processor allocation, e.g. sometimes exceeding 100% of capacity, which is possible with an uncapped SPLPAR


Topic: How LPARs are represented for Modeling/Sizing

› How is CPU reported for LPARs for modeling/sizing from the CDB?

For capacity planning, viewing partition use expressed as SPECints, particularly across physical systems, is the perfect way to see exactly how much is being consumed and what size physical system will be needed to support any particular combination of LPARs.



Topic: ‘What-if’ of non-partitioned to partitioned configuration: Strategy

› Is modeling or sizing required for the CPU? – High level

• CPU utilization projection required• No response time projections required• No growth or overhead projections required

– Mid level• CPU utilization projection required• System growth projections required• Overhead projections required

– Low level• Workload response time projection required• System/workload growth projections required

Use Sizing

Use Modeling

Use Modeling

Specify workloads


Topic: ‘What-if’ of non-partitioned to partitioned configuration: Sizing

› Sizing Technique 1. Determine required CPU utilization objective

• Specify desired CPU utilization threshold (.e.g. 80%)• Can reduce that further to account for

– System growth – General “headroom”

2. Determine SPECints available on proposed CPU configuration(s)• Example shown on the next slide

3. Multiply CPU utilization objective by the available SPECints4. Compare the available SPECints to the CDB graph showing the required

SPECints for the proposed combined systems


‘What-if’ of non-partitioned to partitioned configuration: Sizing

Using the Predictor GUI to determine available SPECInts

Create a new Physical System

Specify the desired rating basis

Filter by desired hardware vendor

Performance rating allows you to choose appropriately powered CPUs

Select the desired CPU



Using the Predictor GUI to determine available SPECInts (continued)

Select the desired number of processors

View the resulting total SPECints available



› Will the proposed CPU accommodate this grouping of systems? YES

The selected CPU has 34.4 SPECInts available (from previous slide)

Using a CPU objective of 80%, that would be .80 * 34.4 = 27.5

Measured maximum usage is about 27 SPECints, so that just fits in 27.5



Topic: ‘What-if’ of adding standalone AIX systems onto existing partitioned configuration: Sizing

› Will the current Physical Frame accommodate these additional systems?

The selected Physical Frame has 280 SPECints available

Construct a Visualizer “group” containing the existing LPARs on the frame, plus the proposed standalone AIX systems, then …



‘What-if’ of adding standalone AIX systems onto existing partitioned configuration: Sizing

› Will the current Physical Frame accommodate these additional systems? YES

The selected Physical Frame has 280 SPECInts available (from previous slide)

Using a CPU objective of 80%, that would be .80 * 280 = 224 SPECint capacity is available

Measured maximum usage is about 150 SPECints, so that easily fits in 224



‘What-if’ of adding standalone AIX systems onto existing partitioned configuration: Sizing

› Sizing can be done for metrics in addition to the CPU•CPU sizing by SPECint (technique shown previous slides)

•I/O in MB/sec

•Disk Space Used in GB (and Disk Space Configured)

•Memory Used in GB, non-FileSystem Memory Used (and Memory Configured)

•Network traffic in Mb/sec



Topic: ‘What-if’ changes to current partitioned configuration: Modeling

› Modeling process steps– Build baseline model for current partitioned system

• Use ‘Best Practices’ implementation– What-ifs can include

• Revise CPU configuration of the partitioned system • Revise configuration of individual DLPARs and SPLPARs• Move additional work onto the physical system

– Move entire computers and/or– Import desired Workload(s) from systems not in this model

• Revise CPU utilization objectives• System growth (use Planner facilities)

– Evaluate the model and observe messages/reports for warnings that a CPU objective has been exceeded


Topic: Implementation ‘Best Practices’ for AIX Reporting and Modeling

(1) Confirm that the baseline analysis contains all the important LPARs and that they have been properly instrumented:

1) Instrument all important LPARs with a 7.3.00 (or later) collector• This ensures proper identification of SMT as well as access to the new metrics

required to properly report on LPAR resource consumption2) Ensure that at least one LPAR per physical system is collecting the HMC

configuration data• page 81,

http://documents.bmc.com/supportu/documents/88/98/68898/68898.pdf3) LPARs may be processed in separate Manager domains/policies, but must

be populated to the same Visualizer database • If you choose to split LPARs from the same physical system across multiple

domains/policies, item (2) requires that a least one LPAR per physical system per domain/policy is collecting the HMC configuration

Missing LPARs can compromise having complete results!



Implementation ‘Best Practices’ for AIX Reporting and Modeling

› Every properly instrumented LPAR will appear under the appropriate physical system name, designated as DLPAR or SPLPAR (CDB graphics)

Physical system name with properly recognized LPARs indented below

With 7.4/4.2.02 DLPAR and SPLPAR groupings are shown



Topic: Implementation ‘Best Practices’ for AIX Sizing/Modeling

(2) Confirm that the CPU model is properly identified for both the physical system (Analyze - Physical System report) and the logical systems (Analyze - Hardware Summary report)

Should be -SMT model for most partitioned systems

Always requires User Translation Table entry:

Name="IBM,9117-570",VendorID=IBM,Rate=1650,Alias=p5-570-SMT@1650,FreqTolerance=1

See FAQ for AIX Partitions paper or contact BMC Customer Support for details


Topic: General ‘Best Practices’ for Reporting/Sizing/Modeling

› Check periodically for product updates– console patches ftp://ftp.bmc.com/pub/perform/resolutions/174545.pdf– applicable collector patches ftp://ftp.bmc.com/pub/perform/resolutions/174545.pdf– hardware table http://documents.bmc.com/supportu/documents/04/70/70470/70470.pdf

› Periodically review articles in the Knowledge Base (http://support.bmc.com/arsys/apps/remcspenu.bmc.com/SSP/BMC%3ASSP%3AAdvancedSearchPage/AutonomySrchMain/?cacheid=25e396a9)

– For most AIX environments, the DISREGARD-DISK Analyzer override should be used. This ensures that only physical disks are reported and logical disks are not (ftp://ftp.bmc.com/pub/perform/resolutions/205628.pdf)

– For all AIX environments, ensure that the MAINTAIN DISK I/O HISTORY has been set properly (ftp://ftp.bmc.com/pub/perform/resolutions/136554.pdf). This is automatic with 7.4 collector.

– For all environments, setting CUT-DISK in Manager (or in Analyzer for interactive processing) improves the usability of the model and Visualizer results by eliminating information about disks with minimal activity (ftp://ftp.bmc.com/pub/perform/resolutions/180331.pdf)

› Sign up for proactive notification so that any important problems and their solutions are automatically sent to you

ftp://ftp.bmc.com/pub/perform/resolutions/174545.pdf



http://support.bmc.com/arsys/apps/remcspenu.bmc.com/SSP/BMC%3ASSP%3AAdvancedSearchPage/AutonomySrchMain/?cacheid=25e396a9

http://support.bmc.com/arsys/apps/remcspenu.bmc.com/SSP/BMC%3ASSP%3AAdvancedSearchPage/AutonomySrchMain/?cacheid=25e396a9





Topic: ‘What-if’ of consolidating to VMware host configuration: Strategy

› Techniques for sizing/modeling VMware follow the same overall strategy as for AIX partitions (see slides Strategy for ‘What-if’ of non-partitioned to partitioned)

– Sizing techniques are identical (see slides Sizing technique using Visualizer)– Modeling techniques are somewhat different because the modeling construct

is different for a VMware host• Documentation available (page 48,

http://documents.bmc.com/supportu/documents/88/98/68898/68898.pdf)• 7.4 Windows console offers new time-saving features for this modeling

– Will be shown in the next slides



Topic: ‘What-if’ of consolidating to VMware host configuration: Modeling

› Modeling process steps (GUI samples shown next slides)– Build baseline model for work to be moved

• Recommend “zzz” workload only• Set Baseline ON• Optionally Export desired Workload(s) from systems for transfer to another model

– Specify proposed CPU configuration of the VMware Host system• Use New VMware Host System under Computers

– Specify VMware overhead utilization– Specify required CPU utilization objective

• Define new Virtual Systems representing each of the proposed systems on the host– Move work onto the new VMware host system

• Optionally Import desired Workload(s) from systems not in this model• Move existing Workload(s) onto specific Virtual System

– Model• System growth (use Scenario Planner)

– Evaluate the model and observe messages/reports for warnings that a CPU objective has been exceeded

• Additional information (e.g. FAQs for VMware white paper) is available


Modeling Techniques Catalog:Moving Workloads (Export)

› If the workload is not in the desired model, it can be transferred from one model to another

• Begin with the model which contains the workload to be moved in its baseline configuration

– Be sure the model has been previously baselined– For each workload to be moved

» Export (Windows console) or Save to Library (UNIX console)• Now use the target ‘what-if’ model

– Be sure the model has been previously baselined– For each workload to add

» Import (Windows console) or Add from Library (UNIX console)» Specify the target server

› GUI examples shown next slides


Modeling Techniques Catalog: Moving Workloads (Export)

› Workload Export (Windows console shown)


› Define new VMware Host System (under Computers) (first in 7.3)– Specify CPU, Memory, System Services overhead, CPU Utilization Objectives

Modeling Techniques Catalog:VMware Host System


› Configure one or more New Virtual Systems for the VMware Host– Specify Operating system, virtual processors, Memory, CPU shares

Modeling Techniques Catalog:VMware Host System

NO NEED TO FILL THIS IN


› Windows 7.4 console has new “Move Workloads” feature– Select Computer Types of Virtual Systems

Modeling Techniques Catalog:Moving Workloads onto a Virtual System

1. Get Workloads from another model (optional)

2. Select one or all workloads

3. Select the virtual system

4. Add


Modeling Techniques Catalog:VMware Modeling results

› Predict-VMware Summary report displays complete results – CPU utilization, CPU response time effects (total wait time and VM wait time)


Topic: VMware performance reporting challenges

› What is unique about performance reporting for VMware?– Similar to other partitioned Virtualization architectures, CPU capacity depends

on two levels of available CPU resources• Each Virtual System requires adequate

– (1) Virtual processors AND– (2) VMware host processors

• While workloads are changing– Performance measurements from the VMware host are generally reliable

• ESX Server 3.01: Use 7.3 patch or 7.4 collector• ESX Server 2.5.3: Use 7.3 production (or later) collector• Ongoing research into issues caused by taking a Virtual system offline, stopping

the bgsagent, etc.– Performance measurements from the virtual systems themselves are not

generally reliable• Many guest OS do not have virtualized performance statistics• Collector overhead is generated for EACH guest


Topic: HP partition performance reporting challenges

› What is unique about performance reporting for HP partitions?– Similar to other partitioned Virtualization architectures, CPU capacity depends

on three levels of available CPU resources• (1) Total physical processors• (2) Processors assigned to each nPar• (2) Each standalone nPar requires adequate processors• (3) Each vPar within an nPar requires adequate

– Virtual processors AND– nPar processors

• While workloads are changing


Topic: How to report CPU for a Physical Frame

› How is CPU reported for the physical system from the CDB?

Shows total % of physical system capacity used

Shows % of physical system capacity used by partition


Topic: How to report CPU for each Partition

› How is CPU allocation for each partition reported from the CDB?

Shows total number of processors accessed per nPar or per nPar/vPar (100% = 1 processor)

Dynamic shifting of processors between vPars within an nPar causes processor allocation to vary by interval


How to report CPU for each Partition

› How is CPU usage for each partition reported from the CDB?

Shows % of allocated capacity (processors) used


Topic: Implementation ‘Best Practices’ for HP Reporting and Modeling

(1) Confirm that the baseline analysis contains all the important partitions and that they have been properly instrumented:

a) Instrument all important partitions with a 7.3.00 (or later) collector• This ensures access to the new metrics required to properly report on partition

resource consumption• Ensure that at least one partition per nPar has a collector (required to obtain

complete partition configuration data)b) Partitions may be processed in separate Manager domains/policies, but

must be populated to the same Visualizer database

Missing partitions will compromise having complete results!


Implementation ‘Best Practices’ for HP Reporting and Modeling

› Each instrumented partition will appear under the appropriate physical system name, designated as NPAR or NPARVPAR (CDB graphics)

Physical system name with properly recognized partitions indented one level below


Topic: Implementation ‘Best Practices’ for HP Sizing/Modeling

(2) Confirm that the CPU model is properly identified for both the physical system (Analyze - Physical System report) and the logical systems (Analyze - Hardware Summary report)

Follow “General ‘Best Practices’” (shown earlier for AIX)


Topic: Solaris Container Modeling - Instrumentation

› What is unique about performance reporting for Solaris Containers?– Similar to other partitioned Virtualization architectures, CPU capacity depends

on two levels of available CPU resources• Each Zone requires adequate

– (1) Virtual processors AND– (2) Solaris hardware domain (DSD) processors

– Performance measurements are obtained from a single collector running in the Global Zone• Use 7.3 patch or 7.4 Solaris collector

– Specify how you want to view the work within the hardware domain: by zone, by project, or by “regular” workload using Analyzer (use 7.4 console) • For Windows, do Properties, select Advanced Properties, Build workload by• For UNIX, use File -> Properties, Build workloads by

– The DSD is represented by a Computer, each Zone by a Workload


Topic: Solaris Container Modeling – Reporting CPU Usage by Zone

› How to report CPU usage by zone (CDB graphics)

Shows total % of DSD CPU used

Shows % of DSD used by Zone


Topic: Solaris Container Modeling – Specifying Projected Growth

› How to model increasing load using Scenario PlannerSpecify overall % growth rate (shown, 25%, then 100%)

Optionally, can specify different growth for each zone, use Plan Workload Override option (not shown)

Run Plan, activating Generate Visualizer File option, and populate into Visualizer modeling DB


Topic: Solaris Container Modeling – Reporting Projected CPU Usage by Zone

› How to report projected CPU usage by zone (CDB graphics)Shows total % of DSD CPU used for

• baseline

• each growth period

Also shows % of DSD used by Zone

Note: Baseline model interval was peak prime shift period (20% of available capacity)


Summary

› Follow implementation ‘Best Practices’– Use Support website Knowledge articles– Open a case with Customer Support for customized assistance with implementation – Allow extra time for the first implementation for a particular type of Virtualization

› Apply tools specifically designed to support Virtualization– Generally require at least 7.3 collector and 7.3 console

› Use CDB for a complete view of CPU Usage (historical and future)– Measurements from physical systems (AIX and HP)

• Three graphs per physical system gives a complete reporting solution• More detailed graphs can be used for specific analysis projects• Management only needs to see “% Capacity Used” for the physical system – everything else

is a “tuning” problem!– Modeling results for virtual systems

• View physical systems and individual partitions• View workload performance results within the partitions (HP and AIX only)

– Predictor Scenario Planner results can be shown, too


Summary

› When using Predictor– Use the appropriate modeling construct for each type of Virtualization (e.g.

Physical System, VMware Host System, Computer)– Use the corresponding specific modeling report for each type of Virtualization

(e.g. Physical System Summary, VMware Summary, Computer Workload)› Can use CDB graphics for Sizing when Predictor Modeling isn’t required

– Use “custom” Visualizer graphics to show CPU, I/O, Memory, Disk Space, and Network results

Debbie Sheetz Sr. Staff Consultant BMC Software, Customer Support/Waltham, MA Top Ten Topics in...

Documents

Transcript of Debbie Sheetz Sr. Staff Consultant BMC Software, Customer Support/Waltham, MA Top Ten Topics in...