Performance testing virtualized systems v5

Dan DowningPrincipal ConsultantDion Johnson

2Performance Testing Virtualized Systems

Agenda

Introduction• State of adoption of server virtualization• Operational barriers and concerns• Virtualization technologies

The six critical success factors a performance tester needs to know

• Anatomy if a virtual system• Principles of virtual workload modeling• Shortcomings and key bottlenecks of virtual systems• System resources: what to measure and how• Analyzing and presenting compelling results

Case Study• How a provider of insurance market intelligence went virtual, and how

they proved performance


State of Adoption of Server Virtualization


Operational Challenges / Barriers


Server Virtualization Technologies

Five key technologiesTechnology Example 1 Example 2 Differentiators

Virtualization Software

WMware’s vSphere/ESXi

Microsoft’s Hyper-V (W2008 Server)

Type 1 Native OS hypervisor vs. Type 2 “guest OS”

Broad OS supportProcessor architecture

Intel Xeon 7500 8-core

AMD Opteron 6000 12-core

Core density Threads, DIMMs per socket

Disk IO Interface

iSCSI FCoE

Protocol to minimize cpu overhead, making network-attached storage perform like attached storage

Standard Ethernet cabling vs. Fibre

Network Interface

Broadcom iSoE

Intel I/OAT TCP headers processed on NIC Multiple NICs with “Teaming”

Management Utilities

VMware’s vCenter

MS Virtual Machine Manager

Model virtual workload Monitor host and VM resourcesThe goals: Enable more VMs on a single chip, minimize

cpu overhead for IO, and enable monitoring & management

The goals: Enable more VMs on a single chip, minimize cpu overhead for IO, and enable monitoring & management


Six Critical Factors for Testers

1. Anatomy of a Virtual System2. Mapping workloads from dedicated to

virtual3. Key bottlenecks of virtual systems4. Effective comparative testing

techniques5. System resources to measure6. Results analysis and presentation


1 - Anatomy of a Virtual System

“The abstraction of computer resources” (Wikipedia) Type 1: Native OS, runs as hypervisor on the host, provides peripheral

drivers Type 2: Guest OS that runs under main host OS and relies on host OS drivers

VMs with HW resources allocated by VM software

VM software running as native OS (“Hypervisor”)

Hardware (multi-cpu, multi-core, multi-threaded, lots of memory, multi-NIC)

Network Attached Storage (NAS) or Storage Area Network (NAS) accessed over standard cabling & cpu-optimized protocols

Type 1 Virtualized Server


2 - Virtual Workload Modeling

Web

App

Files

DB

Hardware

1.How quantify the dedicated server workloads?

2.Host of what configuration required to handle the total workload?

3.How many VMs are needed to handle the workload?

4.Which workloads mapped to which VMs?

4 Key Questions:


2a - VMware Capacity Planner

Monitor resources on each tier for a week (cpu, memory, disk & network IO)

Enter configuration and role of each server Run the modeling tool, analyze the proposed host & VM mapping Build prototype and test


3 - VM Bottlenecks & Vulnerabilities #1 limiting resource: Disk IO

• Thus high IO servers (DB) not good candidates for virtualization

# 1 vulnerability: Redundancy / load balancing• Like services should be distributed on more than one

VM to ensure that system can continue to operate if a VM fails

App servers on different hosts


4 - Performance Testing Techniques

Test in parallel on dedicated and VM environments• Eliminates time-of-day differences due to loads on shared

components Test a light workload first

• Establish a baseline for both systems at low/no contention Test a 50% workload

• Compare performance scalability under reasonable load, without potentially overwhelming the DB

Test fail-over of VMs• Validate system recovery / availability /lost transactions if a

virtual web / app / file server fails


5 - System Resources to Measure

VM resources• Cpu, memory, disk IO, network IO (Unix: Load Average)• Web server: queued requests• JVM: Heap utilization

Host resources• Cpu, memory, disk IO (% disk busy and IOPS), network IO• DB server: lock waits, deadlocks

Monitoring utilities• Windows: perfmon• Unix/Linux: nmon, sar


5a - Monitoring Host Resources

VCenter monitoring host resources kry, in addition to monitoring resources on individual VMs, to id IO path bottlenecks


6 – Presenting Comparative Results

Color Measurement Min. Ave. Max. SD Min/Max

INT_END_TO_END 92.3 532.5 1111.6 308.7 12.0x

INTVERT_END_TO_END 105.7 370.8 1099.2 213.4 10.4x

Load ( Vusers int) 30

Load ( Vusers intvert) 30

Business process end-end response over load on same graphBusiness process end-end response over load on same graph

Overall response profiles btwn dedicated and virtual platforms similar


6a – Comparative Page Times

Differences btwn dedicated and Virtual page response times over loadDifferences btwn dedicated and Virtual page response times over load

Average difference: Virtual 7% faster on

average

Max difference: “a wash” (Virtual 1% faster on

average)


6 b – Comparative CPU

VM utilizations higher than on dedicated servers, but still lowVM utilizations higher than on dedicated servers, but still low


6c – Comparative Bandwidth

Color Measurement Avg MAX

Throughput:adv_vm_test_v10_int (bytes/sec) 19,905 115,576

Throughput:adv_vm_test_v10_int_1 (bytes/sec) 21,140 99,477

Throughput:adv_vm_test_v10_intvrt (bytes/sec) 20,187 95,006

Throughput:adv_vm_test_v10_intvrt_1 (bytes/sec) 20,886 101,809

Throughput Total INT Mbps 0.3 1.7 Throughput Total INTVERT Mbps 0.3 1.6

Load ( Vusers int) 30

Load ( Vusers intvert) 30

Bandwidth usage similar in magnitude, profile


Case Study: SAMPLE

Consolidated 18 dedicated servers into 3 virtual hosts

Reduced rack space, power & cooling footprint and maintenance costs

Performance-tested and showed improved performance and increased efficiency

Preeminent provider of market intelligence to the commercial insurance industry• Corporate, legal, financial and regulatory data for hundreds of thousands of

public and private companies• Competitive information for underwriters, brokers and risk managers• Analytic tools for benchmarking and comparative analysis

HQ in NYC

Solution

Business Challenge Needed to contain operating costs while assuring their customers 24/7 availability and performance

Have 18 dedicated, underutilized servers in a managed hosting Tier 4 data center


Environment Migration

18 dedicated, underutilized servers…

…migrated to 3 Virtual Hosts…

…with proven performance & reliability


Customer Benefits

Test results enabled him to gain management confidence that the new solution was sound

Saved several $K/month in environment facility and hardware maintenance costs

Maintained performance and reliability


In Conclusion

As testers, we need to stay ahead of the technologies curve to continue adding value to our organizations

Virtualized systems are becoming the norm While the fundamentals are the same, performance

testing them require new understanding and know-how

Performance testing virtualized systems v5

Technology

Transcript of Performance testing virtualized systems v5