a presentation

www.PerfTestPlus.com

© 2008 PerfTestPlus, Inc. All rights reserved.

Performance Tuning in a Virtual Environment

StickyMinds.com and Better Software magazine presents…

Whose Environment is It Anyway?

Performance Tuning in a Virtual Environment.

Sponsored by Empirix

Non-streaming participants should call 1-866-761-8643International Non-streaming participants should call 1-904-596-2362




Scott Barber, (A.K.A. “The Perf Guy”)[email protected]

Chief Technologist, PerfTestPlus, Inc.www.perftestplus.com

Vice President & Executive Director, Association for Software Testingwww.associationforsoftwaretesting.org

Co-Author, Performance Testing Guidance for Web Applications www.amazon.com/gp/product/0735625700

A Heuristic Approach




Developers write code

Developers release code for test

Testers test code, often in various environments

Testers assess system performance in not very production-like environments

Managers make decisions based on test results

Some defects get fixed before release

The team gets surprised by defects in production

The team puts in lots of overtime to fix the defects

Testing in a Non-Virtual World




Wikipedia lists 8 categories of platform virtualization and 4 categories of resource virtualization

Platform Virtualization• Software simulates a computer environment on a host

machine, usually called “virtual machines”

• Applications run on “virtual machines” as if on a stand-alone hardware platform

• As if stand-alone ≠ Isolated

• Platform virtualization software has overhead

(http://en.wikipedia.org/wiki/Virtualization)

What is Virtualization?




Resource Virtualization• Identify a resource and exclusively assign it’s use to a

specific application

• Sometimes partitions single resources, sometimes spans multiple resources

• Resources may or may not be co-located on a physical machine

• Exclusively assigned ≠ Isolated

• Resource virtualization has overhead

(http://en.wikipedia.org/wiki/Virtualization)

What is Virtualization?




Developers write code

Developers release code for test

Testers test code, often in various configurations

Testers assess system performance in not very production-like environments

Managers make decisions based on test results

Some defects get fixed before release

The team gets surprised by defects in production

The team puts in lots of overtime to fix the defects

Testing in a Virtual World




Virtualization, if done well, can:• Dramatically reduce the overhead involved in managing

test environments

• Create opportunities for unique, targeted, ad-hoc environments without additional hardware costs

• Simplify configuration and release management

• Put the test team in control of their test environments

But what about performance testing?

What’s the Difference?




“With an order of magnitude fewer variablesperformance testing could be a science,

but for now,

performance testing is at best a scientific art.”

--Scott Barber




Test environments are still test environments

Virtual test environments are less likely to match virtual production environments

There are more places for performance issues to hide in virtual environments• Exclusively assigned ≠ Isolated

• As if stand-alone ≠ Isolated

• Virtualization adds overhead

Study: “virtualized servers can handle 14% lower [load] than traditional server configurations” (http://www.sys-con.com/read/503412.htm)

Testing Virtual Performance?!?




“Linear extrapolation of performance test results is,

at best, black magic.

Don’t do it.

(Unless you learned from Connie Smith, PhD or Daniel Menasce, PhD)”

--Scott Barber




If done well, there can be benefits:• Dramatically reduce the overhead involved in managing

test environments

• Easier for performance testing to have it’s own test environment

• Enable collaborative tuning without having to “borrow” hardware resources

• Put the test team in control of their test environments

Virtualization does not change the core principles of successful performance testing

Testing Virtual Performance?!?




“Only performance testing at the conclusion of system or functional testing

is like

ordering a diagnostic blood test after the patient is dead.”

--Scott Barber




Core Principles

Project context is central to successful performance testing.

Business, project, system, & user success criteria.

Identify system usage, and key metrics; plan & design tests.

Install & prepare environments, tools, & resource monitors.

Script the performance tests as designed.

Run and monitor tests. Validate tests, test data, and results.

Analyze the data individually and as a cross-functional team.

Consolidate and share results, customized by audience.

"Lather, rinse, repeat" as necessary.




Contact Info

Scott Barber

Chief TechnologistPerfTestPlus, Inc

E-mail:

[email protected] Web Site:www.PerfTestPlus.com

Whose Environment Is It Anyway?Whose Environment Is It Anyway?Dan Koloski

Web BU CTO and Director of Strategy

Empirix

IMAGE

Empirix – At a GlanceEmpirix – At a Glance

Testing & Monitoring Solutions for:

Privately held, founded 2000

380+ employees

Worldwide presence; HQ in Boston & offices in Japan, UK, Germany and South Korea

Patents in test, monitoring and VoIP diagnostics methodology

Web Applications & SOA

Web Applications & SOA

Contact Center & CRMContact Center & CRM

Carrier-Class VoIP & TelcoCarrier-Class VoIP & Telco

Empirix in the Quality EcosystemEmpirix in the Quality Ecosystem

Empirix Offerings Delivered OutcomeEmpirix Offerings Delivered Outcome

PRODUCTSPRODUCTS

MANAGED SERVICESMANAGED SERVICESMANAGED SERVICESMANAGED SERVICES

EXPERT SERVICESEXPERT SERVICESEXPERT SERVICESEXPERT SERVICESOnsite Consulting, Training, Mentoring

e-TEST suite™OneSight™

e-LoadExpert™

Do-it-yourself

Remote ValidationPerformance Tuning

Supplement Your TeamWe Do it for You

A few points to add to Scott’s presentation…A few points to add to Scott’s presentation…

Scott said…– Virtualization does not change the core principles of

successful performance testing

Why?– Hardware is not virtual

– Whose environment is it, anyway?

– Rapid Bottleneck Identification Tuning a Virtualized Context

Hardware is not virtualHardware is not virtual

Virtualization is app/OS or hardware emulation specific

Underneath it all is a fixed pool of HW resources!

Performance Testing is designed to understand the impact on the hardware – as seen from the perspective of the user

Common Virtualization Types. Source: QAZone Whitepaper, Virtualization from the perspective of a QA/IT Operation Professional, http://qazone,empirix.com.

The Best Performance Data in a Virtual EnvironmentThe Best Performance Data in a Virtual Environment

Understand the impact on the hardware…– Requires accurate workload stimulus representing activity on and

through all VMs associated with that device

– Requires measurement of hardware characteristics at the base layer as well as OS and app-level characteristics in each VM

– Requires real-time collaboration during load testing to differentiate intra-VM bottlenecks (business logic) vs Hardware bottlenecks (CPU, Memory, NIC)

…as seen from the perspective of the user!– Your users don’t care about Virtualization

– Metrics that matter are end-user perceived performance

– Performance testing from inside the VM helps create stimulus but doesn’t report on end-user-perceived performance

Rapid Bottleneck Identification Load Testing in a Virtual EnvironmentRapid Bottleneck Identification Load Testing in a Virtual Environment

What is Load testing?

“Testing conducted to isolate and identify the system and application issues (bottlenecks) that will keep the application from scaling to meet its performance requirements”

What is a Bottleneck?What is a Bottleneck?

Any resource (hardware, software or bandwidth) that places defining limits on data flow or processing speed: your application is only as efficient as its least efficient element

On the Web, bottlenecks directly affect performance and scalability

Most untested systems have more than one bottleneck, but they can only be identified and resolved one at a time.

Where are the Bottlenecks?Where are the Bottlenecks?

The vast majority of bottlenecks are

caused by throughput

limitations – not concurrency.

Throughput80%

Concurrency20%

Source: Empirix

Sample throughput limitations in virtual environmentsSample throughput limitations in virtual environments

Disk I/O– Multiple applications accessing shared disk resources

– Virtual Memory Paging on shared disk resources

NIC– Multiple VMs shipping data through the same physical NIC

CPU– Processor-intensive (i.e., database) applications will compete for shared

CPU time.

Memory– Depends on type of VM configuration how much of an issue this can be

Whose Environment Is It Anyway?Whose Environment Is It Anyway?

Virtualization blurs the line between Dev/Test/Ops environments and ownership– Can exacerbate “silo” problems between Dev/Test/Ops

However, there is an opportunity…

…A common set of measurements can un-blur the line– Test data should inform production SLAs

– Common measurement/metrics can bridge the gap

– Measurement consistency is the cornerstone of ALM

Load Testing Tools DON’T CollaborateLoad Testing Tools DON’T Collaborate

Load Testing Tools Apply Load and Make Measurements

People (and their expertise & experience) Find Bottlenecks & Resolve Them

Collaboration With Experts Resolves Issues Faster & Cheaper

Key to Successful Collaboration = Access to People & Access to Measurements

Tools Should Support Collaboration

e-TEST suite is designed for collaboratione-TEST suite is designed for collaboration

Distributed, high-scale load controlMulti-user Web UI for real-time individual analysis

Each user can view results from the same load test as it runs

SummarySummary

Hardware is not virtual – virtualization does NOT change core performance test best practice

However…– Virtualization introduces new test planning challenges to cover

– Workload/stimulus

– Measurement of impact on end-user

In a world of scarce time, focus efforts on throughput-related workload to identify most common bottlenecks

Visit QAZone for the slidesVisit QAZone for the slides

http://qazone.empirix.com

Free resource, open to all (Empirix & non-Empirix users)– Forums (both QA-centric and

Product related discussions)

– Knowledge Base

– Resource Center

– Events, Announcements, Community & Industry News

Allows members to interact with peers, industry experts and a larger Empirix audience




Have a question for the speakers? Ask now.

Q & A

a presentation

Documents

Transcript of a presentation