Best Practices for Testing In Virtual

Environments

Martin Klaus, Vmware, USA

Europe’s Premier Software Testing EventWorld Forum Convention Centre, The Hague, Netherlands

WWW.QUALTECHCONFERENCES.COM

“The Future of Software Testing”

Best Practices for Testing in Virtual Environments

Martin Klaus

Agenda

Virtualization Basics

Why is Virtualization such a big trend?

Taking Advantage of Virtual Labs for Testing and QA

Architecting Virtual Labs

Best Practices for Testing in Virtual Labs

Virtual Labs at VMware

Performance Testing Considerations

Summary and Next Steps

Virtualization Basics

What is Virtualization?

Virtual MachinePhysical Machine

Virtualization packages hardware + OS + applications into a portable

virtual machine package

Key Properties of Virtual Machines

Partitioning

Run multiple operating systems on one physical machine

Divide system resources between virtual machines

Isolation

Fault and security isolation at the hardware level

Advanced resource controls preserve performance

Encapsulation

Entire state of the virtual machine can be saved to files

Move and copy virtual machines as easily as moving and copying files

Hardware-Independence

Provision or migrate any virtual machine to any similar or different physical server

Run several operating systems

on a single machine.

Physical – Hosted – Server-based Virtualization

Create shared pools of resources to

optimize your infrastructure.

Why is Virtualization such a big trend?

CapacityOn Demand

Hypervisor

Management

VirtualInfrastructure

AutomateSeparate Consolidate Aggregate

Self-Managing Datacenter

Server Consolidation

Test and Development

Computing Clouds

Hypervisor Hypervisor

Management

Hypervisor

Management

VirtualInfrastructure

Automation

Liberate

CapEx

Savings

OpEx

Savings

Business

Agility

Platform for Business Critical Applications

CapacityOn Demand

Self-Managing Datacenter

Server Consolidation

Test and Development

Computing Clouds

Business-Critical Application Momentum

>100,000 companies worldwide virtualize their datacenter

54% deploy enterprise applications on virtual infrastructure

46% standardizing on virtual infrastructure

63% plan to virtualize more than 50% of their x86

infrastructure in 3 years

“In a recent ESG survey, nearly half of the current users

consider themselves to be running mission critical

applications on virtual machines.”ESG Research, December 2007

Virtualizing critical applications in production

92% of our customers are confident that their mission-critical apps run well on VI

Source: VMware customer survey, October 2007, sample size 500

Percent customers running these applications in production

MSFT Exchange 36%

MSFT SharePoint 54%

MSFT SQL Server 62%

ORCL Application Server 31%

ORCL Database 32%

IBM Domino 37%

IBM DB2 20%

IBM WebSphere 44%

All Databases 42%

Taking Advantage of Virtual Labs for Testing and QA

Common Test Lab Challenges

Inefficient provisioning of test environments

Setup, teardown, reset and configuration time

Process, time and budget dependency on IT

Customer Example: Average Machine Utilization

Maintenance Tasks – Non-test related tasks such as Imaging, Installing,

Cleanup, or Locked for Triage.

Testing Tasks – Test execution

0%

10%

20%

30%

40%

50%

60%

70%

80%

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov

Maintenance Tasks Testing Tasks

On Average, test

execution is only

10% of the total

machine

utilization

Image Machine Install Software Run Tests Await Triage Release for new test

20 Minutes 5 Minutes 20-120 Minutes 1-12 Hours!!

Common Test Lab Challenges

Inefficient provisioning of test environments

Setup, teardown, reset and configuration time

Process, time and budget dependency on IT

Constraints of physical hardware

Test matrix limited by available lab resources

Little asset sharing leads to server sprawl

Customer Example: Physical Test Lab Challenges

Resources locked until test run

complete

Every task must go through imaging

and product install

No reuse of common configurations

Remote access issues

Requires hardware specific Images

Consumes lots of power and space

Common Test Lab Challenges

Inefficient provisioning of test environments

Setup, teardown, reset and configuration time

Process, time and budget dependency on IT

Resource requirements

Test matrix limited by available equipment

Little asset sharing leads to server sprawl

Collaboration of development and testing teams

Reproducing environments to troubleshoot problems

Customer Example: Before Virtual Test Lab

“Before” Virtual Test Lab

Number of provisioning requests

and slow IT response time (days)

High temperatures and power

consumption of 120 server spread

across multiple labs

Significant number of machines

reaching the end of their life cycle

causing unexpected downtime

Time and complexity required to

manage and maintain system

configurations

Inability to provide service for

remote users without manual

intervention

Customer Example: Virtual Test Lab

“After” Virtualizing theTest Lab

7 Blade Servers running 20-50 VMs

each

Supports 120 concurrent lab users in

multiple locations

Increased lab capacity by an additional

200 virtual machines

Improved IT response time from days

to minutes with a self-service portal

Fewer machines and dramatically

reduced hardware replacement cost

Customer Example: Test Automation

Infrastructure developed over

6 years

Roughly 100 unique users per

day and over 300 per month

Support team of 8 full-time

engineers

Mostly automated GUI testing

but also includes

programmatic SDK + API tests

Averaging roughly 86%

increase in testing year

over year

Annual Release Cycle – Launch in Fall

0

10000

20000

30000

40000

50000

60000

70000

80000

Feb Mar Apr May Jun Jul Aug Sep

Test Executions by Release

06 Release 07 Release 08 Release

Architecting Virtual Labs

How Virtual Lab Infrastructure Works

Virtual Lab Portal Image Storage Library

Virtual Infrastructure

Virtualized Server Pool

Automated Virtual Lab

LAN/SAN

QA Engineer

User selects a multi-machine configuration, clicks deploy.

Virtual machines stored in the imagelibrary are deployed on the virtualizedserver pool

Once deployed, user directly interacts with the machines, as if sitting at each console

Elapsed Time: 1-2 minutes

Virtualized Server Pool Sizing

Infrastructure Requirements

Server-based virtualization typically achieve

consolidation ratios of 10:1 or higher

QA and test labs often achieve 25:1 or more

Use 4-way servers for higher VM density

Basic sizing formula:

Users * Machines

Consolidation Ratio

Number of Virtualized

Servers needed=

Storage Sizing

Image Library

Efficient use of storage through

linked clones and delta disks

Recommendations

Storage need depends on the

number of base images +

length of linked clones chain

Allocate several large VMFS

data stores (300GB - 500GB

each) for initial VM storage

Minimum 200GB disk space for

media storage

Storage Server

A B C

A’ B’ C’

A’’ B’’ C’’

A’’’ B’’’ C’’’A’’’ B’’’ C’’’

WinXP

base disk

Win2k3

base disk

Test Automation in Virtual Labs

Use Test Automation Tools

Most vendors have plug-ins and integrations with virtual lab

automation products

Build your own automation scripts specific to your environment

and processes, e.g. automated regression testing after check-in

Capture Test Environments

Snapshot virtual machines if tests fail – linked clones reduce

storage overhead

Discard virtual machines if tests succeed

Best Practices for Testing in Virtual Labs

Problem: Test Matrix Management

Test matrix is becoming too difficult to manage

Host operating system, browser, database, application server,

localization, JRE, etc

Trade-off between test matrix, release schedules and quality

must be made

Solution: Using Templates and Configurations

Create multi-level templates

Level 1: OS

Level 2: OS + Patches

Level 3: OS + Patches + Apps

Tests are downloaded to Level 3 configurations

Build VM Configurations from Templates

Preserve “Gold master” images

Instantiate templates in multi-tier configurations

Image library stores all test environment configurations

Selectively deploy, combine or split VM configurations for combinatorial use cases

Problem: Test Resource Collisions

Different types of testing cannot be executed simultaneously in same

environment

User Acceptance Testing

Regression and

Integration Testing

Installation and Upgrade

Testing

Test Environment

Solution: Clones and Network Fencing

Deploy multiple identical environments simultaneously

Linked clones and delta disks reduce storage costs

Regression and

Integration Testing

Installation and Upgrade

Testing

Configuration A

Original

Configuration C

Clone of A

Configuration B

Clone of A

ESX Server Pool

User Acceptance Testing

Problem: Modified Test Environments

Environment Impact Tests

Installation, Upgrade and Patch testing

Test Data Management

Maintain test data through data destructive test cycles

Snapshots and Revert Points

Capture to Library and Checkout of Library

Solution: Store Application State

Pristine Data

State

Test Execution

Unknown Data

State

Revert to Snapshot or

Checkout Library Configuration

Problem: Not-Reproducible Defects

Non-determinism is source of many bugs

Race conditions in multithreaded code

Application behavior dependent on user input, network, etc

Non-reproducible defects get dismissed and shipped

Post-mortem debugging

Applications don’t always crash

Post mortem represents a system snapshot at the time of the

crash but doesn’t capture behavior leading up to the crash

Solution: Record and Replay

Record and replay whole VM execution

i.e., all threads, all processes, and operating system

Instruction exact

Recorded session encapsulated in a file

User

Network

Disk

Record and Replay Benefits

Instruction exact recording of VM execution

Inexpensive enough to use pervasively (won’t miss bug)

Use “go-live” to evaluate what-if scenarios

Replay bug over and over

Records all asynchronous data (user input, network, etc)

Minimal execution dilation (won’t mask bug)

Memory leaks

Race conditions

Minimal probing effects during debugging

Record bug with low overhead

Attach debugger while replaying

Control/inspect program in usual ways: breakpoints, watch points, single stepping, etc.

Virtual Labs at VMware

How VMware QA uses virtualization

Virtual Development Environments

VDE contains IDE, compilers, build tools and system paths

Switch between VDE to get different version build tools

Quickly provisioning VDE for new hires and contractors

Virtual Test Environments

VTE contains deployment environment and application build

Manual and Automated Testing team access VTEs

We also use test and build automation tools

Development and QA Collaboration

Code check-ins trigger automated smoke testing

Continuous integration testing against main branch

Virtual Lab Automation in Development and Test

Build

System

SQA

System

Lab

Manager

QA/ Dev

1. Daily build produced by build System

2. VMware Lab Manager SOAP API used

to checkout reference platforms, install

build, and download and run latest

smoke test scripts

3. Bugs are sent to dev team and filed in

bug tracking system using LiveLinks

4. All results are checked into the Lab

Manager Library. Leases used to keep

library clean.

5. Fixes checked into build system for

next day’s build

Tests

Fixes

Performance Testing Considerations

Performance Expectations for Virtual Infrastructure

Expect CPU overhead to be in the order of 5-20%

This is what we see consistently in partner benchmarks

Expect Storage latency overhead to be negligible

ESX Layer adds about 0.2ms per I/O, and is capable of >100,000iops

NFS and iSCSI protocols often provide sufficient performance, but

can have slightly higher CPU overhead

VMware ESX Scales Out Well

Adding VMs scales well: over 95% scaling factor

Single VM, CPU Intensive Performance

VMware ESX 3.x compared to NativeSPECcpu results covered by O.Agesen and K.Adams PaperWebsphere results published jointly by IBM/VMwareSPECjbb results from recent internal measurements

Measuring the Performance of DB Virtualization

Throughput

Delivered

Minimal

Overheads

Throughput scales with the size of the VMO

racle

10G

R2 S

win

gbench

Thro

ughput

(txns/s

ec)

Number of virtual CPUs in Database/Guest

Java Virtual Machines

Java on VMwareJava on Linux

To

tal S

yste

m T

hro

ug

hp

ut (o

ps/s

)

Tota

l S

yste

m T

hro

ughput (o

ps/s

)

31 2 4 5 6 7 8

Java Virtual Machines

• 3-Tier client-server Java benchmark measuring num. of business transactions per sec.

• Intel Xeon 3.2 GHz, 2 GB RAM, VMware ESX 3.0, BEA LiquidVM 1.1, BEA JRockit R27.3, BEA WLS 9.2 MP2, RHEL 4.0.

• Each VM allocated 1 vCPU and 1 GB vMem. JVM -Xmx=800MB, 135 MB live data.

31 2 4 5 6 7 8

Application Tier Scales Better in Virtual Machines

Summary and Next Steps

Building your own Virtual Test Lab

Implement proof of concept prior to broader deployment

Get your IT department involved

Virtualize one or two QA test lab servers to gain experience

Scale out your virtual lab over time

Online Resources

VMware Web site: http://www.vmware.com/solutions/vla/

White papers, Technical articles, community forums, blogs,

customer references

Free Tools

VMware Converter, VMware ESXi, VMware Server

Conclusion

Virtualize your test lab

Start by leveraging existing servers and local storage

Import physical test configurations as virtual machines

Use test automation tools

Included automated provisioning and capture of test

environments in your tests

From 2007 Customer Survey

Time is the biggest item “saved” through virtual lab automation

50% increase in defects found prior to production/shipment