BenchmarkQA Performance Testing Quality Forum March 2012

Performance Testing Fundamentals for Managers

Presented by:

Jeff RothVP of Consulting Services

BenchmarkQA

Date:

March 22, 2012

Agenda

• Performance Testing 101

• Components of Performance Testing

• Performance Testing Lifecycle

© 2012 BenchmarkQA Slide 2

What is Performance Testing?

© 2012 BenchmarkQA

LoadVolume

Stress

ScalabilitySystem Benchmark

Durability

System TuningDesktop Benchmark

FailoverSystem Optimization

Baseline

Infrastructure Validation

Batch Jobs

Failure Point

Slide 3

Business Goals• Will my system meet my goals?

– User expectations (response time, accuracy)– Load capacity (user volume, cost of configuration)– Data integrity (at normal loads, peak loads, and failure)– Stand up over time (durability)– Failover capability (continuity through failure) – System growth (scalability)

• When will my system degrade and when will it fail?

• What do I need (infrastructure) for effective performance?


Please excuse me while I shout….

PERFORMANCE TESTING IS DEFINED BY WHAT YOU WANT TO ACCOMPLISH

NOT WHAT TOOLS YOU OWN


Before we answer this question, let’s start with a little anatomy of IT systems…

What Can Performance Testing Do?


• Input – Keyboard, Touch Screen, Voice, Camera, Hand Scanner, etc.

• Output – Screen, Printer, Sound, etc.

• Processing – CPU, GPU

• Storage – RAM, Disk, SDD, etc.

• Connectivity – LAN, WAN, WiFi, 3G/4G

User Interfaces


Performance Testing Devices

Business Goals

• Configuration Optimization– Best hardware for the job– OS/Software configuration

• Baseline Device Performance– Embedded devices – software dependent upon hardware– With and without connectivity


Performance Testing Systems

• Include devices or not

• Real vs. simulated integration points

• Isolate risk areas

• Iterative process

• Load testing tools…


COMPONENTS OF A PERFORMANCE TEST

© 2012 BenchmarkQA

What Makes up a Performance Test

© 2012 BenchmarkQA

Web Farm Mainframe

Databases

Target of TestLoad Generator

Controller

Slide 11


Virtual User Script

• Activity/Business process

© 2012 BenchmarkQA

Search ItemHome Page Select Item

Slide 12


Technical Details of a Virtual User Script

• Protocol - HTTP, AJAX, SOAP, SAP, CITRIX, etc.

• Uniqueness– Session variables– User IDs– Test Data

• Files• Correlate within the script• Made up

• Think time


Performance Test Scenario

• One or more test scripts

• Quantity of VUsers per script

• Velocity of activity per user

• Designate ramp up & test schedule based on goal


Ramp Up And Failure

© 2012 BenchmarkQA

Errors begin to occur @ 400VUsers

Number of VUsers AxisPercent of

Errors Axis

Slide 15

PERFORMANCE TEST LIFECYCLE

© 2012 BenchmarkQA

© 2012 BenchmarkQA

PMBOK - PROJECT MANAGEMENT PHASES

SELECTION INITIATION PLANNING EXECUTION & CONTROL CLOSURE

Project Definition Requirements Analysis & Design

Technical Analysis & Design

Build/Integrate & Test

User Acceptance Test

Deployment & Wrap-Up

Design

Build

Execute

Analyze

Initial Planning InitialPlanning

Slide 17

Softw

are

Dev

elop

men

t Life

cycl

e (S

DLC

)Pe

rfor

man

ce T

estin

g Li

fecy

cle

Performance Testing Lifecycle

Design

Build

Execute

Analyze

Initial Planning

The Performance Test

lifecycle requires some initial

upfront planning work, and

becomes a cyclical effort to

achieve your performance

goals.

Typical engagements are 7 to 9 weeks including initial planning.


Participants

ØProject StakeholdersØProject ManagerØSystem ArchitectsØKey SMEsØPerformance Architect

Duration

1 to 4 Weeks

4 Test Effort Scope4 Software Components4 Target System Architecture4 High Volume Business Processes and Resource

Intensive System Activities4 Workload Profiles4 Performance Test Goals

4 Resources4 Target Test System Hardware4 Project Personnel and Test Effort Roles4 Test Data Management Plan4 Test Tools (load injection hardware and software)

4 Test Schedule4 Project Schedule4 Project Dependencies4 Performance Test Work Breakdown Structure


Participants

ØBusiness SMEsØSystem SMEsØPerformance ArchitectØPerformance

Engineers

Duration

1 to 4 Weeks

4 Load Test Tools4 Test Script IDE4 Load Generation Farm4 Protocol Licensing (Web, Winsock, Citrix, etc.)

4 Business Process4 Screen by Screen Walk Through4 Usage Frequency and Pacing4 Test Data & Test User IDs

4 System Intensive Processes4 Batch Jobs4 Reporting Systems4 Interfaces/Feeds

4 Test Monitoring Strategy4 Servers/Databases/Network4 Integrate with Load Test Tool (Security

Permissions)4 Data Center/ SME System Monitoring


Participants

ØBusiness SMEsØSystem SMEsØPerformance

Engineers

Duration

2 to 12 Weeks

4 Test Scripts4 Record Business Processes4 Correlate Data4 Parameterized Data4 Pacing4 End User Response Time Measurements4 Typically 4 to 8 Scripts

4 Test Scenarios

4 User Ramp Up Scenario

4 Frequency of Business Process Activities

4 User Managed Activities

4 Results Storage

4 Verify Test Readiness

4 User Concurrency

4 Multi-Script Concurrency

4 Load Generators© 2012 BenchmarkQA Slide 21

Participants

ØProject StakeholdersØProject ManagerØSystem ArchitectØKey SMEsØPerformance

Engineers

Duration

1 to 8 Weeks

4 Schedule Execution4 No Users in Target System4 Anti-Virus Scans, Automated Backups4 Off Hours Scheduling

4 Monitor Execution4 Load Test Progress4 Load Test System4 Real Time Response Time Metrics4 Target System Utilization Under Load

4 General Execution Approach4 Small Load First To Confirm System Is Stable4 Reasonable User Load Ramp Up4 More Than One Test Execution To Confirm

Goals


Participants

ØProject StakeholdersØProject ManagerØSystem ArchitectØBusiness SMEsØSystem SMEsØPerformance Architect

& Engineers

Duration

1 to 4 Weeks

4 Test Goal Assessment4 Response Time Performance4 System Utilization

4 Bottleneck Identification4 Correlate Response Metrics with System

Utilization Metrics4 Isolate Causal System/User Activity

4 Remediation Strategy4 Configuration Changes4 System Changes4 Hardware Changes4 Test Script Changes4 Load Scenario Change4 Monitoring Changes

Repeat Cycle Until Goals Achieved© 2012 BenchmarkQA Slide 24

Things to Remember• Set clear objectives

– To ensure all team members agree on the questions performance testing will answer

• non-functional requirements• solution requirements specification

– To understand the expectations of the business owners– To know when testing is complete

• Strive for realism– Subject the system under test to conditions that match production as

closely as possible– If possible use the actual production hardware for the performance test


Things to Remember• Understand what happens

– Every component of a system must be monitored and data collected– Subject matter experts analyze collected data

• Provide meaningful recommendations– Must improve the end-user experience– Must be based on a cost-benefit analysis


Thank You!For additional information on Consulting Services with

BenchmarkQA, please contact:

Jeff Roth952.392.2384

[email protected]

© 2012 BenchmarkQA

BenchmarkQA Performance Testing Quality Forum March 2012

Technology

Transcript of BenchmarkQA Performance Testing Quality Forum March 2012