Fundamentals of Application Performance Testing

Practical Performance Analyst – 28th July 2012

http://www.practicalperformanceanalyst.com

Agenda Performance Engineering Life Cycle

What is Proactive Performance Management

Holistic View of Performance

What is Performance Testing

Why is Performance Testing Important

Aspects to consider while Performance Testing

Pitfalls to avoid during Performance Testing

Performance Testing Process

Challenges involved in Performance Testing

Deliverables for the Performance Testing process

Resources & tools for Performance Testing

Performance Engineering Life Cycle

Software Development Life Cycle

Functional Requirements Gathering

Architecture & Design

Build Application

System Test,

System Integrated Test & UAT

Deploy Into Production

Performance Engineering Life Cycle

Non Functional Requirements Gathering

Design for Performance &

Performance Modelling

Unit Performance Test &

Code Optimization

Performance Test

Monitoring & Capacity Management

Proactive Performance Management

Performance Requirements

Analysis

Performance Modelling &

Capacity Planning

Build & Optimization

Performance Testing

Performance Monitoring

Capacity Management

Proactive Performance Management helps by –

Identifying performance issues early in the design

Validate suitability of your application architecture for your documented NFR’s

Validate your application capacity requirements for the documented NFR’s

Advocates early SVT to identify potential bottlenecks early in the build and development cycle

Reduces potential for major rework later in the program

Advocates using modelling techniques to predict performance impact for growing workload

Txn Performance

- Response Times, etc.

Application Performance – Operations/Sec, Messages/Sec,

Transactions/Sec, etc.

Infrastructure Performance – CPU Utilization, Memory Utilization, Disk IOPS, etc.

Network Performance – Packet Loss, Jitter, Packet Re-ordering, Delay, etc.

Holistic View of Performance

What Is Performance Testing

Performance Testing is quite often confused with Performance Engineering. Although Performance Testing is an integral and important part of Performance Engineering its usefulness is confined to validating Application Performance later in the SDLC (Software Development Life Cycle).

Performance Engineering as we’ve defined it in the past consists of a set of roles, responsibilities, processes and deliverables across the SDLC whose objective is to build applications that meet their NFR’s.

Performance Testing can be defined as the process of validating your application performance against the set of defined Non Functional Requirements.

The objectives of Performance Testing are generally a combination of the following:

Validate Application Scalability – Performance Testing

Validate Application Performance at Peak Workload – Stress Testing

Validate Application Performance over a extended duration of time – Soak Testing

Establish Application Breaking Point – Break Point Testing

Performance Testing has traditionally been and continues to be a proactive process. Programs and project generally lack both the capability and willingness to consider Performance early in the SDLC.

While Performance Testing allows for validation of Application Performance later in the SDLC, a Performance Engineer can use a combination of different modelling techniques to validate application designs, infrastructure capacity and application performance early in the SDLC and mitigate risk accordingly.

Why Is Performance Testing Important

Performance Testing is important for the following reasons –

Identify bottlenecks across the application and highlight opportunities for tuning

Validate your design & infrastructure capacity assumptions made early at the design stage

To validate if your applications meet the program Non Functional Requirements

Validate that the application is able to deliver the expected End User Experience while consuming resources optimally across the infrastructure and network tier

To determine the breaking point of your applications. Knowing the breaking point of your applications helps you understand the maximum Business Workload the application can process before performance starts to degrade.

Aspects to consider while Performance Testing

Following are the aspect you should keep in mind when planning Performance Testing –

Performance Testing should ideally start through setting up of the program Performance Engineering team at the Requirements Gathering & Design Stage. At this stage you should be working on closing of the following documents:

Program Performance Engineering Strategy

Performance Testing Workload

Non Functional Requirements

Performance Testing Strategy

Start early and don’t wait until the code base completely stabilizes. I encourage you to start Performance Testing early, identify opportunities for tuning and go back with feedback to the development teams.

Work closely with the development teams to meet your documented NFR’s. Make sure you have the buy in of your program leads, environment leads and development team leads. With the appropriate buy in issues get addressed a lot faster.

Discuss and document your environment requirements. Anything lesser than 100% capacity of production will require you to use some black magic (aka. Performance Modelling Techniques) to extrapolate performance characteristics

Discuss and document your data requirements (Script data requirements, application data requirements, etc.). Data preparation can be one of the most challenging tasks on a program.

Discuss & document your Performance Testing and Performance Monitoring (Application + Infrastructure) tool requirements. You will also need to discuss and work out the cost implications due to Virtual Users (VU) licensing.

Make sure everyone the program understands the NFR’s and overall SVT Business Workload very clearly. Ambiguity in any of these areas will land you in serious trouble later in the program.

Track performance issues closely. Use your program risks & issues register to track performance bottlenecks that are preventing you from progressing with SVT.

Aspects to consider while Performance Testing

Consider having a good balance of Performance Engineers, Performance Testers, Diagnostics resources & Application Experts. To be successful you should plan to build a Performance Testing team that doesn’t just execute but also knows how to identify relevant defects, tune the application/database and work with the development teams in coming up with fixes to relevant Performance issues.

Plan your resourcing model early on. To obtain a good balance of Performance Engineers & Performance Testers you will need a good head start. Obtaining Performance Engineering skills is generally tough and you might want to look at sourcing them from a niche consulting outfit.

Don’t wait till for SIT & UAT to complete for you to start planning for Performance Test. Performance Testing just before go live is the best way to ensure that your application fails to meet the documented NFR’s (if they even exist i.e.).

Pitfalls to avoid during Performance Testing

Ensure that you’ve got well defined and clearly articulated Non Functional Requirements. Your Non Functional Requirements should be signed of by business and program management.

Ensure that you’ve got well defined and clearly articulated Performance Testing Workload & Performance Testing Strategy. Your Workload & Strategy should be signed of by business and program management.

Ensure that you’ve got the appropriate tools for Performance Testing, Diagnostics & Performance Monitoring. Avoid recommending or using Open Source tool as far as Performance Testing is concerned. There are quite a few affordable tools (including cloud based solutions) for Performance Testing.

Push for an Performance Testing environment that’s equivalent to production. This will ensure that you don’t have to extrapolate performance of the application from your testing environment into the production environment.

Model application performance early on in the SDLC using modelling techniques described earlier. You can use a combination of analytical and simulation modelling techniques to validate application performance for the given combination of application architecture and infrastructure capacity.

Update your Business Workload for Performance Testing as your business grows and the environment business works in changes. Change in Business Workload would require updating of your NFR’s and rebase lining of your Workload for Performance Test. It changes your requirements and can have considerable impact on tool requirements, infrastructure requirements, resource requirements, etc. but is an important that that should be tracked through the life of the program.

Performance Testing Process

Document Non Functional

Requirements

Document Performance

Testing Workload

Document Performance

Testing Strategy

Document tooling, licensing, data, environment requirements

Document resource requirements

Setup Performance Testing Team

Setup Environment for Performance

Testing

Create Performance

Testing Scripts

Create Data for Performance

Testing

Setup Performance Testing Workload Scenarios within

Tesing Tool

Setup Monitoring & Diagnostics tools

Execute Performance Tests

( A )

Collect test data from testing, monitoring &

diagnostics tool

Identify performance

bottlenecks & tune the application

Exit if we’ve met our NFRs else loop

back to Step

( A )

Challenges involved in Performance Testing

Challenges convincing the delivery teams to focus on Performance Engineering & Performance Testing activities early in the SDLC

Challenges obtaining funding for the appropriate performance testing, monitoring & diagnostics tools

Challenges obtaining the right combination of resources to address your Performance Testing needs

Challenges obtaining clean, transformed and scaled data for your Performance Testing needs

Challenges getting access to Business SME for duration of Performance Test

Challenges getting access to the application SME’s during the course of the program

Challenges getting buy in from all the development and support teams to support Performance Testing activities

Challenges convincing the client to invest in Performance Testing infrastructure equivalent to production

Challenges due to lack of Non Functional Requirements

Challenges due to lack of understanding of Business workload

Deliverables – Performance Testing

Following would be the deliverables for Performance Testing -

Non Functional Requirements document (If not created during the initial PE phase)

Overall Business Workload to be processed by the application (If not created during the initial PE phase)

Overall Performance Testing approach

Tooling requirements

Data requirements

Resource requirements

Infrastructure requirements

Network requirements

Resources & Tools

HP Load Runner - http://www8.hp.com/us/en/software-solutions/software.html?compURI=1175451

IBM Rational Performance Tester - http://www-01.ibm.com/software/awdtools/tester/performance/

Microsoft Visual Studio Ultimate Edition - http://www.microsoft.com/visualstudio/en-gb/products

Webload - http://radview.com/

Neoload - http://www.neotys.com/

Load Storm - http://loadstorm.com/performance-testing-tool

Micro Focus Silk Performer - http://www.microfocus.com/products/silk/silkperformer/index.aspx

SOASTA Cloud Performance Test - http://www.soasta.com/

Thank You

Please support us by taking a moment and sharing this content using the Social Media Links at Practical Performance Analyst

trevor@practicalperformanceanalyst.com