Top ten secret weapons for performance testing in an agile environment

patrick.kua@thoughtworks.com(and alistair.jones@thoughtworks.com)

© ThoughtWorks 2011

SECRET

WEAPON

#

Make Performance Explicit

© ThoughtWorks 2011

1

SECRET

WEAPON

# So that I can make better investment decisionsAs an investorI want to see the value of my portfolio presented on a single web pagemust have “good” performance, less than 0.2s

page load for about 10,000 concurrent users

© ThoughtWorks 2011

© ThoughtWorks 2011

So that investors have a high-quality experience as the business growsAs the Operations ManagerI want the portfolio value page to render within 0.2s when 10,000 users are logged in

SECRET

WEAPON

#

One Team

© ThoughtWorks 2011

2

© ThoughtWorks 2011

Team Dynamics

© ThoughtWorks 2011

Performance Testers Part of Team

© ThoughtWorks 2011

© ThoughtWorks 2011

Performance Testers Part of Team

© ThoughtWorks 2011

Pair on Performance Test Stories

© ThoughtWorks 2011

Rotate Pairs

SECRET

WEAPON

#

Customer Driven

© ThoughtWorks 2011

3

© ThoughtWorks 2011

What was a good source of requirements?

© ThoughtWorks 2011

Existing Pain Points

© ThoughtWorks 2011

An example...

© ThoughtWorks 2011

So that we can budget for future hardware needs as we growAs the data centre managerI want to know how much traffic we can handle now

© ThoughtWorks 2011

Another example

© ThoughtWorks 2011

So that we have confidence in meeting our SLAAs the Operations ManagerI want to ensure that a sustained peak load does not take out our service

© ThoughtWorks 2011

Personas

© ThoughtWorks 2011

Who is the customer?End Users

OperationsPower Users

Marketing

Investors

SECRET

WEAPON

#

Discipline

© ThoughtWorks 2011

4

© ThoughtWorks 2011

Is the hypothesis

valid?

Change the application

code

Observe test results

Formulate an hypothesis

Design an experiment

Run the experiment

What do you see?

Why is it doing that?

How can I prove that’s what’s happening?

Take the time to gather the evidence.

Safe in the knowledge that I’m making it faster

© ThoughtWorks 2011

© ThoughtWorks 2011

Saw tooth pattern (1 minute intervals)

Directory structure of (yyyy/mm/minuteofday)?. Slow down due to # of files in directory?

1 directory should result in even worse performance...

We ran the test…

Is the hypothesis

valid?

Change the application

code

Observe test results

Formulate an hypothesis

Design an experiment

Run the experiment

One Directory

© ThoughtWorks 2011

SECRET

WEAPON

#

Play Performance Early

© ThoughtWorks 2011

5

© ThoughtWorks 2011

End

Start End

Other projects start performance testing

here

Start

Agile projects start performance testing as

early as possible

SECRET

WEAPON

#

Iterate Don’t (Just) Increment

© ThoughtWorks 2011

6

© ThoughtWorks 2011

© ThoughtWorks 2011

We Sashimi

© ThoughtWorks 2011

Sashimi Slice By... Presentation

© ThoughtWorks 2011

So that I can better see trends in performanceAs the Operations ManagerI want a graph of requests per second

© ThoughtWorks 2011

So that I can better see trends in performanceAs the Operations ManagerI want a graph of average latency per second

© ThoughtWorks 2011

So that I can easily scan results at a single glanceAs the Operations ManagerI want a one page showing all results

© ThoughtWorks 2011

Sashimi Slice By... Scenario

© ThoughtWorks 2011

So that we never have a day like “October 10”As the Operations ManagerI want to ensure that a sustained peak load does not take out our service

© ThoughtWorks 2011

So that we never have a day like “November 12”As the Operations ManagerI want to ensure that an escalating load up to xxx requests/second does not take out our service

SECRET

WEAPON

#

Automate, Automate, Automate

© ThoughtWorks 2011

7

© ThoughtWorks 2011

AutomatedCompilation

AutomatedTests

AutomatedPackaging

AutomatedDeployment

Automation => Reproducible and ConsistentAutomation => Faster FeedbackAutomation => Higher Productivity

Why Automation?

© ThoughtWorks 2011

© ThoughtWorks 2011

AutomatedApplication Deployment

AutomatedLoad

Generation

AutomatedTest

Orchestration

AutomatedAnalysis

Automated Scheduling

Automated Result Archiving

SECRET

WEAPON

#

Continuous Performance Testing

© ThoughtWorks 2011

8

© ThoughtWorks 2011

Application

Build Pipelines

© ThoughtWorks 2011

Performance

© ThoughtWorks 2011

SECRET

WEAPON

#

Test Drive Your Performance Test Code

© ThoughtWorks 2011

9

© ThoughtWorks 2011

V Model Testing

http://en.wikipedia.org/wiki/V-Model_(software_development)

Performance Testing

Slower + Longer

Fast

Speed

© ThoughtWorks 2011

We make mistakes

© ThoughtWorks 2011

V Model Testing

http://en.wikipedia.org/wiki/V-Model_(software_development)

Performance Testing

Slower + Longer

Fast

Speed

Unit test performance

code to fail faster

© ThoughtWorks 2011

Fail Fast!

Fast feedback!

Faster learning

Faster results

© ThoughtWorks 2011

Classic Performance Areas to Test

Analysis

Information Collection

Visualisation

Publishing

Presentation

SECRET

WEAPON

#

Get Feedback

© ThoughtWorks 2011

10

© ThoughtWorks 2011

Frequently (Weekly) Showcase

Here is what we learned this week....

© ThoughtWorks 2011

Frequently (Weekly) Showcase

And based on this... We changed our directory structure.

© ThoughtWorks 2011

Frequently (Weekly) Showcase

Should we do something different knowing this new information?

© ThoughtWorks 2011

List of All Secret Weapons

1. Make Performance Explicit2. One Team3. Customer Driven4. Discipline5. Play Performance Early6. Iterate Don't (Just) Increment7. Automate, Automate, Automate 8. Test Drive Your Performance Code9. Continuous Performance Testing10. Get Feedback

© ThoughtWorks 2011

Photo Credits (Creative Commons licence)• Barbed wire picture: http://www.flickr.com/photos/lapideo/446201948/• Eternal clock: http://www.flickr.com/photos/robbie73/3387189144/• Sashimi from http://www.flickr.com/photos/mac-ash/3719114621/

Questions