Top ten secret weapons for performance testing in an agile environment

Top ten secret weapons for agile performance testingby Patrick Kuapatrick.kua@thoughtworks.comhttp://www.thekua.com/atwork/presentations-and-papers/ ThoughtWorks 2010

Make Performance Explicit ThoughtWorks 2010 1

SECRETWEAPON#

So that I can make better investment decisionsAs an investorI want to see the value of my portfolio presented on a single web pagemust have good performance, less than 0.2s page load for about 10,000 concurrent users ThoughtWorks 2010

SECRETWEAPON#

In a conventional project, we focus on the functionality that needs to be delivered.Performance might be important, but performance requirements are considered quite separate from functional requirements.One approach is to attach conditions to story cards, i.e. this functionality must handle a certain load.In our experience, where performance is of critical conern, pull out the performance requirement as its own story 3

ThoughtWorks 2010

So that investors have a high-quality experience as the business growsAs the Operations ManagerI want the portfolio value page to render within 0.2s when 10,000 users are logged in

Calling out performance requirements as their own stories allows you to:validate the benefit you expect from delivering the performance-prioritise performance work against other requirements-know when youre done4

One Team ThoughtWorks 2010 2

SECRETWEAPON#

Team Dynamics ThoughtWorks 2010

Performance Testers Part of Team ThoughtWorks 2010

ThoughtWorks 2010

not sure if you like this picture, I was really looking for a good shot looking out over no-mans land at the Berlin wall.I want the idea of divisions along skill lines breading hostility and un-cooperation.8

Performance Testers Part of Team ThoughtWorks 2010

Pair on Performance Test Stories ThoughtWorks 2010

Rotate Pairs ThoughtWorks 2010

Customer Driven ThoughtWorks 2010 3

SECRETWEAPON#

Everything should be based on some foreseeable scenario, and who benefits from it

Harder to do without repetition (involvement and feedback) [not sure if this makes sense anymore]

Extremely important to keep people focused as its easy to drift

Capture different profiles

Separation simulation from optimisation -> Problem Identification vs Problem Resolution (or broken down further Solution Brainstorm -> Solution Investigation)

Linking back to why is even more essential -> map to existing problems or fears

Latency vs throughput -> determine what is the most useful metric and define service level agreements12

What was a good source of requirements? ThoughtWorks 2010

ThoughtWorks 2010

Existing Pain Points

http://www.flickr.com/photos/denniskatinas/2183690848/

Not sure which one you like better14

An example... ThoughtWorks 2010

So that we can budget for future hardware needs as we growAs the data centre managerI want to know how much traffic we can handle now

ThoughtWorks 2010

Heres an example... (in the style of Feature Injection) Whats our upper limit?

16

Another example ThoughtWorks 2010

17

ThoughtWorks 2010

So that we have confidence in meeting our SLAAs the Operations ManagerI want to ensure that a sustained peak load does not take out our service

Heres another example... (in the style of Feature Injection), Can we handle peaks in traffic again?

So that we have confidence in meeting our SLAAs the Operations ManagerI want to ensure that a sustained peak load does not take out our service

18

Personas ThoughtWorks 2010

19

Who is the customer? ThoughtWorks 2010

End Users

Operations

Power UsersMarketing

Investors

It helps to be clear about who is going to benefit from any performance testing (tuning and optimisation) that is going to take place. Ensure that they get a stake on prioritisation that will help with the next point...20

Discipline ThoughtWorks 2010 4

SECRETWEAPON#

ThoughtWorks 2010Is the hypothesis valid?Change the application codeObserve test resultsFormulate an hypothesisDesign an experimentRun the experimentWhat do you see?Why is it doing that?How can I prove thats whats happening?Take the time to gather the evidence.Safe in the knowledge that Im making it faster

Evidence-based decision-making. Dont commit to a code change until you know its the right thing to do.22

?????????? ThoughtWorks 2010

23

ThoughtWorks 2010Saw tooth pattern (1 minute intervals)Directory structure of (yyyy/mm/minuteofday)?. Slow down due to # of files in directory? 1 directory should result in even worse performance...We ran the testIs the hypothesis valid?Change the application codeObserve test resultsFormulate an hypothesisDesign an experimentRun the experiment

Evidence-based decision-making. Dont commit to a code change until you know its the right thing to do.24

One Directory ThoughtWorks 2010

Play Performance Early ThoughtWorks 2010 5

SECRETWEAPON#

It helps to have the customer (mentioned in the previous slide) be a key stakeholder to prioritise. 26

ThoughtWorks 2010EndStartEndOther projects start performance testing hereStartAgile projects start performance testing as early as possible

Application supports better ability to be performance tested easierLike TDD changes the design/architecture of a systemNeed to find reference for this

Measuring it early helps raise what changes contribute to slowness

Performance work takes longerLead times potentially large and long lead time (sequential) think of where gantt chart may actually be usefulRun it as a parallel track of work to normal functionality (not sequential)

Minimal environment availability (expensive, non concurrent use)

Need minimal functionality or at least clearly defined interfaces to operate against

Want to have some time to respond to feedback -> work that into the process as early as possible and potentially change architecture/design

27

Iterate Dont (Just) Increment ThoughtWorks 2010 6

SECRETWEAPON#

Start with the simplest performance test scenarios -> Sanity test/smoke test-> Hit all aspects-> Use to drive out automated deployment (environment limitations, configuration issues, minimal set of reporting needs green/red)-> Hit integration boundaries but with a small problem rather than everything

Next story might be a more complex script or something that drives out more of the infrastrcutre

Performance stories should not be :-> Build out tasks-> Does not enhance anything without other stories

Log files -> Contents early. Consumer Driven. Contracts for analysis. Keep around. Keep notes around what was varied

INVEST storiesAvoid the large performance test storySeparate types of storiesOptimise vs MeasureOptimise is riskier components. Less known. Done is difficult to estimateMeasure is clearer. Allows you to make better informed choicesKnow when to stopWhen enough is enough

28

ThoughtWorks 2010

The best lessons are learned from iterating, not from incrementing. Iterate over your performance test harness, framework and test fixtures. Make it easier to increment into new areas by incrementing in a different direction each time.

- Start with simple performance test scenarios - Dont build too much infrastructure at once - Refine the test harness and things used to create more tests - Should always be delivering value - Identify useful features in performance testing and involve the stakeholder(s) to help prioritise them in

Prioritise and schedule in analysis stories (metrics and graphs)

Some of this work will still be big29

We Sashimi ThoughtWorks 2010

Sashimi is nice and bite sized. You dont eat the entire fish at once. Youre eating a part of it. Sashimi slices are nice and thin. There are a couple of different strategies linking this in.

Think of sashimi as the thinnest possible slice.30

Sashimi Slice By... Presentation ThoughtWorks 2010

ThoughtWorks 2010

So that I can better see trends in performanceAs the Operations ManagerI want a graph of requests per second

Number of requests over time

32

ThoughtWorks 2010

So that I can better see trends in performanceAs the Operations ManagerI want a graph of average latency per second

Latency over time

33

ThoughtWorks 2010

So that I can easily scan results at a single glanceAs the Operations ManagerI want a one page showing all results

I dont want to click through to each graph

34

Sashimi Slice By... Scenario ThoughtWorks 2010

ThoughtWorks 2010

So that we never have a day like October 10As the Operations ManagerI want to ensure that a sustained peak load does not take out our service

I dont want to click through to each graph

36

ThoughtWorks 2010

So that we never have a day like November 12As the Operations ManagerI want to ensure that an escalating load up to xxx requests/second does not take out our service

I dont want to click through to each graph

37

Automate, Automate, Automate ThoughtWorks 2010 7

SECRETWEAPON#

ThoughtWorks 2010AutomatedCompilationAutomatedTests

AutomatedPackaging

AutomatedDeployment

Automated build is a key XP practice.The first stage of automating a build is often to automate compilationHowever, for a typical project, we go on after compilation to run tests, as another automated step.In fact we may have a whole series of automted steps that chain on after each other, automating many aspects of the development process, all the way from compiling source to to deploying a complete application into the production environment.39

Automation => Reproducible and ConsistentAutomation => Faster FeedbackAutomation => Higher ProductivityWhy Automation? ThoughtWorks 2010

Automation is powerful lever in software projects because:it gives us reproducable, consistent processesWe get faster feedback when something goes wrongOverall higher productivity we can repeat an automated build much more often than we could if it was manual40

ThoughtWorks 2010AutomatedApplication DeploymentAutomatedLoad Generation

AutomatedTest Orchestration

AutomatedAnalysis

Automated SchedulingAutomated Result Archiving

In performance testing we can use automate many of the common tasks in a similar way to how we automate a software build.For any performance test, there is a linear series of activities that can be automated (first row of slide)In our recent projects weve been using the build tool ant for most of performance scripting. You could use any scripting language, but here are some very basic scripts to show you the kind of thing we mean [possibly animate transitions to the 4 following slides]Once weve auomted the running of a single test, we can move on even more aspects of automation such as scheduling and result archiving, whch lead us intoContinuous Performance testing.41

Continuous Performance Testing ThoughtWorks 2010 8

SECRETWEAPON#

ApplicationBuild Pipelines ThoughtWorks 2010Performance

For a faster feedback, set up your CI server so that performance tests are always running against the latest version of the application.

43

ThoughtWorks 2010

Test Drive Your Performance Test Code ThoughtWorks 2010 9

SECRETWEAPON#

V Model Testing ThoughtWorks 2010http://en.wikipedia.org/wiki/V-Model_(software_development)

Performance TestingSlower + LongerFastSpeed

We make mistakes ThoughtWorks 2010

V Model Testing ThoughtWorks 2010http://en.wikipedia.org/wiki/V-Model_(software_development)

Performance TestingSlower + LongerFastSpeedUnit test performance code to fail faster

Classic Performance Areas to Test ThoughtWorks 2010AnalysisInformation CollectionVisualisationPublishingPresentation

Get Feedback ThoughtWorks 201010

SECRETWEAPON#

Frequently (Weekly) Showcase ThoughtWorks 2010

Here is what we learned this week....

Frequently (Weekly) Showcase ThoughtWorks 2010

And based on this... We changed our directory structure.

Frequently (Weekly) Showcase ThoughtWorks 2010

Should we do something different knowing this new information?

List of All Secret WeaponsMake Performance ExplicitOne TeamCustomer DrivenDisciplinePlay Performance EarlyIterate Don't (Just) IncrementAutomate, Automate, Automate Test Drive Your Performance CodeContinuous Performance TestingGet Feedback

ThoughtWorks 2010

Photo Credits (Creative Commons licence)Barbed wire picture: http://www.flickr.com/photos/lapideo/446201948/Eternal clock: http://www.flickr.com/photos/robbie73/3387189144/Sashimi from http://www.flickr.com/photos/mac-ash/3719114621/Questions? ThoughtWorks 2010