Paula O’Grady

Prioritising Tests - Why?You never have enough time to do all the

testing you want/need to do.Allocated testing time may ‘shrink’.How to decide which tests take precedence?Attach a severity or priority, or a calculation of

the two.Other labour-intensive methods.Limitations with this approach in my

experience!

Prioritising Tests – Issues?Ironically, the traditional prioritising techniques

you use may deplete available time even more.Like buying an expensive wallet and not having

enough money left to put into it!Perhaps we need to examine the methods we

use to prioritise tests to maximise efficiency.In addition, alternative techniques may provide

insight, and a new viewpoint.Historically, the best ideas have come from the

most unlikely sources.

Possible SolutionsLateral-thinking approach.As with most ‘radical’ methods, they work best

when used in tandem with the tried-and-tested traditional methods.

Betting & weighting approaches involving teams guessing where most bugs will be found.

Dynamic Betting Exchanges.Split-Second Decision Making and Rapid-

Cognition.Gut-Instinct.

Betting ApproachDivide the product into functional/test areas – this

is the difficult part in my experience.Give each person in the group a virtual €100 to

‘bet’, using as many denominations as they wish.The non-technical team members were especially

encouraged to provide gut-instinct feedback, independent of ‘development\test’ thinking.

In theory, participants can change their bets on the fly and as regularly as they wish – exchange aspect.

No need for sophisticated tools.

Betting Approach – Case StudyCollate data:

    JW FM EMG DMW PD SOS POG CK CON DMC AK RR MM AC Total %1Create Webs 10 2 5 10 2 2.62Alerts and Notifications 10

2 1.13Documentation 5 10 15 10 10 5 15 5 6.84Export Report and Generate Report 5

3 2 10 5 4 2.65Globalisation & Localisation

5 5 5 1.46Kit – Install and Uninstall

5 10 10 5 5 3.27Kit – Upgrade 5 15 10 10 5 5 4.58Project Schedule List 10 33 40 35 10 15 20 15 10 17.09All Other Lists and Libraries

0.010Usability 5 0.511Security 10 2 5 1.512 Load and Performance 2 10 10 5 2.1 2.613BrightWork Reporter & Resource Reporting* 50

5 30 10 30 30 15 20 10 27.9 20.714Gantt Web Part 1 3 2 25 2.815*Chart Web Part 5 10 10 30 15 20 15 8 20 12.116Agile Template 1 5 3 2 1.017Sure Step Templates 2 5 5 3 2 10 2.418All Other Templates 10 20 10 10 5 5.019All Other Web Parts 4 0.420Choice Indicator Icon Column*

2 5 10 5 1 5 2.521Project Calendar* 5 20 30 10 10 2 25 9.222Other – Please Specify 0.0

Betting Approach – Case StudyThe bets were charted. Scale runs from 0% to 25 %.

Betting Approach – Case StudyAdvantages DisadvantagesQuick.Fun.Give people a chance to

get involved (who otherwise wouldn’t).

Less weighted by the thinking of the core technical group.

Allows for ‘unexpected’ anomalies to be found.

Difficult to define the ‘areas’ to bet on.

May not be taken seriously.

Cannot be used on its own.

People may try to ‘follow the crowd’ and bet safely – this defeats the purpose.

Needs management buy-in.

Rapid CognitionLooking away from the arena of Software

Testing.Books that provide insight into how to make

effective decisions quickly:Blink – Malcolm GladwellThe Gift of Fear – Gavin deBeckerOther Books

Not written specifically for a Software Test audience, but the principles and how they apply to decision-making are universal.

Rapid CognitionWhat can these books teach us about

prioritising tests?The lessons here are universal.In times of stress, decisions need to be made.How firemen, policemen & doctors make life-

changing and life-saving decisions.Is there such a thing as ‘Too Much

Information’ when making decisions?

Gut InstinctDoes “Gut feeling = Irrational Decision Making”?In testing terms, using gut instinct could be

viewed as analogous to Exploratory Test Techniques for software.

Not having to explain each small decision and just going with your instinct.

Not necessarily being psychic – you may feel a piece of code has a bug in it because of something that happened the day it was being developed.

The feeling may not necessarily be fully articulated.

ConclusionsThe techniques described in this presentation

should be used in addition to traditional methods

As Agile methods become more commonplace, we need to look at modernising all activities to do with Test Planning.

The bugs that make it into production are generally the ones ‘that we didn’t expect’.

Less traditional methods of prioritising may help mitigate this risk.

www.brightwork.com

Copyright of BrightWork 2009