Top Agile Metrics

Agile Metrics

XBOSo'

Founded in 2006 Dedicated to software quality

Software QA Consulting Software Testing Offices in San Francisco, Beijing and Amsterdam

XBOSo', Inc. All Rights Reserved. 2

House Rules

•  Everyone except the speakers is muted •  QuesDons via gotowebinar control panel OR via TwiHer @XBOSo' using #Top10AgileMetrics

•  QuesDons can be asked throughout the webinar, and will be answered if possible

•  You’ll receive informaDon for recording a'er the webinar


Meet Our Speakers •  CEO and Founder of XBOSoft •  Relevant specialties and passions

–  Software quality process, evaluation, measurement and improvement

–  Mobile User Experience and usability

–  Cycling and travel


Philip Lew

•  CEO of Project Realms & TeamQualityPro

•  A big fan of automated dashboard solutions

•  Over 14 years in software testing and business intelligence

Jay Philips

Sabrina Gasson •  Marketing Manager of

XBOSoft •  Emails you all regularly

to join our industry hot topic webinars

•  And invites you all to download our latest trends in software testing whitepapers.

WHAT IS ONE OF THE MOST FAMOUS THINGS HE SAID?


“LIFE IS LIKE A BOX OF CHOCOLATES, YOU NEVER KNOW WHAT YOU’RE GOING TO GET”

Agile Metrics

By using metrics, we can know what we are going to get!

Agenda

1. Agile, Frameworks and GQM 2. Applying GQM to Agile 3. Top 10 Agile Metrics


Poll#1


What We Can Agree On •  Want to minimize mistakes and doing things over (rework)

•  Want to uDlize resources effecDvely •  Want to predict when things could go wrong •  Want to do what we say we will do, when we say -‐ Dmelines

•  Fix defects early – later is more expensive •  ProducDon defects are bad news


What We Can Agree On About Metrics

•  Want easy to get/capture •  Should be communicated and understood across the organizaDon

•  Should have a valuable purpose toward improvement – Evaluate – Guide – Insighcul – AcDonable


Agile – How Can We Improve?

•  Some say no metrics needed •  Is velocity all you track? •  What is sacrificed for velocity? •  How can we know what and how we are doing or where to improve?

•  How do we know we are spending our effort, resources and Dme in the right place?


Using Frameworks to Organizing Our Thoughts

•  Putting metrics into a framework helps us to categorize and organize

•  Helps us to make sure we have all functions in our organization and process covered

•  Helps us to develop relationships between different elements in our process


Poll #2

Applying Basili’s GQM •  Many so'ware metrics programs have failed because they

had poorly defined, or even non-‐existent objecDves.

Goal

QuesDon Metric


Stakeholders and GQM

Goal 1 Goal 2 Goal 3

QuesDon 1

QuesDon 2

QuesDon 3

QuesDon 4

QuesDon 5

Metric 1

Metric 2

Metric 3

Metric 4

Metric 5

Metric 6

Metric 7

QA Manager Dev Mgr CTO CEO


GQM - Example

•  Goals –  Reduce total cost of development –  Reduce total cost of testing-effort –  Reduce the number of new feature bugs

•  Questions –  Which areas have the highest re-work rates? –  Which functional areas have the most defects? –  How long does it take to repair defects? –  How complete are my requirements? –  How much time do I spend testing?


Metric Q&A Worksheet Category QuesGon Metric Tool to Use Needed

Customer SaDsfacDon

Are we meeDng schedule?

Schedule Variance

Project Management Tool

None

So'ware TesDng Efficiency

When are we finding defects?

So'ware Quality

What kinds of defects are most common?

Usability Are end users gemng lost, understand our UI

So'ware Test EffecDveness

?


Agile ConsideraDons

Waterfall •  Speed •  Quality •  Cost

Agile •  Speed •  Quality •  Cost


RETURN TO BASICS Agile Manifesto


Excerpts from the Agile Manifesto 1.  Working so'ware over comprehensive documentaDon 2.  Our highest priority is to saDsfy the customer

through early and conDnuous delivery of valuable so'ware. 3.  Deliver working so'ware frequently, from a couple of weeks to a

couple of months, with a preference to the shorter Dmescale. 4.  Business people and developers must work together daily

throughout the project. 5.  Working so'ware is the primary measure of progress. 6.  Agile processes promote sustainable development. The sponsors,

developers, and users should be able to maintain a constant pace indefinitely.

7.  ConDnuous aHenDon to technical excellence and good design enhances agility.


Agile – Understanding Where We Came From?


What are the Goals of Agile?

Speed Quality

Customer-‐User Sa.sfac.on


Matching Agile CharacterisDcs and ObjecDves

Agile CharacterisGc

Speed Quality Customer and End User SaGsfacGon

Other

CollaboraDve x x Working So'ware

x

Adapt to changing requirements

x x

From the agile manifesto


What QuesDons Tell Me If I’m Reaching Those ObjecDves?

Speed Quality

Owner ParDcipaDon

Requirements

Working Product

Planned versus unplanned

Schedule

Technical debt



Speed

Requirements

Did I get them in Dme?

Do I understand them?

Did we deliver working product?

Schedule

Am I on schedule?

Can I be more producDve?

How much wasted Dme? Rework?

Unplanned versus planned

Did I esDmate well?

Did I finish what I said I would?

Do I make mistakes that cause more (unplanned) work?



Quality

Product owner

Did my end users find defects?

Did product owner give feedback regularly?

Are there less defects than before?

Requirements

Are my requirements delivered when needed?

Did the delivered product match the requirement? Or reqt. wrong? Do engineers have trouble

understanding the requirements?

Technical Debt

Is my so'ware performance gemng worse and worse?

Am I prepared if my lead engineer leaves?

Do I mess things up when fixing things?


METRICS THAT HELP US ANSWER THESE QUESTIONS!

ARE YOU READY FOR THE COUNTDOWN?


VELOCITY

10.


Am I on schedule? Did I estimate well? Am I getting faster or slower?

Velocity •  Tracking velocity over Dme tells us our progress and capacity and helps us forecast based on our capabiliDes (now and in the future).

•  Measurement – Sum of all the approved esDmates of the stories

•  One point in Dme measurement has liHle meaning, so need to track over Dme or w.r.t. another metric

•  Metric – Velocity (T)-‐(T-‐1)/Velocity (T) XBOSo', Inc. All Rights Reserved. 29

Velocity


-‐25%

-‐20%

-‐15%

-‐10%

-‐5%

0%

5%

10%

15%

20%

25%

30%

1 2 3 4 5 6 7 8 9

Velocity Variance

Burn Down – (Distance over Time)


Poll#3


DEFECT REMOVAL EFFECTIVENESS

9.


Did my end users find defects? Are there less defects than before?

Defect Removal EffecDveness •  We don’t want the end user to find defects •  Any defect found by the user influences quality

•  Metric – Defects in found in ProducDon (DFP) during (90 day) period post release versus Pre-‐Release (DPR) – DPR/(DFP + DPR)


What Should Your Goal Be For DRE

•  EsDmated (by Capers Jones) that defect removal effecDveness differs by different levels of process capability maturity levels: – Level 1: 85% – Level 2: 89% – Level 3: 91% – Level 4: 93% – Level 5: 95%


DRE DistribuDon


C

A

ADDED WORK

8.


Am I on schedule? Am I overcommitting? Is my rate of work sustainable?

Added Work

•  This will help us see if we are making promises that maybe we should not – overcommimng and unsustainable.

•  Measurement – Sum of work (new stories added) over and above the original plan.

•  Metric – New stories added/Original Stories Planned


Piling On


0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9

Revised Planned

Original Plan

WORK CATEGORY ALLOCATION

7.


Am I allocating resources optimally? Do I have trouble understanding requirements?

Work Category AllocaDon •  This will help us see where we are spending our Dme and adjust our prioriDes.

•  Measurement – Sum of work (for each category) Metric – Divide by total work to get % allocaDon

•  Categories could include: –  Technical debt – refactor -‐ redo –  Regression tesDng –  New feature development –  Research –  Requirements review –  Defect fixing –  Other


Is Effort in Alignment?


0%

5%

10%

15%

20%

25%

30%

Are we spending enough time showing our stakeholders what we’ve done?

Are we fixing things too much or making lots of mistakes?

WORK SUSTAINABILITY

6.


Do I have the right people on my team? Did I estimate work well? Can I attain quality sustainably?

Work Sustainability •  “Sustainable” -‐ We can’t sustain our effort if we are conDnuously pumng in overDme

•  OverDme is necessary, but needs to be managed and causes mistakes

•  Mistakes manifest themselves in many ways, such as technical debt or defects

•  OverDme o'en results from poor work esDmates and directly proporDonal to technical debt

•  Metric – Unplanned OverDme Work/Total Work


Unplanned OverDme %


0%

2%

4%

6%

8%

10%

12%

14%

16%

1 2 3 4 5 6 7 8 9

Sprint

What caused this?

Unplanned OverDme and Defects


0

1

2

3

4

5

6

7

1 2 3 4 5 6 7

Defects Found

Unplanned OT

DEFECT REWORK RATE

5.


Is my technical debt accumulating? Am I wasting time?

Defect Rework Rate

•  We can’t get faster if we are always fixing things.

•  Depends on valid data whereby people are logging real hours to tasks they take on.

•  Metric -‐ Fix Effort RaDo : Time to fix defects/Total effort expended (hours)


Defect ReWork %


0%

5%

10%

15%

20%

25%

30%

35%

1 2 3 4 5 6 7 8 9

Accumulated Debt


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5 6 7 8 9

Defect Rework%

Unfixed %

TOTAL UNEXPECTED WORK

4.


What is throwing me off schedule? Am I biting off too much?

Total Unexpected Work •  Measurement – Sum of work done above the original esDmates for those stories in the plan.

•  Are we underesDmaDng difficulty? •  Many reasons:

–  Incomplete user stories – Misunderstood user stories – Bigger and more complex than I thought? – Not enough Dme for research – Too hard…

•  Metric – (Total work done – Planned work)/Planned work


Are We EsDmaDng Well?


0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9

Planned

Unplanned

Per Person/Week During the Sprint

What are we doing that is unplanned?


Research Requirements Coding TesDng Other

29%

9%

19%

23%

20%

DEFECTS IN PRODUCTION AVG. FIX TIME

3.


Are my customers/end users happy? Is the architecture prohibiting me?

Defect in ProducDon Avg. Fix Time •  If defects make it to producDon, that’s bad ! •  But what is worse is if an important defect stays unfixed.

•  Defects that take a long Dme to fix represent either technical debt or poor process

•  Metric – Time to fix Defects (released to producDon-‐usually P1 defects found) / Total P1 Defects (post producDon)


Defect aging?

Don’t Make Them Wait


0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9

#P1 Defects Found

Average Fix Time (Hrs)

Hey, we’re improving! WooHoo!

DEFECT CATEGORIZATION ALLOCATION %

2.


Why are things breaking? Can I get faster?

Defect CategorizaDon AllocaDon

•  Reduce rework = Increase speed •  Discover when defects are injected •  Reduce and prevent defects from happening in the first place.

•  Drive down the Fix Effort RaDo leading to beHer quality and resulDng in higher velocity.

•  Examine defect rates per category by %


Defect Rework AllocaDon


Requirements Coding Environment TesDng

38%

29%

21%

13%

What’s going on here?

Rework-‐Requirements Errors

Requirements •  Change in requirements •  Don’t understand requirements •  Incomplete requirement •  Requirement does not exist-‐ missing

– Didn’t think about a scenario •  ? ‘oh, I meant this…’


WORKING SOFTWARE DELIVERY RATE

1.


Am I delivering what I said I would? Am I getting feedback from the PO? Will I meet the release date?

Working So'ware Delivery Rate •  Does the so'ware work or not? •  Working does not mean perfect •  Working means can show in order to get feedback •  Metric -‐ # story points delivered that ‘work’ / # story points that were to be shown to the PO but could not be shown to work – Many variaDons


Am I Gemng BeHer?


0

5

10

15

20

25

1 2 3 4 5 6 7 8 9

Planning Working Visible

Actual Working

Am I Delivering So I can Get Feedback?


0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8 9

% Working

Be

Do

Have XBOSo', Inc. All Rights Reserved. 66

Process •  Iterative (sprints) •  Daily standups •  Face to face communication •  Post mortem – end of sprint •  Delivery meeting – end of sprint •  Planning meeting – before sprint •  Self organizing

People •  Communicative •  Collaborative/Cooperative •  Flexible and willing •  Knowledgeable-multi •  Initiative/responsible •  Responsive/adaptive

Results •  Speed •  Quality

Agile Is a Way of Thinking “Be-‐Do-‐Have”

JCARD

How to View it all? Example Dashboard


Am I improving? What should I plan?

JCard •  Used to show side by side comparison of improvement for various areas (hours, defects, tesDng)

•  The data is shown in a table format to show data and visual icons, which allows a user to automaDcally see if there was an improvement or not.

•  How do you know if you’re improving if you can’t look back at what’s already happened?

•  Allows you to plan resources for future releases


Am I Improving?


• What and why

•  Goals and objecDves

Understand Ask

yourself quesDons

Evaluate

Measure to answer those

quesDons and take acDon

Improve

TAKE AWAYS-‐SUMMARY


Post your questions on Twitter and we'll answer them @XBOSoft Join us to keep updated on all our webinars, reports and whitepapers: facebook.com/xbosoft +xbosoft linkedin.com/company/xbosoft We post regularly on our blog – check us out! http://xbosoft.com/software-quality-blog/

Why not download our free Agile Metrics Whitepapers, available here: http://xbosoft.com/go/agile-quality-metrics-whitepaper/ http://xbosoft.com/go/agile-velocity-testing-metrics-whitepaper/

You will receive an email with information on slides and recording. Any further queries regarding our services or ideas for future webinars please email us! [email protected]

Q+A

www.xbosoft.com

Participate in our Agile Testing Survey!

http://xbosoft.com/agile-testing-survey/

Top Agile Metrics

Software

Transcript of Top Agile Metrics