Agile Metrics

Santhana Krishnan

Mar 20, 2009

• What is Metrics?

• A Good Agile Metric or Diagnostic…

• Metric / Diagnostic Evaluation Checklist

• Metrics and Examples

• References

Contents

• Quantitative measures of performance or production used to indicate progress or achievement against strategic goals - www.s3co.com/glossary/

• Measurable element of a service, process or function. The real value of metrics is seen in their change over time. Reliance on a single metric is not advised, especially if it has the potential to affect User behaviour in an undesirable way - www.infodiv.unimelb.edu.au/knowledgebase/itservices/a-z/m.html

What is Metrics?

The Iron Triangle

Sco

peTim

e

Cost

Quality

Affirms and reinforces Agile principlesSupports the customer-intimate and value focused traits that reinforce Agile principles. This requires that people who

understand Agile participate in metrics design. The truism "you get what you measure“ reminds us that counterproductive

behaviors may ensue if you reinforce the wrong things (ex: overtime, % utilization, paperwork)

Follows trends, not numbersMeasure "one level up“ to ensure you measure aggregated information, not sub-optimized parts of a whole. Aggregate

above the individual team level for upper management use. To promote process health, do not track at levels more

granular than “a team”, and “an iteration”.

Belongs to a small set of metrics and diagnosticsA "just enough" metrics approach is recommended: too much information can obscure important trends.

A Good Agile Metric or Diagnostic… (1 of 3)

Measures outcome, not outputIn an Agile environment where simplicity or "maximizing the amount of work not done" is promoted, the most

spectacular outcome might be achieved by reducing planned output while maximizing delivered value. Outcomes

are measured in terms of delivered Customer value

Is easy to collectFor team-level diagnostics the ideal is "one button" automation - where data is drawn from operational tools (i.e.

the Product Backlog, acceptance test tools, code analyzers). For management use, avoid rework (ex:

powerpoints) and manipulation of lower level data, aggregation is preferable.

Reveals, rather than conceals, its context and significant variablesShould be visibly accompanied by notes on significant influencing factors, to discourage false assumptions and

facilitate improvement.

A Good Agile Metric or Diagnostic… (2 of 3)

Provides fuel for meaningful conversationFace-to-face conversation is a very useful tool for process improvement. A measurement isolated from its context loses

its meaning. Note: It's a good sign when people talk about what they've learned by using a metric or diagnostic.

Provides feedback on a frequent and regular basisTo amplify learning and accelerate process improvement, metrics should preferably be available at each iteration

retrospective, and at key periodic management meetings.

May measure Value (Product) or ProcessDepending on where problems lie, diagnostics may measure anything suspected of inhibiting effectiveness. Consider

the appropriate audience for each metric, and document its context/assumptions to encourage proper use of its content.

And remember: you get what you measure!

Encourages "good-enough" qualityThe definition of what's "good enough" in a given context must come from that context's Business Customer or their

proxy, not the developers.

A Good Agile Metric or Diagnostic… (3 of 3)

Metric / Diagnostic Evaluation ChecklistName This should be well chosen to avoid ambiguity, confusion, over simplification

Question It should answer a specific, clear question for a particular role of group. If there are multiple questions, design other metrics.

Basis of Measurement

Clearly state what is being measured, including units. Labeling of graph axes must be clear rather than brief

Assumptions Should be identified to ensure clear understanding of data represented

Level and Usage Indicate intended usages at various levels of the organization. Indicate limits on usage, if any

Expected Trend the designers of the metric should have some idea of what they expect to see happen. Once the metric is proven, document common trends

When to Use It what prompted creation or use of this metric? How has it historically been used?

When to Stop Using It when will it outlive its usefulness, become misleading or extra baggage? Design this in from the start

How to Game It think through the natural ways people will warp behavior or information to yield more ‘favorable’ outcomes

Warnings recommend balancing metrics, limits on use, and dangers of improper use

Metrics and examples

VelocityName VelocityQuestion How much software can my team deliver per iteration?

Basis of Measurement Story points or “ideal engineering hours”

Assumptions The team is delivering working software every iteration

Level and UsageVelocity is most useful at the project level. It allows the team to forecast how much work they can expect to complete based on prior efforts.

Expected Trend

Velocity can be affected by many things: Changing team members, obstacles, toolsets, difficulty of feature or amount of learning required, etc. will lower the velocity of the team. Barring unexpected obstacles, a stable team on the same project with the required resources will generally gain in velocity during the course of the project, then plateau.

When to Use ItVelocity is a very useful metric for the team, and should be used during the course of the project once work has started.

When to Stop Using It

In a longer project when the team, resources, and technology are all stable, velocity will also become stable. The team may suspend collecting velocity since it is "known."

How to Game ItVelocity is only meaningful to the exact team providing the data - each team will estimate their work differently from other teams.

Warnings

Velocity is not the same as value. A team with excellent velocity could spend months quickly and effectively delivering software that does not have the investment potential. Comparing velocity of teams is problematic (see above) and should be avoided: this diagnostic is a barometer for the team itself, as a unit. Team member velocities are problematic: velocity should be measured at the team level, since that is the unit that must self-organize to produce value

Velocity Chart – an Example

Scope Burn Up – an Example

Story Rate

Average Story Point Per Day Per Developer

0.1

1.1

0.40.6

0.4

1.3

0.81.0

0.9

1.1 1.1

1.4

0.8

2.1

1.3

1.1

1.3 1.31.1

1.3

0.3

1.2 1.1

0.7

0.0

0.5

1.0

1.5

2.0

2.5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Iteration #

Sto

ry P

oin

t

Story Rate 3 per. Mov. Avg. (Story Rate)

Cost per point

Finger Chart

Worm Chart

QA Metrics – A Sample

Summary – Selenium Tests

Tests Failures Errors Success rate

Time (sec)

52 (677 assertions) 1 0 98% 39 mnts

Code Coverage

 

Iteration 17

Iteration 16

Status

Total Coverage 90.3% 88.9%  

Code Coverage 88.5% 86.6%  

Bugs

  Fixed Carried over Status

Iteration 17 15 5  

To Date 207 5  

More Metrics

Build:• Frequency of builds • Average duration of builds• Number of broken builds per

iteration • Average duration of broken build • Number of builds per iteration

Development:• Cyclometric complexity measures • Distribution of method and class

lengths • Rate of change of source (loc

in/out) • Proportion of source code that is

test code

Tests:• Unit tests per story• Functional tests per story• Defects carried over per iteration• Defects per story

Scope:• Scope change (stories removed or added

from scope due to redundancy or rewrite) per iteration

• Scope changes not caused by additional stories per iteration

• User Stories carried forward (hangover) per iteration

• No of stories held in Analysis, Development, Testing per iteration

• One can get completely lost in too many metrics

• A "just enough" metrics approach is recommended: too much information can obscure important trends

• "If you can't measure it, you can't manage it.“ – Peter Drucker

Just make it simple and practical. Measure those metrics you choose and act on them.

Warning!

• Appropriate Agile Measurement: Using Metrics and Diagnostics to Deliver Business Value

Deborah Hartmann, Agile Process Coach, deborah@hartmann.net & Robin Dymond, Agile Management Consultant, robin.dymond@gmail.com

• Cohn, Mike, Agile Estimating and Planning, Prentice Hall, 2006

References