Agile metrics - Agile KC Meeting 9/26/13

41
Agile Metrics Craig Rudman VP Services, Agile Coach

description

Agile KC Presentation on 9/26/13

Transcript of Agile metrics - Agile KC Meeting 9/26/13

Page 1: Agile metrics - Agile KC Meeting 9/26/13

Agile Metrics

Craig RudmanVP Services, Agile Coach

Page 2: Agile metrics - Agile KC Meeting 9/26/13

Purpose

• Greater alignment of strategy to execution• Improve the flow of work• Improve product quality• Support continuous improvement efforts• Improve the quality of life• Increase trust

Discuss ways to go about measuring work and workflow

Page 3: Agile metrics - Agile KC Meeting 9/26/13

Roadmap

Context

• What can we measure?

• What should we measure?

• What principles inform agile metrics?

Systems Thinking

• Who is your customer?

• The nature of demand

• Value creation

Agile at Scale

• Portfolio• Program• Team

Page 4: Agile metrics - Agile KC Meeting 9/26/13

Why Metrics?

• Increase visibility of work• Increase alignment of strategy to

execution• Improve trust• Improved product quality• Improved workflow• Improved performance• Identify and solve problems

Metrics give us insight into the health of our organization, our people, processes, and tools.

We use metrics to help us adapt and improve

Page 5: Agile metrics - Agile KC Meeting 9/26/13

We can quantify those things we can seeWe can qualify the things we can't see

What can we measure?

Quantify• Effort• Time• System Components• Tests• Events• Data

Qualify• Quality• Perceptions• Feelings• Attitudes

Page 6: Agile metrics - Agile KC Meeting 9/26/13

We should measure the things we value

What should we measure?

If your goal is…• Improving visibility of work

• Better alignment• Increasing trust• Improving quality• Improving process• Solving problems

then you should measure…→ work-in-process→ context→ commitments→ breakage and rework→ flow→ root causes

Page 7: Agile metrics - Agile KC Meeting 9/26/13

The Manifesto for Agile Software Development

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools• Working software over comprehensive documentation• Customer collaboration over contract negotiation• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Page 8: Agile metrics - Agile KC Meeting 9/26/13

The Twelve Principles of Agile Software Development • Our highest priority is to satisfy the customer through early and continuous delivery of

valuable software.• Welcome changing requirements, even late in development. Agile processes harness change

for the customer's competitive advantage.• Deliver working software frequently, from a couple of weeks to a couple of months, with a

preference to the shorter timescale.• Business people and developers must work together daily throughout the project.• Build projects around motivated individuals. Give them the environment and support they

need, and trust them to get the job done.• The most efficient and effective method of conveying information to and within a

development team is face-to-face conversation.• Working software is the primary measure of progress.• Agile processes promote sustainable development. The sponsors, developers, and users

should be able to maintain a constant pace indefinitely.• Continuous attention to technical excellence and good design enhances agility.• Simplicity--the art of maximizing the amount of work not done--is essential.• The best architectures, requirements, and designs emerge from self-organizing teams.• At regular intervals, the team reflects on how to become more effective, then tunes and

adjusts its behavior accordingly.

Page 9: Agile metrics - Agile KC Meeting 9/26/13

Agile Metrics Support Agile PrinciplesIf motivated by these principles… Measure for these outcomes…

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Customer satisfaction Cycle time and throughput Product value realized

Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

Changeability of requirements Value of change

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

Frequency of delivery Product stability

Business people and developers must work together daily throughout the project. Frequency/quality of collaboration Delays

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

Alignment of work to motivated people Quality of team support Release predictability

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

Efficiency of communication Effectiveness of communication

Working software is the primary measure of progress. Rate of product adoption Frequency of product usage Failure demand Breakage

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

Burnout, attrition, turnover, morale Stability of cadence and velocity Stability of team organization

Continuous attention to technical excellence and good design enhances agility. Skills growth Increasing rates of reuse Design/code consistency and quality Test coverage/automation

Simplicity--the art of maximizing the amount of work not done--is essential. Low complexity Short work queues

The best architectures, requirements, and designs emerge from self-organizing teams. Internal locus of control High feature/story acceptance rates

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Consistency of cadence Effectiveness of retrospectives Progress towards improvement goals

Page 10: Agile metrics - Agile KC Meeting 9/26/13

Exercise: Five Votes

• Read through the agile principles• Think about where you would like to see

improvement• Cast your five votes accordingly

– You can put them all on one item– You can distribute them across up to five items

• Working as a group, we will discuss a set of metrics to support the highest-priority item

Page 11: Agile metrics - Agile KC Meeting 9/26/13

Customers Suppliers

Assets/ExpensesProfit/LossCash Flow

Market ShareCustomer SatisfactionEmployee SatisfactionShareholder Benefit

Society BenefitKnowledge Creation

Product/Service QualityTime to Market

Demand

Revenue

Resources

Take a Systems View

Demand

Value

Revenue

Metrics help us understand and manage these things

The primary metric for agile is whether or not value is being created. This is determined empirically, by demonstration, at the end of every single iteration.

Your business is a value-creation system

Page 12: Agile metrics - Agile KC Meeting 9/26/13

The Big Blue Company

Information Services

Data Management

21

Application Development218

Infrastructure/Ops77

Services94

Claims (206)

Prov. & Gov't Relations (162)

Sales & Marketing (146)

Finance (106)

All Others (155)

Customer Service (227)

Membership (129)

Who is your customer?

Demand

Value

Revenue

Page 13: Agile metrics - Agile KC Meeting 9/26/13

Mandates

Preven

tive M

aintenance

How much work can you manage?

Initiatives

Production Support

Enhancement Requests

Defects

Rework

Stuff and Nonsense

Page 14: Agile metrics - Agile KC Meeting 9/26/13

74%

18%

8%

How much work can you manage?

Production Support

Enhancement Requests

Defects

Rework

Stuff and Nonsense

Mandates

Preven

tive M

aintenance

Initiatives

209,320

49,78822,150

Total capacity = 281,258 hrs

Page 15: Agile metrics - Agile KC Meeting 9/26/13

Types of Demand

• What is the nature of customer demand?– How much of it is failure demand?– How much of it is mandates?– How much of it is value demand?

• If demand exceeds capacity, what are your options?

“All time spent handling failure demand is waste.”*

*Mary Poppendieck

Page 16: Agile metrics - Agile KC Meeting 9/26/13

Portfolio Management

• Failure Demand (from 74% to 50%)– Defects– Rework– Missing Features

• Value Demand (from 8% to 20%)– Preventive Maintenance– New Features or Services– New Capabilities

• Mandates (from 18% to 30%)– Regulatory compliance

50%

30%

20%

Failure Demand Mandates

Value Demand

Constrain effort to align with strategy

Page 17: Agile metrics - Agile KC Meeting 9/26/13

It’s All About Choices• Failure Demand

– Take a long-term view– Perform root-cause analysis to find the source of greatest demand– Fix it– Repeat the process

• Value Demand– Rank initiatives according to highest value– Estimate the size of the effort– Find the highest value with the least effort– Build that thing with the least delay– Repeat the process

• Mandates– Fix the due date– Prioritize features using a MoSCoW model (Must, Should, Could, Won't)– Evaluate design options and favor the simplest approach, even if sub-optimal– Optimize workflow to maximize throughput

Page 18: Agile metrics - Agile KC Meeting 9/26/13

How we build itWhat we build

Things We Can Measure

Performance• Throughput• Cycle Time• Effort• Schedule• Inventory

– Work Not Started– Work In Progress

Value• ROI• Revenue• Cost• Cash Flow• Market Share• Growth• Quality

Page 19: Agile metrics - Agile KC Meeting 9/26/13

Value Goals

• Increase revenue• Increase market share• Increase customer satisfaction• Reduce cost• Manage risk

Page 20: Agile metrics - Agile KC Meeting 9/26/13

Features Deliver Value…• Improve customer

satisfaction for both members and providers

• Increase the number of claims processed per employee

• Reduce administrative costs as a percentage of revenue

Add claim status inquiry to web

Add additional conditions to claims adjudication so that

fewer claims "PEND"

Reduce dependencies between systems by introducing service

layer to architecture

Page 21: Agile metrics - Agile KC Meeting 9/26/13

… But Only When In Production

• How long will you have to wait before the feature makes it into production?

• How long will the feature have to be in use before it pays for development and begins to add value?

Ideation

Design

Planning

Coding

Testing

Staging

Release

Effort incurs development costs

Time waiting for new features represents lost opportunity costs

Page 22: Agile metrics - Agile KC Meeting 9/26/13

Cost/Benefit Analysis of Features

• Begin by establishing the value proposition• Evaluate your solution options• Calculate development effort and cost of each• Calculate the cost of delay• Track adoption and use of the feature to

measure value

Page 23: Agile metrics - Agile KC Meeting 9/26/13

Example: Claim Status on the Web

Every day, Customer Service handles some number of phone calls with certain percentage of those calls are inquiring into claim status. That leads us to the following analysis:

• Assume 2400 phone calls each day on average• Assume 20% of them on average (480) are claim status inquiries• Assume 10 minutes per call on average• Assume $50 per hour is the average fully-loaded cost of a CSR• 4800 minutes or 80 hours per day on claim status inquiry equates

to $4,000 per day• If there are 253 work days in a year, that equals $1,012,000 spent

responding to claim status inquiry each yearNote: the example shown is purely fictional and for illustration purposes only.

Page 24: Agile metrics - Agile KC Meeting 9/26/13

Evaluating Options

Three solutions are considered:1. Do nothing2. Develop a feature with fully-committed

teams in four months at a cost of $255,0003. Develop the same feature with partially-

committed teams in sixteen months at a cost of $338,000

Page 25: Agile metrics - Agile KC Meeting 9/26/13

Jan-13

Feb-13

Mar-13

Apr-13

May-13

Jun-13Jul-1

3

Aug-13

Sep-13

Oct-13

Nov-13

Dec-13Jan

-14

Feb-14

Mar-14

Apr-14

May-14

Jun-14Jul-1

4

Aug-14

Sep-14

Oct-14

Nov-14

Dec-14Jan

-15

Feb-15

Mar-15

Apr-15

May-15

Jun-15Jul-1

5

Aug-15

Sep-15

Oct-15

Nov-15

Dec-15

$-

$20,000

$40,000

$60,000

$80,000

$100,000

$120,000

$140,000

$160,000

$180,000

Option 1Option 2Option 3

Cost of DelayTotal 3-year cost:1. Do Nothing: $3,040,0002. Dedicated team: $977,0003. Shared team: $2,070,000

$1,093,400

Page 26: Agile metrics - Agile KC Meeting 9/26/13

Performance Goals

• Increase throughput• Reduce cycle time• Improve quality• Eliminate waste• Improve predictability

Page 27: Agile metrics - Agile KC Meeting 9/26/13

Better Processes Improve Performance

• Increase throughput• Decrease schedule

variance• Shorten feature cycle

time

Increase frequency of releases

Negotiate scope

Reduce WIP to move smaller batches through development

Page 28: Agile metrics - Agile KC Meeting 9/26/13

Goal: Improve Throughput

• Features per quarter• Stories per sprint• Issues per day• Value per month

Throughput is the rate at which a system achieves its goal in units of output over time

Page 29: Agile metrics - Agile KC Meeting 9/26/13

High Variability Reduces Flow

Symptoms– Flow becomes

unpredictable– Bottlenecks occur– Cycle time increases– Throughput goes down

Response– Reduce WIP– Increase slack– Swarm around bottlenecks

Knowledge work inherently has a high degree of variability

Page 30: Agile metrics - Agile KC Meeting 9/26/13

Flow

Design Capacity: the maximum amount of work that a system is capable of completing over time

Cycle Time: the amount of time that work spends in the system

Utilization: the amount of capacity being used at a given time (%)

Effective Capacity: the actual amount of work that a system is capable of completing over time

WIP: the amount of work in that is in flight at a given time

Slack: the amount of capacity not being used at a given time (%)

Throughput: the average rate at which completed work exits the system

Constraint: anything that limits throughput

Page 31: Agile metrics - Agile KC Meeting 9/26/13

Collecting Flow Data

• Model value stream• Record transitions• Calculate cycle time• Calculate WIP

Feature Ready Dev/Test Add'l Test DeployCycle Time

(days)1 2/1/13 2/2/13 2/2/13 2/3/13 22 2/1/13 2/2/13 2/3/13 2/3/13 23 2/1/13 2/3/13 2/3/13 2/5/13 44 2/1/13 2/3/13 2/4/13 2/5/13 45 2/1/13 2/3/13 2/6/13 2/7/13 66 2/1/13 2/4/13 2/9/13 2/9/13 87 2/1/13 2/4/13 2/9/13 2/9/13 88 2/1/13 2/6/13 2/9/13 2/10/13 9

Date Ready Dev/Test Add'l Test Deployed Total WIP2/1/13 10 0 0 0 102/2/13 8 1 1 0 102/3/13 5 2 1 2 82/4/13 7 3 2 2 122/5/13 7 3 0 4 102/6/13 6 3 1 4 102/7/13 8 3 0 5 112/8/13 9 4 0 5 132/9/13 10 2 2 7 14

2/10/13 9 1 3 8 13

Page 32: Agile metrics - Agile KC Meeting 9/26/13

Cumulative Flow Diagram

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 2402468

10121416182022242628303234363840

ReadyDevTestDeployed

WIP (9 items)

Throughput (1.25 items/day)

Cycle Time (10 days)

Page 33: Agile metrics - Agile KC Meeting 9/26/13

Agile Teams

• Team: a group of people linked together with common purpose

• Backlog: Work identified, but not yet started• Sprint: a time-boxed interval of time during which work is

completed• Story: a narrative about the feature to be developed with clear

completion criteria• Story Points: an estimate of the size of a story compared to

other stories• Velocity: team throughput, as measured by story points per

sprint

Page 34: Agile metrics - Agile KC Meeting 9/26/13

Velocity Measures

• Burndown: total work remaining

• Burnup: total work completed

• Scope Change: additions or subtractions of work from the backlog

• Glidepath: estimated time remaining until the backlog is depleted based on available data

SprintPts

Completed Burnup BurndownScope

Changes Glidepath0 750 9001 40 40 790 80 9152 90 130 700 8503 10 140 790 100 7844 75 215 765 50 7195 120 335 565 -80 6546 32 367 533 5897 60 427 473 5238 95 522 378 4589 393

10 32811 26212 19713 13214 6715 116 0

Page 35: Agile metrics - Agile KC Meeting 9/26/13

1 2 3 4 5 6 7 80

100

200

300

400

500

600

700

800

900

1000

BurnupBurndownGlidepath

Velocity Reporting

Velocity = 65 pts/sprint

Page 36: Agile metrics - Agile KC Meeting 9/26/13

Velocity Pros and ConsUsed Well• Helps teams make commitments

that they can keep• Helps estimate the depth of a

backlog• Helps product owners know how

much runway to prepare• Helps teams reflect on their

efficiency and identify ways to improve

• Accurately reflects the sustainable pace of a team

• Emphasizes the importance of “done”

Used Poorly• Misunderstood as a measure of

team effectiveness• Misused as a measure of

individual performance • Invites meaningless or damaging

comparisons between teams• Leads to Inflated Velocity

Syndrome (IVS), a misguided attempt to improve throughput by focusing on utilization rather than outcomes.

Page 37: Agile metrics - Agile KC Meeting 9/26/13

Commitment

• Scope is negotiable• Teams are self-governing• Deadlines matter• Done means done

Agility is rooted in commitment

Page 38: Agile metrics - Agile KC Meeting 9/26/13

Commitment Variance

1 2 3 4 5 6 7 80

20

40

60

80

100

120

140 PlanActualRange (-20%)Range (+20%)

Page 39: Agile metrics - Agile KC Meeting 9/26/13

A Good Metric

• Supports your goals• Fits your process• Is fact-based• Difficult to

manipulate

Page 40: Agile metrics - Agile KC Meeting 9/26/13

Inspect and AdaptPortfolio• Inspect: Demand (planned and unplanned)• Adapt: Control allocation of effort to demand categories

Program• Inspect: Workflow; feature acceptance rates; value attainment• Adapt: Reduce WIP; negotiate feature scope and priority; remove

bottlenecks; increase slack; smaller, more frequent releases

Team• Inspect: Team throughput, commitment variance, quality• Adapt: Negotiate story scope and priority, remove impediments,

regular retrospectives, continuous improvement

Page 41: Agile metrics - Agile KC Meeting 9/26/13