Cycle Time Analytics Making decisions using Lead Time and Cycle Time to avoid needing estimates for...

Cycle Time AnalyticsMaking decisions using Lead Time and Cycle Time to avoid needing estimates for every

story

Troy Magennis @t_magennis

Slides at bit.ly/agilesim

LKCE 2013 – Modern Management Methods

2 @t_Magennis slides at bit.ly/agilesim

Actual Maximum

Actual Minimum

1

3

2

4

Q. What is the chance of the 4th sample being between the range seen after the first three samples? (no duplicates, uniform distribution, picked at random)

@t_Magennis slides at bit.ly/agilesim

Actual Maximum

Actual Minimum

1

3

2

4

Highest sample

Lowest sample


?

?

?

?


Actual Maximum

Actual Minimum

1

3

2

4

25% chance higher than highest seen

25% lower than highest and higher than second highest

25% higher than lowest and lower than second lowest

25% lower than lowest seen

Highest sample

Lowest sample


A. 50%

% = (1 - (1 / n – 1)) * 100@t_Magennis slides at bit.ly/agilesim

Actual Maximum

Actual Minimum

1

3

2

12



Highest sample

Lowest sample


?

?

A. 90%

% = (1 - (1 / n – 1)) * 100

4

5

6

7

8

9

10

11


# Prior Samples Prediction Next Sample Within Prior Sample Range

3 50%4 67%5 75%6 80%7 83%8 86%9 88%

10 89%11 90%12 91%13 92%15 93%17 94%20 95%


8

Four people arrange a restaurant booking after work

Q. What is the chance they arrive on-time to be seated?


Commercial in confidence9

Person 1 Person 2 Person 3 Person 41

in 1

6 EV

ERYO

NE

is O

N-T

IME

15 T

IMES

mor

e lik

ely

at le

ast o

n pe

rson

is la

te

Prioritized by Opinion / Mandatory

Team Exists & Fixed in size?

No Estimate!

Date Fixed or New Team?Forecast Team Size/Skillset

Options

Economic Prioritization Used?

Calculated Value / Cost of Delay?

Date Fixed or New Team?

Forecast Team Size/Skillset Options

Team Fixed in size?

Forecast Date, Manage Risks

Foggy Value / Cost of Delay?

Foggy Estimate GUESS!

70% Chance you need to

Forecast(You just performed a probabilistic forecast!

3.5 cases out of 5)

Estimate or #NoEstimate?

16

Forecasts are attempts to answer questions about

future events. They are an estimate within a stated

uncertainty

85% Change of

15th August 2013

Definitely Greater

than $1,000,000

At Least 2 Teams


There is NO single forecast result

Uncertainty In = Uncertainty Out

There will always be manypossible results, some more likely and this is the

key to proper forecasting@t_Magennis slides at bit.ly/agilesim

18

Probabilistic Forecasting combines many uncertain inputs to find many possible outcomes, and what

outcomes are more likely than others

Time to Complete Backlog

50% Possible Outcomes


Like

lihoo

d


19

Did the Obama 2012 Campaign Fund Advertising to Achieve 50% Chance of Re-election?

Time to Complete Backlog


15%

Like

lihoo

d


20

Task Uncertainty – Summing Variance

Source attribution: Aidan Lyon, Department of Philosophy. University of Maryland, College Park. “Why Normal Distributions Occur” http://aidanlyon.com/sites/default/files/Lyon-normal_distributions.pdf

1 2 3 4


http://aidanlyon.com/sites/default/files/Lyon-normal_distributions.pdf

22

Staff

Dev

Cos

t

Cost

of D

elay

Decision Induced Uncertainty

Forecast Completion DateJuly August September October November December

Planned / Due Date

Actual Date

Every choice we make changes the outcome


24

MODELING AND CYCLE TIMEWhat is modelling and how to use cycle time

A model is a tool used to mimic a real world process

Models are tools for low-cost experimentation



Depth of Forecasting modelsPartitioned Cycle Time

Simulated process

Sim

ple

Dia

gnos

tic

28

Simple Cycle Time Model

Amount of Work(# stories)

Lead Time or Cycle Time

ParallelWork in Proc.(WIP)

Random Chance / Risk / Stupidity @t_Magennis slides at bit.ly/agilesim

Use with attribution29

Capturing Cycle Time and WIPStory Start Date Completed

DateCycle Time

(days)1 1 Jan 2013 15 Jan 20132 5 Jan 2013 12 Jan 20133 5 Jan 20134 6 Jan 20135 3 Jan 2013 7 Jan 20136 7 Jan 2013 18 Feb 20137 10 Jan 2013 22 Jan 20138 10 Jan 2013 18 Jan 20139 13 Jan 2013 26 Jan 2013

10 15 Jan 2013

Date “Complete” – Date “Started”

14



DateCycle Time

(days)1 1 Jan 2013 15 Jan 20132 5 Jan 2013 12 Jan 20133 5 Jan 20134 6 Jan 20135 3 Jan 2013 7 Jan 20136 7 Jan 2013 18 Feb 20137 10 Jan 2013 22 Jan 20138 10 Jan 2013 18 Jan 20139 13 Jan 2013 26 Jan 2013

10 15 Jan 2013

Date WIP1 Jan3 Jan4 Jan5 Jan6 Jan7 Jan8 Jan9 Jan

10 Jan…

15 JanCount of Started, but

Not completed

5



DateCycle Time

(days)1 1 Jan 2013 15 Jan 2013 142 5 Jan 2013 12 Jan 2013 7 3 5 Jan 20134 6 Jan 20135 3 Jan 2013 7 Jan 2013 46 7 Jan 2013 18 Feb 2013 427 10 Jan 2013 22 Jan 2013 128 10 Jan 2013 18 Jan 2013 89 13 Jan 2013 26 Jan 2013 13

10 15 Jan 2013

Date WIP1 Jan 13 Jan 24 Jan 25 Jan 36 Jan 47 Jan 58 Jan 59 Jan 5

10 Jan 7… …

15 Jan 7

Sum Random Numbers

25112943342631452227

31436545

87

34735448

1912242721

39

202329

187410295Sum:

…..

Historical Story Cycle Time Trend

Days To Complete

Basic Cycle Time Forecast Monte Carlo Process

1. Gather historical story lead-times2. Build a set of random numbers based on pattern3. Sum a random number for each remaining story to build a potential outcome4. Repeat many times to find the likelihood (odds) to build a pattern of likelihood outcomes

𝑇𝑜𝑡𝑎𝑙𝐷𝑎𝑦𝑠❑=𝑆𝑢𝑚(𝑆𝑡𝑜𝑟𝑦𝑛×𝑅𝑎𝑛𝑑𝑜𝑚𝑛)

𝐸𝑓𝑓𝑜𝑟𝑡

More often

Less often

34

Backlog

Feature 1

Feature 2

Feature 3

1. Historical Cycle Time

3. The Work (Backlog)

2. Planned Resources/ WIP

Monte Carlo Analysis = Process to Combine Multiple Uncertain Measurements / Estimates

4. Historical Scope Creep Rate

(optional) 5. Historical Defect Rate and Cycle Times(optional)

6. Phases


35 @t_Magennis slides at bit.ly/agilesim


How certain based on model

forecast

Further calculations to

make economic tradeoffs

38

Y-Axis: Number of Completed

Stories

Project Complete Likelihood

Range of complete stories probability

0 to 50%

50 to 75%

>75%X-Axis: Date


39

FORECASTING STRATEGIES

Janu

ary

Febr

uary

Mar

ch

April

May

June July

Augus

tSe

ptem

ber

The Future

The Past

1. Model Baseline using historically

known truths

2. Test Model against historically

known truths3. Forecast

When you have historical data

42

Compare Model vs Actual Often


Range of complete probability

Actual results to compare if model is predictable

43

Forecast Trend Report CardWatch GREAT Watch Ok Bad Watch WORRY Watch

Cycle Time Trend versus Model ↘ ↘ ↗ ↗ ↘ ↘ ↗ ↗ACTIVE WIP Actual vs Model ↘ ↗ ↘ ↗ ↘ ↗ ↘ ↗Story Count Trend versus Model ↘ ↘ ↘ ↘ ↗ ↗ ↗ ↗

Arrow Legend: Green: Heading in beneficial direction Red: Heading in damaging direction

For “Watch” – the WIP and story count trends need to be analyzed. They are 1:1 ratio of compensating importance, and may offset each other.


When you have no historical dataJa

nuar

y

Febr

uary

Mar

ch

April

May

June July

Augus

tSe

ptem

ber

The Future


46

Simple Cycle Time Model

Amount of Work(# stories, scope creep, defects)Cycle

Time(range guess or samples)

ParallelWork in Proc.(WIP on board)


48

If we understand how cycle time is statistically distributed, then an

initial guess of maximum allows an inference to be made

Alternatives -

• Borrow a similar project’s data• Borrow industry data• Fake it until you make it… (AKA guess range)


49

Probability Density Function

Histogram Gamma (3P) Lognormal Rayleigh Weibull

x1301201101009080706050403020100-10

f(x)

0.32

0.28

0.24

0.2

0.16

0.12

0.08

0.04

0

Note: Histogram from actual data



Exponential Distribution (Weibull shape = 1)The person who gets the work can complete the workTeams with no external dependenciesTeams doing repetitive work E.g. DevOps, Database teams,


Weibull Distribution (shape = 1.5)Typical dev team ranges between 1.2 and 1.8


Rayleigh Distribution (Weibull shape = 2)Teams with MANY external dependenciesTeams that have many delays and re-work. E.g. Test teams

53


Histogram Weibull

x1201101009080706050403020100

f(x)

0.28

0.24

0.2

0.16

0.12

0.08

0.04

0

Scale – How Wide in Range. Related to the

Upper Bound. *Rough* Guess: (High – Low) / 4

Shape – How Fat the distribution. 1.5 is a good starting point.

Location – The Lower Bound


54

What Distribution To Use...

• No Data at All, or Less than < 11 Samples (why 11?)– Uniform Range with Boundaries Guessed (safest)– Weibull Range with Boundaries Guessed (likely)

• 11 to 30 Samples– Uniform Range with Boundaries at 5th and 95th CI– Weibull Range with Boundaries at 5th and 95th CI

• More than 30 Samples– Use historical data as bootstrap reference– Curve Fitting software


55

Questions…• Download the slides (soon) and software at

http://bit.ly/agilesim• Contact me

– Email: [email protected]– Twitter: @t_Magennis

• Read:

http://bit.ly/agilesim

mailto:[email protected]


CURRENT FORECASTING PRACTICES

Why traditional forecasting practices fail to deliver and why this is the only outcome possible…

Total Story Lead Time

30 days

Development Time5 Days (~ 15%)

Testing Time 2 Days

Defect Rework 2 Days

Release / DevOps Time 1 Day

Blocked and Waiting Time 9 Days

Waiting Time 3 Days

Waiting Time8 Days


30 days

Story / Feature Inception5 Days

Waiting in Backlog25 days

System Regression Testing & Staging 5 Days

Waiting for Release Window5 Days

“Active Development”30 days

Pre Work

30 days

Post Work

10 days


30 days






Pre Work

30 days

Post Work

10 days

9 days (70 total)approx 13%


ESTIMATING DISTRIBUTIONS USING HISTORIC DATA

How to use historic data to estimate the different distributions


What Distribution To Use...

• No Data at All, or Less than < 11 Samples (why 11?)– Uniform Range with Boundaries Guessed (safest)– Weibull Range with Boundaries Guessed (likely)

• 11 to 50 Samples– Uniform Range with Boundaries at 5th and 95th CI– Weibull Range with Boundaries at 5th and 95th CI– Bootstapping (Random Sampling with Replacement)

• More than 100 Samples– Use historical data at random without replacement– Curve Fitting


Tools

• EasyFit from http://mathwave.com $499• R http://cran.r-project.org Free• Statistics feature of KanbanSim

http://focusedobjective.com Free• Excel of course• Monte Carlo features of KanbanSim $495-995• Google for Monte Carlo Simulation Tools

– Oracle, Palisade, HubbardResearch (he may be here, so a big recommendation!)

http://mathwave.com/

http://cran.r-project.org/

http://cran.r-project.org/

http://focusedobjective.com/


Sampling at Random Strategies

• If you pick what samples to use, you bias the prediction…

• Strategies for proper random sampling –– Use something you know is random (dice, darts)– Pick two groups using your chosen technique and

compute your prediction separately and compare– Don’t pre-filter to remove “outliers”– Don’t sort the data, in fact randomize more if

possible


Concurrent WIP Sample : Find the smallest and the biggest or take at least 11 samples to be 90% sure of range

Estimating Concurrent Effort from Cumulative Flow Chart

20% 40%

60% 80%


Scope Creep Over TimeLook at the rate new scope is added over time

PREDICTION INTERVALS

What is the likelihood the next sample I take will be within the range of previous samples

Actual Maximum

Actual Minimum

1

3

2


4

Actual Maximum

Actual Minimum

1

3

2

4

Highest sample

Lowest sample


?

?

?

?

Actual Maximum

Actual Minimum

1

3

2

4


25% lower than highest and higher than second highest

25% higher than lowest and lower than second lowest


Highest sample

Lowest sample


A. 50%

% = (1 - (1 / n – 1)) * 100

Actual Maximum

Actual Minimum

1

3

2

12



Highest sample

Lowest sample


?

?

A. 90%

% = (1 - (1 / n – 1)) * 100

4

5

6

7

8

9

10

11

# Prior Samples Prediction Next Sample Within Prior Sample Range

3 50%4 67%5 75%6 80%7 83%8 86%9 88%

10 89%11 90%12 91%13 92%15 93%17 94%20 95%


THE SHAPE OF CYCLE TIMEWhat distribution fits cycle time data and why…



Histogram Gamma (3P) Lognormal Rayleigh Weibull

x1301201101009080706050403020100-10

f(x)

0.32

0.28

0.24

0.2

0.16

0.12

0.08

0.04

0

Note: Histogram from actual data



Histogram Weibull

x1301201101009080706050403020100-10

f(x)

0.32

0.28

0.24

0.2

0.16

0.12

0.08

0.04

0

Normal, or it will be after a few

thousand more simulations

Base + 1 Delay

Base + 2 Delays

Base + 3 Delays

Base + 4 Delays

Base + 5 Delays


ASSESSING AND COMMUNICATING RISK

Speaking Risk To Executives

• Buy them a copy of “Flaw of Averages”• Show them you are tracking & managing risk• Do

– “We are 95% certain of hitting date x”– “With 1 week of analysis, that may drop to date y”– “We identified risk x, y & z that we will track weekly”

• Don’t – Give them a date without likelihood

• “February 29th 2013”

– Give them a date without risk factors considered• “To do the backlog of features, February 29th, 2013”

We spend all our time estimating here

1 2 3

**Major risk events have the predominate role in deciding where deliver actually occurs **

Plan Performance Issues

External Vendor Delay

Risk likelihood changes constantly

1 2 3

95th Confidence

Interval


Models Start with Assumptions

• Cycle Time– Guessed– Last releases performance– Another teams performance

• Amount of Work– # Stories guessed– Scope creep guessed– Similar sizing distribution to last release assumed

• Concurrent WIP– Guessed– Measured from last release– Similar process to last release assumed


Forecast Completion DateJuly August September October November December

Planned / Due Date

Actual Date CCost to Develop

Staff Option A

Staff Option C

Staff Option B

Staff C : $

Staff B : $$

Staff A : $$$$$$$$Actual Date BActual Date A


Can’t I just use Average Cycle Time?

Forecast (Days)

Median 525Mean 670MC >50% 677MC >85% 793MC >95% 851

Projection of cycle time to complete 25 stories in a sequential fashion based upon historical cycle time data






Tota

l Cyc

le T

ime






Test

Dev

Le

adTi

me


Backlog

Feature 1

Feature 2

Feature 3

1. Historical Cycle Time

2. Planned Resources/ Effort

3. The Work (Backlog)

5. Historical Defect Rate & Cycle Times

4. Historical Scope Creep Rate

A Process to Combine Multiple Uncertain Measurements / Estimates is Needed

(optional)(optional)

Design Develop Test

Design Develop Test

Cycle Time Analytics Making decisions using Lead Time and Cycle Time to avoid needing estimates for...

Documents

Transcript of Cycle Time Analytics Making decisions using Lead Time and Cycle Time to avoid needing estimates for...