No Crystal Ball Gazing -...

[email protected], @djaa_dja

No Crystal Ball Gazing Risk Management & Delivery with Kanban

Using qualitative risk assessment &

probabilistic forecasting for better business

results

Devlin Linkoping, March 2013


Understanding Kanban Systems


H

F F

F F F

F J

I

Pull

Change Requests

Kanban are virtual! Backlog

D

E

A

I

Engin- eering Ready

G

5 Ongoing

Development Testing

Done 3 3

Test Ready

5

PTCs

F

B

C Pull

Pull These are the virtual kanban

*

These are the virtual kanban These are the virtual kanban These are the virtual kanban

The board is a visualization of the workflow process, the work-in-progress and the kanban

Boards are not required to do Kanban!

The first system used database triggers to signal pull. There was no board!

UAT

Deploy-ment

Ready ∞ ∞


Wish to avoid discard after commitment

Commitment is deferred Backlog

H

E

C A

I

Engin- eering Ready

D

5 Ongoing

Development Testing

Done 3 3

Test Ready

5

Commitment point

UAT

Deploy-ment

Ready ∞ ∞

F F

F F F

F F

G

Pull

Pool of

Ideas

Backlog

Items in the backlog remain optional………………………..

Items in the backlog remain optional and unprioritized

We are committing to getting started. We are certain we want to take

delivery.


F F

F F F

F F

Specific delivery commitment may be deferred even later

UAT

H

E

C A

I

Engin- eering Ready

Deploy-ment

Ready

G

D

5 ∞

Pull

Ongoing

Development Testing

Done 3 3

Test Ready

5

PTCs

Change Requests

2nd Commitment

point*

∞

Discarded

I

We are now committing to a specific deployment and delivery date

*This may happen earlier if circumstances demand it

Pool of

Ideas


F F

F F F

F F

Replenishment Cadence UAT

H

E

C A

I

Engin- eering Ready

Deploy-ment

Ready

G

D

5 ∞

Replenishment Ongoing

Development Testing

Done 3 3

Test Ready

5

PTCs

Change Requests

∞

Discarded

I

The frequency of system replenishment should reflect arrival

rate of new information and the transaction & coordination costs of

holding a meeting

Pull

Frequent replenishment is more agile.

On-demand replenishment is

most agile!

Pool of

Ideas


F F

F F F

F F

Delivery Cadence UAT

H

E

C A

I

Engin- eering Ready

Deploy-ment

Ready

G

D

5 ∞

Delivery Ongoing

Development Testing

Done 3 3

Test Ready

5

PTCs

Change Requests

∞

Discarded

I

The frequency of delivery should reflect the transaction & coordination

costs of deployment plus costs & tolerance of customer to take delivery

Pull Deployment buffer size can reduce as frequency of delivery increases

Frequent deployment is more agile.

On-demand deployment is

most agile!

Pool of

Ideas


F F

F F F

F F

Defining Kanban System Lead Time UAT

H

E

C A

I

Engin- eering Ready

Deploy-ment

Ready

G

D

5 ∞

Pull

Ongoing

Development Testing

Done 3 3

Test Ready

5

PTCs

Change Requests

∞

System Lead Time

The clock starts ticking when we accept the customers order, not when

it is placed!

Until then customer orders are merely available options

Lead time ends when the item reaches the

first ∞ queue.

This provides the correct result for Little’s Law and

visualization on a Cumulative Flow

Diagram Discarded

I

Pool of

Ideas


Delivery Rate Lead Time

WIP =

Avg. Lead Time

Avg. Delivery Rate WIP

Pool of

Ideas

Ready To

Deploy

Little’s Law & Cumulative Flow


F F

F F F

F F

Defining Customer Lead Time UAT

H

E

C A

I

Engin- eering Ready

Deploy-ment

Ready

G

D

5 ∞

Pull

Ongoing

Development Testing

Done 3 3

Test Ready

5

PTCs

Change Requests

∞

Customer Lead Time

The clock still starts ticking when we accept the customers order, not when

it is placed!

Discarded

I

Pool of

Ideas

Done

∞

The frequency of delivery cadence will affect customer lead time in

addition to system capability


Flow Efficiency

Done

Pool of

Ideas

F

H E

C A

I

Engin- eering Ready

Deploy-ment

Ready

G D

GY PB

DE MN

2 ∞

P1

AB

Lead Time

Ongoing

Development Testing

Done Verification Acceptance 3 3

Flow efficiency measures the percentage of total lead time is spent actually adding value (or knowledge)

versus waiting

Until then customer orders are merely available options

Waiting Waiting Waiting Working

Flow efficiency = Work Time x 100%

Lead Time Flow efficiencies of 2% have been reported*. 5% -> 15% is normal, >

40% is good!

* Zsolt Fabok, Lean Agile Scotland, Sep 2012, Lean Kanban France, Oct 2012

Working

Multitasking means time spent in working columns is often waiting time


Observe Lead Time Distribution as an enabler of a Probabilistic Approach to Management

Lead Time Distribution

0

0.5

1

1.5

2

2.5

3

3.5

1 8 15 22 29 36 43 50 57 64 71 78 85 92 99 106 113 120 127 134 141 148

Days

CRs &

Bugs

SLA expecta+on of 44 days with 85% on-‐+me

Mean of 31 days

SLA expecta+on of 105 days with 98 % on-‐+me

This is multi-modal data!

The work is of two types: Change Requests (new features); and

Production Defects

This is multi-modal data!

The work is of two types: Change Requests (new features); and

Production Defects


Filter Lead Time data by Type of Work (and Class of Service) to get Single Mode Distributions

85% at 10 days

Mean 5 days

98% at 25 days

Cha

nge

Req

uest

s

Pro

duct

ion

Def

ects

85% at 60 days

Mean 50 days

98% at 150 days


Lead Time

Lead Time

Allocate Capacity to Types of Work

Done

Pool of

Ideas

F

H

E

C

A

I

Engin- eering Ready

Deploy-ment

Ready

G

D

GY

PB DE

MN

2 ∞

P1

AB

Separate understanding of Lead Time for each type of work

Ongoing

Development Testing


Change Requests

Production Defects

Separate understanding of Lead Time for each type of work

4

3

Consistent capacity allocation should bring some consistency to delivery rate

of work of each type

Consistent capacity allocation should bring more consistency to delivery rate

of work of each type


The Optimal Time to Start

impact

When we need it

85th percentile

Ideal Start Here

Commitment point

If we start too early, we forgo the option and opportunity to do

something else that may provide value.

If we start too late we risk incurring

the cost of delay

With a 6 in 7 chance of on-time delivery, we can always expedite to

insure on-time delivery


Metrics for Kanban Systems

Cumulative flow integrates demand, WIP, approx. avg. lead time and delivery rate capabilities Lead time histograms show us actual lead time capability Flow efficiency, value versus failure demand (rework), initial quality, and impact of blocking issues are also useful


Scaling Up (Probabilistic Forecasting)


Scaling Up - Planning a big project Device Management Ike II Cumulative Flow

020406080100120140160180200220240

10-Feb

17-Feb

24-Feb

2-Mar

9-Mar

16-Mar

23-Mar

30-Mar

Time

Feat

ures

Inventory Started Designed Coded Complete

Slope in middle 3.5x -‐ 5x slope

at ends 5x

Required (local average) delivery rate

2006 2008

During the middle 60% of the project schedule we need Throughput (velocity) to average 220 features

per month


Delivery Rate Lead Time

WIP =

Little’s Law

From observed capability

Treat as a fixed

variable

Target to

achieve plan

Calculated based on known lead time

capability & required delivery rate

Determines staffing level

Changing the WIP limit without maintaining the staffing level ratio represents a change to the way of

working. It is a change to the process and will produce a change in the

observed ‘common cause’ capability of the system

Plan based on currently observed capability and current working practices. Do not assume process improvements.

If changing WIP to reduce undesirable

effects (e.g. multitasking), get new sample data (perform a spike) to observe

the new capability


55/week 0.4 weeks

WIP = 22 =

Using Little’s Law

From observed capability

Treat as a fixed

variable

Target to

achieve plan

Calculated based on known lead time

capability & required delivery rate

Determines staffing level

At this point perhaps just a little black magic and experience may be required.

Rounding 22 up to 25 would

conveniently provide for 5 teams with a WIP limit of 5 items each

If our current working practices/process exhibited an average WIP of 1 item per

person then we require 25 people organized in 5 teams of 5 people to

complete the project on-time


1 lane per team

2-tiered board Kanban System within a Kanban System


Lead time

WIP in this area should be 25 items*

*photo taken early in the project before it

was fully staffed/loaded

Median lead time target is 2 days

Alert managers if beyond 5 days


Risks & Qualitative Assessment


A Lean approach to alignment with business risks uses Qualitative Assessment

But how do we determine the risks in a work item that we must manage? We need a fast, cheap, accurate, consensus forming

approach to risk assessment. We need Lean Risk Assessment!

The answer is to use a set of qualitative methods to assess different dimensions of risk such as urgency


Sketch market payoff function R

oom

nig

hts

sold

per

day

Actual rooms sold

Cost of delay

Estimated additional rooms sold

When we need it When it arrived

Cost of delay for an online Easter holiday marketing promotion is difference in integral between the two curves

time


Cost of Delay based on Market Payoff Sketches

Cost of delay function for an online Easter holiday marketing campaign delayed by 1 month from mid-January (based on diff of 2 integrals on previous slide)

Treat as a Standard Class item

2me

impa

ct

Total cost of delay


Establish urgency by qualitative matching of cost of delay sketches

+me

impact

+me

+me

+me

impact

impact

impact

+me

impact

+me

impact

impact

Expedite – critical and immediate cost of delay; can exceed kanban limits (bumps other work)

Fixed date – cost of delay goes up significantly after deadline; Start early enough & dynamically prioritize to insure on-time delivery

Standard - cost of delay is shallow but accelerates before leveling out; provide a reasonable lead-time expectation

Intangible – cost of delay may be significant but is not incurred until much later; important but not urgent

+me


Cost of Delay has a 2nd Dimension

+me

impact

+me

impact

+me

impact

impact

Extinction Level Event – a short delay will completely deplete the working capital of the business

Major Capital – the cost of delay is such that a major initiative or project will be lost from next year’s portfolio or additional capital will need to be raised to fund it

Discretionary Spending – departmental budgets may be cut as a result or our business misses its profit forecasts

Intangible – delay causes embarrassment, loss of political capital, affects brand equity, mindshare, customer confidence, etc

+me

?

Working capital

Working capital


Risk is a multi-dimensional problem

So understanding cost of delay enables us to know what to pull next? Yes, however, it isn’t always relevant! Cost of delay

attaches to a deliverable item. What if that item is large? Whole projects, minimum marketable features (MMFs) or minimum viable products (MVPs) consist of many smaller items. We need to understand the risks in those smaller

items too, if we are to know how to schedule work, replenish our system and make pull decisions wisely


Market Risk of Change

Mar

ket R

isk

Sche

dulin

g

Highly likely to change

Highly unlikely to

change

Start Early

Start Late

Differentiators

Spoilers

Table Stakes

Cost Reducers

Potential Value Profits

Market Share etc

Regulatory Changes

Buy (COTS) Rent (SaaS)

Build (as rapidly as

possible)


Product Lifecycle Risk

Prod

uct R

isk

Inve

stm

ent

Not well understood High demand for innovation &

experimentation

Well understood Low demand for

innovation

Low

Low

Innovative/New

Major Growth Market

Cash Cow

Growth Potential

High

Low

High


Shelf-Life Risk

Short (days, weeks,

months)

Medium (months, quarters,

1-2 years)

Long (years, decades)

Known Expiry Date,

Seasonal (fixed window of

opportunity)

Fashion Craze Fad


Shelf-Life Risk

Sche

dule

Ris

k

Short (days, weeks,

months)

Medium (months, quarters,

1-2 years)

Long (years, decades)

High

Low

Known Expiry Date,

Seasonal (fixed window of

opportunity)

Fashion Craze Fad

Inno

vatio

n High

Low

Num

ber o

f Opt

ions

Many

Few


Shelf-Life Risk

Diff

eren

tiato

r

Short (days, weeks,

months)

Medium (months, quarters, 1-2 years)

Long (years,

decades)

Low

Low

Known Expiry Date,

Seasonal (window of opportunity)

Fashion Craze Fad

Inno

vatio

n

High

Low

Num

ber o

f Opt

ions

Many

Few

Spoi

ler/F

ollo

wer

High

Low

Schedule Risk

If we are market leading our innovations are less time critical


Risk is a multi-dimensional contextual problem

These are just useful examples!

We must develop a set of risk taxonomies that work in context for a

specific business.

We can easily envisage other risk dimensions such as technical risk, vendor dependency risk, organizational

maturity risk and so forth.

It may be necessary to run a workshop with stakeholders to explore and expose the real business risks requiring

management


(Just a taste of) Risk Management with Kanban


How much risk do you want to take?

Given our current capabilities, our desired strategic position and go-to-

market strategies, how much risk do you want to take?

We only have capacity to do so much work. How we allocate that capacity across different risk dimensions will

determine how aggressive we are being from a risk management perspective.

The more aggressive we are in allocating capacity to

riskier work items the less likely it is that the outcome will match our expectations


Hedge Delivery Risk by allocating capacity in the kanban system

Done

F H

E

C

A

I

Engin- eering Ready

Deploy-ment

Ready

G

D

GY

PB MN

2 ∞

P1 AB

Ongoing

Development Testing


Expedite 1

3

Fixed Date

Standard

Intangible

2

3 DE


Aligning with Strategic Position or Go-to-Market Strategy

Done

F

H

E

C

A

I

Engin- eering Ready

Deploy-ment

Ready

G

D

GY

PB MN

2 ∞

P1 AB

Ongoing

Development Testing


Table Stakes 3

1

Cost Reducers

Spoilers

Differentiators

2

1 DE

DA

The concept of a minimum viable product (MVP) will contain the table

stakes for at least 1 market niche Market segmentation can be used to narrow the

necessary table stakes for any given market niche! Enabling early delivery for narrower markets but

potentially including value generating differentiating features


Trade off growing market reach against growing share & profit within a niche

Done

F

H

E

C

A

I

Engin- eering Ready

Deploy-ment

Ready

G

D

GY

PB MN

2 ∞

P1 AB

Ongoing

Development Testing


Table Stakes 3

1

Cost Reducers

Spoilers

Differentiators

2

1 DE

DA

It is important to define a MVP in terms of table stakes and differentiators required to enter a specific market

segment

Capacity allocated to Table Stakes will determine how fast new niches can be developed.

Allocate more to Table Stakes to speed market reach/

breadth.

Allocate more to differentiators to grow market share or profit margins

Allocate more to spoilers to defend market share


An underlying philosophy of pragmatism


Some simple rules to improve delivery forecasting

1.  Limit WIP 2.  Observe what really happens in your own

environment 3.  Measure current system capability as lead

time distribution, throughput rate and flow efficiency

4.  Forecast Probabilistically

Assume that the past is a strong predictor of the future

In low flow efficiency systems, environmental conditions (system factors) outweigh technical performance factors

by up to 20 times in determining the outcome. If the environment isn’t changing neither should results.


Prediction based on qualitative risk assessment

Stop Crystal Ball Gazing!

Do not speculate!

Do not “estimate” the size, weight, complexity of an

item. Instead qualitatively assess the risks inherent in

a work item


Some simple rules to improve risk management

1.  Establish a list of risks that are applicable in your business domain

2.  Create a qualitative taxonomy of 2 to 6 categories for each dimension of risk

3.  Work with things that can be established by consensus as (soft) facts. Do not speculate about the future!

4.  Use meaningful business language

Cost of delay, shelf-life, product adoption lifecycle, market risk of change

All can be established as (soft*) facts. Risks associated with different classifications within these risk dimensions

are understood and the dynamics of how they might affect an outcome are predictable

* Where hard facts cannot be established by measurement or market research, a strong consensus opinion is achieved


Prediction based on qualitative risk assessment

For example, if we load our entire capacity with fixed delivery date

demand then it is highly likely that some items will be delivered late and we will incur a (significant)

cost of delay


Allocate capacity to hedge risks

5.  Our key strategy to manage risk is to allocate capacity in accordance with our capability, risk tolerance and business risks under management

6.  Set kanban limits across risk categories

7.  Allow the kanban to signal what type of risk item to pull next


Defer Commitment. Banish Backlogs

8.  Defer Commitment to manage uncertainty

9.  The Lean concept of “Last Responsible Moment” is at the point of commitment – the point of replenishment

10. “Backlog” implies committed. Uncommitted items are options. Develop a “pool of ideas”

When developing options upstream of the commitment point, classify the item for each dimension of risk under

management.

A good mix of options, providing choices within each risk category is required. The more risks under management the more options will be required. The greater the min-

max upstream kanban limits will need to be


Abandon Prioritization. Banish Priority

11. Prioritization is waste!

Prioritization is an exercise to schedule a sequence of items at a specific point in time. Only at the point of commitment can a proper assessment be made of what to pull next. Filter options based on kanban signals. Select from filtered subset

Priority is a proxy variable for real business risk information.

Do not mask risk behind a proxy. Enable better

governance and better decision making by exposing the business risks under management throughout the

workflow


Abandon Formulas & Calculations

12. Do not try to give relative weight to risk categories or calculate a risk number using a formula

Weightings and formulas mask real risk information and may lead us to unbalanced, misallocated capacity and poor risk management decisions

Without a formula calculating a priority should be impossible!

Embrace the idea that formulas and proxy variables such

as “priority” have no place in sound risk management decision making

Transparently expose business risks throughout the

system


Visualize Risks on the Ticket

Title

Checkboxes… risk 1 risk 2 risk 3 risk 4

req

com

plet

e

Color of the ticket

Typically used to indicated

technical or skillset risks

H

Decorators

Letter

SLA or Target Date

Business risk visualization

highlighted in green


Visualize Risks on the Board

Done

F H

E

C

A

I

Engin- eering Ready

Deploy-ment

Ready

G

D

GY

PB MN

2 ∞

P1 AB

Ongoing

Development Testing


Expedite 1

3

Fixed Date

Standard

Intangible

2

3 DE


Conclusions


Focus on Sources of Delay

In low flow efficiency systems, focusing on sources of delay – queues, blocking issues, rework, provides high leverage improvement

in observed capability Lead time performance is strongly biased to

environmental factors, not technical capabilities


Forecast Probabilistically

Use observed capability data!

Accept that the future is likely to strongly reflect past performance. Use historical

data to forecast probabilistically

Abandon Cartesian decomposition and speculative

attempts to deterministically estimate size, complexity or level

of effort


Qualitative Approaches are Lean

Qualitative approaches to risk assessment are fast, cheap and drive consensus

There is no crystal ball gazing! Risk

analysis is not speculative!

Stop speculating about business value and ROI.

Instead assess real risks and design kanban systems to

manage them!


Kanban enables more predictable delivery and better risk management

Kanban systems address variability in, and focus attention on improving, flow!

Kanban enables predictable delivery

Exploit predictability in delivery with qualitative risk

management.

Stop Crystal Ball Gazing!


Thank you!


About

David Anderson is a thought leader in managing effective software teams. He leads a consulting, training and publishing and event planning business dedicated to developing, promoting and implementing sustainable evolutionary approaches for management of knowledge workers.

He has 30 years experience in the high technology industry starting with computer games in the early 1980’s. He has led software teams delivering superior productivity and quality using innovative agile methods at large companies such as Sprint and Motorola.

David is the pioneer of the Kanban Method an agile and evolutionary approach to change. His latest book is published in June 2012, Lessons in Agile Management – On the Road to Kanban. David is a founder of the Lean Kanban University, a business dedicated to assuring quality of training in Lean and Kanban for knowledge workers throughout the world.


Acknowledgements

Donald Reinertsen directly influenced the adoption of virtual kanban systems and the assessment of cost of delay & shelf-life as criteria for scheduling work into a kanban system. Daniel Vacanti helped with a deeper understanding of Little’s Law and the long term planning approach. Troy Magennis has been inspiring with his work on probabilistic planning, risk management and Monte Carlo simulation. I borrowed the term “Stop Crystal Ball Gazing” from Chris Matts.


David J Anderson & Associates, Inc.


Appendix


Example Distributions

Sta

ndar

d Exp

edite

Inta

ngib

le

Fixe

d D

ate

No Crystal Ball Gazing -...

Documents

Transcript of No Crystal Ball Gazing -...