Modeling, Simulation & Data Mining: Answering Tough Cost, Date

& Staff Forecasts Questions

Troy Magennis (Focused Objective)

Larry Maccerone (Rally)

Pain Point

My Boss “Needs” A Date…

Difficulty Supporting

Decisions with Quantitative

Evidence

Difficulty in Assessing &

Communicating Risk / Uncertainty

Arm my teams (and yours) with the tools and techniques

to solve these problems

My Mission

Demo’s

1. Forecasting date & cost of a new project

2. Effective mining and modeling historical data

3. Finding the few valuable & needed estimates

4. Determining the cost of defects & added scope

5. Finding project staffing options and impact

6. Cost/benefit of various outsourcing models

7. Modeling and communicating risk factors

2 Minutes About Larry

• [some ideas: Experience history, Rally, Passion for better visibility into metrics, and mention your thesis (although we cover this specifically later) – Segway into your first topic]

Flaw of Averages, Risk & Monte Carlo Sim

A model is a tool used to mimic a

real world process

A tool for low-cost experimentation

Monte Carlo Simulation

Monte Carlo Simulation?

Performing a simulation of a model multiple times using

random input conditions and recording the frequency of

each result occurrence

Scrum

Backlog This Iteration Deployed

2 5

8

Run Sim Total Iterations

1 3

2 2

3 5

4 3

5 4

6 2

… …

Kanban

Backlog Design Develop Test Deployed

2

1 – 2 days 1 – 5 days 1 – 2 days

Run Time Total

1 5

2 4

3 3

4 9

5 5

6 6

… …

Result versus Frequency (50 runs)

More Often

Less Often Result Values – For example, Days

15 10 20

Fre

qu

en

cy o

f R

esu

lt

1

5

10

15

20

25

Result versus Frequency (250 runs)

More Often

Less Often Result Values – For example, Days

15 10 20

Fre

qu

en

cy o

f R

esu

lt

1

5

10

15

20

25

Result versus Frequency (1000+ runs)

More Often

Less Often Result Values – For example, Days

15 10 60

Fre

qu

en

cy o

f R

esu

lt

1

5

10

15

20

25

Key Point 1

There is NO single forecast result

There will always be many possible results, some more likely

Time to Complete Backlog

50% Possible Outcomes

50% Possible Outcomes

When pressed for a single number, we often give the average.

Like

liho

od

Key Point 2

“Average” is NEVER an option

But, I.T. gets worse

1 2 3

Planned Backlog

Perf/Memory Issues

Vendor Delay

Time to Delivery

Like

liho

od

Promised New Average

50% Possible Outcomes

Key Point 3

Risks play a BIG role in forecasts

Yes, more than backlog!

PAIN POINT Demo: Forecasting… My Boss “Needs” a Date…

In this demo

• Basic Scrum and Kanban Modeling

• How to build a simple model

– SimML Modeling Language

– Visual checking of models

– Forecasting Date and Cost

– The “Law of Large Numbers”

Metrics & Measurement

Why measure?

• Ask audience

• Show my slide covering what was said

• Explain that they all boil back to feedback/insight, except directly driving

Bad measurement can drive wrong behavior

• NBA sharp

Start with the outcomes

• Measurement -> Insight -> Decisions -> Outcomes

• Work backwards

Balance

• Do it fast(er)

o Productivity

o Time to market

• Do it when expected o Predictability

o Responsiveness

• Do it right o Quality

o Customer satisfaction

• Do it with a smile o Employee satisfaction

Analyze correctly

• Show non-normal cycle time distribution

• Explain risk evaluation error by assuming normal distribution

• Explain value of long tail

Collection costs

• Perceived cost is high

• No need for explicit collection activities

• Plenty to learn in passive data from ALM and other tools

Demo: Using History… Reverse Engineering Data

Probability Density Function

Lognormal (1; 1)

x

121086420

f(x)

0.24

0.22

0.2

0.18

0.16

0.14

0.12

0.1

0.08

0.06

0.04

0.02

0

Probability Density Function

Weibull (1.5; 1)

x

32.521.510.50

f(x)

0.72

0.64

0.56

0.48

0.4

0.32

0.24

0.16

0.08

0

Probability Density Function

Normal (1; 0)

x

3.22.41.60.80-0.8-1.6-2.4-3.2

f(x)

0.4

0.36

0.32

0.28

0.24

0.2

0.16

0.12

0.08

0.04

0

Probability Density Function

Triangular (3; 0; 10)

x

109876543210

f(x)

0.2

0.18

0.16

0.14

0.12

0.1

0.08

0.06

0.04

0.02

0

In this demo

• How to examine the properties of data

• How to use historical data in a model

–How to take samples and build a distribution

–Compare the impact of this distribution on the forecast

Demo: Finding What Matters Most Cost of Defects & Staff Analysis

Actively Manage

Ignore for the moment

Sensitivity Report

Staff Skill Impact Report

Explore what staff changes have the greatest impact

In this demo

• Finding what matters most

– Manual experiments

– Sensitivity Testing

• Finding the next best 3 staff skill hires

• Minimizing and simplifying estimation

– Grouping backlog

– Range Estimates

– Deleting un-important model elements

Demo: Finding the Cost / Benefit of Outsourcing

In this demo

• Model the impact of various outsourcing models

Assessing and Communicating Risk

Speaking Risk To Executives

• Buy them a copy of “Flaw of Averages” • Show them you are tracking & managing risk • Do

– “We are 95% certain of hitting date x” – “With 1 week of analysis, that may drop to date y” – “We identified risk x, y & z that we will track weekly”

• Don’t – Give them a date without likelihood

• “February 29th 2013”

– Give them a date without risk factors considered • “To do the backlog of features, February 29th, 2013”

We spend all our time estimating here

1 2 3

**Major risk events have the predominate role in deciding where deliver actually occurs **

Plan Performance Issues

External Vendor Delay

Risk likelihood changes constantly

1 2 3

95th Confidence

Interval

Risk likelihood changes constantly

1 2 3

95th Confidence

Interval

Risk likelihood changes constantly

1 2 3

95th Confidence

Interval

Risk likelihood changes constantly

1 2 3

95th Confidence

Interval

Key Points

• There is no single release date forecast

• Never use Average as a quoted forecast

• Risk factors play a major role (not just backlog)

• Take care with Non-Normal data

• Measurement -> Insight -> Decisions -> Outcomes : But Backwards!

• Define experiments and prove outcomes

• Communicate Risk early with executive peers

Call to action

• Read these books

• Download the software FocusedObjective.com

• Follow @AgileSimulation

• Learn: http://strategicdecisions.stanford.edu/

Please Submit an Eval Form!

We want to learn too!

BEST PRACTICES

Model (a little)

Visually Test

Monte-Carlo Test

Sensitivity Test

The Model Creation

Cycle

Baseline

Make Single

Change

Compare Results

Make Informed

Decision(s)

The Experiment

Cycle

Best Practice 1

Start simple and add ONE input condition at a time.

Visually / Monte-carlo test

each input to verify it works

Best Practice 2

Find the likelihood of major events and estimate delay E.g. vendor dependencies,

performance/memory issues, third party component

failures.

Best Practice 3

Only obtain and add detailed estimates and opinion to a

model if Sensitivity Analysis says that input is material

Best Practice 4

Use a uniform random input distribution UNTIL sensitivity

analysis says that input is influencing the output

Best Practice 5

Educate your managers’ about risk. They will still want a “single” date for planning, but let them decide 75th or

95th confidence level (average is NEVER an option)

SIMULATION EXAMPLES

Return to main presentation…

unlikely

certain

Forecasts Return to main presentation…

unlikely

certain

Forecasts

50% Possible

Outcomes

50% Possible Outcomes

Return to main presentation…

Actively Manage

Ignore for the moment

Sensitivity Report Return to main presentation…

Staff Skill Impact Report

Explore what staff changes have the greatest impact

Return to main presentation…

Return to main presentation…

Focused Objective

• Risk Tools for Software Dev

• Scrum/Agile Simulation

• Kanban/Lean Simulation

• Forecasting Staff, Date & Cost

• Automated Sensitivity Analysis

• Data Reverse Engineering

• Consulting / Training

• Book

We Use & Recommend: EasyFit

• Mathwave.com

• Invaluable for

– Analyzing data

– Fitting Distributions

– Generating Random Numbers

– Determining Percentiles

Free Images: MorgueFile.com

• http://www.morguefile.com/corporate/about

– Calendar: http://www.opticgroove.com.au

– Calculator: http://www.therising-sun.us

– Dice: souldestine@cox.net