Agile Metrics: Make Better Decisions with Data
Transcript of Agile Metrics: Make Better Decisions with Data
K4 Keynote 11/17/2016 4:15:00 PM
Agile Metrics: Make Better Decisions with Data
Presented by:
Larry Maccherone
AgileCraft
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-‐268-‐8770 ·∙ 904-‐278-‐0524 - [email protected] - http://www.stareast.techwell.com/
Larry Maccherone AgileCraft
An industry-recognized leader in agile, metrics, and visualization, Larry Maccherone currently helps a number of companies with the design of their analytics products including AgileCraft and Pendo.io. Previously, Larry led the insights product line at Rally Software which enabled better decisions with data, leveraged big data techniques to conduct groundbreaking research, and offered the first-ever agile performance benchmarking capability. Before Rally, Larry worked at the Software Engineering Institute for seven years conducting research on software engineering metrics with a particular focus on reintroducing quantitative insight back into the agile communities.
11/3/2016
1
Make BetterMake Better Decisions with Data
Larry MaccheroneLinkedIn.com/in/LarryMaccherone
LinkedIn.com/in/LarryMaccherone
LinkedIn.com/in/LarryMaccherone
11/3/2016
2
What?What?So what?
NOW WHAT?
LinkedIn.com/in/LarryMaccherone
What?
LinkedIn.com/in/LarryMaccherone
11/3/2016
4
Visualization is like photography. Impact is a function of focus, illumination, and perspective.
What?
LinkedIn.com/in/LarryMaccherone
Credit: Edward Tufte
NOW WHAT?
LinkedIn.com/in/LarryMaccherone
11/3/2016
5
D ’tDon’tLaunch!
LinkedIn.com/in/LarryMaccherone
Prevent your own disastrous decisions
LinkedIn.com/in/LarryMaccherone
11/3/2016
6
Larry MaccheroneLinkedIn.com/in/LarryMaccherone
LinkedIn.com/in/LarryMaccherone
Cognitive biasworks against good decisions
LinkedIn.com/in/LarryMaccherone
11/3/2016
7
We don't see things the way they are.
We see things the way we are.
LinkedIn.com/in/LarryMaccherone LinkedIn.com/in/LarryMaccherone
~The Talmud
Next slide is a movieclick to play
LinkedIn.com/in/LarryMaccherone
11/3/2016
8
Denying the Evidence
LinkedIn.com/in/LarryMaccherone
Truths about cognitive bias
1. Very few people are immune to it.
2 We all think that we are part of that2. We all think that we are part of that small group.
3. You can be trained to get much, much better. Douglass Hubbard – How to Measure Anything
4. We do a first-fit pattern match. Not a best-fit pattern match And we only use about 5% of the information
LinkedIn.com/in/LarryMaccherone
match. And we only use about 5% of the information to do the matching.
5. We evolved to be this way (survival trait).
11/3/2016
10
An example of overcoming cognitive bias
LinkedIn.com/in/LarryMaccherone
We are overconfident when assessing our own uncertainty
But, training can “calibrate” people so that of all the times they say they are X% confident, they will be right X% of the time
Trained/Calibrated
Untrained/Uncalibrated
Statistical Error
“Ideal” Confidence
50%
60%
70%
80%
90%
100%
75 71 65 58
21
17
68 15265
4521
Per
cent
Cor
rect
# f R
LinkedIn.com/in/LarryMaccherone
30%
40%
50% 60% 80% 90% 100%
25
70%
Assessed Chance Of Being Correct
P
99 # of Responses
Copyright HDR 2007 [email protected]
11/3/2016
11
Equivalent Bet calibration
What year did Newton published the Universal Laws of Gravitation?
Pick year range that you are 90% certain it would fall within.
Win $1,000:
1. It is within your range; or
2. You spin this wheel and it lands green
10%
LinkedIn.com/in/LarryMaccherone
Adjust your range until 1 and 2 seem equal.
Even pretending to bet money works.
90%
Agile Teams Programs andAgile Teams, Programs, and Portfolios
benefit from similarcalibration exercises
LinkedIn.com/in/LarryMaccherone
11/3/2016
12
Every decision is aforecast!
LinkedIn.com/in/LarryMaccherone
You are forecasting that gyour choice will have better
outcomes than the other alternatives
LinkedIn.com/in/LarryMaccherone
alternatives
11/3/2016
13
How to avoid cognitive bias in decision making
Don't focus on consensus. Ritual dissent is a much more successful approach.
“But that doesn’t explain _______”. An FBI agent knew that some folks were being trained to fly but not take off and land.
Assign someone the role of devil’s advocate
LinkedIn.com/in/LarryMaccherone
Assign someone the role of devil s advocate.Israel’s 10th man.
In other words… Really consider the other ALTERNATIVES
Types of bias
http://srconstantin.wordpress.com/2014/06/09/do-rationalists-exist/
LinkedIn.com/in/LarryMaccherone
11/3/2016
14
LinkedIn.com/in/LarryMaccherone
For a given alternative, let:Pg = Probability of good thing happeningV = “Value” of good thing happening
LinkedIn.com/in/LarryMaccherone
Vg = “Value” of good thing happening
Then:Value of the alternative = Pg × Vg
11/3/2016
15
AA lean/agile product
management example
LinkedIn.com/in/LarryMaccherone
$8M
Best case (25%)
Likely case (50%)
Worst case (25%)
1PW × VW = .25 × -$1.00M = -$0.25M
PL × VL = .50 × $1.00M = $0.50MP × V 25 × $8 00M $2 00M
$1M$1M
1
$2M$2M$1M2
PB × VB = .25 × $8.00M = $2.00M -----------$2.25M
PW × VW = .25 × $1.00M = $0.25MPL × VL = .50 × $2.00M = $1.00MPB × VB = .25 × $2.00M = $0.50M
-----------
LinkedIn.com/in/LarryMaccherone
Which strategy is best……for your company?…for your career?
$1.75M
11/3/2016
16
If you get only 1 project then strategy 2 is better
75% of the time
If you get ∞ projects thenstrategy 1 is better100% of the time
LinkedIn.com/in/LarryMaccherone
How many projects do you need for strategy 1 to be better more often than not?
LinkedIn.com/in/LarryMaccherone
11/3/2016
17
Play with it yourself at:http://jsfiddle.net/lmaccherone/j3wh61r7/
LinkedIn.com/in/LarryMaccherone
LinkedIn.com/in/LarryMaccherone
11/3/2016
18
Emotion and bias plays a part
LinkedIn.com/in/LarryMaccherone
Did any of you get emotional about the $1M loss?
Did any of you want to question the $8M number?
It’s critical to
LinkedIn.com/in/LarryMaccherone
It s critical to… …eliminate fear from the equation
…change the nature of the conversation
11/3/2016
19
Argument is about who is right.Decision making is about what is right.
LinkedIn.com/in/LarryMaccherone
AAnagile delivery date forecast
example
LinkedIn.com/in/LarryMaccherone
11/3/2016
20
Monte Carlo ForecastingWhat it looks likeLive demo: http://lumenize.com (use Chrome)
LinkedIn.com/in/LarryMaccherone
Seek toSeek tochange the nature of
the conversation
LinkedIn.com/in/LarryMaccherone
the conversation
11/3/2016
21
Criteria forgreat visualizationg
Credits:Edward Tufte (mostly)
LinkedIn.com/in/LarryMaccherone
Edward Tufte (mostly) Stephen Few
Gestalt School of Psychology
1. Answers the question, "Compared with what?” (So what?)
LinkedIn.com/in/LarryMaccherone
Trends
Benchmarks
11/3/2016
22
2. Shows causality, or is at least informed by it
The primary chart used by the NASA scientists showed O-ring failure indicators by launch datefailure indicators by launch date.
Tufte's alternative shows the same data by the critical factor, temperature.
The fateful shuttle launch occurred at 31 degree Tufte's visualization
LinkedIn.com/in/LarryMaccherone
at 31 degree. Tufte s visualization makes it obvious that there is great risk for any launch at temperatures below 66 degrees.
3. Tells a story with whatever it takes
Still
Moving
Numbers
Graphics
And …
LinkedIn.com/in/LarryMaccherone
Maybe some fun
11/3/2016
23
4. Is credible
Calculations explained
Sources
Assumptions
Who (name drop?)
Drill-down
LinkedIn.com/in/LarryMaccherone
How?
Etc.
5. Has business value(or value in it’s social context)
like Vic Basili’sGoal-Question-Metric (GQM)
but withoutISO/IEC 15939 baggage
The ODIM framework
D
I N S I G H T
M E A S U R E
THINK
EFFECT
LinkedIn.com/in/LarryMaccherone
O U T C O M E
D E C I S I O NTHINK
11/3/2016
24
6. Shows comparisons easily (1)
aka: Save the “pie” for dessert Credit:
• Stephen Few (Perceptual Edge)• http://www.perceptualedge.com/ar
ticles/08-21-07.pdf
LinkedIn.com/in/LarryMaccherone
6. Shows comparisons easily (2)
Can you compare the market share from one year to the next?
Q i kl Whi h t i i h th f t t?Quickly: Which two companies are growing share the fastest?
LinkedIn.com/in/LarryMaccherone
One pie chart is bad. Multiple pie charts are worse!!!
11/3/2016
25
6. Shows comparisons easily (3)
How about now?
Can you compare the market share from one year to the next?
LinkedIn.com/in/LarryMaccherone
7. Allows you to see the forestANDAND the trees
LinkedIn.com/in/LarryMaccherone
11/3/2016
26
8. Informs along multiple dimensions (1)
LinkedIn.com/in/LarryMaccherone
9. Leaves in the numbers wherewhere possible
LinkedIn.com/in/LarryMaccherone
11/3/2016
27
10. Leaves out glitter
Examples of how NOT to do it.
LinkedIn.com/in/LarryMaccherone
Top 10 criteria for great visualization
1. Answers the question, "Compared with what?”
6. Shows differences
Credits:• Edward Tufte• Stephen Few• Gestalt
(School of Psychology)
(SO What?)
2. Shows causality, or is at least informed by it. (NOW WHAT?)
3. Tells a story with whatever it takes.
easily.
7. Allows you to see the forest AND the trees.
8. Informs along multiple dimensions.
9 Leaves in the numbers where
LinkedIn.com/in/LarryMaccherone
takes.
4. Is credible.
5. Has business value or impact in its social context.
9. Leaves in the numbers where possible.
10. Leaves out glitter.
11. Uses good visual grammar
11/3/2016
28
“They” say…
Nobody knows what’s gonna happen next: not on a freeway, not in an
airplane, not inside our own bodies d t i l t t k ithand certainly not on a racetrack with 40 other infantile egomaniacs.
– Days of Thunder
Trying to predict the future is like trying to drive down a country road at night with no lights while looking
t th b k i d
LinkedIn.com/in/LarryMaccherone
out the back window. – Peter Drucker
Never make predictions, especially about the future.– Casey Stengel
When you come to a fork in the road…
take it!
~Yogi Berra
What?the metrics and analysis
S h t?
LinkedIn.com/in/LarryMaccherone
So what?how it compares/trendswhat it means
NOW WHAT?every decision is a forecast