'Metrics That Matter': Gabrielle Benefield @ Colombo Agile Con 2014

Metrics that matter

Waterfall

Agile

Mobius

Ac#vity based tracking

Output based tracking

Outcome based tracking

What do we measure?

Planning

Development

Design

Testing

Maintain

Launch

OPTION A Improve network

Increase # checkout conversions

Baseline:15% Target:30%

Current:18%

5

Op#on Value Risk Cost Impact

Op#on A

? M M ?

Op#on B

? H H ?

Op#on C

? L L ?

People start the buying process but don’t complete it

HYPOTHESIS Improving the page load speed will help conversions

Increase checkout conversions Improve page

load speed

DATA 85% of users abandon checkout

How to create outcomes case study

OPTION B Build a local datacenter

DATA 86% of users abandon checkout when page loads times = > 4 seconds

CREATE OPTIONS Generate ideas on how to fix the page load speed

INSIGHT The further users are from the Datacenter, the slower the page loads OPTION C

Compress page elements

IMPLEMENT Image compression

EXPERIMENT Test one server with 0.002% split test

RESEARCH Run network diagnostics with a trace route

NEXT IMPROVEMENTS What else can we do to improve the conversion rate?

IMPLEMENT: Add instrumentation and exit survey to pages OPTION A

Fix the network OPTION Find out why people are leaving the service

Tom Gilb’s ‘Ilities’

( Reliability ( Availability ------ ( Maintainability ( ( Integrity ------------ ( Threat ( ( Security ( ( ( ( Connectability ( ( Flexibility------------ ( Tailorability ---( Extendibility

Quality ------------ ( Adaptability ----- ( ( Interchangeability ( ( ( ( Upgradeability ----- ( Installability ( ( Portability ( ( Improvability ( ( (

( Entry level experience

( ( Training attribute ( Usability --------- ( Handling ability ( Likeability ( Demonstratability

( Financial saving Resource saving ---( Time saving

( Effort saving ( Equipment saving

( Throughput Workload capacity ( Response time

( Storage capacity

Quality outcomes

How contracts measure success

PLAN

OUTPUT

ACTIVITIES

OVERARCHING GOAL

CONTR

ACT

CONTR

ACT

Measurement of success

Why there is friction

ACTUAL OUTCOMES

ACTIONS

PLANS

OVERARCHING GOAL

KNOWLEDGE GAP The difference between

what we would like to know and what we actually know

EFFECTS GAP The difference between what we expect our ac#ons to achieve and

what they actually achieve

ALIGNMENT GAP The difference between

what we want people to do and what they actually do

Source: ‘The Art of Ac#on’ by Stephen Bungay

Medium to large projects experience 25-‐35% requirements change. Capers Jones on So,ware Development Our customers say that if it weren’t for the contract making it so expensive and #me-‐consuming, they would have had more changes, and could have built beSer products.

Locking in scope, locks out opportunities

FRAMEWORK AGREEMENT Warran#es IP Constraints ($, Time, Regulatory)

Statement Of Target Outcome

(SOTO)

$

The contracts have to be Agile

Snowden’s Cynefin model

Cause and effect rela#ons repeatable, perceivable and predictable

Cause and effect are only coherent in retrospect and do not repeat

Cause and effect separated over #me and space

CHAOS SIMPLE (Known)

COMPLEX (Unknowable)

COMPLICATED (Knowable)

No cause and effect rela#onships perceivable

Benefits of Agile

Questions?

Nearly 50 countries use Systematic’s solutions Systematic’s offices

Manage complex projects to succes using CMMI, Lean and Scrum hSp://www.systema#c.com

Public Sector Intelligence & National Security

Healthcare Defence

Standard Project Status Report Managing normal projects is a prerequisite to manage large projects CMMI and Lean provides a solid foundation for both

Status Analysis of measures Risk top 3

Ac#ons

Risk Measure

12 std KPI’s

Example: Flow of story implementation and fix-time after failed build

Productivity is optimized: •  Flow of work - maximized •  Fix-time of failed builds - minimal •  Sprint test and release - minimal

à Variance reduced

Measures inspired from Lean

Fix time for failed builds

Flow of story implementation

Questions?

You get what you measure

“Perhaps what you measure is what you get. More likely, what you measure is all you get. What you don’t (or can’t) measure is lost.”

H Thomas Johnson (2006)

Friction

“Fric7on manifests itself when human beings with independent wills try to achieve a collec7ve purpose in a fast-‐changing, complex environment where the future is fundamentally unpredictable.”

‘The Art of Ac#on’ by Stephen Bungay, 2011

•  Fric#on:

Ø  An accumula#on of innumerable peSy circumstances

e.g. uncertain#es, errors, accidents, technical difficul#es, the unforeseen

Ø  … which impact on decisions, morale and ac#ons

Ø  … thereby preven#ng you from achieving your goal

Causes of friction

Human finitude

•  Services are performed by people:

Ø  People are imperfect

Ø  People are intrinsically unpredictable

Ø  People have different interpreta#ons

•  The environment is complex and chao#c:

Ø  The environment is non-‐linear and fundamentally unpredictable

Ø  External factors contribute towards a change in the environment

Ø  Even if the environment is stable, some of the effects of our ac#ons will be unintended.

Pay with a one

step pay button

Step

Search for

book

Step

Decrease time to pay

TO

Decrease time to find

a book

TO

Find one book and get it shipped

Goal

Setup one step pay option

Goal

Call and pay

Step

Enter payment

details

Step Save payment

details

Step

Our team tries to get feedback and hit the outcomes with the least amount of effort. Goal mapping helps us set a goal and find the simplest path to reach it.

22

Outcomes Op#ons In Progress Done

23

Decrease page load #me

Setup local datacenter

Option

Cache page

elements

Option

Decrease #me to find products

Redesign

search results

Option

Clickstream analysis

Research

Datacenter split test Australia

Research

Test caching one page

Research

Split test redesign

Research

Trace route analysis

Research

Teams track outcomes and op#ons visually during delivery.

How much progress did we make?


Baseline: 7 minutes

Target: < 3 minutes

Current: 5 minutes


Baseline: 12 seconds

Target: < 4 seconds

Current #me: 9 seconds

We saved 2 minutes

We reduced page load speed by 3 seconds

24

We are constantly adap#ng our model based on feedback. When we see the value diminish, we can choose other op#ons or outcomes to explore

25

x Point of diminishing returns

Time

Outcomes (value)

26

Value Type

Value Driver

Scale -‐ What to Measure

Scale Defini#ons

Meter -‐ How to

Measure Performance Reliability Mean #me between failure

over #me period Time Period: current month, trailing 12 months

Monitoring system

Performance Responsiveness Elapsed #me between user reques#ng and receiving informa#on for percen#le of transac#ons

Time: seconds ; Percen#le: 90th ; Transac#ons: <Define specific transac#ons>

Performance tes#ng system

Profitability Increase Revenue Total revenue over #me period Time Period: current month, trailing 12 months

Financial system

Delivery Improve Customer Sa#sfac#on

Average scores pertaining to survey ques#ons for current customers…

Survey Ques#ons: <define specific ques#ons> or Net Promoter score

Survey system

Quality Reduce Incident Rate Rate of new incidents over #me period

Time Period: current month, trailing 12 months

Report from issue tracking system

Example

A comparison of the activities

PRODUCT BACKLOG

Waterfall Project

Agile Project

REQUIREMENTS

Discrete modules of somware

Integra#on late in the process

Each sprint/itera#on builds on the earlier sprints/itera#ons

Outcomes are the desired results that deliver value. E.g. Increase checkout conversion rates by 19%

What do you want to improve?

© Gabrielle Benefield 28

Create target outcomes

Set a baseline e.g. Currently only 11% of people

complete checkout

Set a target e.g. 30% of people complete

checkout

Baseline: 11% Target: 30%

Give it a name in verb + noun phrase form

Increase checkout conversion rates

29

Set target outcomes for the ‘ili#es’

30

Scalability Must scale to 3 million users.

Availability System must have a minimum of 99.9 uptime

Usability Must be easy to use and fast to learn.

Interoperability Must integrate with all customer systems.

31

Pay with a one step pay button

Step

Cache page

elements

Step

Search for

book

Step

Decrease time to

pay

TO

Decrease time to

load pages

TO

Decrease time to

find a book

TO

Find one book and get it shipped

Goal

Setup one step pay option

Goal

Call and pay

Step

Read details

Step

Create a Goal map Start with the Goal and work back to find the least amount of steps needed to reach

the goal

Enter payment

details

Step Save payment

details

Step

Decrease the page load times

Goal Setup 1 server and split test

Step

Which op#ons should we do first? We can guess, or we can test with an experiment

?

32



Option

Test caching elements on the payment page

Experiment

Cache page elements

Option

Setup one server test

with 1% of traffic

Experiment

Create experiments

?

33

Take a target outcome

Design experiments to test ideas or de-risk

your work

Experiment

Outcomes Options In Progress Done

Visually track outcomes and op#ons during delivery

© The007model.com 34



Option

Cache page

elements

Option


Redesign search results

Option Clickstream

analysis

Research

Datacenter split test Australia

Research

Test caching one

page

Research

Split test redesign

Research

Trace route analysis

Research

How much progress did we make? Decrease #me to find products

Baseline: 7 minutes

Target:

< 3 minutes

Current: 5 minutes

Reduce customer complaints

Baseline: 250 calls per day

Target:

< 150 calls per day

Current: 190 calls per day

We saved 2 minutes

We reduced calls by 60 per day

© The007model.com 35

Redesign

checkout process

Option

Snowden’s cynefin model

Cause and effect rela#ons repeatable, perceivable and predictable

Cause and effect are only coherent in retrospect and do not repeat

Cause and effect separated over #me and space

CHAOS SIMPLE (Known)

COMPLEX (Unknowable)

COMPLICATED (Knowable)

No cause and effect rela#onships perceivable

Outcomes are the desired results that deliver value. E.g. Increase checkout conversion rates by 19%

What do you want to improve?

© Gabrielle Benefield 37

We are constantly adap#ng our model based on feedback. When we see the value diminish, we can choose other op#ons or outcomes to explore

38

x Point of diminishing returns

Time

Outcomes (value)

READY and competence match When work allocated to sprint is READY, flow and stability is achieved

0,00

20,00

40,00

60,00

80,00

100,00

120,00

140,00

160,00

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35

Actual effort

Ready NOT Ready

0,00%

10,00%

20,00%

30,00%

40,00%

50,00%

60,00%

70,00%

80,00%

90,00%

100,00%

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35

Flow

Ready NOT Ready

Flow Actual effort spent

Projects measures how ”READY” a sprint plan is

Measures of key competencies needed in project

Meeting READY criteria creates stability

Cost of change

Time

Low

High

Plan

Develop

Design

Test

Launch

The point where we have the least informa#on, the contract is signed locking in that ignorance for the whole project.

When the customer sees what has been built, when tes#ng begins. When the cost of change and fixing defects is highest.

'Metrics That Matter': Gabrielle Benefield @ Colombo Agile Con 2014

Software

Transcript of 'Metrics That Matter': Gabrielle Benefield @ Colombo Agile Con 2014