Benchmark METRICS THAT MATTER October 4 2012

METRICS THAT MATTER Challenge Your Current Thinking!

(a.k.a. Selecting Valuable Metrics Instead of Vanity Metrics)

10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved. 1

Presented by: Betty Schaar & Jeff Roth

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

METRICS THAT MATTER

10/5/2012 ©2012 BenchmarkQA, Inc. All rights reserved.

2

“The only metrics that entrepreneurs should invest energy in

collecting are those that help them make decisions. Unfortunately,

the majority of data available in off-the-shelf analytics packages

are what I call ‘Vanity Metrics.’ They might make you feel good,

but they don’t offer clear guidance for what to do.”

Eric Ries

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

MEASUREMENT VS. METRIC

Measurement - The value of a dimension, quantity, or capacity obtained by collecting project data.


3

A metric or group of metrics can be used to make decisions and manage a project.

A single measurement does not provide support for decision-making.

Metric - A comparison, ratio, or plot of a series of measurements or an algorithm of two or more measurements.

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

ISSUES WITH METRICS

vs. Valuable

> Relative vs. Absolute

> Not enough context

> Systems don’t support data collection

> Not enough historical data to be valuable

> Fear causes “skewage” of data

> Outliers cause “skewage” of data

> The numbers can lie! (or can at least be manipulated)


4

DETERMINING METRICS CONTEXT


5

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

CONTEXT TYPES


6

SDLC/Methodology

Environment/Technologies

Team/Organization

In Relation to Other Information

Type of Project

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

CONTEXT – SDLC/METHODOLOGY


7

> Waterfall

> Agile

> Multiple phases/releases

> Pilot or prototype initially

> External constraints/factors impacting delivery

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

AGILE SPECTRUM


8

© 2012 Impressum

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

CONTEXT - TYPE OF PROJECT

> In-house

New development

Maintenance

> COTS

> Outsourced

> SaaS

> New delivery platform, e.g. mobile


9

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

CONTEXT - TEAM/ORGANIZATION

> Co-located team vs. distributed team

> Internal vs. external application users


10

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

CONTEXT – ENVIRONMENT/TECHNOLOGIES

> Regulated vs. non-regulated

> Mature vs. leading-edge technologies


11

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

CONTEXT – IN RELATION TO OTHER INFO > We have no open Severity 1 defects.

Can we release? ∙ What if I told you we also have 250 Severity 2

defects?

> We ran 2,000 test cases. Can we release?

∙ What if I told you we ran out of time to test the most recently added feature?

> We covered all of the critical requirements. Can we release?

∙ What if I told you we only had time to run positive tests?

∙ What if I told you we still have 2 open Critical Severity defects?


12

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

CONTEXT – DON’T OVERTHINK IT


13

> SDLC/methodology

> Project type

> Team/organization

> Environment/ technologies

> Metrics in relation to other information

UNDERSTANDING YOUR AUDIENCE


Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

ME > QA Lead

Test case building progress by tester

Test execution progress by tester

Defect close rate

Test coverage

> Test Analysts Tests to build

Feature coverage

Test to execute

Defects written by me

Defects assigned to me


15

ME

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

YOU


16

Project Manager

Development Lead

Project Sponsor QA Manager

Business Analyst

Developer Product Owner

Customer ME

YOU

YOU YOU

YOU

YOU

YOU YOU

YOU

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

US

> Project team

> Scrum team

> QA team


17

ME

YOU

YOU

YOU

YOU

YOU

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

THEM

> Enterprise

> Execute Suite / C-level

> Divisional Managers

> Customers


18

ME

THEM

THEM THEM

USE METRICS TO ANSWER A QUESTION


Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

ARE WE DONE YET?

> Building Tests

# of Total Tests vs. # Written/Approved

Test Coverage Completeness

> Executing Tests

# of Test Passed/Failed/Blocked/Not Run

Automation vs. Manual

Velocity/Rate of Execution


20

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts


21

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

UAT PROCESS EXECUTION DASHBOARD


22

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

QUALITY CENTER TIP – VELOCITY OF TEST


23

Setup Result

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

ARE WE DONE YET? > Sufficient Quality

# of Active Defects by Severity

Quality Bar


24

Feature Critical High Medium Low Total Ready for

Retest

General 0 0 0 0 0 0

Feature 1 0 0 0 0 0 0

Feature 2 0 1 1 0 2 2

Feature 3 0 0 0 0 0 1

Feature 4 0 0 3 1 4 3

Feature 5 0 1 2 0 3 5

Feature 6 0 0 0 0 0 0

Feature 7 3 1 1 0 5 0

Feature 8 0 0 0 0 0 0

Feature 9 2 3 5 0 10 0

Feature 10 0 1 0 0 1 0

Feature 11 2 3 5 2 12 1

Feature 12 - General 0 1 0 0 1 0

Feature 12: Sub 1 0 1 0 0 1 0

Feature 12: Sub 2 0 0 0 0 0 0

Feature 12: Sub 3 0 0 1 0 1 0

Feature 12: Sub 4 0 0 0 0 0 0

Feature 12: Sub 5 0 0 0 0 0 0

Feature 12: Sub 6 0 0 0 0 0 0

Feature 13 0 5 0 0 5 1

Totals 7 17 18 3 45 13

Closed Since 8/18/2011

Closed Deferred Duplicate Ready for

Retest Reported Since

8/18/2011

18 245 9 0 13 5

# Tests Passed

Total # Tests Executed

QUALITY BAR

65%

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

WAS OUR APP/RELEASE GOOD ENOUGH?

> Warranty Period

Quantity & Severity of Post Release Defects

Time for Post Production Fixes

> Defect Removal Rate

> Cost of Production Defects


25

# of pre-release defects

(# of warranty period defects

+ # of pre-release defects)

# hours for prod defects fixes

X average burden rate $$

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

IS QUALITY IMPROVING?

> Trending Analysis

# of pre- and post-release defects found and fixed per release (root cause distribution analysis by release)

> Retrospectives

Burnup/Burndown over multiple sprints

User Stories/Tasks


26

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

HOW EFFECTIVE ARE WE?

> Sprint delivery effectiveness

Burn down/burn up Velocity by Sprint

Retrospective

> Test coverage

> # Tests per resource per hour


27

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

CAN I QUANTIFY THE IMPACT OF AN ISSUE?

> Frequency of Issue Quantity of occurrences

Timeframe of each occurrence

> Severity of Issue Empirical scales (show stopper, critical, high,

medium, low)

Subjective scales

> Impact of Issue Importance to business (priority/severity)

Cost of not fixing


28

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

AUTOMATION ROI

> Simple ROI: monetary savings due to test automation PROs: Good overview for management

CONs: Oversimplified & need resource costs

> Efficiency Automation: time savings resulting from test automation

PROs: Easy to gather data, shows team impacts

CONs: Oversimplified, assumes 100% test execution each cycle


29

ROI= Gains – Investment Costs

Investment Costs

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

SIMPLE ROI


30

0.0%

50.0%

100.0%

150.0%

200.0%

250.0%

300.0%

1 2 3 4

% ROI

Years

Simple ROI

18 Cycles

24 Cycles

30 Cycles

Task Manual Automation Manual Auto

Qty Rate Factor Total $$ QTY Rate Factor Total Initial Cost

Hardware 1 $ 1,000 - $ 1,000 2 $ 1,000 $ 2,000

Software (Initial Costs) 0 $0 - $ - 2 $ 4,000 $ 8,000 $ 1,000 $ 52,500

Software (Maintenance Costs) N/A $0 - $ - 1 $ 8,000 0.2 $ 1,600

Build 500 Test Scripts N/A $0 - $ - 500 $ 85 1 $ 42,500

Execute & Analyze 500 Test Scripts 500 $60 0.17 $ 5,100 1 $ 85 4 $ 340

Maintain 500 Test Scripts N/A $0 - $ - 1 $ 85 8 $ 680

Execute Manual Test Suite (less Automated)

1000 $0 0.17 Time to Execute Automation

0.03

Total Cost of 24 Cycles of Manual Testing $ 123,400

Total Cost of 24 Cycles of Test Automation $ 78,580

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

EFFICIENCY ROI


31

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

1 2 3 4

% ROI

Years

Efficiency ROI

18 Cycles

24 Cycles

30 Cycles

Task Manual Automation Hours per Day

Qty Rate Factor Total Days QTY Rate Factor Total Days Manual Auto

Hardware 1 $ 1,000 - 2 $ 1,000 8 18

Software (Initial Costs) 0 $0 - 2 $ 4,000

Software (Maintenance Costs) N/A $0 - 1 $ 8,000 0.2

Build 500 Test Scripts N/A $0 - 500 $ 85 1 62.5

Execute & Analyze 500 Test Scripts 500 $60 0.17

10.6 1 $ 85 4

0.5

Maintain 500 Test Scripts N/A $0 - 1 $ 85 8 1.0

Execute Manual Test Suite (less Automated)

1000 $0 0.17

21.3 Time to Execute Automation

0.03 0.8

Total Time in Days of 24 Cycles of Manual Testing

765.0 Total Time in Days of 24 Cycles of Test Automation

628.5

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

ACTUAL VS. PROJECTED ROI


32

-150.0%

-100.0%

-50.0%

0.0%

50.0%

100.0%

150.0%

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Year 1 Year 1

% ROI

Month

Actual vs. Projected ROI

Projected ROI

Actual ROI

• Visibility for ongoing value of automation • Better understand/plan ROI for other

automation efforts • Combat long term automation malaise

Th

e S

of

tw

ar

e Q

ua

lit

y E

xp

er

ts

REFERENCES

> Vanity Metrics vs. Actionable Metrics, http://www.fourhourworkweek.com/blog/2009/05/19/vanity-metrics-vs-actionable-metrics/

May 19, 2009 by Eric Reis

> Dion Johnson “Test Automation ROI” http://www.dijohn-ic.com/test_automation_roi.pdf

February 7, 2006 Whitepaper created for SQE Webinar


33

http://www.fourhourworkweek.com/blog/2009/05/19/vanity-metrics-vs-actionable-metrics/










http://www.dijohn-ic.com/test_automation_roi.pdf




THANK YOU FOR ATTENDING! For more information about BenchmarkQA and the services we offer, please contact:

Molly Decklever

9523.392.2384

[email protected]


Benchmark METRICS THAT MATTER October 4 2012

Technology

Transcript of Benchmark METRICS THAT MATTER October 4 2012