T12 Metrics

10/16/2014 11:15:00 AM

Metrics That Matter

Presented by:

Pablo Garcia

Redmind

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Pablo Garcia

Redmind With nearly twenty years of experience in the IT industry, Pablo Garcia has spent time as a project manager, product manager, head of outsourcing, and agile coach. Pablo has first-hand experience as a tester, test manager, head of QA, and test consultant across a broad range of industries including telecoms, financial, gaming, retail, and broadcasting. In 2011, he formed Redmind, a company specializing in consultancy and training in all aspects of testing, management, and agile. Instrumental in establishing the first Swedish Testing Academy, Pablo is a board member of the Stockholm SIGIST.

Metrics That Matter 2014-10-16

Pablo Garcia Munos

In the testing industry since 1996 Has worked in different areas, Banking, Gaming, Medicin, Insurance, Telecom, Automotive... Founder of Redmind, 30 consultants in Stockholm, Gothenburg and Malmö. Redmind focus: adding value in the long run.

Creation of organizations, optimizing test and QA, Offshoring, requirements, procurement, education, mentoring and Agile coaching.

”Testing in Agile projects” since 2007 ISTQB foundation adv. TM, ReqB foundation, Agile coach, Scrum coach, TPI, TMAP, Exploratory testing, context driven testing, Risk based testing, Structured testing, IEEE829, IEEE830, Certified Scrum Master.

Metrics that matter !!

!  Definition!!  Curve Types!!  5 basic metrics!!  S-curves!!  Other Metrics!!  Conclusion!

Metrics !!!

metrics: A measurement scale and the method used for measurement. [ISO 14598] !

Metrics 1!!

A metric is a standard unit of measure, such as mile or second, or more generally, part of a system of parameters, or systems of measurement, or a set of ways of quantitatively and periodically measuring, assessing, controlling or selecting a person, process, event, or institution, along with the procedures to carry out measurements and the procedures for the interpretation of the assessment in the light of previous or comparable assessments.!

Metrics 2!!

Metrics are usually specialized by the subject area, in which case they are valid only within a certain domain and cannot be directly benchmarked or interpreted outside it. This factor severely limits the applicability of metrics, for instance in comparing performance across domains. The prestige attached to them may be said to relate to a 'quantifiability fallacy', the erroneous belief that if a conclusion is reached by quantitative measurement, it must be vindicated, irrespective of what parameters or purpose the investigation is supposed to have.

Why Metrics!!

!  Ask questions.!!  To measure is to know.!!  Real live status reporting!!  Measure our test capability!!  Improve quality!!  Effectiveness and efficiency of the organisation!!  ……..!

Grouping!!

!  Quality!!  Productivity!!  Cost!

!  But this is a restriction!

10/1/14

Variable

Time

Types of curves 1

o

!!  Ackumulated values!

10/1/14

o

!!  Measured values!

Types of curves 2

Variable

Time

11 10/1/14

today Release

Req in %

Tme

1.Requirement Coverage (ack)

o

!!  All requirements that are dev AND testedOK are included.!!  Provides status reports of how much of the product that is completed.!!  Accuracy according to test methodology and testers capability.!!  Requirement quality, traceability and negative test cases important. !!  Gives credibility to test manager.!

12 10/1/14

2.Open defect reports

!!  All detected defect reports that was

!found but never closed is counted!!  Gives an indication of the products

!maturity level. !!  About 8 – 10 of the top percentage gives !an indication.!!  Divide the defect reports into priorities!

Number of open defect reports

Time

13 10/1/14

Today Release

Defect reports

Time

3.New defect reports (ack)

o

!!  All defect reports are counted!!  Gives an indication to help you figure

!out how close you are to finishing. !!  Look for the ”knee”.!

14 10/1/14

Today Release

Defect reports

Time

3.New defect reports (ack)

o

!!  All defect reports are counted!!  Gives an indication to help you figure

!out how close you are to finishing. !!  Look for the ”knee”.!

15 10/1/14

4.New defect reports

!!  All defect reports are counted.!!  Gives the extra maintaince cost if the

!product is delivered now!!  All defects found after release by the stakeholders, how much more does it cost?!!  Divide defect reports into priorities.!

Number of defect reports

Time

16 10/1/14

Today Release

Test cases in %

Time

5.Test coverage (ack)

o

!!  All testcases that has been executed are

!counted!!  Gives a status report over the test teams

!expected result!!  Could be used when requirements and

!traceability are missing.!!  Doesn’t give as much information about

!the products quality or status.!

Today Release

Expected.number of defects

Defect reports

Closed defect reports (OK)

New Release date Time

“S-curve” 1

!!  All defect reports are counted!!  Gives: open DR and New ack DR.!!  Gives an indication of whether or not

!the product will be ready in time for !release!

!  Can be predicted with a mathematical !algoritm!

Today Release

Planned tests

Executed tests Passed

(OK)

New Release date Time

Number of executed test cases

“S-curve” 2

!!  All testcases that have been executed are

!counted.!!  Gives test coverage and an indication of

!how many test cases that did not pass.!!  Could be used when requirements and

!tracebility are missing. !!  Gives an indication of whether or not

!the product will be ready in time for !release.!

!  Can be predicted with a mathematical !algoritm!

DDP Long-term control

Measurement of defect detection level DDP stands for”Defect Detection Percentage”

Measurement of defect detection level options: Include severity level in calculation.

Measurement depths (Number of test levels to measure)

Tip: Start simple

Defects found in this test level Total number of defects found including found defects after delivery

DDP =

DDP

Functionality Test

6 months

56 defects

140 defects

actual 40%

70 defects

System Test

21 10/1/14

Other metrics - 1 !!  Important to think about is how you present the metrics.!

!

22 10/1/14

Other metrics - 2 !!  Even a lot of information can be presented in a simple way.!

!

23 10/1/14

Other metrics- 3 !!  Colors gives a good overview!

!

Example of defect report status development

0

100

200

300

400

500

600

2007

-06-

01

2007

-06-

08

2007

-06-

15

2007

-06-

22

2007

-06-

29

2007

-07-

06

2007

-07-

13

2007

-07-

20

2007

-07-

27

2007

-08-

03

2007

-08-

10

2007

-08-

17

2007

-08-

24

2007

-08-

31

2007

-09-

07

2007

-09-

14

2007

-09-

21

2007

-09-

28

2007

-10-

05

Unassigned

Assigned

Resolved

Closed

Fix later

Duplicate

Dismissed

Total

Other metrics - 4

Example of how the last weeks changes could be displayed more clearly.

-10

-5

0

5

10

15

20

25

30

Assigned Closed Dismissed Duplicate Fix later Resolved Unassigned Grand Total

Serie1

Other metrics- 5

Conclusion!!

!  Select a metric that suits you.!!  Begin simple and easy!!  Select and add a metric, one by one.!!  Evaluate and control continuesly.!!  Communicate.!!  Metric is only information and should never be used as !a grounds for an evaluation for a certainly individual!

Questions?!