Effective Metrics for Managing test efforts

7/29/2019 Effective Metrics for Managing test efforts

1/23

Effective Metrics for Managing a Test Effort

Presented By:Shaun BradshawDirector of Quality SolutionsQuestcon Technologies

Quality - Innovation - Vision


2/23

Slide 2

Objectives

Why Measure?

Definition

Metrics Philosophy

Types of MetricsInterpreting the Results

Metrics Case Study

Q & A


3/23

Slide 3

Software bugs cost the U.S. economyan estimated $59.5 billion per year.

An estimated $22.2 billion

could be eliminated byimproved testing that enables

earlier and more effective

identification and removal of defects.

- US Department of Commerce (NIST)

Why Measure?


4/23

Slide 4

It is often said,You cannot improve what you

cannot measure.

Why Measure?


5/23

Slide 5

Definition

Test Metrics:

Are a standard of measurement.

Gauge the effectiveness and efficiency of several

software development activities.

Are gathered and interpreted throughout the test

effort.

Provide an objective measurement of the success

of a software project.


6/23

Slide 6

Metrics Philosophy

When tracked and usedproperly, test metrics canaid in software

development processimprovement by providingpragmatic & objective

evidence of processchange initiatives.

Keep It Simple

Make It Meaningful

Track It

Use It


7/23

Slide 7

Metrics Philosophy

Measure the basics first

Clearly define each metric

Get the most bang for your buck

Keep It Simple

Make It Meaningful

Track It

Use It


8/23

Slide 8

Metrics Philosophy

Metrics are useless if they aremeaningless (use GQM model)

Must be able to interpret theresults

Metrics interpretation should beobjective

Make It Meaningful

Keep It Simple

Track It

Use It


9/23

Slide 9

Metrics Philosophy

Incorporate metrics tracking intothe Run Log or defect trackingsystem

Automate tracking process toremove time burdens

Accumulate throughout the testeffort & across multiple projects

Track It

Keep It Simple

Use It

Make It Meaningful


10/23

Slide 10

Metrics Philosophy

Use It

Keep It Simple

Make It Meaningful

Track It

Interpret the results

Provide feedback to the ProjectTeam

Implement changes based onobjective data


11/23

Slide 11

Types of Metrics

Base Metrics Examples

Raw data gathered by Test AnalystsTracked throughout test effort

Used to provide project status andevaluations/feedback

# Test Cases

# Executed

# Passed

# Failed

# Under Investigation# Blocked

# 1st Run Failures

# Re-Executed

Total ExecutionsTotal Passes

Total Failures


12/23

Slide 12

Types of Metrics

# Blocked

Base Metrics Examples

The number of distinct test cases thatcannot be executed during the test effortdue to an application or environmental

constraint.Defines the impact of known systemdefects on the ability to execute specifictest cases

Raw data gathered by Test AnalystTracked throughout test effort

Used to provide project status andevaluations/feedback

# Test Cases

# Executed

# Passed

# Failed

# Under Investigation# Blocked

# 1st Run Failures

# Re-Executed

Total ExecutionsTotal Passes

Total Failures


13/23

Slide 13

Types of Metrics

Calculated Metrics Examples

Tracked by Test Lead/ManagerConverts base metrics to useful data

Combinations of metrics can be used toevaluate process changes

% Complete% Test Coverage

% Test Cases Passed

% Test Cases Blocked

1st

Run Fail RateOverall Fail Rate

% Defects Corrected

% Rework

% Test EffectivenessDefect Discovery Rate


14/23

Slide 14

Types of Metrics

1st Run Fail Rate

Calculated Metrics Examples

The percentage of executed test casesthat failed on their first execution.

Used to determine the effectiveness of

the analysis and development process.Comparing this metric across projectsshows how process changes haveimpacted the quality of the product atthe end of the development phase.

Tracked by Test Lead/ManagerConverts base metrics to useful data

Combinations of metrics can be used toevaluate process changes

% Complete% Test Coverage

% Test Cases Passed

% Test Cases Blocked

1st

Run Fail RateOverall Fail Rate

% Defects Corrected

% Rework

% Test EffectivenessDefect Discovery Rate


15/23

Slide 15

Sample Run Log

Sample System Test

Current # of

TC ID Run Date Actual Results Status Runs

SST-001 01/01/04 Actual results met expected results. P P 1

SST-002 01/01/04 Sample failure F F 1

SST-003 01/02/04 Sample multiple failures F F P P 3



SST-006 01/03/04 Sample Under Investigation U U 1SST-007 01/03/04 Actual results met expected results. P P 1

SST-008 Sample Blocked B B 0

SST-009 Sample Blocked B B 0






SST-015 01/03/04 Actual results met expected results. P P 1SST-016 0

SST-017 0

SST-018 0

SST-019 0

SST-020 0

Run Status


16/23

Slide 16

Sample Run Log

Metric Value Metric Value

Total # of TCs 100 % Complete 11.0%

# Executed 13 % Test Coverage 13.0%

# Passed 11 % TCs Passed 84.6%

# Failed 1 % TCs Blocked 2.0%# UI 1 % 1st Run Failures 15.4%

# Blocked 2 % Failures 20.0%

# Unexecuted 87 % Defects Corrected 66.7%

# Re-executed 1 % Rework 100.0%

Total Executions 15

Total Passes 11

Total Failures 3

1st Run Failures 2

Calculated MetricsBase Metrics


17/23

Slide 17

Issue:The test team tracks and reportsvarious test metrics, but there is noeffort to analyze the data.

Result:Potential improvements are not

implemented leaving process gapsthroughout the SDLC. This reduces theeffectiveness of the project team andthe quality of the applications.

Metrics Program No Analysis


18/23

Slide 18

Solution:

Closely examine all available data

Use the objective information todetermine the root cause

Compare to other projects

Are the current metrics typical of

software projects in yourorganization?

What effect do changes have onthe software developmentprocess?

Metrics Analysis & Interpretation

Result:Future projects benefit from a moreeffective and efficient applicationdevelopment process.


19/23

Slide 19

Volvo IT of North America had little or no testinginvolvement in its IT projects. The organizations projectswere primarily maintenance related and operated in aCOBOL/CICS/Mainframe environment. The organizationhad a desire to migrate to newer technologies and felt thattesting involvement would assure and enhance thistechnological shift.

While establishing a test team we also instituted a metricsprogram to track the benefits of having a QA group.

Metrics Case Study


20/23

Slide 20

Project V

Introduced a test methodology and metrics programProject was 75% complete (development was nearly

finished)Test team developed 355 test scenarios30.7% - 1st Run Fail Rate31.4% - Overall Fail Rate

Defect Repair Costs = $519,000

Metrics Case Study


21/23

Slide 21

Project T

Instituted requirements walkthroughs and design reviewswith test team input

Same resources comprised both project teamsTest team developed 345 test scenarios17.9% - 1st Run Fail Rate18.0% - Overall Fail Rate

Defect Repair Costs = $346,000

Metrics Case Study


22/23

Slide 22

Metrics Case Study

Project V Project T Reduction of

1st Run Fail Rate 30.7% 17.9% 41.7%

Overall Failure Rate 31.4% 18.0% 42.7%

Cost of Defects 519,000.00$ 346,000.00$ 173,000.00$

Reduction of33.3%in the cost of defect repairs

Every project moving forward, using the sameQA principles can achieve the same type of savings.


23/23

Slide 23

Q & A

Effective Metrics for Managing test efforts

Documents

Transcript of Effective Metrics for Managing test efforts