Quality Index: A Composite Metric for the Voice of Testing

©2015, Cognizant Technology solution 1

Quality Index

A Composite Metric for the Voice

of Testing

October 1, 2015

Nirav Patel - Walgreens Boots

Alliance

[email protected]

Sutharson Veeravalli – Cognizant

Technology Solutions

[email protected]

©2015, Cognizant Technology solution 2 2 ©2015, Cognizant Technology solution

Speaker(s) Bio

A director at Cognizant’s Quality Engineering & Assurance (QE&A) practice, Sutharson

Veeravalli manages large delivery organizations and specializes in implementing

transformational initiatives that enhance the delivery ecosystem. Sutharson has twenty years

of experience spanning banking and retail domains, process and domain consulting, large

system delivery, product management, and QA transformation for clients. He brings a diverse

experience in leading multiple transformational initiatives that have delivered multimillion

dollar savings to clients

Nirav Patel, the QA manager of QAE for pharmacy, retail, and health and wellness divisions at

Walgreens, has been in QA for more than fifteen years. Specializing in metrics-driven

governance and financial control, Nirav drives the concept-to-implementation process of the

transformational initiatives across his portfolio. He is an active advocate of quality

engineering to drive consistency and efficiency within the practice.

Nirav

Sutharson

©2015, Cognizant Technology solution 3 3

The Curious case of information overload...

GUARD [on screen]: I've sealed the compartment. It can't get out, that lock's got a billion combinations. DOCTOR: A Dalek's a genius. It can calculate a thousand billion combinations in one second flat. DOCTOR: It's not just energy. That Dalek just absorbed the entire internet. It knows everything Doctor Who - 2005 Series, Episode 6

Blessed are the ones who have Daleks as their project managers… rest of us have to still deal with the information overload in managing projects

How many times have you felt…???

Overwhelmed at the amount of data & metrics produces by the teams. and yet not getting a holistic view on project health ?

Unsure if in-progress & completed programs are on track to deliver the intended business outcomes?

Limited in conveying the project/program status to non-technical business & senior management stakeholder? Challenged in articulating the quality

assurance view to external stakeholders?

©2015, Cognizant Technology solution

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRxqFQoTCOrJvuyxg8cCFYJ2Pgod5tQMAQ&url=http://doctorwhotoys.net/dalekprogenitorguardian.htm&ei=JWC6VaqjDILt-QHmqbMI&bvm=bv.99028883,d.dmo&psig=AFQjCNHtNO7m634p9LoDO4ZgwGEDjj8Tlw&ust=1438364056291593


Why will traditional metrics & articulation not solve our challenges?

Business Expectation

Simple non technical view on program/project status

Standardized, contextual & actionable information

Information aligned to outcomes to be delivered

QA challenges

Delivery metrics requires some level of understanding QA

domain

Plethora of metrics , too difficult to summarize

Voice of QA is limited and mostly overridden by program

pressures

Lack of business outcome view. Focus is more on tactical

delivery

QA should restrict to test execution and achieving end

date

Solution

Synergize bottom up metrics to create a multi dimensional

view that uses a simple composite metric to succinctly

summarize status against expected outcomes,

strengthening the “Voice of QA”

Standardized, business outcomes aligned metrics framework that uses “Quality Index” as voice of QA



Multi-dimensional Metrics Framework

• QUALITY INDEX based reporting • A unified view of quality depicting

the quality level against benchmark

Business outcome aligned

• At highest level metrics are aligned to business outcome • Execution predictability • Product stability • Learning & skills • Financial control

Broader Measures

• Measures not restricted to single phase.

• Specific measure (called quality gauge) defined for each phase of SDLC

Contextual insights

• Metrics collated contextual to stakeholder

• Level 1: for QA stakeholders

• Level 2: SDLC stakeholders

• Level 3: Business stakeholders

Objective : To create a unified metrics framework to • Standardize metrics, data sources, tools and process to capture data • Unified reporting for multi-level (L1, L2 and L3), contextual management status • Facilitate single view of Quality through ‘Quality index (QI)’ (Project, Program and release level) & Centralized dashboard

Single view of quality



Build maturity Requirements maturity QA efficiency QA Quality Upstream quality

Broader Measure & Contextual Insights

Requirements Analysis Test Design Test Execution & Closure Design & Build

Measure

SDLC Stakeholders

QA Stakeholders

Process adherence Automation/tooling

percentage Improvement

defects found in DIT

Build success rate Deployment time

Requirements traceability Test design productivity Rework effort (due to QA) Review Efficiency Test design productivity

(Auto) Performance test design

productivity Build to QA spend ratio

Requirements Implementation index (based on static testing defects

Test execution productivity

Test execution productivity

Performance test execution productivity

Test Efficiency =(1-defect rejection)

Requirement stability index (RSI)

Defect reopen rate Environment

downtime Schedule Variance Business time to

market QA committed

schedule



Business Outcome aligned metrics with QI

Benefits quantified through Single unit of

measure “Y-o-Y Improvement

on Cost/TCP”

Quality Index

Quality Index

Quality Index

Quality Index

Execution Predictability

Learning & Skills

Production Stability

Financial Control

• Requirements implementation index • Requirement stability index (RSI) • Defect reopen rate • Test environment downtime • Schedule Variance • Effort variation (due to QA) • Risk Aligned test coverage • Maturity of build Organization

• Team Competency Index

• Review Efficiency • Innovation Index

• Requirements Traceability • UT/DIT coverage • UT/DIT test results • Defect leakage to production • Application defect density

• Test Design productivity (manual, automation, performance)

• Test Execution productivity (manual, automation, performance)

• Rework Effort(Due to QA) • Test Effectiveness • Build to QA ratio • % CAPEX reduction through virtualization • Automation Coverage

What is the improvement ?


©2015, Cognizant Technology solution 8 ©2015, Cognizant Technology solution 8

Quality Index – Voice of the QA

Quality Index (QI) is the composite metric used to

summarize the quality of each measure

A QI of 1 indicates the measure is in alignment with the

expected benchmark value.

How is QI Calculated ? A measure (e.g. Execution predictability) is linked to influencing metrics Every metric is measured against the expected benchmark The QI index is calculated as 1, if the metric value is equal to the expected benchmark level Based on QI value a RAG status is assigned: QI >1 is Green, > 0.8 and < 1 is Amber and < 0.8 is Red The individual metric level QI is rolled up to measure level by assigning weightages to each metric

How is QI rolled up ? Quality index is calculated at project level and rolled up through business as below

How is QI represented? QI is represented as dial for current value and also trend graph

Project 1 Project 2 Project n Project 3

Release 1 Release 2 Release n

LOB 1 LOB 2 Program

Project 4


Weightage 15%

Benchmark 5%

Weightage 25%

Benchmark 8%

Weightage

10%

Benchmark

1%

Weightage 25%

Benchmark 5%

Weightage 15%

Benchmark 5%

Example of Calculating Execution Predictability

% Rework Effort due to change in

requirements - Measure of the effort going

on in rework of test artifacts

(Total Rework Effort (across all phases)) /

(Actual Effort expended for the project )*100

Tool #1

Tool #2 %

Execution Predictability - This view helps to analyze, evaluate & infer the execution predictability of the Project from QAE point of view. It helps to infer if the execution measures/metrics are on track & to help to change course correction based on deviations, if any. The metrics that are captured as part of this view are as listed below

Defect Reopen rate - Measures the quality

(No of defects reopened post retest/no of valid

defects logged)

Tool #1 %

Test Environment Downtime - Measures availability of test environments to test teams

(# of person days lost due to environment

downtime/ (total planned test execution person

days + planned test data setup person days))*100

Tool #1 %

Schedule Variance - Provides a measure of deviation of the actual

schedule against the planned schedule

Actual end date – Planned end date) / Planned end date – Planned start date) * 100

Tool #2 %

Requirement Stability Index - Measure of the stability of requirements

(Number of Original Requirements + Number of

requirements changed + Number of requirements added + Number

of requirements deleted)/ (Number of Original

Requirements)

Tool #1 #

Effort Variance - Provides a measure of the deviation of the actual effort

expended against the estimated effort

(Actual Effort -- Estimated Effort) / Estimated Effort) *

100 Tool #2 %

Weightage 15%

Metric value is compared against the benchmark. Meaning if “% Rework” value is “9%” and Benchmark was

“4%” then QI is (4% / 9%) * 100% = 44%

Value from Step 1 * Weightage %

= 44% * 15% = 6.6%

Execution predictability = SUMMATION of all Metrics Computed

Proportionally Weightage Index

Ste

p 1

Ste

p 2

Ste

p 3

Compute Base Quality Index

Compute Proportionally Weightage Index

Compute Execution Predictability

Benchmark 4%



Typical QI based dashboard



In Summary

Provide contextual, actionable and relevant information to stakeholders

Align messaging to ‘business outcomes’ as against ‘delivery status”

Have single view of quality using “Quality Index” like composite metric

Scale the framework outside delivery into all QA goals

Business Value of QA can be demonstrated by

Quality Dashboard

Outcome Alignment

Multi-dimensional Metrics Framework

Quality Index



Outcome Based View

Expected outcome Measure Influencing metrics

Programs and projects needs to be

monitored for ON TIME delivery

Execution Predictability

% Rework Effort due to change in requirements

Defect reopen rate

Test environment downtime

Schedule Variance


% Effort Variation

Measure quality of product under

test by BSS QAE

Measure quality of product

deployed post BSS QAE phase

Product Stability - Inflight

Product Stability – Post QA

Requirements traceability

Application Defect Density

UAT Defect Leakage (%)

Production Defect Leakage (%)



QI – Calculation sample

Measure Influencing Metrics Release 1 Release 2 Release 3 Release 4 Weightage Benchmark Metric level

Quality Index

Measure level

Quality Index

Execution

predictability

% Rework Effort due to

change in requirements 0.3% 1.0% 9.0% 1.0% 15% 4% 150.0% 1.04

Defect reopen rate - 10.0% 12.0% 11.0% 20% 5% 0.0%

Test environment

downtime 12.0% 9.0% 9.0% 3.0% 25% 8% 66.7%

Schedule Variance 3.0% 10.0% - - 15% 5% 150.0%

Requirement stability

index (RSI) - 1.10 1.07 1.30 10% 1 0.0%

% Effort Variation -0.4% 56.0% -6.0% -17.0% 15% 5% 122.0%

Production

Stability –

Inflight

Requirements

traceability 100.0% 100.0% 100.0% 100.0% 5% 100% 100.0% 1.21

Application Defect

Density 14.0% 12.0% 8.0% 11.0% 20% 1% 70.0%

Production

Stability - Post

QA

UAT Defect Leakage (%) 1.0% 0.2% 1.0% 0.1% 35% 3% 150.0%

Production Defect

Leakage (%) 2.0% 0.8% 1.0% - 40% 2% 100.0%



Financial

Control

Production

Stability

Execution

Predictability

Learning &

Skills

A View for Agile

Requirements implementation index


Defect reopen rate

Test environment downtime

Schedule Variance

Effort variation

Risk Aligned test coverage

Maturity of Dev Organization

Defect Density (Agile)

% Scope Creep

Testing Efficiency at the Sprint level

Test Design productivity (manual,

automation, performance) Test Execution productivity (manual,

automation, performance) Rework Effort(Due to QA) Test Effectiveness Build to QA ratio Offshore, Near shore leverage % CAPEX reduction through virtualization Velocity of the sprint Velocity rate Story point Productivity

Requirements Traceability UT/DIT coverage UT/DIT test results Defect leakage to production Application defect density Testing Efficiency at the Sprint level

Team Competency Index Review Efficiency Innovation Index


Quality Index: A Composite Metric for the Voice of Testing

Software

Transcript of Quality Index: A Composite Metric for the Voice of Testing