Post on 20-Jan-2015
description
Significance of Test Coverage
Sep 2011
2 © Virtusa Corporation ● Confidential
Contents
Meaning of test coverage
Business importance for test coverage
Types of test coverage
Tools for test coverage
Test coverage in Agile
Test coverage in Risk based testing
Test coverage and Regression test decisions
Interpretation of test coverage numbers
3 © Virtusa Corporation ● Confidential
Meaning of test coverage
Total number of requirements tested against the total number of requirements in scope
Req ID Test case reference Test status
REQ 1 TC x to z Pass/Fail
REQ 2 TC x to z Pass/Fail
REQ 3 TC x to z Pass/Fail
REQ 4 TC x to z Pass/Fail
REQ ….n TC x to z Pass/Fail
4 © Virtusa Corporation ● Confidential
Business importance of test coverage
• Test coverage is the direct indicator of the unknown risks• Sometimes it is a contractual requirement to show 100% test
coverage• Effectiveness of other defect metrics depends on test coverage• Many times 100% test coverage from SIT is the entry criteria for
UAT and for Go-live
5 © Virtusa Corporation ● Confidential
Types of test coverage
Test coverage
ReleaseProject
Design Execution
6 © Virtusa Corporation ● Confidential
Tools for test coverageRequirement traceability matrix
Req ID UC # Unit test case #
Test Scenario #
SIT Test case #
UITTest case #
Test Status
Defect ID
7 © Virtusa Corporation ● Confidential
Test coverage in Agile
Sprint 1 Sprint 2 Sprint 3 Sprint n
Release X
Reg Test coverage
0%
Reg Test coverage
65%
Reg Test coverage
70%
RegTest coverage
85%
Reg Test coverage
100%
8 © Virtusa Corporation ● Confidential
Test coverage in Risk based testing
• Risk based testing is conscious decision to reduce testing time• In this case test coverage cannot be 100%• For such situations, we need to calculate & ensure “Critical”
and “High” complexity scenario coverage separately• Need to intimate customer upfront about the test coverage
9 © Virtusa Corporation ● Confidential
Test coverage and Regression test decisions
Req ID
UC #
Unit test case #
Test Scenario #
SIT Test case #
UITTest case #
Test Status
Defect ID
• We need two matrices to decide regression test scenarios• 100% regression testing for all cycles is impractical• We need functional dependency matrix to do impact analysis• Further we need Req Traceability matrix to decide the test
scenarios
Requirement Traceability MatrixFunctional Dependency Matrix
Functions Fun 1 Fun 2 Fun 3 Fun..n
Fun1
Fun2
Fun3
Req ID Req 1 Req 2 Req 3 Req n
10 © Virtusa Corporation ● Confidential
Interpretation of test coverageDSI trend Vs Test coverage
Project A – DSI increases along with test coverage in cycle 3Project B – DSI decreases over cycle after cycle even if the coverage is increased in the regression test
11 © Virtusa Corporation ● Confidential
Thank You