Metrics that Matter

Measuring Quality in Agile Project

Sreekanth Singaraju, VPSharon Lee, Director of Marketing

2

Agenda

• Agile Overview• Testing in Agile• Importance of Metrics• Key Metrics

– Test and Execution Coverage – Defect identification efficiency – Technical Debt – Test confidence level

• Tools

3

Why Organizations Use Agile

• Develop iteratively with business needs

• Deliver working software rapidly

• Flexibility to change with business needs

• Incorporate end-user feedback

• Highly productive technical teams

Deliver what end-users really want in an iterative fashion

4

Role of Testing in Agile

• Testing may be done by people who aren’t called “QA” or “Tester”; e.g. Business Analysts.

• Some teams may use developers for system or acceptance testing.

• Dedicated testers bring two benefits– Focus on customer usage over technical implementation– Focus on uncovering flaws over confirming completeness

What product or service does testing provide?“Tested, Debugged Software” – Wrong

Testing provides Information about the status of software under development to make informed decisions.

5

Testing in Agile Projects

Agile Testing Quadrants provide a concise way to communicate the different layers of testing needed and how they can be executed

6

Testing in Agile Projects

• Testing is time-boxed like development

• Functionality to be tested is not limited like Dev

• Difficult to achieve “fully-tested” status

• Defects found do not reflect Test Results

• Testing spread across Dev and Test teams

Unique Challenges on Agile Projects

7

Metrics Objectives

8

Metrics Objectives

For Business Stakeholders provide:

• All the information they need to make decisions, and no more.

• Information at the level of detail they can use.

• Information at the scope they care about (team, project, program, line of business, enterprise, industry)

• Information pertaining to the time frame they care about (day, iteration, release, project, strategic milestone)

9

Metrics Objectives

Communicate the following aspects:

• The level of testing that was accomplished in a sprint, release etc.

• The efficiency with which defects have been identified in different testing stages

• The cost of trade-offs being made in each sprint

• The overall confidence level in the testing efforts

10

Coverage Metrics

PrincipleCommunicate to all stakeholders the coverage provided by formal testing

Dimensions• Use Validation points to evaluate application’s compliance with requirements• Weight them by business criticality

DefinitionDesigned Coverage = Designed Validation Points / All Validation Points (Estimated)Executed Coverage = Executed Validation Points / All Validation Points (Estimated)

Best Practices• Keep coverage metrics simple• Introduce granularity to address business needs• Trend to show progress (or declines – great case for additional resources)

11

Defect Identification Efficiency

PrincipleIdentify how defects are identified and created. Evaluate compliance of Agile principles.

Dimensions• Number of defects identified in different stages of software lifecycle• Number of defects created in different stages of software lifecycle• Use key Software stages to draw valuable inferences

DefinitionProportion of defects identified by each stage of a lifecycle – Ex – Testing –56%, UAT – 18%, Production - 4%, other - 22%

Best Practices• Start with defects identified first – then introduce defects created• Use “Other” to catch exceptions and introduce granularity incrementally• Factor in defect severity if that information is readily available

12

Technical Debt

PrincipleProvide a financial impact of technical choices on the overall product resulting from technology, process and implementation choices made during development

Dimensions• Quality of software code• Unresolved defects that are business or quality critical• Impact of technical or architectural choices

DefinitionCost estimates for redressing most critical (Business perspective) defects or technical choices

Best Practices• Explosive- Handle and communicate with care• Be conservative – use a very strong criteria to evaluate “criticality”• Use several available tools to evaluate software code quality

13

Confidence Level

PrincipleDemonstrate the confidence of the testing team in the testing efforts executed on a release with the resources provided for testing.

Informational• Test Execution Coverage• Expertise of resources to test the application• Quantitative results of defects including Defect identification efficiency

DefinitionBased on the evaluation of the 3 dimensions above rank each of them on a scale of 0(Poor) – 1(High). Overall confidence = (3.Coverage.Resources.Results)/(Coverage+Resources+Results)

Best Practices• Be prepared with supporting facts• Define clearly laid out criteria for ranking

14

Key Success Drivers

• Alignment with ultimate stakeholders’ goals

• Won’t be perfect so don’t wait for them to be

• Keep them simple

• Collaborate with other functions

• Trending more valuable than snapshots

Key best practices to succeed at creating 

metrics.

15

Tools

Quality Management Systems• HP Quality Center• Silk Central• Rally

Automation Tools• QTP• Selenium• Silk

Code Quality Analysis• CAST AIP• Coverity• Radar

16

QUESTIONS & DISCUSSION

www.allianceglobalservices.com

17

Thank YouFor more information email us at: info@allianceglobalservices.com