A SIMPLE VIEW OF HOW TO REDUCE TOTAL COST OF QUALITY

PRESENTED BY CATHERINE OEHL

COQAA Monthly Meeting

December 5, 2012

AGENDA

12/5/2012 2

• What is Quality and Why is it important• What is Total Cost of Quality (TCoQ)• What is the purpose and value of

measuring TCoQ• How to measure and reduce TCoQ• Hypothetical Scenario applying TCoQ• Q&A

QUALITY DEFINED• Definition: A measure of excellence or a state of being free from

defects, deficiencies and significant variations. It is brought about by strict and consistent commitment to certain standards that achieve uniformity of a product in order to satisfy specific customer or user requirements.

• ISO 8402-1986 standard defines quality as "the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs."

• If an automobile company finds a defect in one of their cars and makes a product recall, customer reliability and therefore production will decrease because trust will be lost in the car's quality.

12/5/2012 3

Source: http://www.businessdictionary.com/definition/quality.html

CONSEQUENCES OF POOR QUALITY

12/5/2012 4

COMMON STATE OF QUALITYo Current projects spend 40-50% of the effort on avoidable reworko About 80% of avoidable rework comes from 20% of the defectso About 80% of defects comes from 20% of the modules and about half

of the modules are defects freeo 90% of the downtime comes from at most 10% of the defectso Peer Reviews catch 60% of the defectso Perspective-based reviews catch 35% more defects than non directed

reviewso Disciplined personal practices can reduce defect introduction rates by

up to 75%o While several organizations measure and track application total cost of

ownership (TCO), very few measure the total cost of quality

12/5/2012 5

WHAT IS TCOQ

• Total Cost of Quality (TCoQ) consists of the costs associated with preventing, assessing, finding and correcting defective work.

• Note that the TCoQ analysis looks only at our (producers) costs and not the customer’s cost due to poor quality.

12/5/2012 6

Cost of Quality

Cost of Good Quality

Prevention Costs Inspection Costs

Cost of Bad Quality

Internal Failure Costs

External Failure Costs

QUALITY COSTS OF TCOQPrevention cost + Inspection cost + Internal Failure cost + External Failure cost

Prevention costs arise from efforts to keep errors or defects from occurring at all

Inspection costs arise from detecting errors/defects via inspections, tests, audits

Internal failure costs arise from defects caught internally and dealt with by discarding or repairing the defective items

External failure costs arise from defects that actually reach customers. Often other costs are incurred in addition to correcting the failure.

12/5/2012 7

PURPOSE AND VALUE OF TCOQAddresses and measures the following questions:• What is the state of quality of your applications?• How does quality impact your business?• How is quality measured?• When are errors/defects introduced? Why?• When are errors/defects corrected? What are the costs?• What are the processes that impact quality?• How do you quantify the effectiveness of each process?• How much do failures cost?• What is the root cause of the failure?

12/5/2012 8

WHEN AND WHERE ARE DEFECTS INTRODUCED

Source: Writing Testable Requirements, Dick Bender

12/5/2012 9

Re-quire-ments56%

Design27%

Coding7%Unknown

10%

Source: NIST Software Quality Study

WHEN AND WHERE ARE DEFECTS DETECTED

12/5/2012 10

Source: NIST Software Quality Study

COST TO CORRECT DEFECTS

12/5/2012 11

Source: B. Boehm and V. Basili, “Software Defect Reduction Top 10 List”, IEEE Computer

Cost to remove defects is 100 times more

THE QUALITY LEVER

12/5/2012 12

Source: http://en.wikipedia.org/wiki/Quality_costs

Prevention / Reviews

Appraisal /Inspection

Failures / Reaction

Cost Savings

Greatest Leverage Good

Leverage LeastLeverage

PREVENTIVE COSTSGOAL: Prevent defects before they happen

(Work done right the first time)

Recommended Activities:– Formal Requirements Gathering Processes– Requirements Analysis and Management– Early Prototyping– Usability Analysis– Static Testing– Peer Reviews– Release and Configuration Management Processes– Standards for Test Environments– Quality Training– Accurate Internal Communication

12/5/2012 13

INSPECTION COSTSGOAL: Detect the Errors (Find them early)

Recommended Activities:– Code inspections– Design reviews– Glass box and black box testing– Test Automation– Usability Testing– Pre-release / User Acceptance Testing– Structured Personal reviews focused on

verification and validation

12/05/2012 14

INTERNAL FAILURE COSTSGOAL: Report and fix defects before production

release (Fix and Re-Test)

Incurred Activities:• Diagnose Defects• Defect Fixes• Defect Management• Re-Test and Regression Testing• Rework code and recompile• Wait states

12/5/2012 15

EXTERNAL FAILURE COSTSGOAL: Reduce and remediate impact to customers

(Apologize and recover)

Incurred Activities:• Technical Support Calls• Recalls• Customer Service• Scrap• Overtime• Regulatory / Legal Costs

12/5/2012 16

HYPOTHETICAL SCENARIO

• Suppose we have a software product that has one production release per quarter.

• On average each release contains 1000 must-fix defects that will be identified and repaired.

• We will assume developers catch 25% of defects during unit and integration testing at a cost of $10/defect

12/5/2012 17

1 - NO INSPECTION TESTING

12/5/2012 18

Inspection Testing CostsNon Formal Testing

Staff $0 Infrastructure $0 Tools $0 Total Investment $0

 Development  Must-Fix Defects Found 250Fix Cost (Internal Failure - $10/defect) $2,500

 Testing (Pre-Release)  Must-Fix Defects Found 0Fix Cost (Internal Failure - $100/defect) $0

 Customer Support (Production)  Must-Fix Defects Reported 750Fix Cost (External Failure - $1000/defect) $750,000

 Cost of Quality  Conformance (Inspection Costs) $0 NonConformance (Internal Failures + External Failures) $752,500

TCoQ $752,500 ROI N/A

2 - MANUAL INSPECTION TESTING

12/5/2012 19

Inspection Testing CostsNon Formal Testing

Manual Testing

Staff $0 $60,000 Infrastructure $0 $10,000 Tools $0 $0 Total Investment $0 $70,000      Development    Must-Fix Defects Found 250 250Fix Cost (Internal Failure - $10/defect) $2,500 $2,500      Testing (Pre-Release)    Must-Fix Defects Found 0 350Fix Cost (Internal Failure - $100/defect) $0 $35,000      Customer Support (Production)    Must-Fix Defects Reported 750 400Fix Cost (External Failure - $1000/defect) $750,000 $400,000      Cost of Quality    Conformance (Inspection Costs) $0 $70,000 NonConformance (Internal Failures + External Failures) $752,500 $437,500

TCoQ $752,500 $507,500 ROI N/A 350%

3 - MANUAL WITH AUTOMATED INSPECTION TESTING

12/5/2012 20

Inspection Testing CostsNon Formal Testing

Manual Testing

Automated Testing

Staff $0 $60,000 $60,000 Infrastructure $0 $10,000 $10,000 Tools $0 $0 $12,500 Total Investment $0 $70,000 $82,500        Development      Must-Fix Defects Found 250 250 250Fix Cost (Internal Failure - $10/defect) $2,500 $2,500 $2,500        Testing (Pre-Release)      Must-Fix Defects Found 0 350 500Fix Cost (Internal Failure - $100/defect) $0 $35,000 $50,000        Customer Support (Production)      Must-Fix Defects Reported 750 400 250Fix Cost (External Failure - $1000/defect) $750,000 $400,000 $250,000        Cost of Quality      Conformance (Inspection Costs) $0 $70,000 $82,500 NonConformance (Internal Failures + External Failures) $752,500 $437,500 $302,500

TCoQ $752,500 $507,500 $385,000 ROI N/A 350% 445%

What if we added some simple Prevention activities like Static Reviews, Peer Reviews, Change Management ?

4 - ADDING PREVENTION ACTIVITIES

12/5/2012 21

Prevention Costs       Resource CostsStatic Reviews (10 people every 2 weeks reviewing 5 sets of documents x $75/hr)   $97,500Peer Reviews (2 developers every 2 weeks reviewing 10 modules x $75)     $39,000Change Management (10% Configuration Mgr & 10% Requirements Lead & 10% Test Lead)   $46,800

Inspection Testing Costs Non Formal Testing Manual Testing Automated Testing  Staff $0 $60,000 $60,000  Infrastructure $0 $10,000 $10,000  Tools $0 $0 $12,500  Total Investment (Prevention & Inspection costs) $0 $70,000 $82,500 $265,800        Development        Must-Fix Defects Found 250 250 250 500Fix Cost (Internal Failure - $10/defect) $2,500 $2,500 $2,500 $5,000        Testing (Pre-Release)        Must-Fix Defects Found 0 350 500 350Fix Cost (Internal Failure - $100/defect) $0 $35,000 $50,000 $35,000        Customer Support (Production)        Must-Fix Defects Reported 750 400 250 50Fix Cost (External Failure - $1000/defect) $750,000 $400,000 $250,000 $50,000        Cost of Quality        Conformance (Inspection Costs) $0 $70,000 $82,500 $265,800NonConformance (Internal Failures + External Failures) $752,500 $437,500 $302,500 $90,000

TCoQ $752,500 $507,500 $385,000 $355,800 ROI N/A 350% 445% 57%

SUMMARY• Quality creates value and reduces risk exposure• The “cost of quality” isn’t the price of creating a quality product or

service. It’s the cost of NOT creating a quality product or service.• Errors / defects are often introduced early and typically found later• Total Cost of Quality (TCoQ) consists of the costs associated with

preventing, assessing, finding and correcting defective work. • The central theme of quality improvement is that larger investments in

prevention drive even larger savings in quality-related failures and Inspection efforts.

• Any Quality improvement effort should use TCoQ to quantify the effectiveness of the process.

12/5/2012 22

CONTACT INFO AND Q&A

Catherine Oehl

oehlc@yahoo.com

614-440-6477

LinkedIn Profile:

http://lnkd.in/u8kcyy

12/5/2012 23

RESOURCES & REFERENCES Software Defect Reduction Top 10 List by Barry Boehm and Victor Basili Using Cost Benefit Analysis to Develop Software Process Improvement

Strategies, Data Analysis Center of Software Traditional P-A-F method by Juran (1951) and Feigenbaum (1956) –

Prevention Appraisal and Failure costs Phil Crosby’s Model from the book Quality is Free (1979) – conformance to

requirements – cost of conformance (done Right) and non-conformance (done Wrong)

Process Cost Model by Ross (1977) http://asq.org/learn-about-quality/cost-of-quality/overview/overview.html http://www.authorstream.com/Presentation/Hassanasif-493959-cost-of-quali

ty http://www.bexcellence.org/Cost-Of-Quality.html http://www.isixsigma.com/implementation/financial-analysis/cost-quality-not-

only-failure-costs/ http://www.projectconnections.com/articles/051005-koch.html

12/5/2012 24

SUGGESTED TCOQ BENCHMARK MEASUREMENTS

• Process (Productivity & Prevention)– Effectiveness of Test Processes

• Release Mgt, Configuration Mgt

– Efficiency of Test Processes• Defect Mgt, Test Environment Mgt, Test Data Mgt

– Consistency of Test Activities• Test Planning, Test Estimation, Test Execution

• Schedule– Testing Planning

• Duration of lifecycle phases• Duration of Test Phase/Stage

– Test Estimating/Projections• Planning Estimate• Component Based Estimates

12/5/2012 25

COMMON MEASUREMENTS OF FAILURES

• Failures– Volume of Defects

• Open, Closed, New, Test Phase/Stage Contained, Defect Aging (# of days open)

– Severity of Defects• Criticals, Highs, Mediums, Lows, Problem Areas,

Low to all reported ratio

12/5/2012 26

SIMPLE ACTUAL QUALITY COSTS

• Build Test Mgt Costs (P+I+IF)– Manual Test Hour Costs– Automated Test Hour Costs– Test Hour Costs per Test Phase/Stage– % of Total Build Costs

• Breakfix/Maintenance Costs (EF)

12/5/2012 27

VIEW OF TCOQ IN TERMS OF QA

12/5/2012 28