HCLT Research Paper: Cost of Quality

14
Cost of Quality Needs Your Attention Not Sometimes But Always Authors Manas Chakraborty, Symu Koul, & Suresh Sundararajan, ETS-IV&VS HCL Technologies, Chennai Dr Usha Thakur, ATS Technical Research HCL Technologies, Chennai

description

http://www.hcltech.com/enterprise-transformation-services/overview~ More on ETS The purpose of this paper is to illustrate the significance of incorporating CoQ principles in software solutions and to evaluate its impact on business. It also highlights HCL’s capabilities in analyzing IT projects and implementing the CoQ approach across business domains, which bear ample testimony to the fact that CoQ is a business opportunity and not a cost centre. This paper will be of interest to IT professionals who are exploring ways to help their companies speed up time‐to‐market and cut operational cost.

Transcript of HCLT Research Paper: Cost of Quality

Page 1: HCLT Research Paper: Cost of Quality

Cost of Quality

Needs Your Attention Not Sometimes

But Always

Authors

Manas Chakraborty, Symu Koul, & Suresh

Sundararajan, ETS-IV&VS

HCL Technologies, Chennai

Dr Usha Thakur, ATS Technical Research

HCL Technologies, Chennai

Page 2: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 2

CoQ: Needs Your Attention Not Sometimes But Always

© 2010, HCL Technologies Ltd.

June 2010

Page 3: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 3

Contents

Introduction ............................................................................................................................................................ 4

Purpose ................................................................................................................................................................... 4

CoQ and Its Relevance ............................................................................................................................................ 5

CoQ Service from HCL ............................................................................................................................................. 7

CoQ as an Integral Part of SDLC ........................................................................................................................ 12

Benefits ................................................................................................................................................................. 14

Page 4: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 4

Introduction In and of themselves tools and technologies are not very useful and when they end up in the hands of people

who do not understand their significance, they are as good as not having any. By the same token a software

solution is not very valuable in itself, but it can serve as a powerful tool for enabling and transforming business

operations; in fact, it has the potential of giving impressive business results.

Of late, IT roles, activities, solutions, and services have come under the scanner of business executives and are

being assessed for the extent to which they create (business) value i.e., help company executives in achieving

strategic objectives such as giving their company a competitive edge, increasing profit margins, cutting overall

operational costs, retaining existing customers and obtaining new ones, increasing customer satisfaction,

improving internal communication, and speeding up time-to-market for software products and services. It is,

therefore, not surprising that the impact of software solutions and services is increasingly being measured in

terms of business criteria such as Return on Investment (ROI) and Return on Quality (ROQ). Moreover,

businesses are no longer willing to work within the confines of IT silos; they want IT to adapt easily to the

dynamic needs of business without compromising on quality.

If the goal of software solutions is to improve the way a business is run so that it becomes more profitable and

healthy, then Cost of Quality (CoQ) is a key approach for achieving that kind of improvement. The principles of

CoQ are well established in the manufacturing industry, given its long history.1 Their relevance in determining

the quality of software solutions has been understood only in the last couple of decades, mostly in the context

of the costs incurred as a result of delivering poor quality software. The main idea behind a CoQ approach is to

increase revenue and decrease operational cost by reducing rework and maximizing opportunities.

Purpose The purpose of this paper is to illustrate the significance of incorporating CoQ principles in software solutions

and to evaluate its impact on business. It also highlights HCL’s capabilities in analyzing IT projects and

implementing the CoQ approach across business domains, which bear ample testimony to the fact that CoQ is

a business opportunity and not a cost centre. This paper will be of interest to IT professionals who are

exploring ways to help their companies speed up time-to-market and cut operational cost.

1 For an interesting read on this see, Matthew Littlefield, "The Cost of Quality: Benchmarking Enterprise Quality

Management," (2007) http://www.hertzler.com/php/portfolio/white.papers.php [April 2010]->represents when the site

was accessed.

Page 5: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 5

CoQ and Its Relevance

Any discussion on the subject of CoQ leads one to acknowledge the contributions of Juran, Gryna and Crosby2

who emphasized that it is more economical to pay attention to quality early in the lifecycle of a product (by

incurring prevention and appraisal costs) than to wait for products to fail (for non conformance) and then fix

them (thereby incurring failure costs). Realizing its significance, the Software Engineering Institute (SEI) has

made quality an integral part of its Capability Maturity Model Integration (CMMI) best practices.3 Furthermore,

many organizations have made an effort to improve the quality of their software by integrating the principles

of Six Sigma.4

According to Crosby,

Quality means conformance to requirements, not goodness

Quality is achieved by prevention, not appraisal

Quality has a performance standard of Zero Defects, not acceptable quality levels

Quality is measured by the Price of Non-conformance™, not indexes5

In other words, CoQ refers to the cost that companies undertake to rectify the mistake they made and have to

correct it. By the same token CoQ refers to the cost companies incur to ensure that the software solutions they

deliver to their clients are free of defects. To achieve that goal, companies need to incur costs (read ‘invest’) in

the form of resources, tools and activities throughout the lifecycle of software to identify and prevent defects

as well as potential failures. The seriousness of problems arising from software errors was very well highlighted

2 For their earliest contributions the CoQ debate see J. Juran and F. Gryna, Quality Control Handbook 4th ed., McGraw-

Hill, New York, 1988. See also P. Crosby, Quality Is Free: The Art of Making Quality Certain McGraw-Hill, New York, 1979.

3 For an implementation of quality principles, see Dennis R. Goldenson, et al., "Why Make the Switch: Evidence about the

Benefits of CMMI," (2004) http://www.sei.cmu.edu/library/assets/evidence.pdf [April 2010].

4 While adherence to Six Sigma principles in software has its share of critics, they have been used for improving the

quality of software and reducing the overall CoQ. See, for instance, Dr. Bill Eventoff, "Applying Six Sigma to Software

Development: A Practical Guide," (2002) http://www.swqual.com/SQGNE/presentations/2004-

05/Eventoff%20May%202005.ppt [April 2010]. See also Maneesh Aggarwal, "Six Sigma Meets Software Development,"

http://www.isixsigma.com/index.php?option=com_k2&view=item&id=1307:&Itemid=49 [April 2010].

5 Cited in Kevin Weiss, "Biography - Philip B. Crosby: June 18, 1926 – August 18, 2001,"

http://www.philipcrosby.com/25years/crosby.html [April 2010]. For an interesting article on calculating the ROI of CoQ,

see Sandra A Slaughter, et al., "Evaluating the Cost of Software Quality," COMMUNICATIONS OF THE ACM August

1998/Vol. 41, No. 8. This paper is also available at http://www.cse.buffalo.edu/~hungngo/SCE-Papers/p67-slaughter.pdf

[April 2010].

Page 6: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 6

by a study commissioned by the Department of Commerce's National Institute of Standards and Technology

(NIST) in the US in 2002.6 The study acknowledged that

[s]oftware is error-ridden in part because of its growing complexity. The size of software

products is no longer measured in thousands of lines of code, but in millions. Software

developers already spend approximately 80 percent of development costs on identifying and

correcting defects, and yet few products of any type other than software are shipped with such

high levels of errors. Other factors contributing to quality problems include marketing

strategies, limited liability by software vendors, and decreasing returns on testing and

debugging, according to the study. At the core of these issues is difficulty in defining and

measuring software quality. The increasing complexity of software, along with a decreasing

average product life expectancy, has increased the economic costs of errors. The catastrophic

impacts of some failures are well-known. For example, a software failure interrupted the New

York Mercantile Exchange and telephone service to several East Coast cities in February 1998.

But high-profile incidents are only the tip of a pervasive pattern that software developers and

users agree is causing substantial economic losses.7

In 2008 IDC conducted a survey among quality managers of 139 North American companies. The survey

respondents estimated that if 100% of software defects were addressed and remediated prior to production,

they would experience a 32% cost savings.8 In 2008 Original Software conducted a qualitative opinion survey of

senior IT executives and CIOs from organizations with average annual revenue of $5.8billion and an average IT

budget of $764 million. The survey revealed that over 60% of them felt a strong need to improve the quality of

software applications.9

6 See Michael Newman, "Software Errors Cost U.S. Economy $59.5 Billion Annually: NIST Assesses Technical Needs of

Industry to Improve Software-Testing," (2002) http://www.nist.gov/public_affairs/releases/n02-10.htm [April 2010]. See

also Bill Hoffman, "Battling Software Defects One Developer at a Time," (February 23, 2010)

http://www.itbusinessedge.com/cm/community/features/guestopinions/blog/battling-software-defects-one-developer-

at-a-time/?cs=39611 [April 2010].

7 Ibid. Similar concerns were highlighted by Herb Krasner. His research led him to conclude that "[t]ypical manufacturing

CoQ, rang[es] from 5 to 25 percent of company sales, [which] contrasts significantly with CoSQ. With the present state of

software engineering practice we can expect CoSQ to range from 10 to 70 percent of development costs. Even accounting

for the margin between production costs and sales, CoSQ appears to be roughly twice manufacturing CoQ." See Herb

Krasner, "Using the Cost of Quality Approach for Software," (February 2006), p. 7,

http://lifelong.engr.utexas.edu/pdf/sqi/cosq-article-new06.pdf [April 2010].

8 See Melinda-Carol Ballou, "Improving Software Quality to Drive Business Agility," (June 2008)

http://www.coverity.com/library/pdf/IDC_Improving_Software_Quality_June_2008.pdf [April 2010]. This survey was

sponsored by Coverity Inc.

9 See "Software Quality and Testing: A CIO Perspective," http://www.origsoft.com/whitepapers/ [April 2010].

Page 7: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 7

CoQ Service from HCL HCL’s CoQ service is about total quality management of software in the pre or post development phase. It is

most helpful in determining the structure of cost across the lifecycle of software and in developing and

implementing best practices strategies. This service includes:

Thorough assessment of Quality Assurance (QA) activities and cost structure in four areas: planning,

engineering, review and testing, and rework

Detailed gap analysis between corporate vision and existing QA practices in relation to where the

organization concerned wants to be

Root-cause analysis of both costs: good and poor quality

Recommendations for improving CoQ

Identification of CoQ business cases

Implementation of high quality standards

As illustrated in Figure 1, a good part of the CoQ exercise is contingent upon parameters used for determining

the quality of ‘good’ and ‘poor’ quality.

Figure 1: Cost of Quality Parameters

Page 8: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 8

Cost of Good Quality: Appraisal activities, among others, measure, evaluate and audit products or services to

assure conformance to quality standards and performance requirements; their aim is to ensure that software

is free from defects and conforms to customer requirements. Prevention activities ensure that necessary

training, processes, tools, and hardware as well as resources are available for performing appraisal activities.

Since both the activities are pre-emptive measures, they are regarded as the cost of good quality.

Cost of Poor Quality: When detects (arising, for instance, from non-conformance to customer requirements)

are not caught during the development phase of software and are noticed just prior to delivery, they require a

great deal of re-work and re-testing effort. If software defects come to the surface after delivery, the cost of

review and root-cause analysis effort are significantly higher. Irrespective of when the failures come to light

(before or after delivery), they both reflect non-conformance to customer requirements, and are hence

referred to as cost of delivering poor quality software.

The value of being conscious about the cost of good and poor quality is fully realised only when, as shown in

Figure 2, the assessment data obtained from various stakeholders (at a client organization) is entered in HCL’s

CoQ analysis tool.

Figure 2: CoQ Modeling Results

Furthermore, as illustrated in Figure 3 and Figure 4, the HCL analysis tool allows for the quantification and

consolidation of quality parameters. Consequently, senior management at client organizations can then focus

on areas that require further improvement and develop a strategy for keeping the CoQ at acceptable levels.

Page 9: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 9

Figure 3: Quality Parameters Reflecting As-Is Situation

Figure 4: Quality Parameters Reflecting a Reduction in CoQ

Page 10: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 10

The HCL team uses an indexing system to make the results of the CoQ exercise meaningful to client

organizations. For instance, in the sample shown in Figure 5, the index number of 4, (which represents low

quality and high cost) was arrived at following a cost and quality assessment of testing practices at a client

location. The results allowed senior management to identify areas that required attention in order to reduce

cost and improve software quality.

Figure 5: CoQ Indexing

Based on its root cause findings, the HCL CoQ team identifies initiatives for closing the gaps between As-Is and

the desired objectives. As shown in the sample in Figure 6, the proposed initiatives are clubbed into solution

themes (group of initiatives), prioritized for ease of implementation, and then mapped to an IT strategy. The

HCL team also calculates the business benefits of implementing its proposed solution and suggests numerous

options for its implementation such as building the proposed solution in-house organically, partnering with a

third party to obtain the required skills, developing skills with third party via a co-sourcing/out-sourcing

strategy, or out-sourcing the entire solution to another company.

Page 11: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 11

Figure 6: Theme Based Solution Proposition

As illustrated in the sample in Figure 7, the HCL team prioritizes the themes in its proposed solution and maps

them in a 2/2 matrix to show specific business benefits and ease of implementation. This exercise helps client

organizations to focus on recommendations that are not only easy to implement right away but also offer

business benefits (in terms of quality, cost, and productivity). Moreover, as and when required, client

organizations can extend the same recommendations to other projects in phases.

Page 12: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 12

Figure 7: Business Benefits of Proposed Solution

CoQ as an Integral Part of SDLC Although it is widely acknowledged that CoQ principles and practices should be used throughout the lifecycle

of software, most software teams find themselves walking a tightrope between the needs of business teams

(for faster time-to-market) and development teams (who require more time for comprehensive testing).

As shown in Table 1, in cases where development teams have the mandate to integrate CoQ principles and

practices throughout the lifecycle of software, HCL’s CoQ team can assist client organization in several

concrete ways to overcome challenges that are faced by software, testing teams, and allied service teams.

Page 13: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 13

Testing Phases

Phase-Wise Activities Business Benefits to Client

HCL Deliverables Challenges

Planning Requirements analysis and implementation feasibility Identification of tools for test management, defect management, test automation and performance testing

Reduction in effort and cost to fix defects (discovered later) Savings realized on tool cost and rationalization of tools usage across multiple projects

Reports Requirements analysis Tool evaluation Documents Testing estimation Test strategy Microsoft project plan Non functional

requirements Test scenario

identification Technologies and skills

set identification

Ensuring the involvement of the CoQ team right from requirements phase

High Level Test Design

Design analysis and feasibility study Identification of: impact to upstream and

downstream application and aligning testing activities to cover all areas

test environment and test data set-up

System capacity planning and load modeling

Savings in effort and cost related to defect fix at early stage

Reports Detailed design

feasibility analysis High level test

scenario and impact analysis

Documents Requirements

Traceability Matrix Hardware and

software requirements and Test Data control sheet for gathering test data

Understanding software functionality Ensuring the involvement: Domain experts Performance

engineering experts

Detailed Test Design

Detailed test case preparation and review of test cases (by business users/SMEs) to ensure complete coverage of test cases against functionality Performance scenario identification

Avoid spend on rework

Upload test cases in test management tools

Having resources with skills set in testing techniques

Test Execution

Smoke testing to ensure basic functionalities are working such that the entire QA team can be involved for testing all the features System integration testing, regression testing, user acceptance testing, performance testing, Identification of performance bottlenecks and recommendations fixes Script execution in coordination with other resources involved for monitoring

Launch of new features /application within budget and on time

Reports Test summary Daily status Defect status

Having resources who are adept at addressing challenges related to test environment and test data set-up

Page 14: HCLT Research Paper: Cost of Quality

Cost of Quality: Needs Your Attention Not Sometimes But Always Page 14

Testing Phases

Phase-Wise Activities Business Benefits to Client

HCL Deliverables Challenges

Support Defect verification and impact analysis Update test cases related to defect fix Support for enhancement and impact analysis Calculation of Return on Investment ( ROI) based on application readiness, number of release and nature of enhancements Customization of existing HCL test automation framework requiring minimal maintenance

Increase in customer satisfaction resulting from better performance of software Savings in effort and cost due and increase in revenue due to readily available test automation framework and ease of maintenance

Documents ROI based automation Production defect

trend analysis

Collecting data on customer

satisfaction data revenue

generation penalties paid

and support charges

Having resources with advanced knowledge of test automation

Table 1: CoQ and SDLC

Benefits If software products suffer from a high number of defects, low quality, and require a great deal of manual

effort and rework, then it is next to impossible to achieve business objectives such as speed up time-to-market

and lower operational cost. What the CoQ approach from HCL allows client organizations to do is not only

identify problem areas (which result in poor quality of software) but also implement a holistic plan of action

for achieving business objectives and monitoring its progress in a measurable manner on a continuous basis.