The Economics of Software Reliability

1

11/19/03 ©USC-CSE 1

University of Southern CaliforniaCenter for Software EngineeringC S E

USC

Barry Boehm, USC

ISSRE 2003 Keynote Address

November 19, 2003([email protected])

(http://sunset.usc.edu)

The Economics of Software Reliability

11/19/03 ©USC-CSE 2


USC

Outline

• The business case for software reliability– Example: net value of test aids– Value depends on stakeholder value propositions

• What are stakeholders really relying on?– Safety, accuracy, response time, …– Attributes often conflict

• Software dependability in a competitive world• Conclusions

2

11/19/03 ©USC-CSE 3


USC

Software Reliability Business Case• Software reliability investments compete for

resources– With investments in functionality, response time,

adaptability, speed of development, …

• Each investment option generates curves of net value and ROI– Net Value NV = PV(benefit flows-cost flows)– Present Value PV = Flows at future times discounted– Return on Investment ROI = NV/PV(cost flows)

• Value of benefits varies by stakeholder and role

11/19/03 ©USC-CSE 4


USC

Software Testing Business Case

• Vendor proposition– Our test data generator will cut your test costs in

half– We’ll provide it to you for 30% of your test costs– After you run all your tests for 50% of your

original costs, you’re 20% ahead

• Any concerns with vendor proposition?

3

11/19/03 ©USC-CSE 5


USC

Software Testing Business Case

• Vendor proposition– Our test data generator will cut your test costs in half– We’ll provide it to you for 30% of your test costs– After you run all your tests for 50% of your original costs,

you’re 20% ahead

• Any concerns with vendor proposition?– Test data generator is value-neutral*– Every test case, defect is equally important– Usually, 20% of test cases cover 80% of business value

* As are most current software engineering techniques

11/19/03 ©USC-CSE 6


USC

20% of Features Provide 80% of Value: Focus Testing on These (Bullock, 2000)

% of Valuefor

CorrectCustomer

Billing

Customer Type

100

80

60

40

20

5 10 15

Automated test generation tool- all tests have equal value

4

11/19/03 ©USC-CSE 7


USC

Value-Based Testing Provides More Net Value

Net Value

NV

60

40

20

0

-20

20 40 10060 80

-40

(100, 20)

Percent of tests run

Test Data Generator

Value-Based Testing(30, 58)

0100100+2010080100

….….….….….….….

549440-10405040

588830-15304530

557520-20204020

405010-25103510

000-300300

NVValueCostNVValueCost

Value-Based TestingTest Data Generator% Tests

11/19/03 ©USC-CSE 8


USC

There is No Universal Dependability-Value Metric

• Different stakeholders rely on different value attributes– Protection: safety, security, privacy– Robustness: reliability, availability, survivability– Quality of Service: performance, accuracy, ease of use– Adaptability: evolvability, interoperability– Affordability: cost, schedule, reusability

• Value attributes continue to tier down– Performance: response time, resource consumption (CPU, memory,

comm.)

• Value attributes are scenario-dependent– 5 seconds normal response time; 2 seconds in crisis

• Value attributes often conflict– Most often with performance and affordability

5

11/19/03 ©USC-CSE 9


USC

Major Information System Dependability Stakeholders

•Information Suppliers-citizens, companies

•Information Brokers

•Information Consumers

-financial services,

-decisions, education,

•Mission Controllers

Developers, Acquirers, Administrators

•Dependents

Information System

- passengers, patients

news media

entertainment

-pilots, distributioncontrollers

11/19/03 ©USC-CSE 10


USC

Overview of Stakeholder/Value Dependencies

• Strength of direct dependency on value attribute**- Critical ; *-Significant; blank-insignificant or indirect

AttributesStakeholders

**

*

** ** **

**

** **

**

**

**

***

*

*

**

* *

*

Prote

ction

Robus

tness

Quality

of S

ervic

e

Adapta

bility

Afford

abilit

y

Developers, AcquirersMission Controllers, Administrators

Info. ConsumersInfo. Brokers

Info. Suppliers, Dependents

6

11/19/03 ©USC-CSE 11


USC

D-Attr ibu tes

Info

Su

pp

liers

Dep

end

ents

Info

Bro

kers

Co

nsu

mers

Info

Missio

n C

on

trollers

Develo

pers, M

ainten

ance

Ad

min

istrators

Acq

uirers

Mi ssi on -Protection cri t uncri t

Sa fe ty **Securi ty * ** **Pri vacy ** *

RobustrossRe li ab i l i ty ** ** ** **Ava i l ab i l i ty * ** ** ** ** **Survivabi l i ty ** ** ** **

Qual i ty o f ServicePerfo rm ance ** ** ** ** *-**Accuracy, Consi stency ** ** ** * ** *Accessib i l i ty, ease o f use ; ** ** ** ** ** **d iff icul ty o f m isuse

Evo lvab i l i ty ** ** * * ** ** **In teroperab i l i ty ** ** **

Cost * **Schedu le * ** **Reusabi l i ty ** * *

Stakeholders

Elaboration of Stakeholder/Value Dependencies

11/19/03 ©USC-CSE 12


USC

Implications for Dependability Engineering

• There is no universal dependability metric to optimize

• Need to identify system’s success-critical stakeholders– And their dependability priorities

• Need to balance satisfaction of stakeholder dependencies– Stakeholder win-win negotiation– Dependability attribute tradeoff analysis

• Need value-of-dependability models, methods, and tools

7

11/19/03 ©USC-CSE 13


USC

Outline

• The business case for software reliability• What are stakeholders really relying on?• Software dependability in a competitive world

– Is market success a monotone function of dependability?

– Is quality really free?– Is “ faster, better, cheaper” really achievable?– Value-based vs. value-neutral methods

• Conclusions

11/19/03 ©USC-CSE 14


USC

Competing on Cost and Quality- adapted from Michael Porter, Harvard

Too cheap (low Q)

Costleader

(acceptable Q)

Stuck inthe middle Quality

leader(acceptable C)

Over-priced

Priceor CostPoint

ROI

8

11/19/03 ©USC-CSE 15


USC

Competing on Schedule and Quality- A risk analysis approach

• Risk Exposure RE = Prob (Loss) * Size (Loss)

– “ Loss” – financial; reputation; future prospects, …

• For multiple sources of loss:

sourcesRE = ΣΣΣΣ [Prob (Loss) * Size (Loss)]source

11/19/03 ©USC-CSE 16


USC

Example RE Profile: Time to Ship- Loss due to unacceptable dependability

Time to Ship (amount of testing)

RE =P(L) * S(L)

Many defects: high P(L)Crit ical defects: high S(L)

Few defects: low P(L)Minor defects: low S(L)

9

11/19/03 ©USC-CSE 17


USC

Example RE Profile: Time to Ship- Loss due to unacceptable dependability

- Loss due to market share erosion


RE =P(L) * S(L)

Few rivals: low P(L)Weak rivals: low S(L)

Many r ivals: high P(L)Strong r ivals: high S(L)



11/19/03 ©USC-CSE 18


USC

Example RE Profile: Time to Ship- Sum of Risk Exposures


RE =P(L) * S(L)

Few rivals: low P(L)Weak rivals: low S(L)

Many r ivals: high P(L)Strong r ivals: high S(L)

SweetSpot



10

11/19/03 ©USC-CSE 19


USC

Comparative RE Profile: Safety-Critical System


RE =P(L) * S(L)

Mainstream Sweet

Spot

Higher S(L): defects

High-QSweetSpot

11/19/03 ©USC-CSE 20


USC

Comparative RE Profile: Internet Startup


RE =P(L) * S(L)

Mainstream SweetSpot

Higher S(L): delaysLow-TTM

SweetSpot

TTM:Time to Market

11

11/19/03 ©USC-CSE 21


USC

Interim Conclusions• Unwise to try to compete on both

cost/schedule and quality– Some exceptions: major technology or marketplace

edge

• There are no one-size-fits-all cost/schedule/quality strategies

• Risk analysis helps determine how much testing (prototyping, formal verification, etc.) is enough– Buying information to reduce risk

• Often difficult to determine parameter values– Some COCOMO II values discussed next

11/19/03 ©USC-CSE 22


USC

0.8 0.9 1.0 1.1 1.2 1.3

Slight inconvenience

Low, easily recoverable

loss

Moderate recoverable

loss

High Financial

Loss

Loss of Human Life

1 month

1 day

2 years

100 years

Defect Risk Rough MTBF(mean time between failures)

In-house support software1.0

Relative Cost/Source Instruction

Software Development Cost/Quality Tradeoff- COCOMO II calibration to 161 projects

1 hour

High

RELYRating

VeryHigh

Nominal

Low

VeryLow

12

11/19/03 ©USC-CSE 23


USC

0.8 0.9 1.0 1.1 1.2 1.3



loss


loss

High Financial

Loss

Loss of Human Life

1 month

1 day

2 years

100 years


Commercial quality leader

1.10


Commercial cost leader

0.92



1 hour

High

RELYRating

VeryHigh

Nominal

Low

VeryLow

11/19/03 ©USC-CSE 24


USC

0.8 0.9 1.0 1.1 1.2 1.3


(1 hour)


loss


loss

High Financial

Loss

Loss of Human Life

1 month

1 day

2 years

100 years


Commercial quality leader

1.10


Commercial cost leader

0.92

0.82

Startup demo

Safety-critical

1.26



High

RELYRating

VeryHigh

Nominal

Low

VeryLow

13

11/19/03 ©USC-CSE 25


USC

“ Quality is Free”

• Did Philip Crosby’s book get it all wrong?

• Investments in dependable systems

– Cost extra for simple, short-life systems

– Pay off for high-value, long-life systems

11/19/03 ©USC-CSE 26


USC

Software Life-Cycle Cost vs. Dependability

0.8

VeryLow

Low Nominal HighVeryHigh

0.9

1.0

1.1

1.2

1.3

1.4

1.10

1.0

0.92

1.26

0.82

Relative Cost to Develop

COCOMO II RELY Rating

14

11/19/03 ©USC-CSE 27


USC


0.8

VeryLow


0.9

1.0

1.1

1.2

1.3

1.4

1.10

0.92

1.26

0.82

Relative Cost to Develop, Maintain


1.23

1.10

0.99

1.07

11/19/03 ©USC-CSE 28


USC


0.8

VeryLow


0.9

1.0

1.1

1.2

1.3

1.4

1.10

0.92

1.26

0.82

Relative Cost to Develop, Maintain


1.23

1.10

0.99

1.07

1.11

1.05

70% Maint.

1.07

1.20

15

11/19/03 ©USC-CSE 29


USC

Software Ownership Cost vs. Dependability

0.8

VeryLow


0.9

1.0

1.1

1.2

1.3

1.4

1.10

0.92

1.26

0.82

Relative Cost to Develop, Maintain,Own andOperate


1.23

1.10

0.99

1.07

1.11

1.05

70% Maint.

1.07

1.20

0.760.69

VL = 2.55L = 1.52

Operational-defect cost at Nominal dependabili ty= Software life cycle cost

Operational -defect cost = 0

11/19/03 ©USC-CSE 30


USC

Outline• The business case for software reliability• What are stakeholders really relying on?• Software dependability in a competitive world

– Is market success a monotone function of dependability?

– Is quality really free?– Is “ faster, better, cheaper” really achievable?– Value-based vs. value-neutral methods

• Conclusions

16

11/19/03 ©USC-CSE 31


USC

Cost, Schedule, Quality: Pick any Two?

C

QS

11/19/03 ©USC-CSE 32


USC

Cost, Schedule, Quality: Pick any Two?

C

QS

C

QS

• Consider C, S, Q as Independent Variable– Feature Set as Dependent Variable

17

11/19/03 ©USC-CSE 33


USC

C, S, Q as Independent Variable• Determine Desired Delivered Defect Density (D4)

– Or a value-based equivalent• Prioritize desired features

– Via QFD, IPT, stakeholder win-win• Determine Core Capability

– 90% confidence of D4 within cost and schedule– Balance parametric models and expert judgment

• Architect for ease of adding next-priority features– Hide sources of change within modules (Parnas)

• Develop core capability to D4 quality level– Usually in less than available cost and schedule

• Add next priority features as resources permit• Versions used successfully on 32 of 34 USC digital library

projects

11/19/03 ©USC-CSE 34


USC

Value-Based vs. Value Neutral Methods

• Value-based defect reduction• Constructive Quality Model (COQUALMO)• Information Dependability Attribute Value

Estimation (iDAVE) model

18

11/19/03 ©USC-CSE 35


USC

Value-Based Defect Reduction Example:

Goal-Question-Metric (GQM) Approach

Goal: Our supply chain software packages have too many defects. We need to get their defect rates down

Question: ?

11/19/03 ©USC-CSE 36


USC

Value-Based GQM Approach – IQ: How do software defects affect system value

goals?ask why initiative is needed

- Order processing- Too much downtime on operations critical path- Too many defects in operational plans- Too many new-release operational problems

G: New system-level goal: Decreasesoftware-defect-related losses in operationaleffectiveness

- With high-leverage problem areas above as specific subgoals

New Q: ?

19

11/19/03 ©USC-CSE 37


USC

Value-Based GQM Approach – IINew Q: Perform system problem-area root cause analysis:

ask why problems are happening via modelsExample: Downtime on critical path

Order Validateorder

Produce status reports

Schedule packaging,

delivery

Validateitems in

stockitems

Prepare delivery

packages

Deliver

order

• Where are primary software-defect-related delays?• Where are biggest improvement-leverage areas?

– Reducing software defects in Scheduling module– Reducing non-software order-validation delays– Taking Status Reporting off the critical path– Downstream, getting a new Web-based order entry system

• Ask “ why not?” as well as “ why?”

11/19/03 ©USC-CSE 38


USC

Value-Based GQM Results• Defect tracking weighted by system-value priority

– Focuses defect removal on highest-value effort

• Significantly higher effect on bottom-line business value– And on customer satisfaction levels

• Engages software engineers in system issues– Fits increasing system-criticality of software

• Strategies often helped by quantitative models– COQUALMO, iDAVE

20

11/19/03 ©USC-CSE 39


USC

COCOMO II

Current COQUALMO System

COQUALMO

DefectIntroduction

Model

DefectRemoval

Model

Software platform, Project, product and personnel attributes

Software Size Estimate

Defect removal profile levelsAutomation, Reviews, Testing

Software development effort, cost and schedule estimate

Number of residual defectsDefect density per unit of size

11/19/03 ©USC-CSE 40


USC

Defect Removal Rating Scales

Highly advanced

tools, model-based test

More advance test tools,

preparation.Dist-

monitoring

Well-defined test seq. and

basic test coverage tool

system

Basic testTest criteria

based on checklist

Ad-hoc test and debug

No testingExecution Testing and

Tools

Extensive rev iew

checklistStatistical

control

Root cause analysis,

formal followUsing

historical data

Formal review roles and Well-trained people

and basic checklist

Well-defined preparation,

rev iew, minimal

follow-up

Ad-hoc informal walk-

through

No peer rev iew

Peer Reviews

Formalized specification, verification.Advanced

dist-processing

More elaborate

req./designBasic dist-processing

Intermediate-level module

Simple req./design

Compiler extension

Basic req. and design

consistency

Basic compiler capabilit ies

Simple compiler syntax

checking

Automated Analysis

Extra HighVery HighHighNominalLowVery Low

COCOMO II p.263

21

11/19/03 ©USC-CSE 41


USC

Defect Removal Estimates- Nominal Defect Introduction Rates

60

28.5

14.37.5

3.5 1.60

10

20

30

40

50

60

70

VL Low Nom High VH XH

Delivered Defects/ KSLOC

Composite Defect Removal Rating

11/19/03 ©USC-CSE 42


USC

iDAVE Model

22

11/19/03 ©USC-CSE 43


USC

ROI Analysis Results Comparison - Bu sin ess Appl i cat ion vs. Space Appl ication

0. 32

- 0. 98- 2

0

2

4

6

8

10

12

14

16

N- >H H- >VH VH- >XH

Dependability Investment Levels

RO

I

Sierra Order Processing Planetary Rover

11/19/03 ©USC-CSE 44


USC

Conclusions: Software Reliability (SWR)• There is no universal SWR metric to optimize

– Need to balance stakeholder SWR value propositions• Increasing need for value-based approaches to SWR

– Methods and models emerging to address needs• “ Faster, better, cheaper” is feasible

– If feature content can be a dependent variable• “ Quality is free” for stable, high-value, long-life systems

– But not for dynamic, lower-value, short life systems• Future trends intensify SWR needs and challenges

– Criticality, complexity, decreased control, faster change

The Economics of Software Reliability

Documents

Transcript of The Economics of Software Reliability