The Paradox of Software Quality - Why More Bugs Indicate Better Software? · 2013-11-07 · more...

The Paradox of Software Quality

Why More Bugs Indi ate Better Software?

Audris Mo kus

Avaya Labs Resear h

211 Mt Airy Rd

Basking Ridge, NJ 07920

audris�avaya. om

Nov 7, 2013

Motivation

Are there fundamental time-lag relationships among software produ tion

fa tors?

Can they be harnessed to improve software development?

A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 2 / 29

Empiri al Studies of Software Development

Typi al investigated relationships in software, e.g., size and defe ts

◮Are aused by external-to-software fa tors

◮Short term trends: produ t adoption, extent of usage

◮Long term trends: world e onomy, business pra ti e, te hnology

◮Have un lear me hanism of a tion

◮Typi ally not usable in pra ti e


Empiri al Studies of Software Development

Typi al investigated relationships in software, e.g., size and defe ts

◮Are aused by external-to-software fa tors

◮Short term trends: produ t adoption, extent of usage

◮Long term trends: world e onomy, business pra ti e, te hnology

◮Have un lear me hanism of a tion

◮Typi ally not usable in pra ti e

Investigating SW evolution

by observing only software

≡ Divining reality from

shadows on a ave wall


Proposed Solution

◮Investigate short-term and re urring relationships with a lear

me hanism originating from the way software is reated and used

◮Use information from outside software development ave

◮Answer a tual software engineering questions

◮How to to evaluate the e�e tiveness of QA pra ti es?

◮e.g., by omparing two releases of software

◮Do easy-to-get measures, e.g., defe ts, approximate quality?


Approa h

◮Start from lear assumptions

◮Observe fundamental relationships

◮Validate

◮Build more omplex propositions using validated relationships

De�ne: Bug

A user-observed (and reported) program behavior (e.g., failure) that results

in a ode hange.

De�ne: A tion Will Introdu e a Bug

A tion will in rease the han es of a Bug o urring in the future.


Assumed ba kground knowledge

Developers reate software by making hanges to ode

◮All hanges are re orded by a Version Control System

◮A release of software is simply a dynami superposition of hanges

Before:

int i = n;

while(i++)

prinf(" %d", i−−);

After:

//print n integers

int i = n;

while(i++ && i > 0)

prinf(" %d", i−−);

one line deleted

two lines added

two lines un hanged


Assumed ba kground knowledge

Developers reate software by making hanges to ode

◮All hanges are re orded by a Version Control System

◮A release of software is simply a dynami superposition of hanges

Before:

int i = n;

while(i++)

prinf(" %d", i−−);

After:

//print n integers

int i = n;

while(i++ && i > 0)

prinf(" %d", i−−);

one line deleted

two lines added

two lines un hanged

Other attributes: date, developer, defe t number,

submit omment: e.g, �Fix bug 3987 - rashing when menu item is

sele ted�


First Fundamental Law of Software Evolution

Formulation

Code hange will introdu e bugs

Me hanism

◮New ode has defe ts

◮New ode exer ises existing ode di�erently

◮Program behavior hanges

Note: platform hanges ause ode hanges

Eviden e

◮New releases bring new bugs

◮Model: a business-driven feature implementation ode hange leads to

N ∼ Poisson(λ) �xes with delay T ∼ Exp(µ) [1℄


Model predi tion for one release

2001.0 2001.5 2002.0 2002.5

010

2030

4050

Calendar Weeks

Wee

kly C

hang

es (P

erso

n W

eeks

)

New feature changesActual fix changesPredicted fix changes


Model predi tion for 11 releases (using earlier release)

Calendar Weeks

Wee

kly C

hang

es (P

erso

n Wee

ks)

010203040

1994 1996 1998 2000 2002

r1 r7

r2

010203040

r80

10203040

r3 r9

r4

010203040

r100

10203040

r5 r11

r6New feature changesActual fix changesPredicted fix changes


Corollary 1: Need to Normalize by Change to Obtain Quality

How to normalize by hange?

◮Divide by the number of pre-release hanges

◮Divide by the LOC added or hanged

Hypothesis 1

In rease ↑ in the number of ustomer-found defe ts per

pre-release hange (a simple-to-obtain measure) affe ts

users' per eption of software quality negatively ↓


Qualitative eviden e: No

Quotes from a quality manager

�we tried to improve quality: get most experien ed team members to

test, do ode inspe tions, ondu t root ause analysis, ...�

�Did it work? I.e., is this release better than previous one?�

Everyone uses defe t density (e.g., ustomer reported defe ts per 1000

hanges or lines of ode), but �it does not re�e t the feedba k from

ustomers.�


Let's Peek Outside the Software Development Cave


Does the in rease in

the number of users and

the amount of usage

introdu e bugs?


Se ond Fundamental Law of Software Evolution

Formulation

Deploying to more users

will introdu e bugs

Me hanism

◮New use pro�les

◮Di�erent environments

Eviden e

MRs p

er We

ek (P

erson

Mon

ths)

Post Release

0

5

10

15

20

25

30 V 5.6 V 6.0

Release with no users

has no bugs


Third Fundamental Law of Software Evolution

Formulation

Longer (and heavier) use will introdu e bugs

Me hanism

◮New inputs and use ases are

en ountered over longer periods

◮More extreme environmental onditions

happen over longer periods

Eviden e

◮Bugs tend to be en ountered even after year(s) of usage

◮See Commandments below


Third Fundamental Law of Software Evolution

Formulation

Longer (and heavier) use will introdu e bugs

Me hanism

◮New inputs and use ases are

en ountered over longer periods

◮More extreme environmental onditions

happen over longer periods

Eviden e

◮Bugs tend to be en ountered even after year(s) of usage

◮See Commandments below

Does every user and every year of usage introdu e the same

number of bugs?


Commandment 1: Don't Install Right After the Release Date

Formulation

Users who install lose to the release date will

introdu e more bugs

Me hanism

◮Later users get builds with pat hes

◮Servi es team understands better how to install/ on�gure properly

◮Workarounds for many issues are dis overed

Eviden e

0.0 0.2 0.4 0.6 0.8 1.0

0.0

Fraction of customers observing SW issue

Time (years) between launch and deployment

Frac

tion

◮Quality ↑ with time after the laun h,

and is an order of magnitude better

one year later [2℄


Commandment 2: Don't Pani After Install/Upgrade

Formulation

A user will introdu e more bugs lose to their

install/upgrade date

Me hanism

◮Software is not hardware: parts do not wear o�

◮Mis on�guration or in ompatibility with the environment

Eviden e

1st month from install Three months from install

0.0

0.2

0.4

0.6

0.8

1.0

Frac

tion

of C

usto

mer

Issu

es E

ncou

nter

ed

◮Two thirds of ustomer issues (leading

to a software �x) are reported within

three months of install

◮Sample: 87 release/produ t

ombinations


Corollary 1: Customer Quality

Formulation

Software release quality from users perspe tive is the fra tion of:

◮The number of users reporting a bug shortly after the installation over

◮The number of users who install soon after the release date


Corollary 1: Customer Quality

Formulation

Software release quality from users perspe tive is the fra tion of:

◮The number of users reporting a bug shortly after the installation over

◮The number of users who install soon after the release date

�We live or die by this measure�

VP for quality


Testing Hypothesis 1: Defe t Density Re�e ts Customer

Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

M Customer Defects Per Pre−Release change



Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

LC

Customer Defects Per Pre−Release Change% of custmrs with defect within 3m. of install



Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

LC


better

worse



Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

LC


better

worse

better

worse



Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

LC


better

worse

better

worse

worse

better



Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

LC


better

worse

better

worse

worse

better

better

worse



Quality

M

MM

M

M

M

0.00

0.05

0.10

0.15

Qua

ntity

C

C

CC

C

C

r1.1 r1.2 r1.3 r2.0 r2.1 r2.2

LC


better

worse

better

worse

worse

better

better

worse

better

worse

Perfe t anti- orrelation?!


Trying Another Produ t

MM

M M

M

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Qua

ntity

C

C

C

CC

r5 r5.1 r6.0 r7.0 r7.1

MC

Customer Defects Per Pre−Release ChangeCustomer Defects/Installed System

Perfe t anti- orrelation again?!


Why ustomers like high defe t density?



Customers don't are about defe t density

◮Most ustomers try to avoid bugs

◮By not jumping to a major dot zero release

◮By not installing immediately when new release is available

Software salesmen don't are about defe t density

◮They want their ustomers to avoid bugs

◮By warning about produ ts that are likely to ause problems

Software support people don't are about defe t density

◮They want their ustomers to report as few problems as possible

◮By delaying wide installation of new releases


Lemma 1: Major Releases Have Few Customers

Minor releases have two to �ve times more ustomers

12

34

5X

tim

es m

ore

cu

sto

me

rs

Note: based on 38 major and 49 minor releases in 22 produ ts


Commandment 3

Thou Shell Have a Constant Rate of Customer Issues

Me hanism

◮The only thing ustomers like less than a Bug is


Commandment 3


Me hanism


◮The bug that does not get �xed for a long time


Commandment 3


Me hanism


◮The bug that does not get �xed for a long time

◮Team handling ustomer issues an not expand and ollapse

instantaneously and has limited throughput

Eviden e

2011−07 2011−12 2012−05 2012−10 2013−03

Numb

er of n

ew iss

ues for

one p

roduct

050

100150

Monthly numbers of

new ustomer issues is

relatively onstant


Law of Minor Release

Formulation

Minor releases have high defe t density but low han es

that any given ustomer will observe a defe t

De�nition

Major Releases Have More Code Change

Me hanism


Dis ussion

◮There exist Laws of Software Evolution, but

◮Fo us on short-term, repeating relationships with a lear me hanism

◮Look outside SW ave to observe them

◮Chose pra ti al questions

◮Pra ti e hints

◮Development pro ess view does not represent ustomer views

◮Maintenan e � the most important quality improvement a tivity


Referen es I

Audris Mo kus, David M. Weiss, and Ping Zhang.

Understanding and predi ting e�ort in software proje ts.

In 2003 International Conferen e on Software Engineering, pages 274�284, Portland, Oregon, May 3-10 2003.

ACM Press.

Audris Mo kus, Ping Zhang, and Paul Li.

Drivers for ustomer per eived software quality.

In ICSE 2005, pages 225�233, St Louis, Missouri, May 2005. ACM Press.


Abstra t I

The traditional view of software quality fo uses on ounting bugs � issues that are observed and reported by users

and implemented as hanges to the sour e ode. Fewer bugs intuitively (and obviously) imply higher software

quality. This hasty on lusion, however, ignores omplex equilibrium resulting from a tions of di�erent groups of

parti ipants in software produ tion: developers, users, support, and sales. For example, users improve software

quality by dis overing and reporting defe ts that are too ostly to be dis overed otherwise. As new fun tionality is

delivered in major releases, quality ons ious users often stay on the sidelines until a se ond minor release delivers

properly working features, bug �xes, and stability improvements. The major releases, being of lower quality, have

fewer users and, onsequently, fewer bugs. I will dis uss several fundamental laws of software produ tion system that

explain this paradox in a quantitative manner. Ea h law has a lear me hanism of a tion, is grounded in resour e

and physi al onstraints, and is empiri ally validated. The laws provide guidelines on how to measure, understand,

and improve quality of software.


Audris Mo kus

Avaya Labs Resear h

233 Mt. Airy Road

Basking Ridge, NJ 07920

ph: +1 908 696 5608, fax:+1 908 696 5402

http://mo kus.org, mailto:audris�mo kus.org

Audris Mo kus is interested in quantifying, modeling, and improving software development. He

designs data mining methods to summarize and augment software hange data, intera tive

visualization te hniques to inspe t, present, and ontrol the development pro ess, and statisti al

models and optimization te hniques to understand the relationships among people,

organizations, and hara teristi s of a software produ t. Audris Mo kus re eived B.S. and M.S.

in Applied Mathemati s from Mos ow Institute of Physi s and Te hnology in 1988. In 1991 he

re eived M.S. and in 1994 he re eived Ph.D. in Statisti s from Carnegie Mellon University. He

works at Avaya Labs Resear h. Previously he worked in the Software Produ tion Resear h

Department of Bell Labs.


The Paradox of Software Quality - Why More Bugs Indicate Better Software? · 2013-11-07 · more...

Documents

Transcript of The Paradox of Software Quality - Why More Bugs Indicate Better Software? · 2013-11-07 · more...