Post on 06-Aug-2020
The Paradox of Software Quality
Why More Bugs Indi ate Better Software?
Audris Mo kus
Avaya Labs Resear h
211 Mt Airy Rd
Basking Ridge, NJ 07920
audris�avaya. om
Nov 7, 2013
Motivation
Are there fundamental time-lag relationships among software produ tion
fa tors?
Can they be harnessed to improve software development?
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 2 / 29
Empiri al Studies of Software Development
Typi al investigated relationships in software, e.g., size and defe ts
◮Are aused by external-to-software fa tors
◮Short term trends: produ t adoption, extent of usage
◮Long term trends: world e onomy, business pra ti e, te hnology
◮Have un lear me hanism of a tion
◮Typi ally not usable in pra ti e
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 3 / 29
Empiri al Studies of Software Development
Typi al investigated relationships in software, e.g., size and defe ts
◮Are aused by external-to-software fa tors
◮Short term trends: produ t adoption, extent of usage
◮Long term trends: world e onomy, business pra ti e, te hnology
◮Have un lear me hanism of a tion
◮Typi ally not usable in pra ti e
Investigating SW evolution
by observing only software
≡ Divining reality from
shadows on a ave wall
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 3 / 29
Proposed Solution
◮Investigate short-term and re urring relationships with a lear
me hanism originating from the way software is reated and used
◮Use information from outside software development ave
◮Answer a tual software engineering questions
◮How to to evaluate the e�e tiveness of QA pra ti es?
◮e.g., by omparing two releases of software
◮Do easy-to-get measures, e.g., defe ts, approximate quality?
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 4 / 29
Approa h
◮Start from lear assumptions
◮Observe fundamental relationships
◮Validate
◮Build more omplex propositions using validated relationships
De�ne: Bug
A user-observed (and reported) program behavior (e.g., failure) that results
in a ode hange.
De�ne: A tion Will Introdu e a Bug
A tion will in rease the han es of a Bug o urring in the future.
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 5 / 29
Assumed ba kground knowledge
Developers reate software by making hanges to ode
◮All hanges are re orded by a Version Control System
◮A release of software is simply a dynami superposition of hanges
Before:
int i = n;
while(i++)
prinf(" %d", i−−);
After:
//print n integers
int i = n;
while(i++ && i > 0)
prinf(" %d", i−−);
one line deleted
two lines added
two lines un hanged
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 6 / 29
Assumed ba kground knowledge
Developers reate software by making hanges to ode
◮All hanges are re orded by a Version Control System
◮A release of software is simply a dynami superposition of hanges
Before:
int i = n;
while(i++)
prinf(" %d", i−−);
After:
//print n integers
int i = n;
while(i++ && i > 0)
prinf(" %d", i−−);
one line deleted
two lines added
two lines un hanged
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 6 / 29
Assumed ba kground knowledge
Developers reate software by making hanges to ode
◮All hanges are re orded by a Version Control System
◮A release of software is simply a dynami superposition of hanges
Before:
int i = n;
while(i++)
prinf(" %d", i−−);
After:
//print n integers
int i = n;
while(i++ && i > 0)
prinf(" %d", i−−);
one line deleted
two lines added
two lines un hanged
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 6 / 29
Assumed ba kground knowledge
Developers reate software by making hanges to ode
◮All hanges are re orded by a Version Control System
◮A release of software is simply a dynami superposition of hanges
Before:
int i = n;
while(i++)
prinf(" %d", i−−);
After:
//print n integers
int i = n;
while(i++ && i > 0)
prinf(" %d", i−−);
one line deleted
two lines added
two lines un hanged
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 6 / 29
Assumed ba kground knowledge
Developers reate software by making hanges to ode
◮All hanges are re orded by a Version Control System
◮A release of software is simply a dynami superposition of hanges
Before:
int i = n;
while(i++)
prinf(" %d", i−−);
After:
//print n integers
int i = n;
while(i++ && i > 0)
prinf(" %d", i−−);
one line deleted
two lines added
two lines un hanged
Other attributes: date, developer, defe t number,
submit omment: e.g, �Fix bug 3987 - rashing when menu item is
sele ted�
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 6 / 29
First Fundamental Law of Software Evolution
Formulation
Code hange will introdu e bugs
Me hanism
◮New ode has defe ts
◮New ode exer ises existing ode di�erently
◮Program behavior hanges
Note: platform hanges ause ode hanges
Eviden e
◮New releases bring new bugs
◮Model: a business-driven feature implementation ode hange leads to
N ∼ Poisson(λ) �xes with delay T ∼ Exp(µ) [1℄
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 7 / 29
Model predi tion for one release
2001.0 2001.5 2002.0 2002.5
010
2030
4050
Calendar Weeks
Wee
kly C
hang
es (P
erso
n W
eeks
)
New feature changesActual fix changesPredicted fix changes
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 8 / 29
Model predi tion for 11 releases (using earlier release)
Calendar Weeks
Wee
kly C
hang
es (P
erso
n Wee
ks)
010203040
1994 1996 1998 2000 2002
r1 r7
r2
010203040
r80
10203040
r3 r9
r4
010203040
r100
10203040
r5 r11
r6New feature changesActual fix changesPredicted fix changes
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 9 / 29
Corollary 1: Need to Normalize by Change to Obtain Quality
How to normalize by hange?
◮Divide by the number of pre-release hanges
◮Divide by the LOC added or hanged
Hypothesis 1
In rease ↑ in the number of ustomer-found defe ts per
pre-release hange (a simple-to-obtain measure) affe ts
users' per eption of software quality negatively ↓
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 10 / 29
Qualitative eviden e: No
Quotes from a quality manager
�we tried to improve quality: get most experien ed team members to
test, do ode inspe tions, ondu t root ause analysis, ...�
�Did it work? I.e., is this release better than previous one?�
Everyone uses defe t density (e.g., ustomer reported defe ts per 1000
hanges or lines of ode), but �it does not re�e t the feedba k from
ustomers.�
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 11 / 29
Let's Peek Outside the Software Development Cave
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 12 / 29
Does the in rease in
the number of users and
the amount of usage
introdu e bugs?
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 13 / 29
Se ond Fundamental Law of Software Evolution
Formulation
Deploying to more users
will introdu e bugs
Me hanism
◮New use pro�les
◮Di�erent environments
Eviden e
MRs p
er We
ek (P
erson
Mon
ths)
Post Release
0
5
10
15
20
25
30 V 5.6 V 6.0
Release with no users
has no bugs
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 14 / 29
Third Fundamental Law of Software Evolution
Formulation
Longer (and heavier) use will introdu e bugs
Me hanism
◮New inputs and use ases are
en ountered over longer periods
◮More extreme environmental onditions
happen over longer periods
Eviden e
◮Bugs tend to be en ountered even after year(s) of usage
◮See Commandments below
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 15 / 29
Third Fundamental Law of Software Evolution
Formulation
Longer (and heavier) use will introdu e bugs
Me hanism
◮New inputs and use ases are
en ountered over longer periods
◮More extreme environmental onditions
happen over longer periods
Eviden e
◮Bugs tend to be en ountered even after year(s) of usage
◮See Commandments below
Does every user and every year of usage introdu e the same
number of bugs?
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 15 / 29
Commandment 1: Don't Install Right After the Release Date
Formulation
Users who install lose to the release date will
introdu e more bugs
Me hanism
◮Later users get builds with pat hes
◮Servi es team understands better how to install/ on�gure properly
◮Workarounds for many issues are dis overed
Eviden e
0.0 0.2 0.4 0.6 0.8 1.0
0.0
Fraction of customers observing SW issue
Time (years) between launch and deployment
Frac
tion
◮Quality ↑ with time after the laun h,
and is an order of magnitude better
one year later [2℄
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 16 / 29
Commandment 2: Don't Pani After Install/Upgrade
Formulation
A user will introdu e more bugs lose to their
install/upgrade date
Me hanism
◮Software is not hardware: parts do not wear o�
◮Mis on�guration or in ompatibility with the environment
Eviden e
1st month from install Three months from install
0.0
0.2
0.4
0.6
0.8
1.0
Frac
tion
of C
usto
mer
Issu
es E
ncou
nter
ed
◮Two thirds of ustomer issues (leading
to a software �x) are reported within
three months of install
◮Sample: 87 release/produ t
ombinations
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 17 / 29
Corollary 1: Customer Quality
Formulation
Software release quality from users perspe tive is the fra tion of:
◮The number of users reporting a bug shortly after the installation over
◮The number of users who install soon after the release date
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 18 / 29
Corollary 1: Customer Quality
Formulation
Software release quality from users perspe tive is the fra tion of:
◮The number of users reporting a bug shortly after the installation over
◮The number of users who install soon after the release date
�We live or die by this measure�
VP for quality
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 18 / 29
Testing Hypothesis 1: Defe t Density Re�e ts Customer
Quality
M
MM
M
M
M
0.00
0.05
0.10
0.15
Qua
ntity
C
C
CC
C
C
r1.1 r1.2 r1.3 r2.0 r2.1 r2.2
M Customer Defects Per Pre−Release change
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 19 / 29
Testing Hypothesis 1: Defe t Density Re�e ts Customer
Quality
M
MM
M
M
M
0.00
0.05
0.10
0.15
Qua
ntity
C
C
CC
C
C
r1.1 r1.2 r1.3 r2.0 r2.1 r2.2
LC
Customer Defects Per Pre−Release Change% of custmrs with defect within 3m. of install
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 19 / 29
Testing Hypothesis 1: Defe t Density Re�e ts Customer
Quality
M
MM
M
M
M
0.00
0.05
0.10
0.15
Qua
ntity
C
C
CC
C
C
r1.1 r1.2 r1.3 r2.0 r2.1 r2.2
LC
Customer Defects Per Pre−Release Change% of custmrs with defect within 3m. of install
better
worse
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 19 / 29
Testing Hypothesis 1: Defe t Density Re�e ts Customer
Quality
M
MM
M
M
M
0.00
0.05
0.10
0.15
Qua
ntity
C
C
CC
C
C
r1.1 r1.2 r1.3 r2.0 r2.1 r2.2
LC
Customer Defects Per Pre−Release Change% of custmrs with defect within 3m. of install
better
worse
better
worse
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 19 / 29
Testing Hypothesis 1: Defe t Density Re�e ts Customer
Quality
M
MM
M
M
M
0.00
0.05
0.10
0.15
Qua
ntity
C
C
CC
C
C
r1.1 r1.2 r1.3 r2.0 r2.1 r2.2
LC
Customer Defects Per Pre−Release Change% of custmrs with defect within 3m. of install
better
worse
better
worse
worse
better
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 19 / 29
Testing Hypothesis 1: Defe t Density Re�e ts Customer
Quality
M
MM
M
M
M
0.00
0.05
0.10
0.15
Qua
ntity
C
C
CC
C
C
r1.1 r1.2 r1.3 r2.0 r2.1 r2.2
LC
Customer Defects Per Pre−Release Change% of custmrs with defect within 3m. of install
better
worse
better
worse
worse
better
better
worse
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 19 / 29
Testing Hypothesis 1: Defe t Density Re�e ts Customer
Quality
M
MM
M
M
M
0.00
0.05
0.10
0.15
Qua
ntity
C
C
CC
C
C
r1.1 r1.2 r1.3 r2.0 r2.1 r2.2
LC
Customer Defects Per Pre−Release Change% of custmrs with defect within 3m. of install
better
worse
better
worse
worse
better
better
worse
better
worse
Perfe t anti- orrelation?!
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 19 / 29
Trying Another Produ t
MM
M M
M
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Qua
ntity
C
C
C
CC
r5 r5.1 r6.0 r7.0 r7.1
MC
Customer Defects Per Pre−Release ChangeCustomer Defects/Installed System
Perfe t anti- orrelation again?!
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 20 / 29
Why ustomers like high defe t density?
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 21 / 29
Why ustomers like high defe t density?
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 22 / 29
Why ustomers like high defe t density?
Customers don't are about defe t density
◮Most ustomers try to avoid bugs
◮By not jumping to a major dot zero release
◮By not installing immediately when new release is available
Software salesmen don't are about defe t density
◮They want their ustomers to avoid bugs
◮By warning about produ ts that are likely to ause problems
Software support people don't are about defe t density
◮They want their ustomers to report as few problems as possible
◮By delaying wide installation of new releases
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 22 / 29
Lemma 1: Major Releases Have Few Customers
Minor releases have two to �ve times more ustomers
12
34
5X
tim
es m
ore
cu
sto
me
rs
Note: based on 38 major and 49 minor releases in 22 produ ts
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 23 / 29
Commandment 3
Thou Shell Have a Constant Rate of Customer Issues
Me hanism
◮The only thing ustomers like less than a Bug is
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 24 / 29
Commandment 3
Thou Shell Have a Constant Rate of Customer Issues
Me hanism
◮The only thing ustomers like less than a Bug is
◮The bug that does not get �xed for a long time
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 24 / 29
Commandment 3
Thou Shell Have a Constant Rate of Customer Issues
Me hanism
◮The only thing ustomers like less than a Bug is
◮The bug that does not get �xed for a long time
◮Team handling ustomer issues an not expand and ollapse
instantaneously and has limited throughput
Eviden e
2011−07 2011−12 2012−05 2012−10 2013−03
Numb
er of n
ew iss
ues for
one p
roduct
050
100150
Monthly numbers of
new ustomer issues is
relatively onstant
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 24 / 29
Law of Minor Release
Formulation
Minor releases have high defe t density but low han es
that any given ustomer will observe a defe t
De�nition
Major Releases Have More Code Change
Me hanism
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 25 / 29
Law of Minor Release
Formulation
Minor releases have high defe t density but low han es
that any given ustomer will observe a defe t
De�nition
Major Releases Have More Code Change
Me hanism
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 25 / 29
Law of Minor Release
Formulation
Minor releases have high defe t density but low han es
that any given ustomer will observe a defe t
De�nition
Major Releases Have More Code Change
Me hanism
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 25 / 29
Law of Minor Release
Formulation
Minor releases have high defe t density but low han es
that any given ustomer will observe a defe t
De�nition
Major Releases Have More Code Change
Me hanism
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 25 / 29
Law of Minor Release
Formulation
Minor releases have high defe t density but low han es
that any given ustomer will observe a defe t
De�nition
Major Releases Have More Code Change
Me hanism
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 25 / 29
Law of Minor Release
Formulation
Minor releases have high defe t density but low han es
that any given ustomer will observe a defe t
De�nition
Major Releases Have More Code Change
Me hanism
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 25 / 29
Law of Minor Release
Formulation
Minor releases have high defe t density but low han es
that any given ustomer will observe a defe t
De�nition
Major Releases Have More Code Change
Me hanism
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 25 / 29
Dis ussion
◮There exist Laws of Software Evolution, but
◮Fo us on short-term, repeating relationships with a lear me hanism
◮Look outside SW ave to observe them
◮Chose pra ti al questions
◮Pra ti e hints
◮Development pro ess view does not represent ustomer views
◮Maintenan e � the most important quality improvement a tivity
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 26 / 29
Referen es I
Audris Mo kus, David M. Weiss, and Ping Zhang.
Understanding and predi ting e�ort in software proje ts.
In 2003 International Conferen e on Software Engineering, pages 274�284, Portland, Oregon, May 3-10 2003.
ACM Press.
Audris Mo kus, Ping Zhang, and Paul Li.
Drivers for ustomer per eived software quality.
In ICSE 2005, pages 225�233, St Louis, Missouri, May 2005. ACM Press.
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 27 / 29
Abstra t I
The traditional view of software quality fo uses on ounting bugs � issues that are observed and reported by users
and implemented as hanges to the sour e ode. Fewer bugs intuitively (and obviously) imply higher software
quality. This hasty on lusion, however, ignores omplex equilibrium resulting from a tions of di�erent groups of
parti ipants in software produ tion: developers, users, support, and sales. For example, users improve software
quality by dis overing and reporting defe ts that are too ostly to be dis overed otherwise. As new fun tionality is
delivered in major releases, quality ons ious users often stay on the sidelines until a se ond minor release delivers
properly working features, bug �xes, and stability improvements. The major releases, being of lower quality, have
fewer users and, onsequently, fewer bugs. I will dis uss several fundamental laws of software produ tion system that
explain this paradox in a quantitative manner. Ea h law has a lear me hanism of a tion, is grounded in resour e
and physi al onstraints, and is empiri ally validated. The laws provide guidelines on how to measure, understand,
and improve quality of software.
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 28 / 29
Audris Mo kus
Avaya Labs Resear h
233 Mt. Airy Road
Basking Ridge, NJ 07920
ph: +1 908 696 5608, fax:+1 908 696 5402
http://mo kus.org, mailto:audris�mo kus.org
Audris Mo kus is interested in quantifying, modeling, and improving software development. He
designs data mining methods to summarize and augment software hange data, intera tive
visualization te hniques to inspe t, present, and ontrol the development pro ess, and statisti al
models and optimization te hniques to understand the relationships among people,
organizations, and hara teristi s of a software produ t. Audris Mo kus re eived B.S. and M.S.
in Applied Mathemati s from Mos ow Institute of Physi s and Te hnology in 1988. In 1991 he
re eived M.S. and in 1994 he re eived Ph.D. in Statisti s from Carnegie Mellon University. He
works at Avaya Labs Resear h. Previously he worked in the Software Produ tion Resear h
Department of Bell Labs.
A. Mo kus (Avaya Labs Resear h) Quality Paradox Nov 2013 29 / 29