Good & Bad Software Projects...© Quantitative Software Management, Inc. Good and Bad Software...
Transcript of Good & Bad Software Projects...© Quantitative Software Management, Inc. Good and Bad Software...
-
© Quantitative Software Management, Inc.
Good and Bad Software Projects: Good and Bad Software Projects: by the Numbersby the Numbers
Quantitative Software Management, Inc.Quantitative Software Management, Inc.2000 Corporate Ridge, Suite 9002000 Corporate Ridge, Suite 900
Mclean, VA 22102Mclean, VA 22102Tel: 703 790Tel: 703 790--0055, Fax 703 7490055, Fax 703 749--37953795
Email: Email: [email protected][email protected]: Web: www.qsm.comwww.qsm.com
-
© Quantitative Software Management, Inc. #2
Framework
• Correlation is not Causation100% of Convicted Felons Have/Had a Mother and a FatherNo Causal Relationship Between Parenthood and Felony
• “There are three kinds of lies: lies, damned lies, and statistics” – Benjamin Disraeli
• What is Measured Tends to be OptimizedFocusing on a Single Factor (Variable) May Lead to Unintended and Undesirable Results
-
© Quantitative Software Management, Inc. #3
Quotation
The Government is extremely fond of amassing great quantities of statistics. These are raised to the nth degree, the cube roots are extracted and the results are arranged into elaborate and impressive displays. What must be kept in mind, however, is that in every case, the figures are first put down by a village watchman and he puts down anything he damn well pleases.
Sir Josiah Stamp, Her Majesty’s (Queen Victoria) Collector of Inland Revenues, more than a century ago.
-
© Quantitative Software Management, Inc. #4
Outline
• Project Selection• “Normal” Variability for Schedule & Effort• Best Performing & Worst Performing
Projects Defined• Best & Worst Projects Compared• Differentiators• Non-differentiators• Conclusion• Questions?
-
© Quantitative Software Management, Inc. #5
Project Selection
• Business IT (Information Technology) Projects
• Completed in Last 5 Years• Confidence Level of Average or Better in
Quality of Metrics• 1509 Projects• 66 Distinct Organizations (Many Divisions
within Companies)
-
© Quantitative Software Management, Inc. #6
Normal Variability, Time/Cost Trade-off
Service Time Frame Cost
1st Class 1 to 3 days $0.42Priority Mail 2 days $4.80Express Mail Overnight $14.55 - $23.40
U.S. Postal Service 1 Ounce Letter
• You pay a premium for guaranteed quick delivery
• Software functions the same way:
• The relationship between Cost/Effort and Schedule is non-linear
-
© Quantitative Software Management, Inc. #7
Normal VariabilityProject Duration vs Size
1 10 100 1,000Size (thousands)
0.01
0.1
1
10
100
Sch
edule
(Month
s)
Project Effort vs Size
1 10 100 1,0000.01
0.1
1
10
100
1,000
10,000
Perso
n M
onth
s
Size
Dur
atio
nE
ffort
-
© Quantitative Software Management, Inc. #8
Normal Variability, Schedule
Project Duration vs Size
1 10 100 1 0000.01
0.1
1
10
100
+1σ
Average
-1σ
2/3 projects between boundaries
4.52 months
16.69 months
Schedule varies 369% from -1σ to +1σ
Size
Schedule
-
© Quantitative Software Management, Inc. #9
Normal Variability, Effort
Project Effort vs Size
1 10 100 1 0000.01
0.1
1
10
100
1,000
10,00
+1σ
Average
-1σ
25.8 effort months
231.3 effort months
Effort (cost) varies 890% from -1σ to +1σ
Effort
Size
-
© Quantitative Software Management, Inc. #10
Best & Worst Defined
• Projects Often Optimize Schedule at the Expense of Effort (Cost) or Vice-Versa
Time/Effort Trade-off
• Some Projects Optimize Both; Others Fail on Both Counts
• Best Projects are Defined as Being One Standard Deviation or More Better than Average for Both Schedule and Effort
Worst Projects are Just the Opposite
-
© Quantitative Software Management, Inc. #11
Best & Worst DefinedProject Duration vs Size
1 10 100 1,000Size (thousands)
0.01
0.1
1
10
100
Sch
edule
(Month
s)
Project Effort vs Size
1 10 100 1,000Size (thousands)
0.01
0.1
1
10
100
1,000
10,000
Perso
n M
onth
s
Best projects 5.4% of total
Worst projects 4.5% of total
-
© Quantitative Software Management, Inc. #12
Best & Worst Compared: Cost/Effort• For a 10,000 line of code project, the Worst projects average 30.6 times as much effort
For a 100,000 line of code project the Worst projects average 23 times as much effort
• Effort is usually the largest cost component in software development
Project Effort vs Size
1 10 100 1,000
Size (thousands)
0.001
0.01
0.1
1
10
100
1,000
10,000
Pe
rson
Mon
ths
-
© Quantitative Software Management, Inc. #13
Best & Worst Compared: Duration• Worst projects’schedules are 7.3 times as long for a 10,000 line of code project
5.8 times as long for a 100,000 line of code project
Project Duration vs Size
1 10 100 1,000
Size (thousands)
0.01
0.1
1
10
100
Sch
edu
le (M
on
ths)
-
© Quantitative Software Management, Inc. #14
Best & Worst Compared: Avg. Staff• Worst projects had significantly higher staffing levels
4.2 times greater for a 10,000 line of code project3.8 times greater for a 100,000 line of code project
• All Best Projects had average staff less than 6
Average Staff vs. Size
1 10 100 1,000
Size (thousands)
0.001
0.01
0.1
1
10
100
1,000
Avera
ge S
taff
-
© Quantitative Software Management, Inc. #15
Best & Worst Compared: Quality 1• Worst projects have far more defects
13.3 times as many for a 10,000 line of code project29.5 times as many for a 100,000 line of code project
• Quality difference increases with project size
Pre-Implementation Defects
0.1 1 10 100 1,000
Effective SLOC (thousands)
0.001
0.01
0.1
1
10
100
1,000
10,000
100,000
Defe
ct Cou
nt
-
© Quantitative Software Management, Inc. #16
Best & Worst Compared: Quality 2• Few Worst Projects report post-implementation defects• Best Projects trend parallels entire data set; but is slightly better
MTTD (Post-Implementation Quality)
1 10 100 1,000
Effective SLOC (thousands)
0.001
0.01
0.1
1
10
100
1,000
MTT
D 1
st Mo
nth
(Da
ys)
-
© Quantitative Software Management, Inc. #17
Best & Worst Development Type
% Best Projects by Development Classification
(Unknown)
Conversion (
-
© Quantitative Software Management, Inc. #18
Best & Worst Development Type• Best projects likely to be New
Development or Minor Enhancements• Worst projects are disproportionately
Major Enhancements
-
© Quantitative Software Management, Inc. #19
Best & Worst Phase Effort Percentages
Best Projects Effort % by Phase
30.9%
69.1%
0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0%
Analysis/Design
Build & Test
Worst Projects Effort % by Phase
10.4%
89.6%
0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0%
Analysis/Design
Build & Test
• Best projects allocate nearly 3 times as much effort to Analysis & Design on a percentage basis
-
© Quantitative Software Management, Inc. #20
Best & Worst: Difficulty
Average Value of Metrics
Overall Difficulty
Data Complexity
Customer Interface
Ext Sys Interface
Integration Complex.
Doc Rqmts
H/W Stability
Sys s/w stability
0 1 2 3 4 5 6 7 8 9 10
Average
5.3
5.7
3.1
2.3
4.0
4.4
9.0
8.4
7.8
5.2
5.8
3.7
4.2
7.2
8.7
7.2
-
© Quantitative Software Management, Inc. #21
Best & Worst: Personnel
Average Value of Metrics
Overall People
Mgmt Eff.
Training Avail.
Staff Turnover
Dev Team Skill
Knowledge
App. Experience
Motivation
Cohesiveness
Communication
0 1 2 3 4 5 6 7 8 9 10
Average
6.9
7.5
4.0
1.8
7.4
7.5
6.3
8.7
7.8
3.3
6.8
5.2
5.3
4.8
5.3
5.3
5.2
5.0
5.5
5.7
-
© Quantitative Software Management, Inc. #22
Best & Worst:ToolsAverage Value of Metrics
Overall Tools
Testing Tools Cap.
Program. Tools Cap.
Config Mgt Tools Cap
Proj Mgmt Tools Cap.
Dev Std Robustness
Dev Std Adherence
Dev Std Exp.
Dev Std Adaptability
0 1 2 3 4 5 6 7 8 9 10
Average
6.0
4.3
8.2
5.3
5.9
5.6
7.2
7.0
7.6
5.8
4.7
6.2
6.8
6.8
6.7
5.5
6.5
6.0
-
© Quantitative Software Management, Inc. #23
DifferentiatorsAverage Value of Metrics
Mgmt Eff.
Staff Turnover
Dev Team Skill
Motivation
Cohesiveness
Communication
Knowledge
Overall Difficulty
Customer Interface
Doc Rqmts
Program. Tools Cap.
0 1 2 3 4 5 6 7 8 9 10
Average
7.5
1.8
7.4
8.7
7.8
3.3
7.5
5.3
3.1
4.4
8.2
5.2
4.8
5.3
5.0
5.5
5.7
5.3
7.8
5.8
7.2
6.2
People, Communication, Knowledge
Complexity
Tools
-
© Quantitative Software Management, Inc. #24
Non-Differentiators
Average Value of Metrics
Data Complexity
Integration Complex.
H/W Stability
Sys s/w stability
Overall Tools
Proj Mgmt Tools Cap.
Dev Std Exp.
0 1 2 3 4 5 6 7 8 9 10
Average
5.7
4.0
9.0
8.4
6.0
5.9
7.0
5.2
4.2
8.7
7.2
5.8
6.8
6.5
-
© Quantitative Software Management, Inc. #25
Non-DifferentiatorsNumber of Projects vs Primary Lang
(Unknown)ABAP
ACCESSAdvantage:Gen
ASPASSEMBLER
C#C++
COBOLCold Fusion
DCLDELPHI
HTMLINGRES
JAVAJava Script
LOTUS NOTELOTUS ScriptManual Test
ORACLEORACLE DEV 2000
PL/1PL/SQL
SQLTRANSACT
VB in ACCESSVBScript
Visual BasicVisual Page
VPF
Prim
ary
La
ng
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Number of Projects
8511111
6 13611 31 31
321111 3311
2 612
Number of Projects vs Primary Lang(Unknown)
ASPBaanC++
COBOLCOGEN
IEFIEF/COOL:GEN
JAVAJCLKSH
MONKOpenROAD
ORACLEORACLE DEV 2000
PACBASESMALLTALKSource File
SQLVB
Visual Basic
Prim
ary
Lan
g
0 5 10 15 20 25 30 35 40
Number of Projects
3611
37
211
411111111111
2
-
© Quantitative Software Management, Inc. #26
Conclusions
• Projects of the same size and complexity can vary dramatically in cost and schedule
• Major enhancements are a “mine field”: comprising half of worst performing projects
• The amount of effort spent in Analysis & Design is a key differentiator between Best and Worst projects
• Social and leadership factors seem to contribute more to project success or failure than technical ones
• Programming language is not a key differentiator
-
© Quantitative Software Management, Inc. #27
Questions??
Good and Bad Software Projects: by the NumbersFrameworkQuotationOutlineProject SelectionNormal Variability, Time/Cost Trade-offNormal VariabilityNormal Variability, ScheduleNormal Variability, EffortBest & Worst DefinedBest & Worst DefinedBest & Worst Compared: Cost/EffortBest & Worst Compared: DurationBest & Worst Compared: Avg. StaffBest & Worst Compared: Quality 1Best & Worst Compared: Quality 2Best & Worst Development TypeBest & Worst Development TypeBest & Worst Phase Effort PercentagesBest & Worst: DifficultyBest & Worst: PersonnelBest & Worst:ToolsDifferentiatorsNon-DifferentiatorsNon-DifferentiatorsConclusionsQuestions??