Guidelines for the application of Data Envelopment Analysis to assess evolving software Alexander...
-
Upload
paulina-hardeman -
Category
Documents
-
view
216 -
download
3
Transcript of Guidelines for the application of Data Envelopment Analysis to assess evolving software Alexander...
Guidelines for the application of Data Envelopment Analysis
to assess evolving software
Alexander Chatzigeorgiou
University of Macedonia
Thessaloniki, Greece
6th International Workshop on Foundations and Techniques for OSS Certification
Efficiency
1,000,000 €
25,000 €
Efficiency =profit
#employees
Analogy between companies and software
v1.0
v1.2
v2.0
vx.y
. . .
size properties
size properties
size properties
size properties
metrics
metrics
metrics
metrics
Software Context
CBO
LCOM
WMC
DIT
. . .
Software Context (2)
version i
version i+1
Complexity = 5
Complexity = 5
Inputs
The ideal input variable would be a measure of functionality (e.g. FPs)
S o f t w a r e A n a l y s i s a n d D e s i g n
AnalysisPreliminary
DesignDesign...
Functional Requirements ...
software metrics...
(most)
method/attributes
Size metrics can be used as a substitute
How DEA works…
v1
v2
v3
v4
v5
v6
v7
v8
v.1 v.2 v.3 v.4 v.5 v.6 v.7 v.8
NOO (x) 130 150 200 230 300 320 400 450
fan-in (y1) 10.6 11.2 13 13.6 15 15.1 15.8 14
cohesion
(y2)9 12.5 10.5 11 12 10 9 13
v.1 v.2 v.3 v.4 v.5 v.6 v.7 v.8
NOO (x) 1 1 1 1 1 1 1 1
fan-in (y1) 0.08 0.07 0.07 0.06 0.05 0.05 0.04 0.03
cohesion
(y2)0.07 0.08 0.05 0.05 0.04 0.03 0.02 0.03
How DEA works…(2)
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
fan-in coupling / NOO
coh
esio
n /
NO
O
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.10
0
v.2
v.1
v.3v.4
v.5
v.6v.8
v.7
Efficient Frontier
A’
A
Efficiency =0A
0A’v.4
Added value of DEA
01002003004005006007008009001000
8
9
10
11
12
13
14
15
16
0.9.
00.
9.1
0.9.
20.
9.3
0.9.
40.
9.5
0.9.
60.
9.7
0.9.
80.
9.9
0.9.
100.
9.11
0.9.
120.
9.13
0.9.
140.
9.15
0.9.
160.
9.17
0.9.
180.
9.19
0.9.
200.
9.21
fan-in fan-out cohesion NOC
NO
C
fan
-in
, fan
-ou
t, c
oh
esio
n
Versions
improving qualitydegrading quality
improving quality
Added value of DEA (2)
01002003004005006007008009001000
8
9
10
11
12
13
14
15
16
0.9.
00.
9.1
0.9.
20.
9.3
0.9.
40.
9.5
0.9.
60.
9.7
0.9.
80.
9.9
0.9.
100.
9.11
0.9.
120.
9.13
0.9.
140.
9.15
0.9.
160.
9.17
0.9.
180.
9.19
0.9.
200.
9.21
fan-in fan-out cohesion NOC
NO
C
fan
-in
, fan
-ou
t, c
oh
esio
n
Versions
DEA offers a unified view by combining selected design metrics and by normalizing over inputs
DEA is particularly useful when the trends of individual metrics are highly convoluted
Case Study 2
Project: JDeodorant (20 revisions, 4 years of development)Evaluation of: changeability (CBO, LCOM*, DIT **)
** Samoladas I, Gousios G, Spinellis D, Stamelos I. The SQO-OSS quality model: Measurement based open source software evaluation. 4th International Conference on Open Source Systems (OSS'2008).
0
10
20
30
40
50
60
70
80
0
200
400
600
800
1000
1200
1400
1600
1800
2000
1 54 84 152 194 208 223 227 232 244 250 261 266 275 285 295 325 340 343 360
CBO
(10x
), LC
OM
*, D
IT (
10x)
NO
O, N
OA
CBO (10x)
NOO
LCOM*
NOADIT (10x)
Revisions
Case Study 2
is the evolution of DEA scores depicting the truth ?
• If a metrics combination is successful in capturing several aspects associated with the same property (e.g. changeability),
• then it should be demonstrated that the combined value is highly correlated to the desired property.
• However, quality sub-characteristics are not directly quantifiable; that is why a number of metrics are required to assess them.
• To confront this problem, we employed secondary measures, which are related to some extent to the property of interest.
Case Study 2 - JDeodorant
is the evolution of DEA scores depicting the truth ?
0
5
10
15
20
25
30
35
40
Add
ed/M
odifi
ed LO
C to
fix
a si
ngle
bug
Revisions5 14 26 4250
53
51
59 79 94
96
98
99
103 129
133
140
142
163
165
170 190
195
204
209
210 235 248 272259 302 326 369
366
364
Case Study 3
Project: Industrial Telecom System (12 versions, 30 years of development)Evaluation of: quality in terms of the number of faults
0
10
20
30
40
50
60
70
80
90
100
0
50
100
150
200
250
300
1 2 3 4 5 6 7 8 9 10 11 12
Faul
ts
Inte
rfac
es
Interfaces
Programming Faults
Design Faults
Versions
Case Study 3
is the evolution of DEA scores depicting the truth ?
• there is no single property that can be related to the number of design and programming faults,
• the engineers suggested to assess the validity of the extracted scores by means of the number of faults having a priority beyond a certain level (trouble reports with priority A or B).
Case Study 3 - Telecom
is the evolution of DEA scores depicting the truth ?
0
2
4
6
8
10
12
14
1 2 3 4 5 6 7 8 9 10 11 12Faul
ts w
ith
high
est p
rior
ity
leve
l
Versions
Super-efficiency
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
fan-in coupling / NOO
coh
esio
n /
NO
O
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.10
0
v.2
v.1
v.3v.4
v.5
v.6v.8
v.7
Efficient Frontier
A’
A
Efficiencyv.2 > 1
Super-efficiency (2)
0
0.2
0.4
0.6
0.8
1
1.2
1 2 3 4 5 6 7 8 9 10
DEA
Effi
cien
cy S
core
Versions
Telecom
When the basic DEA model leads to multiple versions with score equal to one, the super-efficiency DEA model should be used instead.
Selection of outputs
DE
A E
ffic
ien
cy S
core
versions
fan-in, fan-out, cohesion
Output variables can have varying impact on the extracted DEA scores. When not sure about the necessity to consider a metric, experimentation could highlight whether its impact is significant or not.
No Inputs
Inputs can be neglected by providing a constant input for all DMUs
0
0.2
0.4
0.6
0.8
1
1.20.
9.0
0.9.
10.
9.2
0.9.
30.
9.4
0.9.
50.
9.6
0.9.
70.
9.8
0.9.
90.
9.10
0.9.
110.
9.12
0.9.
130.
9.14
0.9.
150.
9.16
0.9.
170.
9.18
0.9.
190.
9.20
0.9.
21
Effici
ency
Sco
re
Versions
considering input
excluding input
Conclusions
• Main benefits from the application of DEA:
• the ability to provide a unified view of several metrics
• the normalization of the derived scores over size properties of the examined systems
• Eventually, DEA leads to an efficiency score for each examined project enabling the ranking of various versions and the analysis of the evolution.
Thank you for your attention
6th International Workshop on Foundations and Techniques for OSS Certification
How DEA works (math…)
The efficiency score of each version (DMU), is given by maximizing the weighted ratio of all outputs over all inputs, subject to the constraints that the efficiency scores for each DMU must be less than or equal to unity:
4.4.
4.24.24.14.1
,,max
4.4.24.1 vv
vvvv
vuu xv
yuyu
vvv
subject to
14.
24.214.1
iv
iviv
xv
yuyu}8.,...,1.{ vvi
0,, 4.4.24.1 vvv vuu