Multi-criteria Decision Analysis for Multi-criteria Decision Analysis for Customization of Estimation by Customization of Estimation by

AnalogyAnalogyMethod AQUAMethod AQUA++

Jingzhou LiJingzhou Li

Guenther RuheGuenther Ruhe

University of Calgary, CanadaUniversity of Calgary, Canada

PROMISE’08, May 13, 2008PROMISE’08, May 13, 2008

22/14/14

Why this Paper?Why this Paper?

Practitioners need better advice on how and whenPractitioners need better advice on how and when

to use methodologiesto use methodologies Universal, project-independent methodologies Universal, project-independent methodologies

are characterized as “weak” in the field of problem solvingare characterized as “weak” in the field of problem solving

(Robert Glass)(Robert Glass) EBA is no exception in that respect!EBA is no exception in that respect! But: How to figure out which variant works best when?But: How to figure out which variant works best when? We do NOT claim to “solve” this problemWe do NOT claim to “solve” this problem The paper describes an approach to make progressThe paper describes an approach to make progress

on the question of customizationon the question of customization Approach is: Multi-criteria decision analysisApproach is: Multi-criteria decision analysis

33/14/14

AQUAExisting EBA

Predicting

Phase2Effort

estimates

Objects under

estimation

Learning

Phase1

Prediction accuracy distributio

n

1. Proposed EBA method AQUA1. Proposed EBA method AQUA++——ArchitectureArchitecture

Data set for

AQUA+

44/14/14

AQUA+

AQUAExisting EBA

Learning

Phase1

Predicting

Phase2Effort

estimates

Prediction accuracy distributio

n

Attribute weighting and

selection

Phase0

Objects under

estimation

Attributes & weights

Predicting

Phase2Effort

estimates

Objects under

estimation

Data set for

AQUA+

1. Proposed EBA method AQUA1. Proposed EBA method AQUA++——ArchitectureArchitecture

Raw historica

l dataDetermining

attribute types

Pre-Phase

• Supports non-quantitative attributes• Tolerates missing values• Determines the number of analogies for adaptation by learning

• Proposes new evaluation criteria

• Attribute weighting and selection using RSA

• Four heuristics: H1 to H4

55/14/14

2. Decision-centric process model of 2. Decision-centric process model of EBAEBA

D8.Determining

closest analogs

Processed

Historical Data

D2.Dealing with

missing values

D1.Impact

analysis of missing values

D7.Retrieving

analogs

Objects Under

Estimation

Effort Estimates

D9.Analogy

adaptation

D11. Comparing EBA methods in generalD10. Choosing evaluation criteria

D6.Determining

similarity measures

Raw Historical

Data

D3.Object

selection

D5.Attribute

weighting & selection

D4. Discretization of attributes

66/14/14

EBA(DB) = C(D, DB, Ch)

Data set

type 1

Data set

type 2

Data set

type k

Customization 1

Customization 2

Customization k

……

Cla

ssifica

tion a

ccord

ing to

chara

cteristics o

f the

data

sets

Si.j for Di ?

3. Customization of EBA 3. Customization of EBA — why?— why?

D = {D1, D2, …, D11},

Di = {Si.j | solution alternatives of task Di}

DB: a historical data set for EBA

Ch: a set of characteristics describing DB

77/14/14

USP05-RQ

USP05-FT

ISBSG04-2Kem87

Desh89

Mends03

-5

15

35

55

75

95

-2 3 8 13 18 23 28

%Missing values

%N

on-q

uan

tita

tive

att

rib

ute

s

New Data Set

Which heuristic

should be used?

4. Customization of EBA 4. Customization of EBA — how?— how?

-1

-0.5

0

0.5

1

1.5

2USP05-RQ

USP05-FT

ISBSG04-2

Mends03

Kem87

Desh89

H0

H1

H3

H4

Empirical knowledge gained from empirical studies.

88/14/14

Decision problemDecision problem: The selection of attribute weighting : The selection of attribute weighting heuristic expected to be the best for a data set of given heuristic expected to be the best for a data set of given characteristicscharacteristics

Decision alternativesDecision alternatives: Attribute weighting heuristics : Attribute weighting heuristics known from literature. Six heuristics were studied. known from literature. Six heuristics were studied.

Evaluation methodEvaluation method: The alternative heuristics are : The alternative heuristics are evaluated by applying them to different data sets for evaluated by applying them to different data sets for AQUAAQUA++. Six publicly available data sets were used. . Six publicly available data sets were used.

Evaluation criteriaEvaluation criteria: : MMREMMRE, , Pred Pred [6], and [6], and Strength Strength [3]. [3]. In order to keep the criteria consistent for minimization, In order to keep the criteria consistent for minimization, MMREMMRE, 1-, 1-PredPred, and 1-, and 1-StrengthStrength were used. were used.

Decision objectiveDecision objective: Determine solution alternatives : Determine solution alternatives (heuristics) such that evaluation criteria (heuristics) such that evaluation criteria MMREMMRE, 1-, 1-PredPred, , and 1-and 1-StrengthStrength get minimized in a balanced manner. get minimized in a balanced manner.

5. Multi-criteria Decision Problem5. Multi-criteria Decision Problem

99/14/14

Definition 1Definition 1. . MREMRE((rkrk)—Magnitude of Relative Error [6])—Magnitude of Relative Error [6]

DefinitionDefinition 22. . MMRE(NMMRE(N,, T) T)—Mean Magnitude of Relative Error [6] —Mean Magnitude of Relative Error [6]

DefinitionDefinition 33. . Pred Pred ((αα,, N N,, T T)—prediction at level )—prediction at level αα [6] , [6] , α = 0.25α = 0.25N - number of analogs, T – similarity thresholdN - number of analogs, T – similarity threshold

DefinitionDefinition 44. . Strength(NStrength(N,, T) T) Support(NSupport(N,, T) T) is the number of objects in is the number of objects in RR that can be estimated that can be estimated

with a given values of (with a given values of (NN, , TT). ). Strength(NStrength(N,, T) T) is then defined as the is then defined as the ratio of ratio of SupportSupport to the total number of objects in to the total number of objects in RR. .

6. Definition of Decision Criteria 6. Definition of Decision Criteria

²( ) ( )

( )

k k

k

EffortEffort r r

Effort r

( )1

k

k

MRE r

r Rn

Pred(α, N, T)= Pred(α, N, T)=

MMRE(NMMRE(N,, T) T) ==

MREMRE((rkrk)=)=

1010/14/14

Data Sets #Objects #Attributes%Missing

Values

%Non-Quantitative Attributes

Source

USP05-RQ 121 14 2.54 71 Li et al., 2005

USP05-FT 76 14 6.8 71 Li et al., 2005

ISBSG04-2 158 24 27.24 63 ISBSG, 2004

Kem87 15 5 0 40Kemerer et al.,

1987

Mends03 34 6 0 0Mendes et al.,

2003

Desh89 81 10 0.006 20Shepperd et al.,

1997

7. Data sets 7. Data sets

1111/14/14

8. Decision Analysis Using ELECTRE8. Decision Analysis Using ELECTRE

Heuristic MMRE Pred(0.25)

H0 0.62 0.44

H1 0.61 0.44

H3 0.6 0.42

H4 0.59 0.42

CfsSubset (Cfs)

0.52 0.4

Wrapper (Wp)

0.66 0.43

Outranking graph and analysis data for Desh89

(an example)

1212/14/14

9. Pareto Analysis Results9. Pareto Analysis Results

Clusters of Pareto frontier of Desh89 with three Clusters of Pareto frontier of Desh89 with three criteriacriteria

ID MMRE 1-Pred(0.25) 1-Strength Heuristic Cluster26 0.23 0.21 0.83 H1 08 0.11 0.17 0.93 H0 1

24 0.16 0.14 0.91 H1 112 0.27 0.31 0.64 H0 213 0.26 0.27 0.73 H0 216 0.61 0.56 0 H1 328 0.58 0.53 0.05 H1 346 0.59 0.58 0 H4 358 0.55 0.56 0.04 H4 359 0.58 0.54 0.02 H4 361 0.52 0.6 0 CfsSubset 317 0 0 0.99 H1 419 0.08 0 0.95 H1 462 0.02 0 0.98 CfsSubset 4

1313/14/14

9. Pareto Analysis Results9. Pareto Analysis Results

Clusters of Pareto frontier of Desh89 with cirteria 1-Clusters of Pareto frontier of Desh89 with cirteria 1-Pred(25) and 1-StrengthPred(25) and 1-Strength

1414/14/14

10. Conclusions and Future Work10. Conclusions and Future Work

Analysis MethodAnalysis

toolNumber of

alternatives

Number of data points for each

alternative

Number of criteria

Expert preference

ELECTREOutranking

relationsmall Small Multiple

Easy to apply

Pareto analysis and clustering

Pareto frontier and clustering

large large MultipleEasy to apply

Future work:Future work: Use PROMISE data base for benchmarking analysis To broaden the scope from EBA method AQUA+ and its

weighting attributes heuristics to other classes of decision and prediction problems

To study more weighting heuristics over additional available data sets

To investigate other aspects of EBA customization

1515/14/14

Discussion and questions?Discussion and questions?

1616/14/14

Major referencesMajor references M. Shepperd, C. Schofield, “Estimating Software Project Effort M. Shepperd, C. Schofield, “Estimating Software Project Effort

Using Analogies”, Using Analogies”, IEEE Transactions on Software EngineeringIEEE Transactions on Software Engineering, , 23(1997) 736-743.23(1997) 736-743.

G. Ruhe, "Software Engineering Decision Support and Empirical G. Ruhe, "Software Engineering Decision Support and Empirical Investigations - A Proposed Marriage", Investigations - A Proposed Marriage", The Future of Empirical The Future of Empirical Studies in Software EngineeringStudies in Software Engineering (A. Jedlitschka, M. Ciolkowski, (A. Jedlitschka, M. Ciolkowski, Eds.), Workshop Serious on Empirical Studies in Software Eds.), Workshop Serious on Empirical Studies in Software Engineering, Vol. 2, 2003, pp 25-34.Engineering, Vol. 2, 2003, pp 25-34.

T. Menzies, Z.H. Chen, J. Hihn, and K. Lum, "Selecting Best T. Menzies, Z.H. Chen, J. Hihn, and K. Lum, "Selecting Best Practices for Effort Estimation", Practices for Effort Estimation", IEEE Transactions on Software IEEE Transactions on Software EngineeringEngineering, Vol. 32, No. 11, 2006, pp 1-13. , Vol. 32, No. 11, 2006, pp 1-13.

R. Glass, "Matching methodology to problem domain", R. Glass, "Matching methodology to problem domain", Communications of the ACMCommunications of the ACM, 47 (5), 19-21. , 47 (5), 19-21.

J.Z. Li, G. Ruhe, A. Al-Emran, and M.M. Ritcher, "A Flexible Method J.Z. Li, G. Ruhe, A. Al-Emran, and M.M. Ritcher, "A Flexible Method for Effort Estimation by Analogy", for Effort Estimation by Analogy", Empirical Software EngineeringEmpirical Software Engineering, , Vol. 12, No. 1, 2007, pp 65-106. Vol. 12, No. 1, 2007, pp 65-106.

J.Z. Li, G. Ruhe, "Decision Support Analysis for Software Effort J.Z. Li, G. Ruhe, "Decision Support Analysis for Software Effort Estimation by Analogy", Estimation by Analogy", Proceedings of ICSE 2007 Workshop on Proceedings of ICSE 2007 Workshop on Predictor Models in Software Engineering (PROMISE'07)Predictor Models in Software Engineering (PROMISE'07) , USA, May , USA, May 2007. 2007.

J.Z. Li, A. Ahmed, G. Ruhe, "Impact Analysis of Missing Values on J.Z. Li, A. Ahmed, G. Ruhe, "Impact Analysis of Missing Values on the Prediction Accuracy of Analogy-based Software Estimation the Prediction Accuracy of Analogy-based Software Estimation Method AQUA", ESEM’07, Madrid, Spain, September 2007.Method AQUA", ESEM’07, Madrid, Spain, September 2007.