/ W&I / MDSE PAGE 023-4-2012

Metrics are usually computed at a low level:

classes, methods, …

Multitude of data values obscures a general

picture of the system maintainability

/W&I / MDSE PAGE 123-4-2012

That we are actually interested in!

/W&I / MDSE PAGE 223-4-2012

You Can't Control the Unfamiliar:

A Study on the Relations

Between Aggregation

Techniques for Software Metrics

Bogdan Vasilescu

Alexander Serebrenik

Mark van den Brand

Two kinds of aggregation

Same artifact, different

metrics

Same metrics, different

artifacts

/W&I / MDSE PAGE 423-4-2012

Various techniques can be

found in the literature

Same metrics, different

artifacts

/W&I / MDSE PAGE 523-4-2012

Traditional: mean,

median, sum, …

Econometric

inequality indices:

Gini, Theil, Hoover,

Kolm, Atkinson

Various techniques can be

found in the literature

Same metrics, different

artifacts

/W&I / MDSE PAGE 623-4-2012

Traditional: mean,

median, sum, …

Econometric

inequality indices:

Gini, Theil, Hoover,

Kolm, Atkinson

Which

aggregation

technique

should we

use?

Questions

1. Which and to what extent do the different

aggregation techniques agree?

2. What is the nature of the relation between the

various aggregation techniques?

3. How does the correlation coefficient change as the

systems evolve?

/W&I / MDSE PAGE 723-4-2012

Qualitas Corpus 20101126

/W&I / MDSE PAGE 823-4-2012

• Qualitas Corpus 20101126r, 106 systems

• FitJava v1.1, 2 packages, 2240 SLOC

• NetBeans v6.9.1, 3373 packages 1890536 SLOC.

1) Agreement between diff techniques

• Agreement:

• Aggregation: Class SLOC Package

• Techniques agree if they rank the packages similarly

/W&I / MDSE PAGE 923-4-2012

We use rank-based correlation coefficient: Kendall’s

1) Agreement: different inequality indices?

• Gini, Theil, Hoover, Atkinson – agree

• aggregates obtained convey the same information

• Kolm does not!

/W&I / MDSE PAGE 1023-4-2012

1) Agreement: traditional and ineq indices?

• mean

• Kolm: strong (0,8) and statistically significant (92%)

• median, standard deviation, and variance

• sum

• does not correlate with any other aggregation technique

/W&I / MDSE PAGE 1123-4-2012

2) Nature of the relation: Typical patterns

• Theil is known to be more

sensitive to the rich

• Theil increases faster

when Gini increases

/W&I / MDSE PAGE 1223-4-2012

• Linear relation with a “fat”

head

Which aggregation technique? (1)

• Theil, Hoover, Gini and Atkinson agree

• Any can be chosen from the correlation point of view

• Some might be “better” in each specific case

• easy to interpret: Gini [0,1]

• provide additional insights: Theil (explanation)

• negative values: Gini, Hoover

− affects the domain!

• sensitive for high values: Theil, Atkinson

• deviations from uniformity: Gini, Hoover

/ W&I / MDSE PAGE 1323-4-2012

Which aggregation technique? (2)

• Kolm and mean agree

• Kolm is reliable for skewed distributions

− better alternative (“by no means”)

• Not in the paper:

− agreement observed for NOC

− but not for DIT!

/ W&I / MDSE PAGE 1423-4-2012

Conclusions

/W&I / MDSE PAGE 1523-4-2012