Measuring Software Redundancy...Software Redundancy Version 1 Version 2 Version n Selection...

Measuring Software Redundancy

Antonio Carzaniga, Andrea Mattavelli, Mauro Pezzè

Università della Svizzera italiana (USI), Switzerland

Redundancy

Software Redundancy

Software Redundancy

Version 1

Version 2

Version n

SelectionAlgorithm

...

Input Output

N-version

Software Redundancy

Version 1

Version 2

Version n

SelectionAlgorithm

...

Input Output

N-version

B

AFault

Workaround

Automatic workarounds

Software Redundancy

Version 1

Version 2

Version n

SelectionAlgorithm

...

Input Output

N-version

MultiMap m = new MultiMap();//…m.put(key, value);

//workarounds for put m.putAll(key, new List().add(value)) m.entrySet().add(new Entry(key, value))

Google Guava


Software Redundancy

Version 1

Version 2

Version n

SelectionAlgorithm

...

Input Output

N-version

?Google Guava




Software Redundancy

Version 1

Version 2

Version n

SelectionAlgorithm

...

Input Output

N-version

?Google Guava




Failures are correlated[Knight et al.]

Software Redundancy

N-version

How much codedo they share?

Failures are correlated[Knight et al.]

?Google GuavaVersion 1

Version 2

Version n

SelectionAlgorithm

...

Input Output




Software Redundancy

N-version

Measuring software redundancy


Version 2

Version n

SelectionAlgorithm

...

Input Output




Two fragments are redundant when they are functionally equivalent and at the same time their executions are different.“

Informal Definition of Redundancy

S0

Sn

CA


Application state space

CB

CA

Functional Equivalence

CB

CA

Functional Equivalence

Observational Equivalence

OCP

OCP


CP

CP

CP

OCP

OCP

CP


CP

OCP

OCP

CP

OCP2

OCP2

CP2

CP2

OCP4

OCP4

CP4

CP4

OCP3

OCP3

CP3

CP3


CB

CA

CB

CA

Execution Diversity

Execution CA

Execution CB

Execution Diversity

actionA actionB actionC actionD …Execution CA

Execution CB

actionZ actionB actionC actionD …

Execution Diversity

actionA actionB actionC actionD …

actionZ actionB actionC actionD …

Execution CA

Execution CB

Software Redundancy

ObservationalEquivalence

Execution Diversity&&

ObservationalEquivalence

Execution Diversity

Software Redundancy

• Binary measure • Not practical

&&

A Measure of Redundancy

Degree of Equivalence

Degree of DiversityR=$f$$ ),(


eS, dS ∈ [0,1]RS = eS(CA,CB) × dS(CA,CB)


R = AGGREGATE(RS)CA,CB


Measure of RedundancyA



Measure of RedundancyA Practical

A Practical Measure of Redundancy



Sample the state space


Observational equivalencemeasure




Difference between executions





Aggregate the redundancy measure


Sampling the State Space

ArrayListMultimap var0 = ArrayListMultimap.create();var0.clear();ArrayListMultimap var3 = ArrayListMultimap.create();var3.clear();boolean var5 = var3.isEmpty();ArrayListMultimap var6 = ArrayListMultimap.create();var6.clear();boolean var8 = var6.isEmpty();boolean var9 = var3.putAll((Multimap) var6);java.util.List var11 = var3.removeAll("hi!");boolean var12 = var0.putAll((short) (-1), (java.lang.Iterable) var11);var0.clear();ArrayListMultimap var14 = ArrayListMultimap.create((Multimap) var0);ArrayListMultimap var17 = ArrayListMultimap.create(1, 10);var17.clear();ArrayListMultimap var19 = ArrayListMultimap.create();var19.clear();ArrayListMultimap var21 = ArrayListMultimap.create((Multimap) var19);boolean var22 = var14.put(var17, var19); // Code fragment A

Sampling the State Space

ArrayListMultimap var0 = ArrayListMultimap.create();var0.clear();ArrayListMultimap var3 = ArrayListMultimap.create();var3.clear();boolean var5 = var3.isEmpty();ArrayListMultimap var6 = ArrayListMultimap.create();var6.clear();boolean var8 = var6.isEmpty();boolean var9 = var3.putAll((Multimap) var6);java.util.List var11 = var3.removeAll("hi!");boolean var12 = var0.putAll((short) (-1), (java.lang.Iterable) var11);var0.clear();ArrayListMultimap var14 = ArrayListMultimap.create((Multimap) var0);ArrayListMultimap var17 = ArrayListMultimap.create(1, 10);var17.clear();ArrayListMultimap var19 = ArrayListMultimap.create();var19.clear();ArrayListMultimap var21 = ArrayListMultimap.create((Multimap) var19);boolean var22 = var14.put(var17, var19); // Code fragment A

Measuring Equivalence

ArrayListMultimap var14 = ArrayListMultimap.create((Multimap) var0);ArrayListMultimap var17 = ArrayListMultimap.create(1, 10);var17.clear();ArrayListMultimap var19 = ArrayListMultimap.create();var19.clear();ArrayListMultimap var21 = ArrayListMultimap.create((Multimap) var19);boolean var22 = var14.put(var17, var19); // Code fragment A



Linkage: boolean var22; ArrayListMultimap var14; Object var17, var19

boolean var22 = var14.put(var17, var19); // Code fragment A

// generated probing code: System.out.println(var22); boolean x0 = var14.isEmpty();System.out.println(x0);var14.clear(); java.util.Map x1 = var14.asMap();int x2 = var14.size(); System.out.println(x2); int x3 = x1.size(); System.out.println(x3); java.util.Set x4 = x1.entrySet();java.util.Iterator x5 = x4.iterator();boolean x6 = x4.isEmpty(); System.out.println(x6);// ... probing code continues



boolean var22 = var14.put(var17, var19); // Code fragment A

List list = new List(); list.add(var19);boolean var22 = var14.putAll(var17, list);

boolean var22 = var14.put(var17, var19);// Code fragment A // Code fragment B




List list = new List(); list.add(var19);boolean var22 = var14.putAll(var17, list);

boolean var22 = var14.put(var17, var19);// Code fragment A // Code fragment B


true

false

1

1

false…

true

false

1

1

false…



eS(CA,CB) = totalsuccessful

CP1CP2CP3CP4CP5CP6CP7CP8CP9

CP10

eS(CA,CB) = 1.0

CP1CP2CP3CP4CP5CP6CP7CP8CP9

CP10

eS(CA,CB) = 0.7


Measuring Diversityboolean var22 = var14.put(var17, var19); // Code fragment A

Measuring Diversityboolean var22 = var14.put(var17, var19); // Execution of code fragment A

Measuring Diversity

Projection

boolean var22 = var14.put(var17, var19); // Execution of code fragment A

Measuring Diversity

ProjectionCode Data


Measuring Diversity

Projection

ArrayListMultimap.put(LObject;LObject;)[email protected](LObject;LObject;)[email protected](LObject;LObject;)Z@200

Statement

3:ArrayListMultimap.put(LObject;LObject;)Z@664:AbstractListMultimap.put(LObject;LObject;)Z@955:AbstractMultimap.put(LObject;LObject;)Z@200

Statement, Depth

Code Data


Measuring Diversity

Projection

Ljava/util/Map;⟶{} Ljava/util/Set;⟶[] Ljava/util/HashMap;⟶{} I⟶1 I⟵1

Type, Value

AbstractMultimap.map⟶{}HashMap.entrySet⟶[]HashMap$EntrySet.this$0⟶{}HashMap$HashIterator.modCount⟶1HashMap$HashIterator.expectedModCount⟵1

Class, Field, Value

Code Data


Measuring Diversityboolean var22 = var14.put(var17, var19);

// Code fragment AList list = new List(); list.add(var19);boolean var22 = var14.putAll(var17, list);

// Code fragment B

Code Projection

ArrayListMultimap.put(LObject;LObject;)[email protected](LObject;LObject;)[email protected](LObject;LObject;)[email protected]()[email protected]()[email protected]()[email protected]()[email protected]()LSet;@953HashMap.entrySet0()LSet;@957HashMap.entrySet0()LSet;@958

ArrayListMultimap.putAll(LObject;LIterable;)[email protected](LObject;LIterable;)[email protected]()LIterator;@774ArrayList$Itr.<init>(LArrayList;LArrayList$1;)V@780ArrayList$Itr.<init>(LArrayList;)V@780ArrayList$Itr.<init>(LArrayList;)V@782ArrayList$Itr.<init>(LArrayList;)V@783ArrayList$Itr.hasNext()[email protected]$100(LArrayList;)[email protected](LObject;LIterable;)[email protected](LObject;)LC;@219HashMap.get(LObject;)LObject;@315HashMap.get(LObject;)LObject;@317HashMap.hash(I)[email protected](I)[email protected](LObject;)LObject;@318HashMap.indexFor(II)I@276



// Code fragment B

Code Projection

ArrayListMultimap.put(LObject;LObject;)[email protected](LObject;LObject;)[email protected](LObject;LObject;)[email protected]()[email protected]()[email protected]()[email protected]()[email protected]()LSet;@953HashMap.entrySet0()LSet;@957HashMap.entrySet0()LSet;@958



// Code fragment B

Code Projection

dS(CA,CB) = 1 - SIMILARITY(PS,A, PS,B)

ArrayListMultimap.putAll(LObject;LIterable;)[email protected](LObject;LIterable;)[email protected]()LIterator;@774ArrayList$Itr.<init>(LArrayList;LArrayList$1;)V@780ArrayList$Itr.<init>(LArrayList;)V@780ArrayList$Itr.<init>(LArrayList;)V@782ArrayList$Itr.<init>(LArrayList;)V@783ArrayList$Itr.hasNext()[email protected]$100(LArrayList;)[email protected](LObject;LIterable;)[email protected](LObject;)LC;@219HashMap.get(LObject;)LObject;@315HashMap.get(LObject;)LObject;@317HashMap.hash(I)[email protected](I)[email protected](LObject;)LObject;@318HashMap.indexFor(II)I@276

es ds RsS0 1.0 0.32989693 0.32989693S1 1.0 0.51781228 0.51781228S2 1.0 0.32989693 0.32989693S3 1.0 0.51781228 0.51781228S4 1.0 0.51781228 0.51781228S5 1.0 0.32989693 0.32989693S6 1.0 0.32989693 0.32989693S7 1.0 0.51781228 0.51781228S8 0.9 0.61892315 0.55703083S9 1.0 0.32989693 0.32989693S10 1.0 0.32989693 0.32989693


es ds RsS0 1.0 0.32989693 0.32989693S1 1.0 0.51781228 0.51781228S2 1.0 0.32989693 0.32989693S3 1.0 0.51781228 0.51781228S4 1.0 0.51781228 0.51781228S5 1.0 0.32989693 0.32989693S6 1.0 0.32989693 0.32989693S7 1.0 0.51781228 0.51781228S8 0.9 0.61892315 0.55703083S9 1.0 0.32989693 0.32989693S10 1.0 0.32989693 0.32989693

R$= AVG(Rs) = 0.418 ± 0.10


Evaluation

Consistency

Significance and usefulness

Evaluation

Consistency

Are the measurements significant and useful?1. Non-reflexivity2. Stability3. Equivalence measure

Consistency

Are the measurements significant and useful?1. Non-reflexivity2. Stability3. Equivalence measure

Evaluation

Stability

Stability

Stability

Algorithm # Impl.Binary search 4Linear search 4Bubble sort 7Insertion sort 3Merge sort 4Quicksort 3

Sear

chSo

rtin

g

StabilityCode Projections

0

0.2

0.4

0.6

0.8

1

ADice Man Cos DamLev Dice Euclid Jaccard Jaro JaroW Lev MC Need Ovlp qGrams SmithW SmithG

Redundancy

Extract to local variable Change name Inline expression Extract method Equivalent input

Data Projections

0

0.2

0.4

0.6

0.8

1


Redundancy


Evaluation

Significance and usefulness Consistency

Significance and usefulness1. Low-level vs high-level2. Predictive ability

Low-level vs High-levelCode Redundancy vs Algorithmic Redundancy


Algorithm # Impl.Binary search 4Linear search 4Bubble sort 7Insertion sort 3Merge sort 4Quicksort 3

Sear

chSo

rtin

g

Binary search Linear search

Bubble sort Insertion sort

Same algorithm, different implementation

0

0.2

0.4

0.6

0.8

1

DamLev Lev Need SmithW SmithG

Redundancy

0

0.2

0.4

0.6

0.8

1


Redundancy

0

0.2

0.4

0.6

0.8

1


Redundancy

0

0.2

0.4

0.6

0.8

1


Redundancy




Same algorithm, different implementation

0

0.2

0.4

0.6

0.8

1


Redundancy

0

0.2

0.4

0.6

0.8

1


Redundancy

0

0.2

0.4

0.6

0.8

1


Redundancy

0

0.2

0.4

0.6

0.8

1


Redundancy


0

00

00 0 0 0 0 0 0 0

00 00 00 00

Different algorithm



0

0.2

0.4

0.6

0.8

1


Redundancy

0

0.2

0.4

0.6

0.8

1


Redundancy

0

0.2

0.4

0.6

0.8

1


Redundancy

0

0.2

0.4

0.6

0.8

1


Redundancy


Predictive Ability

Predictive Ability

B

A

Fault

Workaround


Does redundancy correlate with success?

Predictive Ability

B

A

Fault

Workaround


Predictive Ability

System

Caliper

Carrot2

Predictive Ability

System Method (CA) Workaround (CB)

Caliper

Iterators.forArray(a)LinkedHashMultiset.retainAll(Collection c)ArrayListMultimap.putAll(Object k,…) LinkedHashMultimap.putAll(Object k,…)LinkedHashMultimap.create() LinkedHashMultimap.create(int,int) LinkedHashMultimap.isEmpty()

Arrays.asList(a).iterator()foreach(o in m) if(o not in c) m.remove(o);foreach(o in c) put(k,o);foreach(o in c) put(k,o);create(100,100) create()size() == 0 ? true : false

Carrot2

ImmutableMultiset.of(Object..c) ImmutableMultiset.of(Object..c) ArrayListMultimap.putAll(Object k,…) ImmutableMultiset.of(Object o) Lists.newArrayList()Lists.newArrayList() Lists.newArrayListWithCapacity(int c)Lists.newArrayListWithCapacity(int c) Maps.newHashMap() Maps.newHashMap() Maps.newHashMap()

foreach(o in c) build().setCount(o,count(o)) builder().add(..c).build() foreach(o in c) put(k,o);builder().add(o).build()new ArrayList()new ArrayList(10)new ArrayList()new ArrayList(c) Maps.newHashMapWithExpectedSize(16) new HashMap()new HashMap(16)

Predictive Ability

System Method (CA) Workaround (CB)Success

Rate

Caliper



Carrot2


(100%)(50%)(20%)

(0%)(0%)(0%)(0%)

3/31/28/410/10/2070/2020/34

(59%)(37%)

(8%)(0%)(0%)(0%)(0%)(0%)(0%)(0%)(0%)

13/227/191/130/10/240/240/200/200/540/540/54


Predictive Ability

System Method (CA) Workaround (CB) Redundancy

Caliper



Carrot2


(100%)(50%)(20%)

(0%)(0%)(0%)(0%)

3/31/28/410/10/2070/2020/34

(59%)(37%)

(8%)(0%)(0%)(0%)(0%)(0%)(0%)(0%)(0%)

13/227/191/130/10/240/240/200/200/540/540/54


1.00 ± 0.000.61 ± 0.010.37 ± 0.320.00 ± 0.000.12 ± 0.150.12 ± 0.150.00 ± 0.00

0.56 ± 0.070.24 ± 0.120.37 ± 0.320.32 ± 0.140.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.00

SuccessRate

Predictive Ability

System

Caliper

Carrot2

Method (CA) Workaround (CB) RedundancyIterators.forArray(a)LinkedHashMultiset.retainAll(Collection c)ArrayListMultimap.putAll(Object k,…) LinkedHashMultimap.putAll(Object k,…)LinkedHashMultimap.create() LinkedHashMultimap.create(int,int) LinkedHashMultimap.isEmpty()


1.00 ± 0.000.61 ± 0.010.37 ± 0.320.00 ± 0.000.12 ± 0.150.12 ± 0.150.00 ± 0.00


(100%)(50%)(20%)

(0%)(0%)(0%)(0%)

3/31/28/410/10/2070/2020/34

(59%)(37%)

(8%)(0%)(0%)(0%)(0%)(0%)(0%)(0%)(0%)

13/227/191/130/10/240/240/200/200/540/540/54


0.56 ± 0.070.24 ± 0.120.37 ± 0.320.32 ± 0.140.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.00

SuccessRate

Predictive Ability


Caliper



1.00 ± 0.000.61 ± 0.010.37 ± 0.320.00 ± 0.000.12 ± 0.150.12 ± 0.150.00 ± 0.00

Carrot2



(100%)(50%)(20%)

(0%)(0%)(0%)(0%)

3/31/28/410/10/2070/2020/34

(59%)(37%)

(8%)(0%)(0%)(0%)(0%)(0%)(0%)(0%)(0%)

13/227/191/130/10/240/240/200/200/540/540/54

0.56 ± 0.070.24 ± 0.120.37 ± 0.320.32 ± 0.140.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.00

SuccessRate


Predictive Ability


Caliper



1.00 ± 0.000.61 ± 0.010.37 ± 0.320.00 ± 0.000.12 ± 0.150.12 ± 0.150.00 ± 0.00

Carrot2


(100%)(50%)(20%)

(0%)(0%)(0%)(0%)

3/31/28/410/10/2070/2020/34

(59%)(37%)

(8%)(0%)(0%)(0%)(0%)(0%)(0%)(0%)(0%)

13/227/191/130/10/240/240/200/200/540/540/54

Correlation: 0.940.56 ± 0.070.24 ± 0.120.37 ± 0.320.32 ± 0.140.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.00

SuccessRate

Software Redundancy

N-version



Version 2

Version n

SelectionAlgorithm

...

Input Output





Informal Definition of RedundancySoftware Redundancy

N-version



Version 2

Version n

SelectionAlgorithm

...

Input Output








A Practical Measure of RedundancySoftware Redundancy

N-version



Version 2

Version n

SelectionAlgorithm

...

Input Output










0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy


Data Projections

0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy


Software Redundancy

N-version



Version 2

Version n

SelectionAlgorithm

...

Input Output










0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy


Data Projections

0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy


Different algorithm



0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy

0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy

0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy

0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy


Software Redundancy

N-version



Version 2

Version n

SelectionAlgorithm

...

Input Output









Predictive Ability

System Method (CA) Workaround (CB)SuccessRatio Redundancy

Caliper


Arrays.asList(a).iterator()foreach(o in m) if(o not in c) m.remove(o);foreach(o in c) put(k,o);foreach(o in c) put(k,o);create(100,100)create()size() == 0 ? true : false

1.00 ± 0.000.61 ± 0.010.37 ± 0.320.00 ± 0.000.12 ± 0.150.12 ± 0.150.00 ± 0.00

Carrot2


foreach(o in c) build().setCount(o,count(o)) builder().add(..c).build()foreach(o in c) put(k,o);builder().add(o).build()new ArrayList()new ArrayList(10)new ArrayList()new ArrayList(c) Maps.newHashMapWithExpectedSize(16)new HashMap()new HashMap(16)

0.56 ± 0.070.24 ± 0.120.37 ± 0.320.32 ± 0.140.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.000.00 ± 0.00

(100%)(50%)(20%)

(0%)(0%)(0%)(0%)

3/31/28/410/10/2070/2020/34

(59%)(37%)

(8%)(0%)(0%)(0%)(0%)(0%)(0%)(0%)(0%)

13/227/191/130/10/240/240/200/200/540/540/54

Correlation: 0.94


0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy


Data Projections

0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy


Different algorithm



0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy

0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy

0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy

0

0.2

0.4

0.6

0.8

1


Re

du

nd

an

cy


Software Redundancy

N-version



Version 2

Version n

SelectionAlgorithm

...

Input Output




Measuring Software Redundancy...Software Redundancy Version 1 Version 2 Version n Selection...

Documents

Transcript of Measuring Software Redundancy...Software Redundancy Version 1 Version 2 Version n Selection...