Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A...
Transcript of Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A...
![Page 1: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/1.jpg)
Efficient JavaScript Mutation
Testing
University of British Columbia
Shabnam Mirshokraie
Ali Mesbah
Karthik Pattabiraman
1
![Page 2: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/2.jpg)
• A fault-based technique to assess and improve
the quality of a test suite
• Creates modified versions of the program
2
Mutation Testing
Original Program Mutant
![Page 3: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/3.jpg)
The number of killed mutants by a test
suite as a measure of its effectiveness
3
![Page 4: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/4.jpg)
Mutation Testing Challenges
• High computational cost
• Equivalent mutants– Syntactically different but semantically
identical to the original program
– 10 to 40 percent of mutants are equivalent
4
![Page 5: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/5.jpg)
• Evolutionary techniques (GECCO’04, IST’11)
• Impact on the application’s expected
behaviour (ISSTA’09, ICST’10)
• First generates mutants and then examins for
equivalency
– Computationally expensive and inefficient
5
Prior Approaches
![Page 6: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/6.jpg)
Our Main Goal:
Avoid generating equivalent mutants
How? Narrows down the scope of the mutation
process to behavioural affecting parts of
the code that
6
![Page 7: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/7.jpg)
Our Approach
• Static and dynamic analysis
• A generic mutation testing that guides the mutation
generation process:
– Fewer but more effective mutations
– Mutations with clear impact on the program’s
behaviour
7
![Page 8: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/8.jpg)
Server
Intercept and
Instrument JS
code
Browser
Exec
Trace
Variable
Usage Freq
Dynamic
Invariants
Function
Call Graph
Cyclomatic
complexity
Variable
Selection
Function
Rank
Variable
Mutation
Branch
Mutation
JS Specific
Mutation
Static
JS
Code
Run
8
![Page 9: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/9.jpg)
Server
Intercept and
Instrument JS
code
Browser
Exec
Trace
Variable
Usage Freq
Dynamic
Invariants
Function
Call Graph
Cyclomatic
complexity
Variable
Selection
Function
Rank
Variable
Mutation
Branch
Mutation
JS Specific
Mutation
Static
JS
Code
Run
Intercepts the JavaScript code by setting up a proxy and
instrumenting the code
9
![Page 10: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/10.jpg)
Server
Intercept and
Instrument JS
code
Browser
Exec
Trace
Variable
Usage Freq
Dynamic
Invariants
Function
Call Graph
Cyclomatic
complexity
Variable
Selection
Function
Rank
Variable
Mutation
Branch
Mutation
JS Specific
Mutation
Static
JS
Code
Run
Executes the instrumented program by:
• Crawling the application automatically
• Running the existing test suite
• Combination of the two
10
![Page 11: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/11.jpg)
Server
Intercept and
Instrument JS
code
Browser
Exec
Trace
Variable
Usage Freq
Dynamic
Invariants
Function
Call Graph
Cyclomatic
complexity
Variable
Selection
Function
Rank
Variable
Mutation
Branch
Mutation
JS Specific
Mutation
Static
JS
Code
Run
Gathers detailed execution traces of the application under test
11
![Page 12: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/12.jpg)
Server
Intercept and
Instrument JS
code
Browser
Exec
Trace
(3) Variable
Usage Freq
(2)Dynamic
Invariants
(1) Function
Call Graph
Cyclomatic
complexity
Variable
Selection
Function
Rank
Variable
Mutation
Branch
Mutation
JS Specific
Mutation
Static
JS
Code
Run
Information extraction from the execution traces:
(1) Dynamic call graph of the application
12
![Page 13: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/13.jpg)
startPlay
setup endGamegetDim
13
![Page 14: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/14.jpg)
startPlay
setup endGamegetDim
for(i=0; i<$(".allCells").get().length; i++)
setup($(".allCells").get(i).prop('tagName'));
endGame();
13
![Page 15: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/15.jpg)
startPlay
setup endGamegetDim
if($(cellTag).get().length == 0)
endGame();
for(i=0; i<$(cellTag).get().length; i++)
dimension=
getDim($(cellTag).get(i).width(),
$(cellTag).get(i).height());
$(cellTag).get(i).css('height',
dimension+'px'); 13
![Page 16: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/16.jpg)
startPlay
setup endGamegetDim
var w = width*2, h = height*4, v = w/h;
if(v > 1)
return (v);
else
return (1/v); 13
![Page 17: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/17.jpg)
startPlay
setup endGamegetDim
$(#end).css('height', getDim($('body').width(),
$('body').height())+'px');
13
![Page 18: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/18.jpg)
startPlay
setup endGamegetDim
14
for(i=0; i<$(".allCells").get().length; i++)
setup($(".allCells").get(i).prop('tagName'));
endGame(); 9 elements
![Page 19: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/19.jpg)
startPlay
setup endGame
0.9
0.1
for(i=0; i<$(".allCells").get().length; i++)
setup($(".allCells").get(i).prop('tagName'));
endGame(); 9 elements
getDim
14
![Page 20: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/20.jpg)
startPlay
setup endGamegetDim
0.9
0.1
if($(cellTag).get().length == 0)
endGame();
for(i=0; i<$(cellTag).get().length; i++)
dimension=
getDim($(cellTag).get(i).width(),
$(cellTag).get(i).height());
$(cellTag).get(i).css('height',
dimension+'px');
9 elements
14
![Page 21: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/21.jpg)
startPlay
setup endGamegetDim
0.9
0.9
0.1
0.1
if($(cellTag).get().length == 0)
endGame();
for(i=0; i<$(cellTag).get().length; i++)
dimension=
getDim($(cellTag).get(i).width(),
$(cellTag).get(i).height());
$(cellTag).get(i).css('height',
dimension+'px');
9 elements
14
![Page 22: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/22.jpg)
Server
Intercept and
Instrument JS
code
Browser
Exec
Trace
(3) Variable
Usage Freq
(2)Dynamic
Invariants
(1) Function
Call Graph
Cyclomatic
complexity
Variable
Selection
Function
Rank
Variable
Mutation
Branch
Mutation
JS Specific
Mutation
Static
JS
Code
Run
Information extraction from the execution traces:
(1) Dynamic call graph of the application � FunctionRank
15
![Page 23: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/23.jpg)
• Ranks and select functions for generating variable mutations
• FunctionRank
– # dynamic calls to a given function
– Measures the relative importance of each function at runtime
16
![Page 24: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/24.jpg)
Function called by several functions with high
FunctionRank and high call frequency receives a
high rank itself
17
![Page 25: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/25.jpg)
startPlay
setup endGamegetDim
18
0.9
0.9
0.1
0.1
![Page 26: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/26.jpg)
startPlay
setup endGamegetDim
0.9
0.9
0.1
0.1
1
4
18
32
![Page 27: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/27.jpg)
Server
Intercept and
Instrument JS
code
Browser
Exec
Trace
(3) Variable
Usage Freq
(2)Dynamic
Invariants
(1) Function
Call Graph
Cyclomatic
complexity
Variable
Selection
Function
Rank
Variable
Mutation
Branch
Mutation
JS Specific
Mutation
Static
JS
Code
Run
Information extraction from the execution traces:
(1) Dynamic call graph of the application � FunctionRank
(2) Dynamic invariants
19
![Page 28: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/28.jpg)
20
var w = width*2, h = height*4, v = w/h;
if(v > 1)
return (v);
else
return (1/v);
width>height
startPlay
setup endGamegetDim
![Page 29: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/29.jpg)
Server
Intercept and
Instrument JS
code
Browser
Exec
Trace
(3) Variable
Usage Freq
(2)Dynamic
Invariants
(1) Function
Call Graph
Cyclomatic
complexity
Variable
Selection
Function
Rank
Variable
Mutation
Branch
Mutation
JS Specific
Mutation
Static
JS
Code
Run
Information extraction from the execution traces:
(1) Dynamic call graph of the application � FunctionRank
(2) Dynamic invariants
(3) Variable usage frequency
21
![Page 30: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/30.jpg)
22
var w = width*2, h = height*4, v = w/h;
if(v > 1)
return (v);
else
return (1/v); startPlay
setup endGamegetDim
![Page 31: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/31.jpg)
Server
Intercept and
Instrument JS
code
Browser
Exec
Trace
Variable
Usage Freq
Dynamic
Invariants
Function
Call Graph
Cyclomatic
complexity
Variable
Selection
Function
Rank
Variable
Mutation
Branch
Mutation
JS Specific
Mutation
Static
JS
Code
Run
Variables with a significant impact on the function’s outcome based
on:
• Usage frequency
• Dynamic invariants
23
![Page 32: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/32.jpg)
24
var w = width*2, h = height*4, v = w/h;
if(v > 1)
return (v);
else
return (1/v); startPlay
setup endGamegetDim
width>height
![Page 33: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/33.jpg)
24
var w = width*2, h = height*4, v = w/h;
if(v > 1)
return (v);
else
return (1/v); startPlay
setup endGamegetDim
![Page 34: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/34.jpg)
Server
Intercept and
Instrument JS
code
Browser
Exec
Trace
Variable
Usage Freq
Dynamic
Invariants
Function
Call Graph
Cyclomatic
complexity
Variable
Selection
Function
Rank
Variable
Mutation
Branch
Mutation
JS Specific
Mutation
Static
JS
Code
Run
• Cyclomatic complexity: Statically analyzing the code
• Branch mutation on the highly ranked functions with high
cyclomatic complexity
25
![Page 35: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/35.jpg)
Server
Intercept and
Instrument JS
code
Browser
Exec
Trace
Variable
Usage Freq
Dynamic
Invariants
Function
Call Graph
Cyclomatic
complexity
Variable
Selection
Function
Rank
Variable
Mutation
Branch
Mutation
JS Specific
Mutation
Static
JS
Code
Run
Consider a number of JavaScript specific operators:
• Common mistakes that JavaScript programmers make
• Collected from various resources
26
![Page 36: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/36.jpg)
Tool Implementation
• Implementation of the approach in an open-source tool called Mutandis
• JavaScript dynamic invariants: JSART (ICWE’12)
• Execution trace profiler: Collecting data by exercising the application
– Exhaustive automatic navigation: Crawljax
– Execution of existing test cases
– Combination of crawling and test suite execution
27
![Page 37: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/37.jpg)
Evaluation
• How efficient is Mutandis in generating non-
equivalent mutants?
• How effective is Mutandis in injecting critical
behaviour-affecting faults?
• How useful is Mutandis in assessing existing
test cases of a given web application?
28
![Page 38: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/38.jpg)
Evaluation
• How efficient is Mutandis in generating non-
equivalent mutants?
• How effective is Mutandis in injecting critical
behaviour-affecting faults?
• How useful is Mutandis in assessing existing
test cases of a given web application?
28
![Page 39: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/39.jpg)
• Objects: 5 open-source web applications
• Inject 200 faults for the five objects
– Automatically generating mutants using Mutandis
• Examine output for equivalent mutants
29
Efficiency in generating non-equivalent
mutants
![Page 40: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/40.jpg)
Results:
On average, the percentage of equivalent
mutants generated is 7%
– On average, 10 to 40 percent of equivalent
mutants � Efficiency of Mutandis in generating
non-equivalent mutants
30
![Page 41: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/41.jpg)
Evaluation
• How efficient is Mutandis in generating non-
equivalent mutants?
• How effective is Mutandis in injecting critical
behaviour-affecting faults?
• How useful is Mutandis in assessing existing
test cases of a given web application?
31
![Page 42: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/42.jpg)
• Use the bug severity ranks used by the Bugzilla bug
tracking system
• Choose non-equivalent mutants and assigning a bug
score according to the ranks
Bug Severity Description Rank
Critical Crashes, data loss 5
Major Major loss of functionality 4
Normal Some loss of functionality,
regular issues
3
Minor Minor loss of functionality 2
Trivial Cosmetic issue 1
32
Effectiveness in injecting critical
behaviour-affecting faults
![Page 43: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/43.jpg)
• The average bug severity rank across all applications is 3.6
• The injected faults cause normal to major loss of functionality.
• More than 70% of the faults with major loss of functionality, are in the top 20% of important functions in terms of FunctionRank– Importance of FunctionRank in the fault seeding process
33
![Page 44: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/44.jpg)
Results:
Mutandis is effective in generating mutants
that cause nontrivial errors
34
![Page 45: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/45.jpg)
Evaluation
• How efficient is Mutandis in generating non-
equivalent mutants?
• How effective is Mutandis in injecting critical
behaviour-affecting faults?
• How useful is Mutandis in assessing existing
test cases of a given web application?
35
![Page 46: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/46.jpg)
• Run Mutandis on two JavaScript libraries
– Available Qunit test cases
• Generate 120 mutants for each library
• Determine the usefulness of our approach
based on the number of:
– Non-equivalent generated mutants
– Non-equivalent surviving mutants
36
Usefulness in assessing existing test cases
![Page 47: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/47.jpg)
• Less than 3% of the mutants generated by Mutandis are equivalents
• All the non-equivalent mutants that are not killed by the test suites, are in the top 30% of the important functions in terms of FunctionRank
– The importance of FunctionRank in test case generation
37
![Page 48: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/48.jpg)
Results:
Guides testers towards designing test cases for
important portions of the code
38
![Page 49: Efficient JavaScript Mutation Testingblogs.ubc.ca/karthik/files/2013/03/icst13-talk.pdf · • A generic mutation testing that guides the mutation generation process: –Fewer but](https://reader030.fdocuments.us/reader030/viewer/2022040611/5ed417f88d46b66d226373c5/html5/thumbnails/49.jpg)
Conclusion
• Mutation testing technique that leverages dynamic and static
characteristics
• Selectively mutates portions of the code that exhibit a high
probability of being error-prone and affecting the observable
behaviour
• Implementation of the approach in an open-source tool called
Mutandis (https://github.com/saltlab/mutandis)
• The evaluation of Mutandis
– Efficacy of the approach
39