Amin Milani Fard: Directed Model Inference for Testing and Analysis of Web Applications
Transcript of Amin Milani Fard: Directed Model Inference for Testing and Analysis of Web Applications
Amin Milani Fard
Directed Model Inference for Testing and Analysis of Web
Applications
University of British ColumbiaOct 2015
http://www.knowdiff.net/
Modern Web Applications
Manual Testing and Maintenance
Tedious
Incomplete
Test Model Generation
Test Case Generation
Unit Test Fixture Generation
Code Maintenan
ce
P1. Test Model Generation• In model-based testing, models of program
behaviour are used to generate test cases.
• Dynamic analysis and exploration (crawling) derives test models for many automated testing techniques.
Most industrial web applications have a huge state-space.
Exhaustive crawling (BFS, DFS, or random search), can cause the state explosion problem.
Given a limited time, exhaustive crawlers can become mired in specific parts of the application, yielding poor coverage.
Covering the whole app is infeasible in a limited time, so …
RQ1. How can we derive test models for web applications more effectively compared to exhaustive exploration methods?
Amin Milani Fard, Ali Mesbah
Feedback-Directed Exploration of Web Applications to Derive Test
Models
24th IEEE International Symposium on Software Reliability Engineering (ISSRE),
2013
Test Model GenerationWe consider 4 properties for a test model:• Functionality Coverage: The amount of code coverage• Navigational Coverage: The amount of covering different
navigational branches • Page Structural Coverage: The amount of covering
heterogeneous DOM structures• Size: The number of edges in the SFG
How can we infer a model satisfying all these?
Feedback-directed Exploration• FeedEx uses the feedback obtained to predict
(1) which states should be expanded next(2) in which order events should be executed
• Repeat until a time/state limit• Take the crawler to state s with highest state score• Execute the fittest event on s based on event score
State score is a combination of• Code Coverage Impact: The amount of code coverage
increase• Path Diversity: The diversity (not sharing) of two paths • DOM Diversity: The diversity (dissimilarity) of two DOM
trees
Event Score• Event Productivity Ratio: Unexecuted events first.
Penalize events that result in an already discovered state, e.g. self-loops.
Evaluation and Results• Objects: 6 open-source JavaScript web apps• Fixed time: 5 minutes, no limitations on: depth, number
of states
Coverage0%
12%24%36%48%60%
DOM Diversity00000
Path Diversity00001
Test Model Size Test Suite Size0
150300450600750
DFS BFS RND FeedEx
10-28% imp 7-4000% imp 23-130% imp
38-86% imp 42-61% imp
DOM-based Testing
Crawling automates testing by exploring more states, but is limited in:• Proper input values• Choosing paths to explore• Generating effective assertions
RQ2. Can we utilize the knowledge in existing tests to generate new tests?
P2. Test Case Generation
Amin Milani Fard, Mehdi Mirzaaghaei, Ali Mesbah
Leveraging Existing Tests in Automated
Test Generation for Web Applications
29th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2014
Combining Manual and Automated Tests
- Input data and sequence- DOM elements to be asserted
- Automated crawling- Automated test case generation
Generated test cases
Testilizer idea
Extended State-Flow Graph
Initial State-Flow GraphHuman-
written test cases
Exploring Alternative Paths• Remain close to the manual-test paths
Initial State-Flow Graph
Extended State-Flow Graph
Regenerating Assertions(1) Reusing the same assertion
(2) Regenerating assertions for exact DOM element/region match
(3) Generating assertions for similar DOM region match
Assertion ReuseAn assertion on a shared state can be reused for a new test case.
Assertion RegenerationRepetition-based assertion regeneration
1. exact element-based assertions2. exact region-based assertions
Checked Element
Checked Element Region
Generating Similar Region Assertions
Inexact element/region repetition of a checked element/region can also be important for testing.
A classification problem: • Is a block level DOM element important to be checked by an
assertion?
<div id="header">
<div id="nav">
<div id="footer">
<div id="article"> <div id="sidebar
"><div id="section">
<div id="header">
<div id="nav">
<div id="footer">
<span id="main"> <span id="men
ue"><span id="content">
• 150% imp over the original test suite• 170% imp over the random assertion (RND) & exploration (RAND)• 37% imp over the random assertion (RND)
While code coverage was not our main goal• 30% imp over the original test suite• 18% imp over the random exploration
Fault Detection Rate0%7%
13%20%26%33%
ORIG RAND + RND EXND + RND Testilizer
Evaluation and Results
P3. Unit Test Fixture Generation
If an expected DOM element (test fixture) is not present, a JavaScript unit test may throw an exception or produce an incorrect result.
Test fixtures define states of the test environment before the test.
Proper DOM-based fixtures are required to achieve high coverage.
RQ3. How can we automate fixture generation for unit testing?
Challenges(1) DOM-related variables(2) Hierarchical DOM relations
Amin Milani Fard, Ali Mesbah, Eric Wohlstadter
Generating Fixtures for JavaScript Unit Testing
30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2015
Apply the new fixture on DOMUnit Test
ConFix ideaInstrument the code Instrumented
CodeJavaScript Code
+Function under
test (FUT)
Solve constraints and generate a
fixture
Collect exec trace and deduce DOM-based constraints
Execute the FUT
Instrumentationtrace = [];function confixWrapper(statementType, statement, varList, varValueList, enclosingFunction, actualStatement) { trace.push({statementType: statementType, statement: statement, varList: varList, varValueList: varValueList, enclosingFunction: enclosingFunction, actualStatement: actualStatement}); return actualStatement;}function getConfixTrace() { return trace;}function dg(x) { return confixWrapper("return", "return confixWrapper(\"functionCall\", \"document.getElementById(x)\", [\"x\"], [x], \"dg\", document.getElementById(x));", [""], [], "dg", confixWrapper("functionCall", "document.getElementById(x)", ["x"], [x], "dg", document.getElementById(x)));}function sumTotalPrice() { sum = confixWrapper("infix", "sum = 0", [""], [], "sumTotalPrice", 0); itemList = confixWrapper("infix", "itemList = confixWrapper(\"functionCall\", \"dg('items')\", [\"items\"], ['items'], \"sumTotalPrice\", dg('items'))", [""], [], "sumTotalPrice", confixWrapper("functionCall", "dg('items')", ["items"], ['items'], "sumTotalPrice", dg('items'))); if (confixWrapper("condition", "itemList.children.length === 0", [""], [], "sumTotalPrice", itemList.children.length === 0)) confixWrapper("functionCall", "dg('message')", ["message"], ['message'], "sumTotalPrice", dg('message')).innerHTML = confixWrapper("infix", "confixWrapper(\"functionCall\", \"dg('message')\", [\"message\"], ['message'], \"sumTotalPrice\", dg('message')).innerHTML = \"Item list is empty!\"", [""], [], "sumTotalPrice", "Item list is empty!"); else { for (i = confixWrapper("infix", "i = 0", [""], [], "sumTotalPrice", 0); confixWrapper("loopCondition", "i < itemList.children.length", ["i", "itemList"], [i, itemList], "sumTotalPrice", i < itemList.children.length); i++) { p = confixWrapper("infix", "p = itemList.children[i].value", ["itemList.children[i]"], [itemList.children[i]], "sumTotalPrice", itemList.children[i].value); if (confixWrapper("condition", "p > 0", [""], [], "sumTotalPrice", p > 0)) sum += p; else confixWrapper("functionCall", "dg('message')", ["message"], ['message'], "sumTotalPrice", dg('message')).innerHTML += " Wrong value for item " + i; } confixWrapper("functionCall", "dg('total')", ["total"], ['total'], "sumTotalPrice", dg('total')).value = confixWrapper("infix", "confixWrapper(\"functionCall\", \"dg('total')\", [\"total\"], ['total'], \"sumTotalPrice\", dg('total')).value = sum", [""], [], "sumTotalPrice", sum); }
(1) Extract DOM-based Constraints
(2) Transform them into XPath
(3) Solve it using an available XML XPath solver
Evaluation and Result
• Up to 67% imp in statement coverage
• Up to 300% imp in branch coverage
P4. Code Maintenance• JavaScript is challenging to
maintain.
• Code smells adversely influence program comprehension and maintainability.
• Code smells detection is time consuming.RQ4. Which JavaScript code smells are prevalent in
practice and how can we support automated code refactoring?
Amin Milani Fard, Ali Mesbah
JSNose: Detecting JavaScript Code Smells
13th IEEE International Conference on Source Code Analysis and Manipulation (SCAM), 2013
JavaScript Code Smell DetectionStatic and dynamic code analysis is used to monitor and infer information about objects, functions, and code blocks.
Evaluation and Results
• Evaluated 11 JavaScript/Ajax web applications.
• Effective code smells detection (93% precision, 98% recall)
• Lazy object, long method/function, closure smells, JS/HTML/CSS coupling, and excessive global variables, are the most prevalent smells.
• Strong, and significant +correlation between {LOC, # functions, # JS files, and CC} and the types of smells, and weaker correlation with the # smell instances.
Some topics to work on• Applying learned tests on similar applications
• Generating unit tests based on client-side code similarity and DOM-based tests based on DOM sate similarity
• JavaScript code refactoring• Suggesting and applying refactoring for code
smells
• Understanding test failures and root causes
• Generating integrated tests using existing unit tests
Test Model Generation
Test Case Generation
Unit Test Fixture Generation
Code Maintenan
ce