Slide 1

Mahadevan Subramaniam and Bo Guo University of Nebraska at Omaha An Approach for Selecting Tests with Provable Guarantees Slide 2 Multi-version QA 11/30/2012 2 UNO Requirements Development Team Testing Team codes bugs R1 No bug claimed Testing is very costly and time consuming [NIST] 1) in critical path for release 2) development deadline slips can reduce test time 3) development team needs time to fix bugs Changes Development Team Testing Team Modified codes bugs R2 No bug claimed Two or Three weeks Slide 3 Multi-version QA Scope 11/30/2012 Predictive Changes: new tests to validate new features Corrective Changes: Regression testing of existing features Problems Regression Test Minimization remove redundant tests Regression Test Prioritization fault-finding tests first Regression Test Selection (RTS) select relevant tests 3 UNO Slide 4 Regression Test Selection -- Motivation 11/30/2012 Regression testing is most frequent [NIST] and has significant impact on product quality. System behavior re-validated in practice using large test suites. Re-running the entire test suite each time is impractical. Regression test selection identifies tests from a test suite to validate a change in an evolution step. 4 UNO Slide 5 Current State of Art Code based 11/30/2012 5 UNO Slide 6 Current State of Art Model-based 11/30/2012 6 UNO Slide 7 Current State of Art 11/30/2012 Store all test traces on original version Static analysis Compare new and modified Control Flow Graphs If a trace includes the differences, select test including the trace. Use data flow graphs and program dependency graphs 7 UNO Slide 8 Problems 11/30/2012 Storing all traces is expensive Control flow graphs comparing may involve untested parts of programs No guarantee on the regression test suite Relevant tests may be skipped (Incomplete) Irrelevant tests may be selected 8 UNO Slide 9 Goal 11/30/2012 Develop an automatic approach for regression test selection to create a regression test suite with provable guarantees and without using any history information. Main Challenges Provably predict test execution behavior without actually executing the test ? Keep regression test selection economical Selection cost + regression test cost EFSM extended finite state machine 11/30/2012 UNO 11 s0 s1 1 6 2 3 s2 3 4 5 Initially a = 0; state is s0 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0 v0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {}, 6. close, true, s1 s0, {}, ack(a). List of states The machine is in only one state at a time Triggering condition for each transition Slide 12 Overview of the Approach Analyze a given test description to determine if the test execution will exercise a given change and select the test. Identify descriptions having sufficient information about their execution (fully-observable tests) Analyze such descriptions to determine if the change will be executed. Patch other descriptions to obtain information about their execution. Changes add / delete / replace transitions. Test description is a sequence of input and output messages with parameters with values over booleans, integers, arrays, queues, and records. 12 11/30/2012 12 UNO Slide 13 Overview of the Approach (Contd.) From a given test description and a change find sets of matching transitions that can process inputs in the description descriptions having a matching change transition are candidates determine feasibility of an execution path formed by matching transitions automatically attempt to patch infeasible paths to obtain an execution path Test selected if the change transition appears in the feasible execution path. 13 11/30/2012 13 UNO Slide 14 Tests 1: open(100)/ack(100), deposit(50)/ack(150), close/ack(150) 2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) 3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) A Simple Example 11/30/2012 Initially a = 0; state is s0 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0 v0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {}, 6. close, true, s1 s0, {}, ack(a). 7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a). Not Selected 14 UNO s0 s1 1 6 2 3 s2 3 4 5 7 Slide 15 2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) A Simple Example 2 11/30/2012 Find transitions matching each input of description 2 { 1 }{ 2 }{ 3, 7 }{ 6 } = [] 15 UNO 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0 v0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {}, 6. close, true, s1 s0, {}, ack(a). 7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a). s0 s1 1 6 2 3 s2 3 4 5 7 Slide 16 A Simple Example 2 (conti) 11/30/2012 16 UNO 2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) = [ {1} {2} {3, 7} {6} ] s0 s1 1 6 2 3 s2 3 4 5 7 Transition 2 can immediately follow 1 to process next input of 2 First test input of 2 must be processed by transition 1 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0 v0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {}, 6. close, true, s1 s0, {}, ack(a). 7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a). Slide 17 A Simple Example 2 (conti) 11/30/2012 17 UNO 2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) = [ {1} {2} {3, 7} {6} ] s0 s1 1 6 2 3 s2 3 4 5 7 Transition 7 can immediately follow 2 since (160 > 0) (160 > 150) satisfiable. Selected 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0 v0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {}, 6. close, true, s1 s0, {}, ack(a). 7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a). Slide 18 11/30/2012 A Simple Example 3 18 UNO s0 s1 1 6 2 3 s2 3 4 5 7 3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) = [ {1} {2} {3, 7} {3, 7} {6} ] First two inputs processed by transition 1 followed by transition 2. Transition 7 cannot immediately follow 2 since (100 > 0) ^ (100 > 150) is unsatisfiable. 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0 v0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {}, 6. close, true, s1 s0, {}, ack(a). 7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a). Slide 19 11/30/2012 A Simple Example 3 (conti..) 19 UNO s0 s1 1 6 2 3 s2 3 4 5 7 3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) = [ {1} {2} {3, 7} {3, 7} {6} ] But 3 can immediately follow 2 since (100 > 0) (1000, s0 s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0 v0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {}, 6. close, true, s1 s0, {}, ack(a). 7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a). Slide 20 11/30/2012 A Simple Example 3 (conti..) 20 UNO s0 s1 1 6 2 3 s2 3 4 5 7 3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) = [ {1} {2} {3, 7} {3, 7} {6} ] Neither 7 nor 3 can immediately follow transition 3. 7 can follow 3 after some transitions not using test inputs. ? Can we patch 3 to obtain a path with transition 7? 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0 v0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {}, 6. close, true, s1 s0, {}, ack(a). 7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a). Slide 21 11/30/2012 A Simple Example 3 (conti..) 21 UNO s0 s1 1 6 2 3 s2 3 4 5 7 3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) = [ {1} {2} {3, 7} {3, 7} {6} ] 4 cannot patch since (50 > 0) ^ (50 < 50) is unsatisfiable. 5 can patch since (50 >= 50) is satisfiable and 7 can immediately follow 5. Selected ? 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0 v0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {}, 6. close, true, s1 s0, {}, ack(a). 7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a). Slide 22 Salient Aspects 11/30/2012 Regression test selection for test descriptions with rich data types. Fully-observable tests: descriptions with sufficient information. A simple structural invariant to identify fully-observable tests. Automatically analyze failure of invariant to patch test descriptions. Procedures to select tests guaranteed to exercise changes to EFSMs. Experiments with 10 web services and protocols Study costs of running the full test case and selective test cases. Proposed approach reduces test running times in all examples. 22 UNO Slide 23 Outline 11/30/2012 Preliminaries Fully-Observable Tests Tests with Non-Observable Regions Experiments Conclusion 23 UNO Slide 24 EFSMs 11/30/2012 E = (I i, O i, S i, V i, T i ) I i, O i : input and output messages S i : local states V i : Data and queue variables T i : Deterministic transition relation Transition t: m j, P t, s t q t, m l, A t m i m l : parameterized input and output messages P t : a predicate over variables from V i s t, q t : states from S i A t : ordered sequence of assignments to variables from V i Explicit transition is transition having both input and output messages 24 UNO Slide 25 Test Descriptions and Changes 11/30/2012 Test description = g 0 is concrete initial global state Finite sequence of input/output elements i k /o k Test run r = g 0 t 0 t m g m g 0 is an EFSM run produced by applying to the EFSM in state g 0. All the global states in r are concrete global states. Changes to the EFSM Performed at the transition level =, sign { +, - } (addition, deletion) = (replacement) 25 UNO Slide 26 Outline Preliminaries Fully-Observable Tests Tests with Non-Observable Regions Experiments Conclusion 26 11/30/2012 26 UNO Slide 27 11/30/2012 Transitions Matching a Test Description Transition matches a test if an input-output element of test is an instance of input-output message of the transition and satisfies the input condition of the transition. Transition 3. withdraw(v), v > 0 v Initially a = 0; state is s0 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0 v0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {}, 6. close, true, s1 s0, {}, ack(a). 7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a). 12 3 6 7 11/30/2012 Identifying Fully-Observable Tests 29 UNO 12 2: open(100)/ack(100), deposit(50)/ack(150), withdraw(160)/ack(150), close/ack(150) = [ {1} {2} {3, 7} {6} ] 3 7 1 : conjunct 0: (a 0 == 0) (100 > 0) 1 : conjunct 1: (a 0 == 0) (100 > 0) (a 1 == a 0 +100) (50 > 0) 1 : conjunct 2: (a 0 == 0) (100 > 0) (a 1 == a 0 +100) (50 > 0) (a 2 == a 1 +50) (160 > 0) (1600) (a 1 == a 0 +100) (50 > 0) (a 2 == a 1 +50) (160 > 0) (160 > a 2 ) Slide 30 Outline Preliminaries Fully-Observable Tests Tests with Non-Observable Regions Preliminary Experiments Conclusion 30 11/30/2012 30 UNO Slide 31 Tests with Non-Observable Regions 31 11/30/2012 31 UNO Identify a non-observable region (t k, t k+1 ) for a conjunct that fails to extend to transition t k+1 EFSM paths from output state of t k to input state of t k+1. All intermediate transitions in all paths have no test inputs. Eliminating regions without loops Merge the intermediate transitions with t k to t merge Patch conjunct C = Pos([t 1,, t merge ], g 0 ) Pre(t k+1 ) and check if satisfiable. Fails if patched conjuncts for all paths in region unsatisfiable. Slide 32 Patching Failures Example 32 11/30/2012 32 UNO 3: open(100)/ack(100), deposit(50)/ack(150), withdraw(100)/ack(50), withdraw(60)/ack(50), close/ack(50) s0 s1 1 6 2 3 s2 3 4 5 7 1 2 37 7 ? 3 5 7 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0 v0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {}, 6. close, true, s1 s0, {}, ack(a). 7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a). ? ignore Slide 33 Regions with Terminating Loops 33 11/30/2012 33 UNO Regions having loops cannot be eliminated by simple merging. Each loop iteration requires additional test input. Merge intermediate transitions in the path until the loop end- points and attempt to construct a path by executing loop transitions. Process terminates since all loops are terminating over concrete global states. Slide 34 7 Patching Involving Loop Example 34 11/30/2012 34 UNO 4: open(100)/ack(100), deposit(50)/ack(150), withdraw(50)/ack(100), withdraw(60)/ack(40), withdraw(60)/ack(50), close/ack(50) 1 2 3 7 3 5 3 s0 s1 1 6 2 3 s2 3 4 5 7 ? 7?? ignore Patch fails ? 1. open(v), v > 0, s0 s1, {a = a + v}, ack(a), 2. deposit(v), v > 0, s1 s1, {a = a + v}, ack(a), 3. withdraw(v), v > 0 v0 a < 50, s2 s2, {a = a + 1}, 5. a >= 50, s2 s1, {}, 6. close, true, s1 s0, {}, ack(a). 7. withdraw(v), v > 0 v > a, s1 s1, {}, ack(a). 7 4 5 7 = [ {1} {2} {3, 7} {3, 7} {3, 7} {6} ] Slide 35 Outline Preliminaries Fully-Observable Tests Tests with Non-Observable Regions Experiments Conclusion 35 11/30/2012 35 UNO Slide 36 Experiments Regression Cost Model [Leung and White 91] C1: Cost of running the full test suite C2: Cost of running the selected tests C3: Cost of analyzing the regression test selection Atm: automatic teller machine; Thp: third party call; Bnk: back web services; Ven: a vending machine (Cmp, Tcp, Cnf) : completion, two-phase commit, and conference protocol 36 - > Slide 37 Time Savings across Multiple Changes 37 11/30/2012 UNO Slide 38 Test Selected across Multiple Changes 38 11/30/2012 UNO Slide 39 SPG with Code-based approach 11/30/2012 39 UNO P1: incr1 (x) { p1 while(++x SPG with Code-based approach 11/30/2012 41 UNO P1: incr1 (x) { p1 while(++x 11/30/2012 Pre- and Post-Images of Transitions Pre-image of t: a symbolic global state that triggers the transition t. 3. withdraw(v), v > 0 v