Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and...
-
Upload
hilda-craig -
Category
Documents
-
view
240 -
download
1
Transcript of Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and...
![Page 1: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/1.jpg)
Chapter 13: Regression Testing
Omar Meqdadi
SE 3860 Lecture 13Department of Computer Science and Software Engineering
University of Wisconsin-Platteville
![Page 2: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/2.jpg)
2
Topic Covered
What is Regression Testing? Why Regression testing? Regression Testing Process Object-Oriented Regression Testing
![Page 3: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/3.jpg)
3
What is Regression Testing?
Testing activities occur after software changes During software maintenance phase
Execution of a set of test cases on a program in order to ensure that implemented changes do not produce unintended faults
![Page 4: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/4.jpg)
Regression Testing Levels
Unit Testing Integration Testing Acceptance Testing
![Page 5: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/5.jpg)
Who Perform Regression Testing
Developers: regression testing at the unit level or integration
Test Engineers : regression testing at the function level
Quality Engineers: regression testing at the system level
![Page 6: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/6.jpg)
Why Regression Testing?
Validate and Verify the modified software Validate and verify the new features Ensure existing features are still working(change impact)
Small changes in one part of a program may have indirect undesired effects in other unrelated parts of the program
Fixing could introduce new bugs
![Page 7: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/7.jpg)
Why Regression Testing?
![Page 8: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/8.jpg)
8
Regression Testing Process
Input: Given program P, its modified version P’, and a test set T
that was used to previously test P Goal:
Find a way to utilize T to gain sufficient confidence in the correctness of P’
![Page 9: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/9.jpg)
Regression Testing Process
Activities: Regression Test Selection
Select T’ ⊆ T, a set of test cases to execute on P’ Test Suite Execution
Test P’ with T’, establishing correctness of P’
![Page 10: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/10.jpg)
Regression Test Selection
Selecting a subset of the existing test suite based on information about: Original Version (P) Modified Version (P’) Original Test Suite (T)
Approaches: Select All Modification-Based Test Selection Coverage-Based Test Selection
Test Set Minimization Test Case Prioritization
![Page 11: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/11.jpg)
Select All
Traditional approach Select all the test cases in T to test P’ Advantage:
Safe Problem:
Too expensive Note: the new and the old programs could produce
different outputs
![Page 12: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/12.jpg)
Modification-Based Test Selection
Select set of test cases T that covers only modified or affected portions of P
Based on Module-Based Firewall A changed software module and a closure of all possible affected
modules and related integration links in a program based on the static and dynamic analyses (e.g.: a control-flow graph).
We can come out different kinds of firewalls based on various test models such as: Control Firewall Data Firewall Function (method) Calling Firewall Class Firewall (Object-Oriented Regression Testing)
![Page 13: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/13.jpg)
Modification-Based Test Selection
Test selection concept based om Module Firewall re-test of the changed module and its affected modules
within the firewall re-test all of related integration links in the firewall
![Page 14: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/14.jpg)
Main
M1 M3M2
M7M6M5M4
M8
Modification-Based Test Selection
: Control Flow
Mi : Module
![Page 15: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/15.jpg)
Main
M1 M3M2
M7M6M5M4
M8
Changed Module
Modification-Based Test Selection
![Page 16: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/16.jpg)
Main
M1 M3M2
M7M6M5M4
M8
Changed Module
Modification-Based Test Selection
A control based module firewall: M5, M1, M8 , and Main
![Page 17: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/17.jpg)
Main
M1 M3M2
M7M6M5M4
M8
1
2
3
Changed Module
Modification-Based Test Selection
A control based module firewall: M5, M1, M8 , and MainRe-testing at the unit level: M5
Re-testing at the integration level: 1, 2, 3
![Page 18: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/18.jpg)
Modification-Based Test Selection
Problems: Still too many test cases
Need to further reduce the number of regression tests Coverage: the risk of missing test cases that are able to
reveal defect not yet discovered
![Page 19: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/19.jpg)
Test Set Minimization
The test suite is reduced to a minimal subset that maintains the same coverage as the original test suite with respect to a given coverage criterion
Goal: Find a minimal subset T’of T such that each testable entity (function, statement,…) is covered by at least one test in T’
What we need ? Coverage (Execution Flow Trace) per test case
![Page 20: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/20.jpg)
Test Set Minimization
Steps: Step1: From old version P : determine the coverage for
each test t ⊆ T, determine which of the k testable entities is covered.
Step2: Find a minimal subset T’ of T such that each testable entity (function, statement,…) is covered by at least one test in T’by using Test Case Prioritization
Note: T’is not unique
![Page 21: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/21.jpg)
Test Case Prioritization
Tests are prioritized based on some criteria. Highest priority test are executed earlier Test Prioritization Approaches:
Total statement coverage # of statements executed
Additional statement coverage((Heuristic #1) A higher priority is assigned to a test that covers the higher
number of not yet executed statements Total function coverage
# of functions executed Additional function coverage
![Page 22: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/22.jpg)
Test Set Minimization
Example1: Test prioritization based on total function coverage Suppose
P contains two functions: main and foo suppose that P is tested using test cases t1 and t2 During testing it was observed that :
t1 causes the execution of main but not of foo t2 causes the execution of both main and foo
Now suppose that P’ is obtained from P by making some modification
Which of the two test cases should be included in the regression test suite?
regression test suite consists of only t2 (minimization)
![Page 23: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/23.jpg)
Test Set Minimization
Example2: Test prioritization based on total statement coverage Assume P contains two functions main and f1, where the
flow charts are as follow
![Page 24: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/24.jpg)
Test Set Minimization
Example2: Assume the following test cases
Test case Coverage
T1 main: {1, 2, 3} f1: {1, 3}T2 main: {1, 3} f1: {1, 3}T3 main: {1, 3} f1: {1, 2, 3}
A minimal test set for regression testing is {t1, t3}
![Page 25: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/25.jpg)
Test Case Prioritization
Additional statement coverage((Heuristic #1) A higher priority is assigned to a test that covers the higher
number of not yet executed statements
![Page 26: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/26.jpg)
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3
t2 : S1, S5, S8, S9
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t8 : S1, S2, S3, S4, S7
t9 : S1, S6
t10 : S1, S2
![Page 27: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/27.jpg)
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3
t2 : S1, S5, S8, S9
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t8 : S1, S2, S3, S4, S7
t9 : S1, S6
t10 : S1, S2
![Page 28: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/28.jpg)
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3
t2 : S1, S5, S8, S9
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t8 : S1, S2, S3, S4, S7
t9 : S1, S6
t10 : S1, S2
S: t8
![Page 29: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/29.jpg)
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3 Covered statements t2 : S1, S5, S8, S9 S1, S2, S3, S4, S7
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t9 : S1, S6
t10 : S1, S2
S: t8
![Page 30: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/30.jpg)
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3 Covered statements t2 : S1, S5, S8, S9 S1, S2, S3, S4, S7
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t9 : S1, S6
t10 : S1, S2
S: t8
![Page 31: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/31.jpg)
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3 Covered statements t2 : S1, S5, S8, S9 S1, S2, S3, S4, S7
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t9 : S1, S6
t10 : S1, S2
S: t8, t2
![Page 32: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/32.jpg)
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3 Covered statements
S1, S2, S3, S4, S5, S7, S8, S9
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t9 : S1, S6
t10 : S1, S2
S: t8, t2
![Page 33: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/33.jpg)
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3 Covered statements S1, S2, S3, S4, S5, S7, S8, S9
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t9 : S1, S6
t10 : S1, S2
S: t8, t2
![Page 34: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/34.jpg)
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3 Covered statements
S1, S2, S3, S4, S5, S7, S8, S9
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t9 : S1, S6
t10 : S1, S2
S: t8, t2 , t9
![Page 35: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/35.jpg)
Test Set Minimization/ Prioritization
Problem:
Test minimization discards test cases.
There is a chance that, if P’ were executed against a
discarded test case, it would reveal an error in the
modification made
![Page 36: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/36.jpg)
Object-Oriented Regression Testing
Typical Cases Inheritance:
Getting a new subclass working without exercising the superclass is usually easy
No need to re-test the superclass We assume that inherited features just work
Component based development (Re-use) OO technologies offer strong support for reuse but they
cannot prevent regression bugs No unit-testing We need integration testing
![Page 37: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/37.jpg)
Object-Oriented Regression Testing
Dependency: Dependencies occur for many reasons and so regression
faults can occur in many ways Assume a dependency is between B and D
D has a side effect on B: B fails because a new action of D is inconsistent with B’s requirements and assumptions with respect to D
D is a client of B: D sends a message that violates B’s invariant/precondition. B is not defensive and accepts the incoming message, leading to failure
![Page 38: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/38.jpg)
Object-Oriented Regression Testing
Dependency (contd.): D is a server of B: B sends a message to D. D has been
changed (or contain bugs). D returns a value that causes a violation of B’s invariant . B fails or returns an invalid value to another baseline component C. C fails
![Page 39: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/39.jpg)
Object-Oriented Regression Testing
OO- Regression Testing Concepts Each class must be tested when it is developed Each class needs a regression test Regression tests need to have standard interfaces Thus, we can build the regression test when building the
class and have a better, more stable product for less work
![Page 40: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/40.jpg)
Object-Oriented Regression Testing
OO- Regression Testing Strategy Based on the Class Firewall Concept
Identify changed classes Identify affected classes (using dependency relation) Apply the Class Test Order strategy to perform class re-
testing at the unit level Use the Class Test Order to re-integrate classes together. Select, reuse, and define test cases
![Page 41: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/41.jpg)
Example: A Changed Class in Class Diagram
X
Y
D
T
S
I
U
A C
B
M
N
R V
L
![Page 42: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/42.jpg)
X
Y
D
T
S
I
U
A C
B
M
N
R V
L
Example: A Class Firewall in Class Diagram
![Page 43: Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.](https://reader030.fdocuments.us/reader030/viewer/2022033017/56649ef05503460f94c0033a/html5/thumbnails/43.jpg)
X
Y
D
T
S
I
U
A C
B
M
N
R V
L
1
1.1
1.2
1.3
1.42.1
2
2.2
3 3.1
3.2
Example: Class Test Order for The Class Firewall
4.1
4