For Developer Testing
description
Transcript of For Developer Testing
![Page 1: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/1.jpg)
For Developer Testing
Separation of Concerns
Tao XiePeking University, China
North Carolina State University, USAIn collaboration with Nikolai Tillmann, Peli de Halleux,
Wolfram Schulte @Microsoft Research and students @NCSU ASE
![Page 2: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/2.jpg)
Background – Separation of ConcernsSeparation of Concerns (SoC)
Aspect-Oriented Software Development (AOSD)
“Killer Apps” for SoC/AOSD? Logging, Security Checking, …
Any Other Application Scenarios in Practice? Tool Support for Software Testing?
![Page 3: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/3.jpg)
Software Testing Setup
=?Outputs
Expected
Outputs
Program
+
Test inputs
Test Oracles
![Page 4: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/4.jpg)
Software Testing Problems
=?Outputs
Expected
Outputs
Program
+
Test inputs
Test Oracles
Test Generation Generating high-quality test inputs (e.g.,
achieving high code coverage)
![Page 5: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/5.jpg)
Software Testing Problems
=?Outputs
Expected
Outputs
Program
+
Test inputs
Test Oracles
Test Generation Generating high-quality test inputs (e.g.,
achieving high code coverage)
Test Oracles Specifying high-quality test oracles (e.g.,
guarding against various faults)
![Page 6: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/6.jpg)
The Recipe of Unit/Dev Testing
var list = new ArrayList(); list.Add(item);
Assert.AreEqual(1, list.Count);}
Three essential ingredients: Data Method Sequence Assertionsvoid TestAdd() { int item = 3;
![Page 7: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/7.jpg)
OutlineParameterized Unit Testing
Separate Test Generation and Behavior Specification
Test Generalization Localize Crosscutting Concerns of Behavior Under
TestObject Creation with Factory Method
Localize Crosscutting Concerns of Object CreationEnvironment Isolation with Moles
Localize Crosscutting Concerns of Environment Dependency
![Page 8: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/8.jpg)
Parameterized Unit Testing
void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
Parameterized Unit Test = Unit Test with Parameters
Separation of concerns Data is generated by a tool Developer can focus on functional
specification
[Tillmann&Schulte ESEC/FSE 05]
![Page 9: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/9.jpg)
Dynamic Symbolic Execution
Code to generate inputs for:
Constraints to solve
a!=null a!=null &&a.Length>0
a!=null &&a.Length>0 &&a[0]==1234567890
void TestMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Observed constraints
a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890
a!=null &&a.Length>0 &&a[0]==1234567890
Datanull
{}
{0}
{123…}a==null
a.Length>0
a[0]==123…T
TF
T
F
F
Execute&MonitorSolveChoose next path
Done: There is no path left.
Negated condition
![Page 10: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/10.jpg)
Pex:Visual Studio Power Tool
Download counts (20 months)(Feb. 2008 - Oct. 2009 )
Academic: 17,366 Devlabs: 13,022 Total: 30,388
http://research.microsoft.com/projects/pex/
Dynamic Symbolic Execution
[Tillmann&de Halleux TAP 08]
![Page 11: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/11.jpg)
Pex for Fun Web-based Learning Tool
669,584 clicked 'Ask Pex!‘ since 2010 summer
Try it at http://www.pexforfun.com/
![Page 12: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/12.jpg)
Parameterized Unit TestingGetting PopularParameterized Unit Tests (PUTs) commonly
supported by various test frameworks .NET: Supported by .NET test frameworks
http://www.mbunit.com/ http://www.nunit.org/ …
Java: Supported by JUnit 4.X http://www.junit.org/
Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex
http://research.microsoft.com/Pex/ Java: Supported by Agitar AgitarOne
http://www.agitar.com/
![Page 13: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/13.jpg)
Parameterized vs. Conventional Unit Tests PUTs vs. CUTs
Benefits over CUTs Help describe behaviours for all test arguments
Address two main issues with CUTs Missing test data (that would exercise
important behavior)▪ Low fault-detection capability
Redundant test data and scenarios (that exercises the same behaviour)▪ Redundant unit tests
![Page 14: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/14.jpg)
OutlineParameterized Unit Testing
Separate Test Generation and Behavior Specification
Test Generalization Localize Crosscutting Concerns of Behavior Under
TestObject Creation with Factory Method
Localize Crosscutting Concerns of Object CreationEnvironment Isolation with Moles
Localize Crosscutting Concerns of Environment Dependency
![Page 15: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/15.jpg)
An Example using IntStack
public void CUT1() { int elem = 1; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count());}
Three CUTs
public void CUT2() { int elem = 30; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count());}
public void CUT3() { int elem1 = 1, elem2 = 30; IntStack stk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count());}
CUT1 and CUT2 exercise push with different test data
CUT3 exercises push when stack is not empty
Two main issues with CUTs: Fault-detection
capability: undetected defect where things go wrong when passing a negative value to push
Redundant tests: CUT2 is redundant with respect to CUT1
![Page 16: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/16.jpg)
Test Generalization: IntStack
public void CUT1() { int elem = 1; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count());}
Three CUTs
public void CUT2() { int elem = 30; IntStack stk = new IntStack(); stk.Push(elem); Assert.AreEqual(1, stk.Count());}
public void CUT3() { int elem1 = 1, elem2 = 30; IntStack stk = new IntStack(); stk.Push(elem1); stk.Push(elem2); Assert.AreEqual(2, stk.Count());}
No need to describe test data• Generated automatically
Single PUT replaces multiple CUTs• With reduced size of test
code
public void PUT(int[] elem) { Assume.IsTrue(elem != null); IntStack stk = new IntStack(); for(int i in elem) stk.push(elem); Assert.AreEqual(elem.Length, stk.count());}
An equivalent PUT
[Thummalapenta et al. FASE 11]
![Page 17: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/17.jpg)
Ring a Bell? Refactoring Clones?
Refactoring clone instances (crosscutting concerns) to a single copy (aspects) Need pointcuts to specify join points, i.e., which
original code locations to weave the aspects
Test generalization is different Original code locations (old CUTs) are thrown away PUTs + Pex generate new CUTs No need of join points or pointcuts
![Page 18: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/18.jpg)
OutlineParameterized Unit Testing
Separate Test Generation and Behavior Specification
Test Generalization Localize Crosscutting Concerns of Behavior Under
TestObject Creation with Factory Method
Localize Crosscutting Concerns of Object CreationEnvironment Isolation with Moles
Localize Crosscutting Concerns of Environment Dependency
![Page 19: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/19.jpg)
Object Creation
void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
![Page 20: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/20.jpg)
Object Creation Pex uses public methods to
configure non-public object fields Heuristics built-in to deal with
common types User can help if needed[PexFactoryMethod(typeof(ArrayList))]public static ArrayList Create(int capacity, object[] items) { var list = new ArrayList(capacity); foreach (var item in items) list.Add(item); return list;}
![Page 21: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/21.jpg)
“Join Point” for Factory Method
[PexFactoryMethod(typeof(ArrayList))]public static ArrayList Create(int capacity, object[] items) { var list = new ArrayList(capacity); foreach (var item in items) list.Add(item); return list;}
void TestAdd(ArrayList list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
![Page 22: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/22.jpg)
OutlineParameterized Unit Testing
Separate Test Generation and Behavior Specification
Test Generalization Localize Crosscutting Concerns of Behavior Under
TestObject Creation with Factory Method
Localize Crosscutting Concerns of Object CreationEnvironment Isolation with Moles
Localize Crosscutting Concerns of Environment Dependency
![Page 23: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/23.jpg)
Unit Testing is Not That Easy
Components depend on other components
Hidden Integration Testsvoid FileExistsTest() { File.Write(“foo.txt”, “”); var result = IsFileEmpty(“foo.txt”) Assert.IsTrue(result);}
bool IsFileEmpty(string file) { var content = File.ReadAllText(file); return content.Length == 0;}
File.ReadAllText(file);
File.Write(“foo.txt”, “”);
![Page 24: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/24.jpg)
Isolation is Critical Slow, complicated setup, non-deterministic
tests Solution: Replace by Simpler Environment
(“mocking”)
Testable Design: Abstraction layer + Dependency Injection + Mocks for testing Simply use virtual methods (i.e., overridenable)
Hard-coded Design: No abstraction layer, static methods, sealed types (i.e., not inheritable).
![Page 25: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/25.jpg)
Stubs and Moles Framework
Replace any .NET method with a Delegate similar to a C++ function pointer
Method can be overridden? Use Stubs Interfaces Abstract classes Virtual methods in non-sealed types
Method cannot be overridden? Use Moles
![Page 26: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/26.jpg)
Hard-coded Design Existing external components cannot
be re-factored SharePoint, ASP .NET, …
Need mechanism to stub non-virtual methods Static methods, methods in sealed types,
constructors
![Page 27: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/27.jpg)
Moles – Delegate Based Detours Method redirected to user delegate, i.e.
moled
Requires code instrumentation
bool result = IsFileEmpty(“foo.txt”);Assert.IsTrue(result);
MFile.ReadAllTextString = file => “”; expression lambda : (input parameters) => expression
[de Halleux&Tillmann TOOLS 10]
![Page 28: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/28.jpg)
Moles under the Hood
File.ReadAllText(string name) {
}
mscorlib.dll
File.ReadAllText(string name) { var d = GetDetour(); if (d != null) return d();
}
push ecxpush edxpush eax
.NET RuntimeJust In Time
Compiler
ExtendedReflection
![Page 29: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/29.jpg)
Isolated Parameterized Unit Testing Automated White box Analysis does not
‘understand’ the environment
Isolate Code using Stubs and Moles
if (DateTime.Now == new DateTime(2000,1,1)) throw new Y2KException();
DateTime.Now
???
public void TestY2k(DateTime dt) { MDateTime.NowGet = () => dt ...}
MDateTime.NowGet = () => dt DateTime.Now ==
dt
![Page 30: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/30.jpg)
Separation of Concerns for Developer TestingParameterized Unit Testing
Separate Test Generation and Behavior Specification
Test Generalization Localize Crosscutting Concerns of Behavior Under
TestObject Creation with Factory Method
Localize Crosscutting Concerns of Object CreationEnvironment Isolation with Moles
Localize Crosscutting Concerns of Environment DependencyApplication Scenarios of Separation of
Concerns in Practice
![Page 31: For Developer Testing](https://reader035.fdocuments.us/reader035/viewer/2022062323/568165f5550346895dd91b6f/html5/thumbnails/31.jpg)
Thank you!
Questions ?
https://sites.google.com/site/asergrp/
http://research.microsoft.com/projects/pex/http://
www.pexforfun.com/
http://research.microsoft.com/en-us/projects/pex/community.aspx#publicationshttp://
pexase.codeplex.com/