QuickCheck@Neilfest(or what I learned from Neil about software testing)
John HughesChalmers University/Quviq AB
• Company founded May 2006– Tools for testing software– Profitable in the first year!
• Customers include…
Soon TBA
QuickCheck
• Tests software automatically against a formal specification
prop_reverse() -> ?FORALL({Xs,Ys},
{list(int()),list(int())}, reverse(Xs++Ys) ==
reverse(Xs)++reverse(Ys)).
Testing…
19> eqc:quickcheck(ex:prop_reverse())..........Failed! After 10 tests.{[2],[3,2]}Shrinking....(4 times){[0],[1]}false
QuickCheck
• Tests software automatically against a formal specification
prop_reverse() -> ?FORALL({Xs,Ys},
{list(int()),list(int())}, reverse(Xs++Ys) ==
reverse(Xs)++reverse(Ys)).
QuickCheck in Brief
Specification Test case Test outcome
Generate Execute
Simplify
Minimal counter- example
e.g. {[0],[1]}
A More Realistic Test Case
• A test case for the Erlang process registry
test_register() -> Pid = spawn(), register(name,Pid), Pid2 = whereis(name), assert(Pid==Pid2), unregister(name).
Spawn a new process (set up test data)
Register it—a side-effect
Inspect the results
Did the test pass?Restore the state, ready for next test
Industrial Test Cases
• A sequence of– Calls to an API under test– Checks on the results
• Not much like {[0],[1]}…
• How can we convert this kind of test case into a logical property?
QuickCheck in Industry
Specification Test case: a program
Test outcome
Generate Execute
Simplify
Minimal program
provoking a fault
Programs as Data Objects
• Why?– So failing test cases can be printed– So failing test cases can be repeated– So failing test cases can be simplified
Can we mix generation and execution?
• Randomly generate and perform each call– Save a test case as a list of functions and actual
parameters?– NO!!!
• When we repeat the test, Pid has a different value– We must use a symbolic variable!
test_register() -> Pid = spawn(), register(name,Pid),…
Test Case Language
• Do we need conditionals?
• Tests should be deterministic– Check the right branch, don’t test
• Users need simplicity—straight-line code is simple, and suffices
X = foo(…),case p(X) of true -> baz(X); false -> bar(X)end
X = foo(…),assert not(p(X)),bar(X)
Generating Test Cases
• How do we know– Which commands are valid at each point?– What test data is available at each point?– What results are expected?
• Track a test case state– Check preconditions before generating each
command– Store available data in the state– Check postconditions wrt test case state
Process Registry State
type state() = record pids::list(pid()), regs::list({atom(),pid()})end.
Available process ids
Currently registered processes
Process Registry State
• Two-level programs!– Static = test case generation time– Dynamic = test execution time
type state() = record pids::list(symbolic(pid())), regs::list({atom(),symbolic(pid())})end.
Available process ids
Currently registered processes
Two-level State Machines
• Preconditions– Checked during generation, purely static
• Postconditions– Checked during execution, purely dynamic
• Next state function– Used at both times, two-level
• Command generator– Used during generation, two-level
Process Registry
• spawn()– Adds a (dynamic) pid to the state
• unregister(Name)– Pre: Name must be registered– Removes Name from the state
• register(Name,Pid)– Adds {Name,Pid} to the state– Post: exception if Name or Pid already registered
A Failing Test Case
[{set,{var,2},{call,…,spawn,[]}}, {set,{var,3},{call,…,register,[a,{var,2}]}}, {set,{var,6},{call,…,register,[b,{var,2}]}}, {set,{var,8},{call,…,spawn,[]}}, {set,{var,9},{call,…,register,[b,{var,8}]}}]
V2=spawn(),register(a,V2),register(b,V2),V8=spawn(),register(b,V8)
Code Spec
Inconsistency!
A Buggy Spec
• spawn()– Adds a (dynamic) pid to the state
• unregister(Name)– Pre: must be registered– Removes Name from the state
• register(Name,Pid)– Adds {Name,Pid} to the state– Post: exception if Name or Pid already registered
…provided there is no exception!
”The Trick”
• What if we know part of the structure?– Check the known part in the postcondition– Add the known part statically to the state
• ”The Trick” is as useful as ever!
[{set,{var,2},{call,…,spawn,[]}}, …]
Added to the state Purely dynamic
Test Case Generator
Two-level languages are helping to find thorny bugs in industrial systems!
Randomized Generating Extension
=
Best Bug!• In Ericsson’s Media Proxy (Multimedia IP-
telephony product, SIP)
• A serious bug• Obtained from a simple specification, by
simplifying 160-command sequence!
Add Add Sub Add Sub Add Sub
Top Related