Property-based testing, race conditions, and …...Property-based testing •Finds bugs in...
Transcript of Property-based testing, race conditions, and …...Property-based testing •Finds bugs in...
![Page 1: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/1.jpg)
Property-based testing, race conditions, and QuickCheck
John Hughes
![Page 2: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/2.jpg)
QuickCheck in a Nutshell
Properties Test case Test case Test case Test case Test case
MinimalTest case
![Page 3: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/3.jpg)
Benefits
• Less time spent writing test code
– One property replaces many tests
• Better testing
– Lots of combinations you’d never test by hand
• Less time spent on diagnosis
– Failures minimized automagically
![Page 4: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/4.jpg)
Tests for Base 64 encoding
base64_encode(Config) when is_list(Config) ->
%% Two pads
<<"QWxhZGRpbjpvcGVuIHNlc2FtZQ==">> =
base64:encode("Aladdin:open sesame"),
%% One pad
<<"SGVsbG8gV29ybGQ=">> = base64:encode(<<"Hello World">>),
%% No pad
"QWxhZGRpbjpvcGVuIHNlc2Ft" =
base64:encode_to_string("Aladdin:open sesam"),
"MDEyMzQ1Njc4OSFAIzBeJiooKTs6PD4sLiBbXXt9" =
base64:encode_to_string(
<<"0123456789!@#0^&*();:<>,. []{}">>),
ok.
Test cases
Expected results
![Page 5: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/5.jpg)
Writing a Property
prop_base64() ->
?FORALL(Data,list(choose(0,255)),
base64:encode(Data)==???).
![Page 6: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/6.jpg)
Back to the tests…
base64_encode(Config) when is_list(Config) ->
%% Two pads
<<"QWxhZGRpbjpvcGVuIHNlc2FtZQ==">> =
base64:encode("Aladdin:open sesame"),
%% One pad
<<"SGVsbG8gV29ybGQ=">> = base64:encode(<<"Hello World">>),
%% No pad
"QWxhZGRpbjpvcGVuIHNlc2Ft" =
base64:encode_to_string("Aladdin:open sesam"),
"MDEyMzQ1Njc4OSFAIzBeJiooKTs6PD4sLiBbXXt9" =
base64:encode_to_string(
<<"0123456789!@#0^&*();:<>,. []{}">>),
ok.
Where did these come
from?
![Page 7: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/7.jpg)
Possibilities
• Someone converted the data by hand
• Another base64 encoder
• The same base64 encoder!
Use the other encoder as an
oracle
Use an old version (or a
simpler version) as an oracle
![Page 8: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/8.jpg)
Round-trip Properties
What does this test?
• NOT a complete test—will not find a consistent misunderstanding of base64
• WILL find mistakes in encoder or decoder
Simple properties find a lot of bugs!
prop_encode_decode() ->
?FORALL(L,list(choose(0,255)),
base64:decode(base64:encode(L))
== list_to_binary(L)).
![Page 9: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/9.jpg)
Let’s test some C!
![Page 10: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/10.jpg)
Modelling in Erlang
API
Calls API
Calls API
Calls API
Calls
Model state
Model state
Model state
Model state
postconditions
![Page 11: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/11.jpg)
Example
put 2
get put 3
get put 1
[1] [1,2] [2] [2,3] []
1 2
![Page 12: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/12.jpg)
Code Fragments
next_state_data(_,_,S,_,{call,_,put,[_,X]}) -> S#state{elements=S#state.elements++[X]};
next_state_data(_,_,S,_,{call,_,get,_}) -> S#state{elements=tl(S#state.elements)};
postcondition(_,_,S,{call,_,get,_},Res) -> Res == hd(S#state.elements);
postcondition(_,_,S,{call,_,size,_},Res) -> Res == length(S#state.elements);
![Page 13: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/13.jpg)
A QuickCheck Property
prop_q() ->
?FORALL(Cmds,commands(?MODULE),
begin
{H,S,Res} = run_commands(?MODULE,Cmds),
Res == ok)
end).
![Page 14: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/14.jpg)
Let’s run some tests…
![Page 15: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/15.jpg)
Lessons
• One property can find many bugs
• Shrinking makes diagnosis very simple
![Page 16: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/16.jpg)
Doing it for real
• AUTOSAR
– Embedded software in cars
– ~50 standard components
• Test code from three suppliers against QuickCheck models
• 95 issues found already!
COM component: 500 pages of documents, 250 pages of C code… 25 pages of QuickCheck specification
![Page 17: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/17.jpg)
![Page 18: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/18.jpg)
![Page 19: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/19.jpg)
"We know there is a lurking bug somewhere in the dets code. We have got 'bad object' and 'premature eof' every other month the last year. We have not been able to track the bug down since the dets files is repaired automatically next time it is opened.“
Tobbe Törnqvist, Klarna, 2007
![Page 20: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/20.jpg)
What is it?
Application
Mnesia
Dets
File system
Invoicing services for web shops
Distributed database: transactions, distribution, replication
Tuple storage
500+ people in 5 years
Race conditions?
![Page 21: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/21.jpg)
Imagine Testing This…
dispenser:take_ticket() dispenser:reset()
![Page 22: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/22.jpg)
ok =
1 =
2 =
3 =
ok =
1 =
A Unit Test in Erlang
test_dispenser() ->
Expected results
BUT…
reset(),
take_ticket(),
take_ticket(),
take_ticket(),
reset(),
take_ticket().
![Page 23: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/23.jpg)
A Parallel Unit Test
• Three possible correct outcomes!
reset
take_ticket
take_ticket
take_ticket
1 2
3
1 3
2
1 2
1
ok
![Page 24: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/24.jpg)
Another Parallel Test
• 42 possible correct outcomes!
reset
take_ticket
take_ticket
take_ticket
take_ticket
reset
![Page 25: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/25.jpg)
Modelling the dispenser
reset take take take
0 0 1 2
ok 1 2 3
![Page 26: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/26.jpg)
Parallel Test Cases
resetok
take1
take3
take2
0 0 1 2
![Page 27: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/27.jpg)
prop_parallel() ->
?FORALL(Cmds,parallel_commands(?MODULE),
begin
start(),
{H,Par,Res} =
run_parallel_commands(?MODULE,Cmds),
Res == ok)
end)).
Generate parallel test cases
Run tests, check for a matching serialization
![Page 28: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/28.jpg)
Let’s run some tests
![Page 29: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/29.jpg)
Prefix: Parallel: 1. take_ticket() --> 1 2. take_ticket() --> 1 Result: no_possible_interleaving
take_ticket() -> N = read(), write(N+1), N+1.
![Page 30: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/30.jpg)
dets
• Tuple store:
{Key, Value1, Value2…}
• Operations:
– insert(Table,ListOfTuples)
– delete(Table,Key)
– insert_new(Table,ListOfTuples)
– …
• Model:
– List of tuples (almost)
![Page 31: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/31.jpg)
QuickCheck Specification
... …
... … <100 LOC
> 6,000 LOC
![Page 32: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/32.jpg)
DEMO
• Sequential tests to validate the model
• Parallel tests to find race conditions
![Page 33: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/33.jpg)
Bug #1
Prefix:
open_file(dets_table,[{type,bag}]) -->
dets_table
Parallel:
1. insert(dets_table,[]) --> ok
2. insert_new(dets_table,[]) --> ok
Result: no_possible_interleaving
insert_new(Name, Objects) -> Bool Types: Name = name() Objects = object() | [object()] Bool = bool()
![Page 34: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/34.jpg)
Bug #2
Prefix:
open_file(dets_table,[{type,set}]) --> dets_table
Parallel:
1. insert(dets_table,{0,0}) --> ok
2. insert_new(dets_table,{0,0}) --> …time out…
=ERROR REPORT==== 4-Oct-2010::17:08:21 === ** dets: Bug was found when accessing table dets_table
![Page 35: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/35.jpg)
Bug #3
Prefix:
open_file(dets_table,[{type,set}]) --> dets_table
Parallel:
1. open_file(dets_table,[{type,set}]) --> dets_table
2. insert(dets_table,{0,0}) --> ok
get_contents(dets_table) --> []
Result: no_possible_interleaving !
![Page 36: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/36.jpg)
Is the file corrupt?
![Page 37: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/37.jpg)
Bug #4
Prefix:
open_file(dets_table,[{type,bag}]) --> dets_table
close(dets_table) --> ok
open_file(dets_table,[{type,bag}]) --> dets_table
Parallel:
1. lookup(dets_table,0) --> []
2. insert(dets_table,{0,0}) --> ok
3. insert(dets_table,{0,0}) --> ok
Result: ok
premature eof
![Page 38: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/38.jpg)
Bug #5
Prefix:
open_file(dets_table,[{type,set}]) --> dets_table
insert(dets_table,[{1,0}]) --> ok
Parallel:
1. lookup(dets_table,0) --> []
delete(dets_table,1) --> ok
2. open_file(dets_table,[{type,set}]) --> dets_table
Result: ok
false
bad object
![Page 39: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/39.jpg)
"We know there is a lurking bug somewhere in the dets code. We have got 'bad object' and 'premature eof' every other month the last year.”
Tobbe Törnqvist, Klarna, 2007
Each bug fixed the day after reporting the failing case
![Page 40: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/40.jpg)
Before
• Files over 1GB?
• Rehashing?
• > 6 weeks of effort!
After
• Database with one record!
• 5 calls to reproduce
• < 1 day to fix
![Page 41: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/41.jpg)
• Particularly good for finding feature interactions—such as race conditions
• 100% code coverage is only the beginning…
Hand-written test suites test one
feature at a time
Generated tests can test many
features, in unexpected
combinations
![Page 42: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/42.jpg)
Property-based testing
• Finds bugs in everything it’s applied to!
![Page 43: Property-based testing, race conditions, and …...Property-based testing •Finds bugs in everything it’s applied to! Better Testing—Less Work Title Better Testing with Less Work:](https://reader034.fdocuments.us/reader034/viewer/2022042303/5ece2e3fee11c142a623df66/html5/thumbnails/43.jpg)
Better Testing—Less Work