QuickCheck decreasing your test effort by a factor...
Transcript of QuickCheck decreasing your test effort by a factor...
![Page 1: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/1.jpg)
QuickCheck decreasing your test effort by
a factor five Thomas Arts
Chalmers
![Page 2: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/2.jpg)
Message
Software systems more complex every day... ... features to be released faster ... quality needs to increase
Traditional testing insufficient to keep up with this We need to change our testing methods!
October 14, 2010 Thomas Arts
![Page 3: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/3.jpg)
October 14, 2010 Thomas Arts
QuickCheck
From test case to property
Instead of specifying one or two test cases to demonstrate that the software fulfills a certain property, we specify the property and have the tests automatically generated!
Approach is based upon: controlled random generation of test cases On different abstraction level
Model based testing
![Page 4: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/4.jpg)
Model
We specify our model of the software in a programming language called Erlang
We specify the property to hold also in Erlang
Test cases are generated and run against C code
Failing tests are automatically reduced to shorter test case
October 14, 2010 Thomas Arts
![Page 5: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/5.jpg)
SMS encode/decode Encoding SMS messages
GSM standard Technical Specification 03.38
Packing of 7 bit characters
10 December 2007 Thomas Arts 5
![Page 6: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/6.jpg)
Example
Interpretation 3 characters containing 7 bits packets 8 bits
a b c d e f g a b c d e f g a b c d e f g
10 December 2007 Thomas Arts 6
![Page 7: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/7.jpg)
Example
0 0 0
a b c d e f a b c d e f f
a b c d e
pack
unpack
1 0 0 1 0 0 0 1 1 0 0 1 0
1
1 1 0 1 0 1 0
1 0 0 1 0 0 0 1 1 0 0 1 0 1 1 1 0 1 0 1 0
“Ȳ^Z” = [200,178,26] “Hej” = [72,101,106]
10 December 2007 Thomas Arts 7
![Page 8: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/8.jpg)
DEMO
October 14, 2010 Thomas Arts
![Page 9: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/9.jpg)
Real software
Real software is not as simple
• Ejabberd (XMPP server) • Ericsson’s Media gateway (IMS) • Ericsson’s SIP stack (IP telephony) • Radio base station for 3G • Embedded printer software • SMS/Email gateway • AUTOSAR components • ....
October 14, 2010 Thomas Arts
Testing effort from 20 to 2 weeks! Model tests more. For next iteration, only 1 or 2 days needed
![Page 10: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/10.jpg)
AUTOSAR component as UML state machine
CAN Network Management
October 14, 2010 Thomas Arts
![Page 11: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/11.jpg)
Scheduling
CanNM is scheduled as one of many tasks
October 14, 2010 Thomas Arts
CanNm invoked by calling C function CanNM_Main()
![Page 12: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/12.jpg)
Scheduling
CanNM is scheduled as one of many tasks
October 14, 2010 Thomas Arts
Assumption: One time unit elapses before CanNm_Main() is called
(In fact, C implementation handles the timers, not the scheduler)
![Page 13: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/13.jpg)
Scheduling
CanNM is scheduled as one of many tasks
October 14, 2010 Thomas Arts
Other tasks communicate by calling CanNM interface functions
These update data structures in memory
CanNm_RxIndication() CanNm_NetworkRequest()
![Page 14: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/14.jpg)
Scheduling
CanNM is scheduled as one of many tasks
October 14, 2010 Thomas Arts
A Test is a sequence of events that are provoked by the test
We generate such tests automatically
CanNm_RxIndication() CanNm_NetworkRequest()
![Page 15: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/15.jpg)
AUTOSAR component as UML state machine
CAN Network Management
Now... make a QuickCheck model from this state machine
![Page 16: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/16.jpg)
QuickCheck model
State transitions as Erlang data structure
bus_sleep_mode(_) -> [ {power_off,{call,?MODULE,powerOff,[]}}, {bus_sleep_mode,{call,?MODULE,main,[]}}, {bus_sleep_mode,{call,?MODULE,'CanNm_RxIndication',[id(),u8()]}}, {repeat_message_state,{call,?MODULE,'Nm_PassiveStartUp',[]}}, {repeat_message_state,{call,?MODULE,'CanNm_NetworkRequest',[]}}].
repeat_message_state(_) -> [ {normal_operation_state,{call,?MODULE,main,[]}}, {ready_sleep_state,{call,?MODULE,main,[]}}, {repeat_message_state,{call,?MODULE,main,[]}}, {repeat_message_state,{call,?MODULE,'CanNm_RxIndication',[id(),u8()]}}, {repeat_message_state,{call,?MODULE,'CanNm_TxConfirmation',[id()]}}].
October 14, 2010 Thomas Arts
![Page 17: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/17.jpg)
QuickCheck model
Model how additional state data changes: timers, network status, ... next_state_data(repeat_message_state,repeat_message_state,S,_V,{_,_,main,_}) -> S#can_nm{repeatMessageTimer = S#can_nm.repeatMessageTimer-1,
nmTimeoutTimer =
case S#can_nm.nmTimeoutTimer of
0 -> ?NMTIMEOUT;
N -> N-1
end};
next_state_data(repeat_message_state,repeat_message_state,S,_V,{_,_,_,_}) -> S#can_nm{repeatMessageTimer = S#can_nm.repeatMessageTimer-1,
nmTimeoutTimer = ?NMTIMEOUT};
October 14, 2010 Thomas Arts
![Page 18: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/18.jpg)
QuickCheck model
Property
For any sequence of permissible events: after each event:
the real system should be in the same state as the model system
October 14, 2010 Thomas Arts
![Page 19: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/19.jpg)
What Makes Quickcheck Great?
• Traditional Testing – Each sequence is
• Hard-coded • Picked manually
• Test cases tend to be large to cover tricky cases
• Quickcheck testing – Each testis
• Randomly chosen • Generic
• Test cases shrink to small failing test cases
October 14, 2010 Thomas Arts
![Page 20: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/20.jpg)
QuickCheck: 3 basic ideas
Write Properties instead of Test Cases
October 14, 2010
Test Suite
Test Suite
Test Suite
Less test code, more tested
Thomas Arts
![Page 21: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/21.jpg)
QuickCheck: 3 basic ideas
Find smaller failing tests automatically
October 14, 2010
less debugging
Thomas Arts
![Page 22: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/22.jpg)
QuickCheck: 3 basic ideas
Testing and Property specification part of Coding
October 14, 2010
Product Code
Properties
Test cases
Get tricky code correct quickly
Thomas Arts
![Page 23: QuickCheck decreasing your test effort by a factor fivesast.se/q-moten/2010/stockholm/q15/SAST_Q15_Arts_QuickCheck.pdf · GSM standard Technical Specification 03.38 Packing of 7 bit](https://reader030.fdocuments.us/reader030/viewer/2022040913/5e893a8513f46577fe591e66/html5/thumbnails/23.jpg)
Five times more productive
QuickCheck shows: With model based testing one can gain at least five times more productivity in testing
Model based testing makes software development more effective (feedback loop)
Models are easier to communicate than test cases
Creating models is a creative exercise and a lot of fun.
October 14, 2010 Thomas Arts