A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification...
Transcript of A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification...
![Page 1: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/1.jpg)
A Coq Framework For Verified Property-Based Testing
(part of QuickChick)
Cătălin Hrițcu INRIA Paris-Rocquencourt
(Prosecco team, Place d’Italie office)
![Page 2: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/2.jpg)
Problem: proving in Coq is very costly
• My proofs are boring, but designing security mechanisms is not
– definitions and properties often broken, and evolve over time
2
![Page 3: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/3.jpg)
Problem: proving in Coq is very costly
• My proofs are boring, but designing security mechanisms is not
– definitions and properties often broken, and evolve over time
• Proving does aid design ... but only at a very high cost
– most enlightenment comes from failed, not from successful proofs
a failed proof attempt is a very costly way to discover a design flaw
fixing flaws not always easy, might require serious redesign
– failed proof attempt will generally not convince an engineer
proving while designing is frustrating, tedious, time consuming
3
![Page 4: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/4.jpg)
Problem: proving in Coq is very costly
• My proofs are boring, but designing security mechanisms is not
– definitions and properties often broken, and evolve over time
• Proving does aid design ... but only at a very high cost
– most enlightenment comes from failed, not from successful proofs
a failed proof attempt is a very costly way to discover a design flaw
fixing flaws not always easy, might require serious redesign
– failed proof attempt will generally not convince an engineer
proving while designing is frustrating, tedious, time consuming
Even when design correct & stable, proving still costly
– countless iterations for discovering lemmas and invariants
– my proofs are often “fragile”, so the cost of each iteration is high
4
![Page 5: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/5.jpg)
Problem: proving in Coq is very costly
• My proofs are boring, but designing security mechanisms is not
– definitions and properties often broken, and evolve over time
• Proving does aid design ... but only at a very high cost
– most enlightenment comes from failed, not from successful proofs
a failed proof attempt is a very costly way to discover a design flaw
fixing flaws not always easy, might require serious redesign
– failed proof attempt will generally not convince an engineer
proving while designing is frustrating, tedious, time consuming
Even when design correct & stable, proving still costly
– countless iterations for discovering lemmas and invariants
– my proofs are often “fragile”, so the cost of each iteration is high
5
• This is the itch I’m trying to scratch
– many people seem to have similar itches though
![Page 6: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/6.jpg)
Could testing help with this problem?
• Can property-based testing
– lower the cost of formal proofs?
– become an important part of the theorem proving process in Coq?
6
![Page 7: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/7.jpg)
Could testing help with this problem?
• Can property-based testing
– lower the cost of formal proofs?
– become an important part of the theorem proving process in Coq?
• Yes, I believe / hope so
own recent positive experience with testing
I’m not the only one (e.g. Isabelle, FocalTest, ...)
7
![Page 8: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/8.jpg)
Could testing help with this problem?
• Can property-based testing
– lower the cost of formal proofs?
– become an important part of the theorem proving process in Coq?
• Yes, I believe / hope so
own recent positive experience with testing
I’m not the only one (e.g. Isabelle, FocalTest, ...)
We are basically just starting on this
– A lot of research & engineering work left
8
![Page 9: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/9.jpg)
Collaborators
9
Maxime Dénès (Inria)
John Hughes (Chalmers)
Leo Lampropoulos (UPenn)
Benjamin Pierce (UPenn)
Antal Spector-Zabusky (UPenn)
Dimitris Vytiniotis (MSR Cambridge)
Zoe Paraskevopoulou (ENS Cachan, MPRI, ..
recent Inria intern)
Arthur Azevedo de Amorim (UPenn, recent Inria intern)
![Page 10: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/10.jpg)
This talk
• Property-based testing with QuickChick
• Our QuickCheck clone for Coq (prototype plugin)
• Everything at https://github.com/QuickChick
• Framework for verified property-based testing
• Other things we are doing that I won’t discuss today
• Case studies: noninterference, security monitors, type-checkers
• Relating executable and declarative artifacts in Coq/SSReflect
• Language for property-based generators
• Evaluating testing quality: polarized mutation testing
10
![Page 11: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/11.jpg)
TESTING RED-BLACK TREES
Property-based testing with QuickChick
11
Maxime Dénès (Inria)
Leo Lampropoulos (UPenn)
Zoe Paraskevopoulou (ENS Cachan, MPRI, ..
recent Inria intern)
![Page 12: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/12.jpg)
12
Red-Black Tree Implementation
![Page 13: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/13.jpg)
Red-Black Trees Implementation
13
![Page 14: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/14.jpg)
Declarative Proposition
14
![Page 15: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/15.jpg)
Executable Definitions
15
![Page 16: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/16.jpg)
Property Checker
16
![Page 17: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/17.jpg)
Custom Generator for Trees
17
![Page 18: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/18.jpg)
Running QuickChick
18
![Page 19: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/19.jpg)
Finding a Bug
19
![Page 20: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/20.jpg)
Property-Based Generator
20
...........
![Page 21: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/21.jpg)
Property-Based Generator at Work
22
in less than 4 seconds
![Page 22: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/22.jpg)
VERIFIED PROPERTY-BASED TESTING
Are we testing the right property?
23
Zoe Paraskevopoulou (ENS Cachan, MPRI, ..
recent Inria intern)
![Page 23: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/23.jpg)
Testing Code Can Be Wrong
• QuickChick user has to write effective checkers and generators by hand
– [working on a new language in which one can write both generator and checker as a single program]
– errors can result in testing the wrong conjecture
– randomness makes finding and fixing errors hard
24
![Page 24: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/24.jpg)
Testing Code Can Be Wrong
• QuickChick user has to write effective checkers and generators by hand
– [working on a new language in which one can write both generator and checker as a single program]
– errors can result in testing the wrong conjecture
– randomness makes finding and fixing errors hard
• User generators and checkers + most of QuickChick itself written in Coq
– Can formally we verify them?
25
![Page 25: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/25.jpg)
Verified Property-Based Testing
• Verification framework on top of QuickChick
• Prove correctness of generators and checkers with respect to their declarative specs
• Main novelty: set of outcomes abstraction
– sem. of generator (Gen A) is an Ensemble (A -> Prop)
• the set of values that can be generated with >0 probability
– semantics of checker is a Coq proposition (Prop)
• internally checkers are also generators (Gen Result)
• all results are successful
26
![Page 26: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/26.jpg)
27
![Page 27: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/27.jpg)
Proving correctness of generators
28
![Page 28: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/28.jpg)
Proving correctness of generators
29
![Page 29: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/29.jpg)
Proving correctness of generators
30
![Page 30: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/30.jpg)
Proving correctness of generators
31
![Page 31: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/31.jpg)
Proving correctness of generators
32
![Page 32: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/32.jpg)
Proving correctness of generators
33
![Page 33: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/33.jpg)
Proving correctness of checkers
34
Relating Executable and Declarative Definitions (SSReflect Style)
![Page 34: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/34.jpg)
35
![Page 35: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/35.jpg)
Axioms for Primitive Combinators
36
![Page 36: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/36.jpg)
Lemmas for Derived Generators
37
![Page 37: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/37.jpg)
Lemmas for Checkers
38
![Page 38: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/38.jpg)
Future Work
• More proof automation and infrastructure – changing to efficient data representations
– SMT-based verif. for set of outcome abstraction?
• The first verified QuickCheck implementation – reduce the number of axioms
– probabilistic verification?
• Verify property-based generator language – in general, manually verify reusable infrastructure
• Motto: premature automation is the root of all evil
39
![Page 39: A Coq Framework For Verified Property-Based …...Verified Property-Based Testing •Verification framework on top of QuickChick •Prove correctness of generators and checkers with](https://reader033.fdocuments.us/reader033/viewer/2022043000/5f75123f95f7ce12f4183eaa/html5/thumbnails/39.jpg)
THANK YOU Code at https://github.com/QuickChick
40