Vinai Kopp:

Property Based Testingin PHP

Other schools of thinking

Lots of inspiration for me

Clojure

Property Based Testing

Where Property Based Testing was invented

Haskell

“Don’t write tests. Generate them.”

Prof. John Hughes, Inventor of QuickCheck

Example Based Testing (EBT)

• Think of an example• Call method(s)• Verify result

Property Based Testing (PBT)

• Think of range of inputs• Let computer generate inputs• Call method(s)• Verify properties of result• If a failure is found, shrink inputs

Property Based Testing?

Advantages of Property Based Testing

• Replace many example-based tests• Find more bugs than example-based tests• Test complex systems• Test “black box” systems• Supports deeper thought about system under development

QuickCheck Implementations PHP

•

composer require --dev steos/quickcheck:dev-master

•

composer require --dev giorgiosironi/eris

What does it look like?

What does it look like?

Failed asserting that property is true.Test runs: 7, seed: 1580127726247, smallest shrunk value(s):array ( 0 => '�' . "\0" . '',)vendor/steos/quickcheck/src/QuickCheck/PHPUnit/PropertyConstraint.php:35test/ExampleStringTest.php:22

Failed asserting that property is true.Test runs: 9, seed: 1580132863563, smallest shrunk value(s):array ( 0 => '' . "\0" . '�' . "\0" . ‘‘,)vendor/steos/quickcheck/src/QuickCheck/PHPUnit/PropertyConstraint.php:35test/ExampleStringTest.php:22

When to use Property Based Testing

• During Design• During Implementation (like TDD)• After Implementation• Reproducing a bug

Figuring out the properties to test

• This is … not the easy thing (at first)

• But there are strategies to follow

Figuring out the properties to test

• reverse?• depend on the order of an input sequence?• depend on grouping of arguments? (Commutative)• depend on the order of arguments? (Associative)• have an identity value?• change it’s output if it is called multiple times? (Idempotent)

Algebraic PropertiesDoes the Algorithm…

Figuring out the properties to test

FunctionalityLike TDD, but better:Generate the input values instead of hardcoding them.

Do not re-implement functionality!

Figuring out the properties to test

1. Generate names for directories and files2. Create directories3. Create files4. Execute command (list files)5. Assert number of files matches created files

FunctionalityExample test for the ls CLI utility:

Figuring out the properties to test

Model ⬄ System

Modelling Stateful SystemsThe model behaves like the System Under Test (SUT),but it doesn’t use persistence or have a REST API.

Figuring out the properties to test

1. Create Model of System2. Generate Actions3. Apply actions to Model and System4. Check Model and System state match

Modelling Stateful SystemsThe model behaves like the System Under Test (SUT),but it doesn’t use persistence or have a REST API.

Figuring out the properties to test

Commerce is very stateful.Shopping Cart customizations tend to be stateful, too.

Mostly stateful

Property Based Testing in the Design Phase

• New Systems:Create model before starting with the real implementation

• Existing systems:The model can be partial (only the functionality to test)

• The model development is guided by tests (TDD like)• Building the model gives me a better understanding of the

real system

Property Based Testing in the Design Phase

Property Based Testing in the Design Phase

My current task:Downloadable Products for Shopware 6

(I’ve used the same approach in the Magento 2 context, too.)

For example...

Property Based Testing in the Design Phase

1. Sketch out operationsAdmin

• Create new• Add file• Replace file• Remove file• Delete product, keep downloads• Delete product, remove downloads

Customer

• Purchase downloadable• List available files• Download file

Reporting• Downloads per file• …

Property Based Testing in the Design Phase

2. Generate Action• Generate data for the initial operation on the system• Encode it as an array• Write test for that operation

Property Based Testing in the Design Phase

2. Generate Action

Property Based Testing in the Design Phase

3. Write Test

Property Based Testing in the Design Phase

4. Build Model

Property Based Testing in the Design Phase

1. Create generator for next action2. Think of way to verify action on model3. Write test4. Implement action on model5. GOTO 1

Build next Action...

Property Based Testing in the Design Phase

When the model is complete

... the real work begins:

Implement System with Property Based Tests(and example based tests where it makes sense).

Finally:Use the model to check the system!

Summary

• Be curious!Look what others are doing that is better than what we are doing!

• New skills require practice and patience. Be kind to yourself if something doesn’t work at first, but persist!

• PBT makes systems robust. It facilitates the thinking process as much as the coding process.

Thank You