Property-based testing of XMPP: generate your tests automatically - ejabberd Workshop #1

Property-based testing of XMPP:generate your tests automatically

Thomas ArtsQuviq AB

Why is testing hard?

n feature

sO(n) test cases

3—4 tests per featurepairs of features

O(n2) test cases

triples of features

O(n3) test cases

race conditions

Don’t write tests!

Generate themfrom properties

Example of properties

Round trip propertieshttp://www.w3.org/TR/exi-evaluation/

A.10. Roundtrip SupportA format supports roundtripping if converting a file from XML to that format and back produces an output equivalent to the original input. ...This property is measured by comparing the data which can be represented in XML with those that can be represented in the EXI format:

• Evaluation of Roundtrip Support (XML to EXI to XML):.... If the transformations to and from the EXI format are byte preserving, the measurement is "Exact Equivalence". Otherwise, the measurement is "Lossless Equivalence".

Advanced Erlang initiative

Paris, Nov 2015

http://www.w3.org/TR/exi-evaluation/

http://www.w3.org/TR/exi-evaluation/

Compress and Uncompress example

property "Compress HTML strings" do forall html <- :eqc_gen_html.html_string do ensure :zlib.uncompress(:zlib.compress(html)) == html endend

DEMO

Running QuickCheck

Failed! After 1 tests.not ensured: "<html> <head> <title> </title>\n </head>\n<body></body></html>\n" == '<html> <head> <title> </title>\n </head>\n<body></body></html>\n'Shrinking ........(8 times)not ensured: "<html><head><title/></head><body/></html>" == '<html><head><title/></head><body/></html>’

1) test Property Compress HTML strings (XmlEqc) test/elixir/xml_eqc.exs:11 forall(html <- :eqc_gen_html.html_string()) do ensure(:zlib.uncompress(:zlib.compress(html)) == html) end Failed for '<html><head><title/></head><body/></html>' Finished in 1.3 seconds (0.07s on load, 1.2s on tests)1 test, 1 failure


Paris, Nov 2015

When everything fails...

The Erlang documentation (R18)


Paris, Nov 2015

Compress and Uncompress example

property "Compress HTML strings" do forall html <- :eqc_gen_html.html_string do ensure :zlib.uncompress(:zlib.compress(html)) == html endend

CONTINUE

DEMO

From Unit test to Property

An example from Ejabberd: Testing an XML parser

We need to be able to read XML data in XMPP communication!

<?xml version='1.0'?><stream:stream to='example.com’ xmlns='jabber:client’ xmlns:stream='http://etherx.jabber.org/streams' version='1.0'>

<message from='[email protected]' to='[email protected]' xml:lang='en'> <body>Neither, fair saint, if either thee dislike.</body> </message>


Paris, Nov 2015

XML parser


<stream:stream to='example.com’ xmlns='jabber:client’ xmlns:stream='http://etherx.jabber.org/streams' version='1.0'>

XMLparser

parse tree

XML

Paris, Nov 2015

Typical Unit tests in language X


Paris, Nov 2015

Unit tests quickly get boring to write

Getting test data

How do we get our input data?

• We can write nifty XML pages ourselves...– what about things we don’t consider ?

• There’s enough XML on the web, take some...– how do we get the other side of assertion ?– how to debug when it fails ?


Paris, Nov 2015

Instead use random XML with shrinking

Property

What property to use?

forall xml_chunk <- xml do ensure parse(xml_chunk) == ???end


Paris, Nov 2015

Can we use a round-trip property?

Inverse of an XML parser


<stream:stream to='example.com’ xmlns='jabber:client’ xmlns:stream='http://etherx.jabber.org/streams' version='1.0'>

???

XML

parse tree

Paris, Nov 2015

pretty printer

Property

Round trip XML -> Parse Tree -> XML

forall xml_chunk <- xml do ensure pp( parse(xml_chunk) ) == xml_chunkend

not ensured: "<html><head><title/></head><body/></html>" == "<html><head><title/></head><body></body></html>"


Paris, Nov 2015

Property

Round trip Parse Tree -> XML -> Parse Tree

forall xml_tree <- xml_el do ensure parse( pp(xml_tree) ) == xml_treeend

Now we need to generate parse trees. This is a user defined format... no standard generators.


Paris, Nov 2015

Typical Unit tests in language X


Paris, Nov 2015

CONTINUE

DEMO

Generating trees

Generate trees with tag

#xml_el{ name = children = }


Paris, Nov 2015

some words, like "root" and "message"

list of #xml_el objects (possibly empty)

or {xmlcdata, text}

Invalid strings

.............................Failed! After 30 tests.<message>x𶹔</message>not ensured: {:error, {4, "not well-formed (invalid token)"}} == {:xmlel, "message", [], [xmlcdata: <<120, 29, 240, 182, 185, 148>>]}Shrinking x....(4 times)<root></root>not ensured: {:error, {4, "not well-formed (invalid token)"}} == {:xmlel, "root", [], [xmlcdata: <<0>>]}

forall xml_tree <- xml_el do when_fail IO.puts(pp(xml_tree)) do ensure parse( pp(xml_tree) ) == xml_tree endend


Paris, Nov 2015

not all UTF8 characters seem to be fine

Not even if they pass String.valid?

Restrict valid input data

Restrict to ASCII text, characters 10, 13, 32..126

Shrinking xxx.xx.x.xx....(7 times)</root>not ensured: {:xmlel, "root", [], [xmlcdata: "\n"]} == {:xmlel, "root", [], [xmlcdata: "\r"]}



Paris, Nov 2015

Surely they cannot be distinguished !

That means that if you send a "\r" you receive a "\

n"... fine.

Restrict valid input data

Restrict to ASCII text, characters 10, 13, 32..126

Passed over 10000 tests!



Paris, Nov 2015

Adding attributes

<root version='*' version='3'/>not ensured: {:error, {8, "duplicate attribute"}} == {:xmlel, "root", [{"version", "*"}, {"version", "3"}], []}

<root xmlns='
'/>not ensured: {:xmlel, "root:", [{"xmlns", "\n"}], []} == {:xmlel, "root", [{"xmlns", "\n"}], []}


Paris, Nov 2015

Fix generator (document)

Weird :due to xmlns and "\n"

Paris, Nov 2015

Property-based testing

Software has certain properties... things that should always hold for that software

A test case is one verification that a certain property hold

Paris, Nov 2015

QuickCheck

In real software, properties are more complex

Real software has state... properties need to be stateful and generate a sequence of commands

Shrinking to minimal failing test case saves a lot of time in analysis

Paris, Nov 2015

Larger examples

Testing Ejabberd hooksagainst Elixir specification

Paris, Nov 2015

Ejabberd example

• Start random number of Ejabberd nodes (shrink to minimum number nodes needed to provoke error)

• Tested: add, delete, remove one on all nodes, run, and get info

• Mocking the actual hooks

Found a few issues, such as problem when handlers have same name, but different arity.

https://github.com/processone/ejabberd for details

https://github.com/processone/ejabberd

https://github.com/processone/ejabberd

Paris, Nov 2015

Benefits

• Less time spent writing test code– One model replaces many tests

• Better testing– Lots of combinations you’d never test by hand

• Less time spent on diagnosis– Failures minimized automatically

Property-based testing of XMPP: generate your tests automatically - ejabberd Workshop #1

Technology

Transcript of Property-based testing of XMPP: generate your tests automatically - ejabberd Workshop #1