Download - Agile Testing Practices

Transcript

Paul King Director, ASERT

@paulk_asert

[email protected]

Leveraging Emerging

Technologies in Agile

Teams … and some testing tools

you should try at home

Topics

Introduction

• Using testing DSLs

• All combinations and all pairs

• Auto generated tests

• Model-based testing

• Business/logic rules

• Example driven testing

• Further information AgileAust 2011 - 2

© A

SE

RT

2006-2

011

AgileAust 2011 - 3

Emerging Technology Areas • Development

– Continuous integration

– Convention over

configuration

– Language evolution

– Framework

evolution

– Concurrency

– Configuration

management

• Testing

– ATDD/BDD

– Testing DSLs

– Scripting

• Deployment & Mgmt

– Continuous deployment

– Build optimisation

– Cloud computing

• A few favourites

– Grails

– Gpars

– Gradle

– Tableaux

© A

SE

RT

2006-2

011

Support Agile

via improved

Management of

Complexity

Advances in the Testing Arena

AgileAust 2011 - 4

© A

SE

RT

2006-2

011

Unit Testing

Mock/interaction testing

State-based testing

Integration Testing

Acceptance Testing

Web drivers

Non-web drivers

Test runners

Techniques

Testing DSLs

ATDD/BDD

Data-driven

Logic-driven

Model-driven

Performance

testing

All-pairs &

combinations

Gpars

Groovy and Testing Tool Spectrum*

AgileAust 2011 - 5

© A

SE

RT

2006-2

011

Database

Drivers

DbUnit

DataSets

SqlUnit

groovy.sql

JPA

JDO

BigTable

JDBC

SOAP /

REST

Drivers

GroovyWS

XML-RPC

CXF

Axis2

JAX-WS

JAX-RS

Utilities

AllPairs, Combinations

Polyglot languages

Logic programming

Threads, Parallel /

Concurrency libraries

Data-driven libraries

Networking libraries

XML Processing

Read/write files /

Excel / Word / CSV

Reporting, Logging

Other

Drivers

FEST

Email

FTP

AntUnit

Telnet

SSH

Exec WindowLicker

Tools

iTest2, SoapUI, Twist,

IDEs, JMeter, Text

editors, Recorders,

Sahi, Build Tools, CI

Web

Drivers

WebTest

WebDriver

JWebUnit

Tellurium

Selenium

HtmlUnit

Watij

HttpBuilder

Cyberneko

Runners

Native Groovy, JUnit, TestNG, Spock, EasyB,

JBehave, Cucumber, Robot Framework, SLIM

* Tools/libraries/frameworks don't always neatly fall into one category – still useful conceptually

AgileAust 2011 - 6

© A

SE

RT

2006-2

011

Application under Test

AgileAust 2011 - 7

© A

SE

RT

2006-2

011

Concept

Driver

Runner <webtest name="myTest">

<steps>

<invoke

description="get Login Page"

url="login" />

<verifyTitle

description="we should see the login title"

text="Login Page" />

</steps>

</webtest>

Web Server

HTTP Request / Response

HTTP Request / Response

Read

Script

Manual

Automated

Driver Category • Real browser invoker

– Runs on platform

supported by real

browser

– May need multiple

platforms, e.g. IE6/IE7

– Uses actual JavaScript

engine

– Can be easier to use

with test recorders

– Automation

capabilities differ

across browsers

– Can typically get to all

aspects of browser

• Browser Emulators

– Can simulate multiple

browsers

– Less platform

restrictions

– Good for CI

– Easier to not download

images, resources

– Ability to optimise

JavaScript interactions

– More extensible

– Ability to disable

JavaScript

– Scope for parallelism

AgileAust 2011 - 8

© A

SE

RT

2006-2

011

AgileAust 2011 - 9

© A

SE

RT

2006-2

011

What is Groovy?

• “Groovy is like a super version

of Java. It can leverage Java's

enterprise capabilities but also

has cool productivity features like closures,

DSL support, builders and dynamic typing.”

Groovy = Java – boiler plate code + mostly dynamic typing + closures + domain specific languages + builders + metaprogramming + GDK library

Topics

• Introduction

Using testing DSLs

• All combinations and all pairs

• Auto generated tests

• Model-based testing

• Business/logic rules

• Example driven testing

• Further information AgileAust 2011 - 10

© A

SE

RT

2006-2

011

Prefer scripted over tool centric/recorders

AgileAust 2011 - 11

© A

SE

RT

2006-2

011

• Scripting

– Choose open

languages

– Developer friendly

– Test-first friendly

– Test using whatever

levels make sense

– Tests with code

– Reduce testing

documentation

– Refactor friendly

– Run with every change

as part of continuous

integration

• Tools/Recorders

– Lock-in to vendor tools

– Testing often done

after development

– Tests separate from

code

– Proprietary scripting

languages

– User-interface

focussed

• Hard-coded details

leads to fragile tests

– No or hard refactoring

Prefer business talk over tech talk • Avoid • Prefer

AgileAust 2011 - 12

© A

SE

RT

2006-2

011

import org.openqa.selenium.By import org.openqa.selenium.htmlunit.HtmlUnitDriver def driver = new HtmlUnitDriver() driver.get('http://localhost:8080/postForm') assert driver.title == 'Welcome to SimpBlog' // fill in query form and submit it driver.findElement(By.name('title')). sendKeys('Bart was here') driver.findElement(By.name('content')). sendKeys('Cowabunga dude!') def select = driver. findElement(By.name('category')) select. findElements(By.tagName("option")). find{ it.text == 'Home' }.setSelected() driver.findElement(By.name('btnPost')).click() ...

So

urc

e: E

vo

lvin

g W

eb

-Ba

sed

Test A

uto

ma

tion

into

Ag

ile B

usin

ess S

pecific

ation

s,

Mu

grid

ge e

t a

l

given we are on the blog entry page when entering 'Bart was here' as the title and entering 'Home' as the category then posting 5 appears

Topics

• Introduction

• Using testing DSLs

All combinations and all pairs

• Auto generated tests

• Model-based testing

• Business/logic rules

• Example driven testing

• Further information AgileAust 2011 - 13

© A

SE

RT

2006-2

011

Workshop • Our blog application sometimes

has errors but only for certain

combinations of Author,

Category and Content.

• How should we test it?

AgileAust 2011 - 14

© A

SE

RT

2006-2

011

Workshop • Our blog application sometimes

has errors but only for certain

combinations of Author,

Category and Content.

• How should we test it?

AgileAust 2011 - 15

© A

SE

RT

2006-2

011

What about:

* When a new

Category or

Author is added

in the future?

* Tests take too

long to run?

All Combinations

• Description – Don't have a bunch

of hard-coded, hard

to maintain manual

test data or even

manually generated

CSV file

– Much better to

generate test cases

from succinct

expressions of

what you are trying

to achieve

AgileAust 2011 - 16

© A

SE

RT

2006-2

011

[ ['MacOS', 'Linux', 'Vista'], ['2G', '4G', '6G', '8G'], ['250G', '350G', '500G'] ].combinations().each{ os, mem, disk -> test(os, mem, disk) }

test('MacOS', '4G', '250G') test('Linux', '4G', '250G') test('Vista', '4G', '250G') test('MacOS', '8G', '500G') test('Linux', '8G', '500G') test('Vista', '8G', '500G') // 30 more rows

All Combinations Case Study

AgileAust 2011 - 17

© A

SE

RT

2006-2

011

def combos = [ ["Bart", "Homer", "Marge", "Lisa", "Maggie"], ["Work", "School", "Home", "Travel", "Food"], ["foo", "bar", "baz"] ].combinations() println "Found ${combos.size()} combos" combos.each { author, category, content -> postAndCheck author, category, content }

Found 75 combos

def postAndCheck(author, category, content) { // ... details not shown ... }

All Pairs

• Description – Sometimes

called

pairwise

testing or

orthogonal

array testing

– Technique

to limit the

explosion of test cases by identifying samples of

important classes of test cases (equivalence classes)

• providing maximum coverage with minimum testing

– Instead of all combinations, systematically use pair-

wise combinations of interactions between objects

• as most faults result from adverse two-way interactions

AgileAust 2011 - 18

© A

SE

RT

2006-2

011

SimpBlog Case Study...

AgileAust 2011 - 19

© A

SE

RT

2006-2

011

def cases = new AllPairs().generate( author: ["Bart", "Homer", "Marge", "Lisa", "Maggie"], category: ["Work", "School", "Home", "Travel", "Food"], content: ["foo", "bar", "baz"]) println "Found ${cases.size()} cases" cases.each { next -> println next // just for debugging purposes postAndCheck next.author, next.category, next.content }

def postAndCheck(author, category, content) { // ... details not shown ... }

...SimpBlog Case Study

AgileAust 2011 - 20

© A

SE

RT

2006-2

011

Found 18 cases [content:bar, category:Food, author:Bart] [content:bar, category:School, author:Homer] [content:foo, category:Work, author:Bart] [content:baz, category:School, author:Homer] [content:bar, category:Home, author:Maggie] [content:foo, category:School, author:Marge] [content:bar, category:Work, author:Bart] [content:baz, category:Travel, author:Bart] [content:foo, category:Home, author:Homer] [content:bar, category:Travel, author:Marge] [content:baz, category:Work, author:Homer] [content:bar, category:Travel, author:Lisa] [content:baz, category:Travel, author:Maggie] [content:baz, category:Home, author:Marge] [content:baz, category:Food, author:Homer] [content:baz, category:Travel, author:Lisa] [content:foo, category:Food, author:Maggie] [content:foo, category:Travel, author:Lisa]

All Combinations/Pairs: Going Further

• Advanced options – N-Wise

– Compulsory combinations

– Excluding combinations

– Prioritising combinations

– Shuffling

• Sites – http://en.wikipedia.org/wiki/All-pairs_testing

– http://www.pairwise.org/

• (content starting to age but still useful)

• Tools – SpecExplorer

– http://code.google.com/p/jwise/ AgileAust 2011 - 21

© A

SE

RT

2006-2

011

Topics

• Introduction

• Using testing DSLs

• All combinations and all pairs

Auto generated tests

• Model-based testing

• Business/logic rules

• Example driven testing

• Further information AgileAust 2011 - 22

© A

SE

RT

2006-2

011

Workshop

• Creating good test data

is taking a long time

• Errors seem to occur

when the system is used

with data that we haven’t

tried before

• What else can we do?

AgileAust 2011 - 23

© A

SE

RT

2006-2

011

Workshop

• Creating good test data

is taking a long time

• Errors seem to occur

when the system is used

with data that we haven’t

tried before

• What else can we do?

AgileAust 2011 - 24

© A

SE

RT

2006-2

011

What about:

* When fields

change in

the future?

* Measuring

coverage?

* Repeatable

tests?

QuickCheck • Test Java/Groovy programs more declaratively

using automatic random test data generation

• Replace manually selected scenario-based tests

with specification-based testing

• Generators available: – primitive types,

collections, POJOs

– value ranges

– lists, sets and array

– distinct values

– distributions

– determinism (random,

deterministic )

– generator strategies (composition of generator values (oneOf,

frequency, list, array, nullsAnd), transformation, mutation)

– value frequencies (frequency, oneOf) AgileAust 2011 - 25

© A

SE

RT

2006-2

011

QuickCheck Samples…

AgileAust 2011 - 26

© A

SE

RT

2006-2

011

682 Ant Bee 141 Jul Dog Dog Dog 801 177 Ant Dog 951 Cat Aug Bee Bee Oct Dog 241

for (words in someNonEmptyLists(strings())) { assert words*.size().sum() == words.sum().size() }

def pets = fixedValues(['Ant', 'Bee', 'Cat', 'Dog']) def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) def months = new Generator<String>() { Generator<Date> genDate = dates() String next() { genDate.next().format("MMM") } } def gen = new DefaultFrequencyGenerator(pets, 50) gen.add(nums, 30) gen.add(months, 20) 20.times { def next = gen.next().toString() println next assert next.size() == 3 }

…QuickCheck Samples…

AgileAust 2011 - 27

© A

SE

RT

2006-2

011

682 Ant Bee 141 Jul Dog Dog Dog 801 177 Ant Dog 951 Cat Aug Bee Bee Oct Dog 241

for (words in someNonEmptyLists(strings())) { assert words*.size().sum() == words.sum().size() }

def pets = fixedValues(['Ant', 'Bee', 'Cat', 'Dog']) def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) def months = new Generator<String>() { Generator<Date> genDate = dates() String next() { genDate.next().format("MMM") } } def gen = new DefaultFrequencyGenerator(pets, 50) gen.add(nums, 30) gen.add(months, 20) 20.times { def next = gen.next().toString() println next assert next.size() == 3 }

…QuickCheck Samples…

AgileAust 2011 - 28

© A

SE

RT

2006-2

011

682 Ant Bee 141 Jul Dog Dog Dog 801 177 Ant Dog 951 Cat Aug Bee Bee Oct Dog 241

for (words in someNonEmptyLists(strings())) { assert words*.size().sum() == words.sum().size() }

def pets = fixedValues(['Ant', 'Bee', 'Cat', 'Dog']) def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) def months = new Generator<String>() { Generator<Date> genDate = dates() String next() { genDate.next().format("MMM") } } def gen = new DefaultFrequencyGenerator(pets, 50) gen.add(nums, 30) gen.add(months, 20) 20.times { def next = gen.next().toString() println next assert next.size() == 3 }

…QuickCheck Samples

AgileAust 2011 - 29

© A

SE

RT

2006-2

011

682 Ant Bee 141 Jul Dog Dog Dog 801 177 Ant Dog 951 Cat Aug Bee Bee Oct Dog 241

for (words in someNonEmptyLists(strings())) { assert words*.size().sum() == words.sum().size() }

def pets = fixedValues(['Ant', 'Bee', 'Cat', 'Dog']) def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) def months = new Generator<String>() { Generator<Date> genDate = dates() String next() { genDate.next().format("MMM") } } def gen = new DefaultFrequencyGenerator(pets, 50) gen.add(nums, 30) gen.add(months, 20) 20.times { def next = gen.next().toString() println next assert next.size() == 3 }

QuickCheck: SimpBlog Case Study…

• Approach – Auto-selected values for author, category

– Auto-generated values for title, content

AgileAust 2011 - 30

© A

SE

RT

2006-2

011

def authors = ["Bart", "Homer", "Lisa", "Marge", "Maggie"] def categories = ["Home", "Work", "Food", "Travel"] 10.times { postAndCheck anyString(), anyFixedValue(categories), anyFixedValue(authors), anyString() }

def postAndCheck(title, category, author, content) { ... }

…QuickCheck: SimpBlog Case Study

AgileAust 2011 - 31

© A

SE

RT

2006-2

011

Screenshot

of fixed

titles version

Test Generation: Going Further

• AutoMocks

• Auto domain classes

• Assertions

• http://en.wikipedia.org/wiki/QuickCheck – Erlang, Scheme, Common Lisp, Perl, Python, Clojure,

Scala, Ruby, Java, F#, Standard ML, JavaScript, C++

AgileAust 2011 - 32

© A

SE

RT

2006-2

011

Topics

• Introduction

• Using testing DSLs

• All combinations and all pairs

• Auto generated tests

Model-based testing

• Business/logic rules

• Example driven testing

• Further information AgileAust 2011 - 33

© A

SE

RT

2006-2

011

Workshop • We recently added AJAX to the

SimpBlog application and some

of our users are reporting issues

• Depending on the order in which

the blog form is filled out there

can sometimes be strange error

messages displayed

• What testing strategies can we

use in this scenario?

AgileAust 2011 - 34

© A

SE

RT

2006-2

011

Workshop • We recently added AJAX to the

SimpBlog application and some

of our users are reporting issues

• Depending on the order in which

the blog form is filled out there

can sometimes be strange error

messages displayed

• What testing strategies can we

use in this scenario?

AgileAust 2011 - 35

© A

SE

RT

2006-2

011

What about:

* When a new

Category or

Author is added

in the future?

* What does

coverage mean?

Model-based testing

• Deriving test suites from source code is

usually deemed impractical

• Instead develop models which describe

certain characteristics of the desired

system behaviour

• May involve: – theorem proving

– constraint logic programming

– model checking

– event-flow models

– markov chain models

AgileAust 2011 - 36

© A

SE

RT

2006-2

011

ModelJUnit...

• Description – Supports model-based testing

– Allows you to write simple finite

state machine (FSM) models or

extended finite state machine

(EFSM) models in Java or Groovy

– You can then generate tests from

those models and measure various

model coverage metrics

AgileAust 2011 - 37

© A

SE

RT

2006-2

011

...ModelJUnit...

AgileAust 2011 - 38

© A

SE

RT

2006-2

011

// require modeljunit.jar import nz.ac.waikato.modeljunit.coverage.* import nz.ac.waikato.modeljunit.*

class VendingMachineModel implements FsmModel { def state = 0 // 0,25,50,75,100 void reset(boolean testing) {state = 0}

boolean vendGuard() {state == 100} @Action void vend() {state = 0}

boolean coin25Guard() {state <= 75} @Action void coin25() {state += 25}

boolean coin50Guard() {state <= 50} @Action void coin50() {state += 50} }

def tester = new RandomTester(new VendingMachineModel()) tester.buildGraph() def metrics = [new ActionCoverage(), new StateCoverage(), new TransitionCoverage(), new TransitionPairCoverage()] metrics.each { tester.addCoverageMetric it }

tester.addListener "verbose" tester.generate 20

println '\nMetrics Summary:' tester.printCoverage()

...ModelJUnit

AgileAust 2011 - 39

© A

SE

RT

2006-2

011

...

done (0, coin50, 50) done (50, coin25, 75) done (75, coin25, 100) done Random reset(true) done (0, coin50, 50) done (50, coin25, 75) done (75, coin25, 100) done (100, vend, 0) done (0, coin50, 50) done (50, coin50, 100) done (100, vend, 0) done (0, coin25, 25) done (25, coin25, 50) done Random reset(true) done (0, coin50, 50) done (50, coin25, 75) done (75, coin25, 100) done (100, vend, 0) done (0, coin50, 50) done (50, coin25, 75) ...

...

Metrics Summary: action coverage: 3/3 state coverage: 5/5 transition coverage: 7/8 transition-pair coverage: 8/12 ...

Interlude: Seven Bridges of Königsberg • Königsberg, Prussia (now Kaliningrad, Russia)

spans the Pregel River including two large

islands connected by 7 bridges

• Find a walk through the city crossing each

bridge once and only once. The islands can not

be reached by any route other than the bridges,

and every bridge must be crossed completely

every time.

AgileAust 2011 - 40

© A

SE

RT

2006-2

011

Source: http://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg

Interlude: Seven Bridges of Königsberg • Königsberg, Prussia (now Kaliningrad, Russia)

spans the Pregel River including two large

islands connected by 7 bridges

• Find a walk through the city crossing each

bridge once and only once. The islands can not

be reached by any route other than the bridges,

and every bridge must be crossed completely

every time. Euler showed there is no solution.

AgileAust 2011 - 41

© A

SE

RT

2006-2

011

Source: http://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg

Interlude: Königsberg Bridges Variation 8. The Blue Prince contrives a stealthy plan to build an

eighth bridge so that he can begin in the evening at

his Schloß, walk the bridges, and end at the Gasthaus

to brag of his victory. Of course, he wants the Red

Prince to be unable to duplicate the feat from the

Red Castle. Where does the Blue Prince build the

eighth bridge?

9. The Red Prince, infuriated by his brother's Gordian

solution to the problem, wants to build a ninth bridge,

enabling him to begin at his Schloß, walk the bridges,

and end at the Gasthaus to rub dirt in his brother's

face. As an extra bit of revenge, his brother should

then no longer be able to walk the bridges starting and

ending at his Schloß as before. Where does the Red

Prince build the ninth bridge?

10.The Bishop has watched this furious bridge building

with dismay. It upsets the town's Weltanschauung

and, worse, contributes to excessive drunkenness.

He wants to build a tenth bridge that allows all the

inhabitants to walk the bridges and return to their own

beds. Where does the Bishop build the tenth bridge?

AgileAust 2011 - 42

© A

SE

RT

2006-2

011

Source: http://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg

ModelJUnit: SimpBlog Case Study...

• Does the order in which form

information is entered affect

the application?

AgileAust 2011 - 43

© A

SE

RT

2006-2

011

// require modeljunit.jar, htmlunit.jar import nz.ac.waikato.modeljunit.coverage.* import nz.ac.waikato.modeljunit.* import com.gargoylesoftware.htmlunit.WebClient class SimpBlogModel implements FsmModel {

boolean authorSelected = false boolean categorySelected = false boolean titleEntered = false boolean contentEntered = false int count = 0 def client, page, form // Special known method, allows equivalence class definition // example states: __ __ __ __, AU __ __ __, AU CA TI CO def getState() { "${authorSelected ? ' AU ' : ' __ '}${categorySelected ? ' CA ' : ' __ '}" + "${titleEntered ? ' TI ' : ' __ '}${contentEntered ? ' CO ' : ' __ '}" } ...

... void reset(boolean testing) { authorSelected = false categorySelected = false titleEntered = false contentEntered = false client = new WebClient() url = 'http://localhost:8080/postForm' page = client.getPage(url) assert 'Welcome to SimpBlog' == page.titleText form = page.getFormByName('post') } ...

...ModelJUnit: SimpBlog Case Study...

AgileAust 2011 - 44

© A

SE

RT

2006-2

011

... boolean "enter title Guard"() { !titleEntered }

@Action void "enter title "() { titleEntered = true form.getInputByName('title').setValueAttribute("Title ${count++}") } boolean enterContentGuard() { !contentEntered }

@Action void enterContent() { contentEntered = true form.getTextAreaByName('content').setText("Content ${count++}") } boolean chooseAuthorGuard() { !authorSelected }

@Action void chooseAuthor() { authorSelected = true // simple version just Lisa form.getSelectByName('author').getOptions().find{ it.text == 'Lisa' }.setSelected(true) } boolean pickCategoryGuard() { !categorySelected }

@Action void pickCategory() { categorySelected = true // simple version just Home form.getSelectByName('category').getOptions().find{ it.text == 'Home' }.setSelected(true) } boolean "submit post Guard"() { categorySelected && authorSelected && titleEntered && contentEntered }

@Action void "submit post "() { def result = form.getInputByName('btnPost').click() assert result.getElementsByTagName('h1').item(0).textContent.matches('Post.*: Title .*') // could do more asserts here reset(true) } } // end of SimpBlogModel class definition ...

...ModelJUnit: SimpBlog Case Study...

AgileAust 2011 - 45

© A

SE

RT

2006-2

011

def tester = new RandomTester(new SimpBlogModel()) tester.buildGraph() def metrics = [ new ActionCoverage(), new StateCoverage(), new TransitionCoverage(), new TransitionPairCoverage() ] metrics.each { tester.addCoverageMetric it }

tester.addListener "verbose" tester.generate 50

println '\nMetrics Summary:' tester.printCoverage()

def graphListener = tester.model.getListener("graph") graphListener.printGraphDot "simpblog.dot" println "\nGraph contains " + graphListener.graph.numVertices() + " states and " + graphListener.graph.numEdges() + " transitions."

...ModelJUnit: SimpBlog Case Study...

AgileAust 2011 - 46

© A

SE

RT

2006-2

011

done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( __ CA __ __ , enterContent, __ CA __ CO ) done ( __ CA __ CO , enter title , __ CA TI CO ) done ( __ CA TI CO , chooseAuthor, AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( __ CA __ __ , chooseAuthor, AU CA __ __ ) done ( AU CA __ __ , enter title , AU CA TI __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , chooseAuthor, AU __ __ __ ) done ( AU __ __ __ , pickCategory, AU CA __ __ ) done ( AU CA __ __ , enter title , AU CA TI __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , enterContent, __ __ __ CO ) done ( __ __ __ CO , pickCategory, __ CA __ CO ) done ( __ CA __ CO , chooseAuthor, AU CA __ CO ) done ( AU CA __ CO , enter title , AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( __ CA __ __ , enter title , __ CA TI __ ) done ( __ CA TI __ , chooseAuthor, AU CA TI __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , chooseAuthor, AU __ __ __ ) done ( AU __ __ __ , pickCategory, AU CA __ __ ) done ( AU CA __ __ , enter title , AU CA TI __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) ...

... done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( __ CA __ __ , enterContent, __ CA __ CO ) done ( __ CA __ CO , chooseAuthor, AU CA __ CO ) done ( AU CA __ CO , enter title , AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , chooseAuthor, AU __ __ __ ) done ( AU __ __ __ , pickCategory, AU CA __ __ ) done ( AU CA __ __ , enterContent, AU CA __ CO ) done ( AU CA __ CO , enter title , AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , chooseAuthor, AU __ __ __ ) done ( AU __ __ __ , enter title , AU __ TI __ ) done ( AU __ TI __ , pickCategory, AU CA TI __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done Random reset(true) done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( __ CA __ __ , enterContent, __ CA __ CO ) done ( __ CA __ CO , enter title , __ CA TI CO ) done ( __ CA TI CO , chooseAuthor, AU CA TI CO ) Metrics Summary: action coverage: 5/5 state coverage: 12/16 transition coverage: 19/33 transition-pair coverage: 26/56 Graph contains 16 states and 33 transitions.

...ModelJUnit: SimpBlog Case Study...

AgileAust 2011 - 47

© A

SE

RT

2006-2

011

Simplified

version

(just Lisa)

Advanced

version

...ModelJUnit: SimpBlog Case Study

AgileAust 2011 - 48

© A

SE

RT

2006-2

011

Model-based testing: Going further

• Model-based testing – http://www.cs.waikato.ac.nz/~marku/mbt/modeljunit/

– http://en.wikipedia.org/wiki/Model_based_testing

– http://home.mit.bme.hu/~micskeiz/pages/modelbased_testing.html

– A Taxonomy of Model-Based Testing http://www.cs.waikato.ac.nz/pubs/wp/2006/uow-cs-wp-2006-04.pdf

• Related algorithms – Job scheduling problem

– Travelling salesman problem

– Chinese postman problem

– Vehicle routing problem

– Spanning trees and Hamiltonian paths

– http://www.nada.kth.se/~viggo/problemlist/compendium.html

– http://en.wikipedia.org/wiki/List_of_terms_relating_to_algorithm

s_and_data_structures AgileAust 2011 - 49

© A

SE

RT

2006-2

011

Topics

• Introduction

• Using testing DSLs

• All combinations and all pairs

• Auto generated tests

• Model-based testing

Business/logic rules

• Example driven testing

• Further information AgileAust 2011 - 50

© A

SE

RT

2006-2

011

Workshop • Set up some test cases

representing the Simpsons

weekly blogging habits:

– They never blog on the same day

– Marge blogs only on a Saturday or

Sunday

– Maggie blogs only on a Tuesday or

Thursday

– Lisa blogs only on a Monday,

Wednesday or Friday

– Bart blogs only on the day after Lisa

– Homer only blogs if noone else

blogged the previous day and

doesn't allow anyone to blog the

next day AgileAust 2011 - 51

© A

SE

RT

2006-2

011

Workshop • Set up some test cases

representing the Simpsons

weekly blogging habits:

– They never blog on the same day

– Marge blogs only on a Saturday or

Sunday

– Maggie blogs only on a Tuesday or

Thursday

– Lisa blogs only on a Monday,

Wednesday or Friday

– Bart blogs only on the day after Lisa

– Homer only blogs if noone else

blogged the previous day and

doesn't allow anyone to blog the

next day AgileAust 2011 - 52

© A

SE

RT

2006-2

011

What about:

* When the

habits change?

Constraint/Logic Programming...

• Description – Style of programming where relations between

variables are stated in the form of constraints

– First made popular by logic programming languages

such as Prolog but the style is now also used outside

logic programming specific languages

– Constraints differ from the common primitives of

other programming languages in that they do not

specify one or more steps to execute but rather the

properties of a solution to be found

– Popular libraries used with Groovy supporting

constraint programming include Gecode/J, Choco

and tuProlog

– We'll look at Choco as an example

AgileAust 2011 - 53

© A

SE

RT

2006-2

011

...Constraint/Logic Programming...

AgileAust 2011 - 54

© A

SE

RT

2006-2

011

Source: http://xkcd.com/287/

...Constraint/Logic Programming...

AgileAust 2011 - 55

© A

SE

RT

2006-2

011

// requires choco 2.1.0-basic.jar from http://choco.emn.fr/ import static choco.Choco.* import choco.kernel.model.variables.integer.IntegerVariable def m = new choco.cp.model.CPModel() def s = new choco.cp.solver.CPSolver() def menu = [ 'Mixed fruit' : 215, 'French fries' : 275, 'Side salad' : 335, 'Hot wings' : 355, 'Mozzarella sticks' : 420, 'Sampler plate' : 580 ] def numOrdered = new IntegerVariable[menu.size()] def priceEach = new int[menu.size()] def sum = 1505 ...

Found a solution: 7 * Mixed fruit Found a solution: 1 * Mixed fruit 2 * Hot wings 1 * Sampler plate

...Constraint/Logic Programming

AgileAust 2011 - 56

© A

SE

RT

2006-2

011

... menu.eachWithIndex { name, price, i -> // number ordered >= 0 // number ordered * price <= sum numOrdered[i] = makeIntVar(name, 0, sum.intdiv(price)) priceEach[i] = price } m.addConstraint(eq(scalar(numOrdered, priceEach), sum)) s.read(m) def more = s.solve() while (more) { println "Found a solution:" numOrdered.each { def v = s.getVar(it) if (v.val) println " $v.val * $v.name" } more = s.nextSolution() }

SimpBlog Case Study...

AgileAust 2011 - 57

© A

SE

RT

2006-2

011

def m = new choco.cp.model.CPModel() def s = new choco.cp.solver.CPSolver() daysOfWeek = ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"] def bart = makeIntVar('Bart', 0, 6) def homer = makeIntVar('Homer', 0, 6) def marge = makeIntVar('Marge', 0, 6) def lisa = makeIntVar('Lisa', 0, 6) def maggie = makeIntVar('Maggie', 0, 6) def simpsons = [bart, homer, marge, lisa, maggie] ...

...SimpBlog Case Study...

AgileAust 2011 - 58

© A

SE

RT

2006-2

011

...

// They never blog on the same day for (i in 0..<simpsons.size()) for (j in 0..<i) m.addConstraint(neq(simpsons[i], simpsons[j]))

// Marge blogs only on a Saturday or Sunday m.addConstraint(or(eq(marge, 0), eq(marge, 6)))

// Maggie blogs only on a Tuesday or Thursday m.addConstraint(or(eq(maggie, 2), eq(maggie, 4)))

// Lisa blogs only on a Monday, Wednesday or Friday m.addConstraint(or(eq(lisa, 1), eq(lisa, 3), eq(lisa, 5)))

// Bart blogs only on the day after Lisa m.addConstraint(eq(plus(lisa, 1), bart))

// Homer only blogs if noone else blogged the previous // day and doesn't allow anyone to blog the next day m.addConstraint(and(distanceNEQ(homer, marge, 1), distanceNEQ(homer, bart, 1), distanceNEQ(homer, maggie, 1), distanceNEQ(homer, lisa, 1))) ...

...SimpBlog Case Study

AgileAust 2011 - 59

© A

SE

RT

2006-2

011

...

s.read(m) def more = s.solve() if (!more) println "No Solutions Found" else println pad("Solutions:") + simpsons.collect{ pad(it.name) }.join() while (more) { print pad("") println simpsons.collect { def v = s.getVar(it) pad(daysOfWeek[v.val]) }.join() more = s.nextSolution() }

def pad(s) { s.padRight(12) }

Solutions: Bart Homer Marge Lisa Maggie Thursday Saturday Sunday Wednesday Tuesday Tuesday Saturday Sunday Monday Thursday Saturday Tuesday Sunday Friday Thursday Thursday Sunday Saturday Wednesday Tuesday

Logic solvers: Going further

AgileAust 2011 - 60

© A

SE

RT

2006-2

011

• Available tools – JSR-331: Java Constraint Programming API

http://jcp.org/en/jsr/detail?id=331

– JaCoP: Java Constraint Programming solver

http://jacop.osolpro.com/

– ChocoSolver

http://www.emn.fr/z-info/choco-solver/

Topics

• Introduction

• Using testing DSLs

• All combinations and all pairs

• Auto generated tests

• Model-based testing

• Business/logic rules

Example driven testing

• Further information AgileAust 2011 - 61

© A

SE

RT

2006-2

011

AgileAust 2011 - 63

© A

SE

RT

2006-2

011

Canoo WebTest • Description – Open source tool for automated testing of web applications

– Declarative approach in XML or testing DSL in Groovy

– Has Test Recorder

– Excellent reporting options

– Ant-based under the covers

<target name="login" >

<testSpec name="normal" >

&config;

<steps>

<invoke stepid="get Login Page"

url="login.jsp" />

<verifytitle stepid="we should see the login title"

text="Login Page" />

<setinputfield stepid="set user name"

name="username"

value="scott" />

<setinputfield stepid="set password"

name="password"

value="tiger" />

<clickbutton stepid="Click the submit button"

label="let me in" />

<verifytitle stepid="Home Page follows if login ok"

text="Home Page" />

</steps>

</testSpec>

</target>

AgileAust 2011 - 64

© A

SE

RT

2006-2

011

Firefox Recorder

EasyB Example ...

• When run will be marked as pending – perfect for ATDD

AgileAust 2011 - 65

© A

SE

RT

2006-2

011

scenario "Bart posts a new blog entry", { given "we are on the create blog entry page" when "I have entered 'Bart was here' as the title" and "I have entered 'Cowabunga Dude!' into the content" and "I have selected 'Home' as the category" and "I have selected 'Bart' as the author" and "I click the 'Create Post' button" then "I expect the entry to be posted" }

...EasyB Example...

AgileAust 2011 - 66

© A

SE

RT

2006-2

011

description "Post Blog Entry Feature" narrative "for feature", { as_a "Blogger" i_want "to be able to post a blog" so_that "I can keep others informed" } before "posting blog", { given "we are on the create blog entry page", { webClient = new com.gargoylesoftware.htmlunit.WebClient() page = webClient.getPage('http://localhost:8080/postForm') } } scenario "Bart was here blog", { when "I have entered 'Bart was here' as the title", { form = page.getFormByName('post') form.getInputByName('title').setValueAttribute( 'Bart was here (and so was EasyB)') } ...

...EasyB Example...

AgileAust 2011 - 67

© A

SE

RT

2006-2

011

... and "I have entered 'Cowabunga Dude!' into the content", { form.getTextAreaByName('content').setText('Cowabunga Dude!') } and "I have selected 'Home' as the category", { form.getSelectByName('category').getOptions().find { it.text == 'Home' }.setSelected(true) } and "I click the 'Create Post' button", { result = form.getInputByName('btnPost').click() } then "I expect the entry to be posted", { // check blog post details assert result.getElementsByTagName('h1').item(0).textContent.matches('Post.*: Bart was here.*') def h3headings = result.getElementsByTagName('h3') assert h3headings.item(1).textContent == 'Category: Home' // traditional style h3headings.item(2).textContent.shouldBe 'Author: Bart' // BDD style // expecting: <table><tr><td><p>Cowabunga Dude!</p></td></tr></table> def cell = result.getByXPath('//TABLE//TR/TD')[0] def para = cell.firstChild assert para.textContent == 'Cowabunga Dude!' // para.shouldHave textContent: 'Cowabunga Dude!' } }

...EasyB Example

AgileAust 2011 - 68

© A

SE

RT

2006-2

011

JBehave Example...

AgileAust 2011 - 69

© A

SE

RT

2006-2

011

Given we are on the create blog entry page When I have entered "Bart was here" as the title And I have entered "Cowabunga Dude!" as the content And I have selected "Home" from the "category" dropdown And I have selected "Bart" from the "author" dropdown And I click the 'Create Post' button Then I should see a heading message matching "Post.*: Bart was here.*"

import org.jbehave.scenario.Scenario import org.jbehave.scenario.steps.Steps class NewPostScenario extends Scenario { NewPostScenario() { super([new CreateBlogSteps()] as Steps[]) } }

Scenario:

Given we are on the create blog entry page

When I have entered "Bart was here" as the title

And I have entered "Cowabunga Dude!" as the content

And I have selected "Home" from the "category" dropdown

And I have selected "Bart" from the "author" dropdown

And I click the 'Create Post' button

Then I should see a heading message matching "Post.*: Bart was here.*"

...JBehave Example...

AgileAust 2011 - 70

© A

SE

RT

2006-2

011

import org.jbehave.scenario.steps.Steps import org.jbehave.scenario.annotations.* import com.gargoylesoftware.htmlunit.WebClient class CreateBlogSteps extends Steps { def page, form, result @Given("we are on the create blog entry page") void gotoEntryPage() { page = new WebClient().getPage('http://localhost:8080/postForm') } @When('I have entered "$title" as the title') void enterTitle(String title) { form = page.getFormByName('post') form.getInputByName('title'). setValueAttribute(title + ' (and so was JBehave)') } @When('I have entered "$content" as the content') void enterContent(String content) { form.getTextAreaByName('content').setText(content) } ...

JBehave Web Runner...

AgileAust 2011 - 71

© A

SE

RT

2006-2

011

Interlude: Einstein’s Riddle…

• Wikipedia: The zebra puzzle is a well-

known logic puzzle

–Often called Einstein's Puzzle or

Einstein's Riddle

–Said to have been invented by Albert

Einstein as a boy

–Claim is that Einstein said “Only 2

percent of the world's population can

solve it.”

AgileAust 2011 - 72

© A

SE

RT

2006-2

011

Interlude: …Einstein’s Riddle

AgileAust 2011 - 73

© A

SE

RT

2006-2

011

• Some premises: – The British person lives in the red house

– The Swede keeps dogs as pets

– The Dane drinks tea

– The green house is on the left of the white house

– The green homeowner drinks coffee

– The man who smokes Pall Mall keeps birds

– The owner of the yellow house smokes Dunhill

– The man living in the center house drinks milk

– The Norwegian lives in the first house

– The man who smokes Blend lives next to the one who keeps cats

– The man who keeps the horse lives next to the man who smokes Dunhill

– The man who smokes Bluemaster drinks beer

– The German smokes Prince

– The Norwegian lives next to the blue house

– The man who smokes Blend has a neighbor who drinks water

• And a question: Who owns the fish?

Einstein’s Riddle : Prolog

© A

SE

RT

2006-2

011

% from http://www.baptiste-wicht.com/2011/09/solve-einsteins-riddle-using-prolog % Preliminary definitions persons(0, []) :- !. persons(N, [(_Men,_Color,_Drink,_Smoke,_Animal)|T]) :- N1 is N-1, persons(N1,T). person(1, [H|_], H) :- !. person(N, [_|T], R) :- N1 is N-1, person(N1, T, R). % The Brit lives in a red house hint1([(brit,red,_, _, _)|_]). hint1([_|T]) :- hint1(T). % The Swede keeps dogs as pets hint2([(swede,_,_,_,dog)|_]). hint2([_|T]) :- hint2(T). % The Dane drinks tea hint3([(dane,_,tea,_,_)|_]). hint3([_|T]) :- hint3(T). % The Green house is on the left of the White house hint4([(_,green,_,_,_),(_,white,_,_,_)|_]). hint4([_|T]) :- hint4(T). % The owner of the Green house drinks coffee. hint5([(_,green,coffee,_,_)|_]). hint5([_|T]) :- hint5(T). ...

AgileAust 2011 - 74

Einstein’s Riddle : Groovy testing DSL

© A

SE

RT

2006-2

011

AgileAust 2011 - 75

the Dane drinks tea the German smokes prince the Swede keeps dogs the Brit has a red house the Norwegian owns the first house the man from the centre house drinks milk the owner of the green house drinks coffee the owner of the yellow house smokes dunhill the person known to smoke pallmall rears birds the man known to smoke bluemaster drinks beer the green house is on the left side of the white house the man known to smoke blends lives next to the one who keeps cats the man known to keep horses lives next to the man who smokes dunhill the man known to smoke blends lives next to the one who drinks water the Norwegian lives next to the blue house

SLIM with Groovy

AgileAust 2011 - 76

© A

SE

RT

2006-2

011

ZiBreve…

AgileAust 2011 - 77

© A

SE

RT

2006-2

011

So

urc

e: E

vo

lvin

g W

eb

-Ba

sed

Test A

uto

ma

tion

into

Ag

ile B

usin

ess S

pecific

ation

s,

Mu

grid

ge e

t a

l

SpiderFixture from FitLibrary (FitNesse) ZiBreve

…ZiBreve

AgileAust 2011 - 78

© A

SE

RT

2006-2

011

Source: Evolving Web-Based Test Automation into Agile Business Specifications, Mugridge et al

Example driven testing

• Bridging the communication Gap – Specification by example and agile acceptance testing by

Gojko Adzic

• Specification by Example – How successful teams deliver the right software by Gojko

Adzic: http://manning.com/adzic/

• Exploration Through Example – http://www.exampler.com/blog/ (Brian Marick)

• Evolving Web-Based Test Automation into

Agile Business Specifications

Rick Mugridge et al – http://www.mdpi.com/1999-5903/3/2/159/pdf

AgileAust 2011 - 79

© A

SE

RT

2006-2

011

Topics

• Introduction

• Using testing DSLs

• All combinations and all pairs

• Auto generated tests

• Model-based testing

• Business/logic rules

• Example driven testing

Further information AgileAust 2011 - 80

© A

SE

RT

2006-2

011

Conclusion

• Don’t be afraid to use technology

– For development, testing, deployment

• Embrace continuous learning

• There are many ways to capture business

requirements

– “Deciding what to build”

– Find the best way to represent your

requirements

– Prefer machine understandable,

executable solutions

AgileAust 2011 - 81

© A

SE

RT

2006-2

011

More Information

82

© A

SE

RT

2006-2

011

About the Author

83

© A

SE

RT

2006-2

011

Dr Paul King leads ASERT, an organisation based in

Brisbane, Australia which provides software development,

training and mentoring services to customers wanting to

embrace new technologies, harness best practices and

innovate. He has been contributing to open source projects

for nearly 20 years and is an active committer on numerous

projects including Groovy. Paul speaks at international

conferences, publishes in software magazines and journals,

and is a co-author of Manning's best-seller: Groovy in Action.

Paul King Director, ASERT

@paulk_asert

[email protected]