Paul King Director, ASERT
@paulk_asert
Leveraging Emerging
Technologies in Agile
Teams … and some testing tools
you should try at home
Topics
Introduction
• Using testing DSLs
• All combinations and all pairs
• Auto generated tests
• Model-based testing
• Business/logic rules
• Example driven testing
• Further information AgileAust 2011 - 2
© A
SE
RT
2006-2
011
AgileAust 2011 - 3
Emerging Technology Areas • Development
– Continuous integration
– Convention over
configuration
– Language evolution
– Framework
evolution
– Concurrency
– Configuration
management
• Testing
– ATDD/BDD
– Testing DSLs
– Scripting
• Deployment & Mgmt
– Continuous deployment
– Build optimisation
– Cloud computing
• A few favourites
– Grails
– Gpars
– Gradle
– Tableaux
© A
SE
RT
2006-2
011
Support Agile
via improved
Management of
Complexity
Advances in the Testing Arena
AgileAust 2011 - 4
© A
SE
RT
2006-2
011
Unit Testing
Mock/interaction testing
State-based testing
Integration Testing
Acceptance Testing
Web drivers
Non-web drivers
Test runners
Techniques
Testing DSLs
ATDD/BDD
Data-driven
Logic-driven
Model-driven
Performance
testing
All-pairs &
combinations
Gpars
Groovy and Testing Tool Spectrum*
AgileAust 2011 - 5
© A
SE
RT
2006-2
011
Database
Drivers
DbUnit
DataSets
SqlUnit
groovy.sql
JPA
JDO
BigTable
JDBC
SOAP /
REST
Drivers
GroovyWS
XML-RPC
CXF
Axis2
JAX-WS
JAX-RS
Utilities
AllPairs, Combinations
Polyglot languages
Logic programming
Threads, Parallel /
Concurrency libraries
Data-driven libraries
Networking libraries
XML Processing
Read/write files /
Excel / Word / CSV
Reporting, Logging
Other
Drivers
FEST
FTP
AntUnit
Telnet
SSH
Exec WindowLicker
Tools
iTest2, SoapUI, Twist,
IDEs, JMeter, Text
editors, Recorders,
Sahi, Build Tools, CI
Web
Drivers
WebTest
WebDriver
JWebUnit
Tellurium
Selenium
HtmlUnit
Watij
HttpBuilder
Cyberneko
Runners
Native Groovy, JUnit, TestNG, Spock, EasyB,
JBehave, Cucumber, Robot Framework, SLIM
* Tools/libraries/frameworks don't always neatly fall into one category – still useful conceptually
AgileAust 2011 - 7
© A
SE
RT
2006-2
011
Concept
Driver
Runner <webtest name="myTest">
<steps>
<invoke
description="get Login Page"
url="login" />
<verifyTitle
description="we should see the login title"
text="Login Page" />
</steps>
</webtest>
Web Server
HTTP Request / Response
HTTP Request / Response
Read
Script
Manual
Automated
Driver Category • Real browser invoker
– Runs on platform
supported by real
browser
– May need multiple
platforms, e.g. IE6/IE7
– Uses actual JavaScript
engine
– Can be easier to use
with test recorders
– Automation
capabilities differ
across browsers
– Can typically get to all
aspects of browser
• Browser Emulators
– Can simulate multiple
browsers
– Less platform
restrictions
– Good for CI
– Easier to not download
images, resources
– Ability to optimise
JavaScript interactions
– More extensible
– Ability to disable
JavaScript
– Scope for parallelism
AgileAust 2011 - 8
© A
SE
RT
2006-2
011
AgileAust 2011 - 9
© A
SE
RT
2006-2
011
What is Groovy?
• “Groovy is like a super version
of Java. It can leverage Java's
enterprise capabilities but also
has cool productivity features like closures,
DSL support, builders and dynamic typing.”
Groovy = Java – boiler plate code + mostly dynamic typing + closures + domain specific languages + builders + metaprogramming + GDK library
Topics
• Introduction
Using testing DSLs
• All combinations and all pairs
• Auto generated tests
• Model-based testing
• Business/logic rules
• Example driven testing
• Further information AgileAust 2011 - 10
© A
SE
RT
2006-2
011
Prefer scripted over tool centric/recorders
AgileAust 2011 - 11
© A
SE
RT
2006-2
011
• Scripting
– Choose open
languages
– Developer friendly
– Test-first friendly
– Test using whatever
levels make sense
– Tests with code
– Reduce testing
documentation
– Refactor friendly
– Run with every change
as part of continuous
integration
• Tools/Recorders
– Lock-in to vendor tools
– Testing often done
after development
– Tests separate from
code
– Proprietary scripting
languages
– User-interface
focussed
• Hard-coded details
leads to fragile tests
– No or hard refactoring
Prefer business talk over tech talk • Avoid • Prefer
AgileAust 2011 - 12
© A
SE
RT
2006-2
011
import org.openqa.selenium.By import org.openqa.selenium.htmlunit.HtmlUnitDriver def driver = new HtmlUnitDriver() driver.get('http://localhost:8080/postForm') assert driver.title == 'Welcome to SimpBlog' // fill in query form and submit it driver.findElement(By.name('title')). sendKeys('Bart was here') driver.findElement(By.name('content')). sendKeys('Cowabunga dude!') def select = driver. findElement(By.name('category')) select. findElements(By.tagName("option")). find{ it.text == 'Home' }.setSelected() driver.findElement(By.name('btnPost')).click() ...
So
urc
e: E
vo
lvin
g W
eb
-Ba
sed
Test A
uto
ma
tion
into
Ag
ile B
usin
ess S
pecific
ation
s,
Mu
grid
ge e
t a
l
given we are on the blog entry page when entering 'Bart was here' as the title and entering 'Home' as the category then posting 5 appears
Topics
• Introduction
• Using testing DSLs
All combinations and all pairs
• Auto generated tests
• Model-based testing
• Business/logic rules
• Example driven testing
• Further information AgileAust 2011 - 13
© A
SE
RT
2006-2
011
Workshop • Our blog application sometimes
has errors but only for certain
combinations of Author,
Category and Content.
• How should we test it?
AgileAust 2011 - 14
© A
SE
RT
2006-2
011
Workshop • Our blog application sometimes
has errors but only for certain
combinations of Author,
Category and Content.
• How should we test it?
AgileAust 2011 - 15
© A
SE
RT
2006-2
011
What about:
* When a new
Category or
Author is added
in the future?
* Tests take too
long to run?
All Combinations
• Description – Don't have a bunch
of hard-coded, hard
to maintain manual
test data or even
manually generated
CSV file
– Much better to
generate test cases
from succinct
expressions of
what you are trying
to achieve
AgileAust 2011 - 16
© A
SE
RT
2006-2
011
[ ['MacOS', 'Linux', 'Vista'], ['2G', '4G', '6G', '8G'], ['250G', '350G', '500G'] ].combinations().each{ os, mem, disk -> test(os, mem, disk) }
test('MacOS', '4G', '250G') test('Linux', '4G', '250G') test('Vista', '4G', '250G') test('MacOS', '8G', '500G') test('Linux', '8G', '500G') test('Vista', '8G', '500G') // 30 more rows
All Combinations Case Study
AgileAust 2011 - 17
© A
SE
RT
2006-2
011
def combos = [ ["Bart", "Homer", "Marge", "Lisa", "Maggie"], ["Work", "School", "Home", "Travel", "Food"], ["foo", "bar", "baz"] ].combinations() println "Found ${combos.size()} combos" combos.each { author, category, content -> postAndCheck author, category, content }
Found 75 combos
def postAndCheck(author, category, content) { // ... details not shown ... }
All Pairs
• Description – Sometimes
called
pairwise
testing or
orthogonal
array testing
– Technique
to limit the
explosion of test cases by identifying samples of
important classes of test cases (equivalence classes)
• providing maximum coverage with minimum testing
– Instead of all combinations, systematically use pair-
wise combinations of interactions between objects
• as most faults result from adverse two-way interactions
AgileAust 2011 - 18
© A
SE
RT
2006-2
011
SimpBlog Case Study...
AgileAust 2011 - 19
© A
SE
RT
2006-2
011
def cases = new AllPairs().generate( author: ["Bart", "Homer", "Marge", "Lisa", "Maggie"], category: ["Work", "School", "Home", "Travel", "Food"], content: ["foo", "bar", "baz"]) println "Found ${cases.size()} cases" cases.each { next -> println next // just for debugging purposes postAndCheck next.author, next.category, next.content }
def postAndCheck(author, category, content) { // ... details not shown ... }
...SimpBlog Case Study
AgileAust 2011 - 20
© A
SE
RT
2006-2
011
Found 18 cases [content:bar, category:Food, author:Bart] [content:bar, category:School, author:Homer] [content:foo, category:Work, author:Bart] [content:baz, category:School, author:Homer] [content:bar, category:Home, author:Maggie] [content:foo, category:School, author:Marge] [content:bar, category:Work, author:Bart] [content:baz, category:Travel, author:Bart] [content:foo, category:Home, author:Homer] [content:bar, category:Travel, author:Marge] [content:baz, category:Work, author:Homer] [content:bar, category:Travel, author:Lisa] [content:baz, category:Travel, author:Maggie] [content:baz, category:Home, author:Marge] [content:baz, category:Food, author:Homer] [content:baz, category:Travel, author:Lisa] [content:foo, category:Food, author:Maggie] [content:foo, category:Travel, author:Lisa]
All Combinations/Pairs: Going Further
• Advanced options – N-Wise
– Compulsory combinations
– Excluding combinations
– Prioritising combinations
– Shuffling
• Sites – http://en.wikipedia.org/wiki/All-pairs_testing
– http://www.pairwise.org/
• (content starting to age but still useful)
• Tools – SpecExplorer
– http://code.google.com/p/jwise/ AgileAust 2011 - 21
© A
SE
RT
2006-2
011
Topics
• Introduction
• Using testing DSLs
• All combinations and all pairs
Auto generated tests
• Model-based testing
• Business/logic rules
• Example driven testing
• Further information AgileAust 2011 - 22
© A
SE
RT
2006-2
011
Workshop
• Creating good test data
is taking a long time
• Errors seem to occur
when the system is used
with data that we haven’t
tried before
• What else can we do?
AgileAust 2011 - 23
© A
SE
RT
2006-2
011
Workshop
• Creating good test data
is taking a long time
• Errors seem to occur
when the system is used
with data that we haven’t
tried before
• What else can we do?
AgileAust 2011 - 24
© A
SE
RT
2006-2
011
What about:
* When fields
change in
the future?
* Measuring
coverage?
* Repeatable
tests?
QuickCheck • Test Java/Groovy programs more declaratively
using automatic random test data generation
• Replace manually selected scenario-based tests
with specification-based testing
• Generators available: – primitive types,
collections, POJOs
– value ranges
– lists, sets and array
– distinct values
– distributions
– determinism (random,
deterministic )
– generator strategies (composition of generator values (oneOf,
frequency, list, array, nullsAnd), transformation, mutation)
– value frequencies (frequency, oneOf) AgileAust 2011 - 25
© A
SE
RT
2006-2
011
QuickCheck Samples…
AgileAust 2011 - 26
© A
SE
RT
2006-2
011
682 Ant Bee 141 Jul Dog Dog Dog 801 177 Ant Dog 951 Cat Aug Bee Bee Oct Dog 241
for (words in someNonEmptyLists(strings())) { assert words*.size().sum() == words.sum().size() }
def pets = fixedValues(['Ant', 'Bee', 'Cat', 'Dog']) def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) def months = new Generator<String>() { Generator<Date> genDate = dates() String next() { genDate.next().format("MMM") } } def gen = new DefaultFrequencyGenerator(pets, 50) gen.add(nums, 30) gen.add(months, 20) 20.times { def next = gen.next().toString() println next assert next.size() == 3 }
…QuickCheck Samples…
AgileAust 2011 - 27
© A
SE
RT
2006-2
011
682 Ant Bee 141 Jul Dog Dog Dog 801 177 Ant Dog 951 Cat Aug Bee Bee Oct Dog 241
for (words in someNonEmptyLists(strings())) { assert words*.size().sum() == words.sum().size() }
def pets = fixedValues(['Ant', 'Bee', 'Cat', 'Dog']) def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) def months = new Generator<String>() { Generator<Date> genDate = dates() String next() { genDate.next().format("MMM") } } def gen = new DefaultFrequencyGenerator(pets, 50) gen.add(nums, 30) gen.add(months, 20) 20.times { def next = gen.next().toString() println next assert next.size() == 3 }
…QuickCheck Samples…
AgileAust 2011 - 28
© A
SE
RT
2006-2
011
682 Ant Bee 141 Jul Dog Dog Dog 801 177 Ant Dog 951 Cat Aug Bee Bee Oct Dog 241
for (words in someNonEmptyLists(strings())) { assert words*.size().sum() == words.sum().size() }
def pets = fixedValues(['Ant', 'Bee', 'Cat', 'Dog']) def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) def months = new Generator<String>() { Generator<Date> genDate = dates() String next() { genDate.next().format("MMM") } } def gen = new DefaultFrequencyGenerator(pets, 50) gen.add(nums, 30) gen.add(months, 20) 20.times { def next = gen.next().toString() println next assert next.size() == 3 }
…QuickCheck Samples
AgileAust 2011 - 29
© A
SE
RT
2006-2
011
682 Ant Bee 141 Jul Dog Dog Dog 801 177 Ant Dog 951 Cat Aug Bee Bee Oct Dog 241
for (words in someNonEmptyLists(strings())) { assert words*.size().sum() == words.sum().size() }
def pets = fixedValues(['Ant', 'Bee', 'Cat', 'Dog']) def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) def months = new Generator<String>() { Generator<Date> genDate = dates() String next() { genDate.next().format("MMM") } } def gen = new DefaultFrequencyGenerator(pets, 50) gen.add(nums, 30) gen.add(months, 20) 20.times { def next = gen.next().toString() println next assert next.size() == 3 }
QuickCheck: SimpBlog Case Study…
• Approach – Auto-selected values for author, category
– Auto-generated values for title, content
AgileAust 2011 - 30
© A
SE
RT
2006-2
011
def authors = ["Bart", "Homer", "Lisa", "Marge", "Maggie"] def categories = ["Home", "Work", "Food", "Travel"] 10.times { postAndCheck anyString(), anyFixedValue(categories), anyFixedValue(authors), anyString() }
def postAndCheck(title, category, author, content) { ... }
…QuickCheck: SimpBlog Case Study
AgileAust 2011 - 31
© A
SE
RT
2006-2
011
Screenshot
of fixed
titles version
Test Generation: Going Further
• AutoMocks
• Auto domain classes
• Assertions
• http://en.wikipedia.org/wiki/QuickCheck – Erlang, Scheme, Common Lisp, Perl, Python, Clojure,
Scala, Ruby, Java, F#, Standard ML, JavaScript, C++
AgileAust 2011 - 32
© A
SE
RT
2006-2
011
Topics
• Introduction
• Using testing DSLs
• All combinations and all pairs
• Auto generated tests
Model-based testing
• Business/logic rules
• Example driven testing
• Further information AgileAust 2011 - 33
© A
SE
RT
2006-2
011
Workshop • We recently added AJAX to the
SimpBlog application and some
of our users are reporting issues
• Depending on the order in which
the blog form is filled out there
can sometimes be strange error
messages displayed
• What testing strategies can we
use in this scenario?
AgileAust 2011 - 34
© A
SE
RT
2006-2
011
Workshop • We recently added AJAX to the
SimpBlog application and some
of our users are reporting issues
• Depending on the order in which
the blog form is filled out there
can sometimes be strange error
messages displayed
• What testing strategies can we
use in this scenario?
AgileAust 2011 - 35
© A
SE
RT
2006-2
011
What about:
* When a new
Category or
Author is added
in the future?
* What does
coverage mean?
Model-based testing
• Deriving test suites from source code is
usually deemed impractical
• Instead develop models which describe
certain characteristics of the desired
system behaviour
• May involve: – theorem proving
– constraint logic programming
– model checking
– event-flow models
– markov chain models
AgileAust 2011 - 36
© A
SE
RT
2006-2
011
ModelJUnit...
• Description – Supports model-based testing
– Allows you to write simple finite
state machine (FSM) models or
extended finite state machine
(EFSM) models in Java or Groovy
– You can then generate tests from
those models and measure various
model coverage metrics
AgileAust 2011 - 37
© A
SE
RT
2006-2
011
...ModelJUnit...
AgileAust 2011 - 38
© A
SE
RT
2006-2
011
// require modeljunit.jar import nz.ac.waikato.modeljunit.coverage.* import nz.ac.waikato.modeljunit.*
class VendingMachineModel implements FsmModel { def state = 0 // 0,25,50,75,100 void reset(boolean testing) {state = 0}
boolean vendGuard() {state == 100} @Action void vend() {state = 0}
boolean coin25Guard() {state <= 75} @Action void coin25() {state += 25}
boolean coin50Guard() {state <= 50} @Action void coin50() {state += 50} }
def tester = new RandomTester(new VendingMachineModel()) tester.buildGraph() def metrics = [new ActionCoverage(), new StateCoverage(), new TransitionCoverage(), new TransitionPairCoverage()] metrics.each { tester.addCoverageMetric it }
tester.addListener "verbose" tester.generate 20
println '\nMetrics Summary:' tester.printCoverage()
...ModelJUnit
AgileAust 2011 - 39
© A
SE
RT
2006-2
011
...
done (0, coin50, 50) done (50, coin25, 75) done (75, coin25, 100) done Random reset(true) done (0, coin50, 50) done (50, coin25, 75) done (75, coin25, 100) done (100, vend, 0) done (0, coin50, 50) done (50, coin50, 100) done (100, vend, 0) done (0, coin25, 25) done (25, coin25, 50) done Random reset(true) done (0, coin50, 50) done (50, coin25, 75) done (75, coin25, 100) done (100, vend, 0) done (0, coin50, 50) done (50, coin25, 75) ...
...
Metrics Summary: action coverage: 3/3 state coverage: 5/5 transition coverage: 7/8 transition-pair coverage: 8/12 ...
Interlude: Seven Bridges of Königsberg • Königsberg, Prussia (now Kaliningrad, Russia)
spans the Pregel River including two large
islands connected by 7 bridges
• Find a walk through the city crossing each
bridge once and only once. The islands can not
be reached by any route other than the bridges,
and every bridge must be crossed completely
every time.
AgileAust 2011 - 40
© A
SE
RT
2006-2
011
Source: http://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg
Interlude: Seven Bridges of Königsberg • Königsberg, Prussia (now Kaliningrad, Russia)
spans the Pregel River including two large
islands connected by 7 bridges
• Find a walk through the city crossing each
bridge once and only once. The islands can not
be reached by any route other than the bridges,
and every bridge must be crossed completely
every time. Euler showed there is no solution.
AgileAust 2011 - 41
© A
SE
RT
2006-2
011
Source: http://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg
Interlude: Königsberg Bridges Variation 8. The Blue Prince contrives a stealthy plan to build an
eighth bridge so that he can begin in the evening at
his Schloß, walk the bridges, and end at the Gasthaus
to brag of his victory. Of course, he wants the Red
Prince to be unable to duplicate the feat from the
Red Castle. Where does the Blue Prince build the
eighth bridge?
9. The Red Prince, infuriated by his brother's Gordian
solution to the problem, wants to build a ninth bridge,
enabling him to begin at his Schloß, walk the bridges,
and end at the Gasthaus to rub dirt in his brother's
face. As an extra bit of revenge, his brother should
then no longer be able to walk the bridges starting and
ending at his Schloß as before. Where does the Red
Prince build the ninth bridge?
10.The Bishop has watched this furious bridge building
with dismay. It upsets the town's Weltanschauung
and, worse, contributes to excessive drunkenness.
He wants to build a tenth bridge that allows all the
inhabitants to walk the bridges and return to their own
beds. Where does the Bishop build the tenth bridge?
AgileAust 2011 - 42
© A
SE
RT
2006-2
011
Source: http://en.wikipedia.org/wiki/Seven_Bridges_of_K%C3%B6nigsberg
ModelJUnit: SimpBlog Case Study...
• Does the order in which form
information is entered affect
the application?
AgileAust 2011 - 43
© A
SE
RT
2006-2
011
// require modeljunit.jar, htmlunit.jar import nz.ac.waikato.modeljunit.coverage.* import nz.ac.waikato.modeljunit.* import com.gargoylesoftware.htmlunit.WebClient class SimpBlogModel implements FsmModel {
boolean authorSelected = false boolean categorySelected = false boolean titleEntered = false boolean contentEntered = false int count = 0 def client, page, form // Special known method, allows equivalence class definition // example states: __ __ __ __, AU __ __ __, AU CA TI CO def getState() { "${authorSelected ? ' AU ' : ' __ '}${categorySelected ? ' CA ' : ' __ '}" + "${titleEntered ? ' TI ' : ' __ '}${contentEntered ? ' CO ' : ' __ '}" } ...
... void reset(boolean testing) { authorSelected = false categorySelected = false titleEntered = false contentEntered = false client = new WebClient() url = 'http://localhost:8080/postForm' page = client.getPage(url) assert 'Welcome to SimpBlog' == page.titleText form = page.getFormByName('post') } ...
...ModelJUnit: SimpBlog Case Study...
AgileAust 2011 - 44
© A
SE
RT
2006-2
011
... boolean "enter title Guard"() { !titleEntered }
@Action void "enter title "() { titleEntered = true form.getInputByName('title').setValueAttribute("Title ${count++}") } boolean enterContentGuard() { !contentEntered }
@Action void enterContent() { contentEntered = true form.getTextAreaByName('content').setText("Content ${count++}") } boolean chooseAuthorGuard() { !authorSelected }
@Action void chooseAuthor() { authorSelected = true // simple version just Lisa form.getSelectByName('author').getOptions().find{ it.text == 'Lisa' }.setSelected(true) } boolean pickCategoryGuard() { !categorySelected }
@Action void pickCategory() { categorySelected = true // simple version just Home form.getSelectByName('category').getOptions().find{ it.text == 'Home' }.setSelected(true) } boolean "submit post Guard"() { categorySelected && authorSelected && titleEntered && contentEntered }
@Action void "submit post "() { def result = form.getInputByName('btnPost').click() assert result.getElementsByTagName('h1').item(0).textContent.matches('Post.*: Title .*') // could do more asserts here reset(true) } } // end of SimpBlogModel class definition ...
...ModelJUnit: SimpBlog Case Study...
AgileAust 2011 - 45
© A
SE
RT
2006-2
011
def tester = new RandomTester(new SimpBlogModel()) tester.buildGraph() def metrics = [ new ActionCoverage(), new StateCoverage(), new TransitionCoverage(), new TransitionPairCoverage() ] metrics.each { tester.addCoverageMetric it }
tester.addListener "verbose" tester.generate 50
println '\nMetrics Summary:' tester.printCoverage()
def graphListener = tester.model.getListener("graph") graphListener.printGraphDot "simpblog.dot" println "\nGraph contains " + graphListener.graph.numVertices() + " states and " + graphListener.graph.numEdges() + " transitions."
...ModelJUnit: SimpBlog Case Study...
AgileAust 2011 - 46
© A
SE
RT
2006-2
011
done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( __ CA __ __ , enterContent, __ CA __ CO ) done ( __ CA __ CO , enter title , __ CA TI CO ) done ( __ CA TI CO , chooseAuthor, AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( __ CA __ __ , chooseAuthor, AU CA __ __ ) done ( AU CA __ __ , enter title , AU CA TI __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , chooseAuthor, AU __ __ __ ) done ( AU __ __ __ , pickCategory, AU CA __ __ ) done ( AU CA __ __ , enter title , AU CA TI __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , enterContent, __ __ __ CO ) done ( __ __ __ CO , pickCategory, __ CA __ CO ) done ( __ CA __ CO , chooseAuthor, AU CA __ CO ) done ( AU CA __ CO , enter title , AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( __ CA __ __ , enter title , __ CA TI __ ) done ( __ CA TI __ , chooseAuthor, AU CA TI __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , chooseAuthor, AU __ __ __ ) done ( AU __ __ __ , pickCategory, AU CA __ __ ) done ( AU CA __ __ , enter title , AU CA TI __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) ...
... done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( __ CA __ __ , enterContent, __ CA __ CO ) done ( __ CA __ CO , chooseAuthor, AU CA __ CO ) done ( AU CA __ CO , enter title , AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , chooseAuthor, AU __ __ __ ) done ( AU __ __ __ , pickCategory, AU CA __ __ ) done ( AU CA __ __ , enterContent, AU CA __ CO ) done ( AU CA __ CO , enter title , AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done ( __ __ __ __ , chooseAuthor, AU __ __ __ ) done ( AU __ __ __ , enter title , AU __ TI __ ) done ( AU __ TI __ , pickCategory, AU CA TI __ ) done ( AU CA TI __ , enterContent, AU CA TI CO ) done ( AU CA TI CO , submit post , __ __ __ __ ) done Random reset(true) done ( __ __ __ __ , pickCategory, __ CA __ __ ) done ( __ CA __ __ , enterContent, __ CA __ CO ) done ( __ CA __ CO , enter title , __ CA TI CO ) done ( __ CA TI CO , chooseAuthor, AU CA TI CO ) Metrics Summary: action coverage: 5/5 state coverage: 12/16 transition coverage: 19/33 transition-pair coverage: 26/56 Graph contains 16 states and 33 transitions.
...ModelJUnit: SimpBlog Case Study...
AgileAust 2011 - 47
© A
SE
RT
2006-2
011
Simplified
version
(just Lisa)
Advanced
version
Model-based testing: Going further
• Model-based testing – http://www.cs.waikato.ac.nz/~marku/mbt/modeljunit/
– http://en.wikipedia.org/wiki/Model_based_testing
– http://home.mit.bme.hu/~micskeiz/pages/modelbased_testing.html
– A Taxonomy of Model-Based Testing http://www.cs.waikato.ac.nz/pubs/wp/2006/uow-cs-wp-2006-04.pdf
• Related algorithms – Job scheduling problem
– Travelling salesman problem
– Chinese postman problem
– Vehicle routing problem
– Spanning trees and Hamiltonian paths
– http://www.nada.kth.se/~viggo/problemlist/compendium.html
– http://en.wikipedia.org/wiki/List_of_terms_relating_to_algorithm
s_and_data_structures AgileAust 2011 - 49
© A
SE
RT
2006-2
011
Topics
• Introduction
• Using testing DSLs
• All combinations and all pairs
• Auto generated tests
• Model-based testing
Business/logic rules
• Example driven testing
• Further information AgileAust 2011 - 50
© A
SE
RT
2006-2
011
Workshop • Set up some test cases
representing the Simpsons
weekly blogging habits:
– They never blog on the same day
– Marge blogs only on a Saturday or
Sunday
– Maggie blogs only on a Tuesday or
Thursday
– Lisa blogs only on a Monday,
Wednesday or Friday
– Bart blogs only on the day after Lisa
– Homer only blogs if noone else
blogged the previous day and
doesn't allow anyone to blog the
next day AgileAust 2011 - 51
© A
SE
RT
2006-2
011
Workshop • Set up some test cases
representing the Simpsons
weekly blogging habits:
– They never blog on the same day
– Marge blogs only on a Saturday or
Sunday
– Maggie blogs only on a Tuesday or
Thursday
– Lisa blogs only on a Monday,
Wednesday or Friday
– Bart blogs only on the day after Lisa
– Homer only blogs if noone else
blogged the previous day and
doesn't allow anyone to blog the
next day AgileAust 2011 - 52
© A
SE
RT
2006-2
011
What about:
* When the
habits change?
Constraint/Logic Programming...
• Description – Style of programming where relations between
variables are stated in the form of constraints
– First made popular by logic programming languages
such as Prolog but the style is now also used outside
logic programming specific languages
– Constraints differ from the common primitives of
other programming languages in that they do not
specify one or more steps to execute but rather the
properties of a solution to be found
– Popular libraries used with Groovy supporting
constraint programming include Gecode/J, Choco
and tuProlog
– We'll look at Choco as an example
AgileAust 2011 - 53
© A
SE
RT
2006-2
011
...Constraint/Logic Programming...
AgileAust 2011 - 54
© A
SE
RT
2006-2
011
Source: http://xkcd.com/287/
...Constraint/Logic Programming...
AgileAust 2011 - 55
© A
SE
RT
2006-2
011
// requires choco 2.1.0-basic.jar from http://choco.emn.fr/ import static choco.Choco.* import choco.kernel.model.variables.integer.IntegerVariable def m = new choco.cp.model.CPModel() def s = new choco.cp.solver.CPSolver() def menu = [ 'Mixed fruit' : 215, 'French fries' : 275, 'Side salad' : 335, 'Hot wings' : 355, 'Mozzarella sticks' : 420, 'Sampler plate' : 580 ] def numOrdered = new IntegerVariable[menu.size()] def priceEach = new int[menu.size()] def sum = 1505 ...
Found a solution: 7 * Mixed fruit Found a solution: 1 * Mixed fruit 2 * Hot wings 1 * Sampler plate
...Constraint/Logic Programming
AgileAust 2011 - 56
© A
SE
RT
2006-2
011
... menu.eachWithIndex { name, price, i -> // number ordered >= 0 // number ordered * price <= sum numOrdered[i] = makeIntVar(name, 0, sum.intdiv(price)) priceEach[i] = price } m.addConstraint(eq(scalar(numOrdered, priceEach), sum)) s.read(m) def more = s.solve() while (more) { println "Found a solution:" numOrdered.each { def v = s.getVar(it) if (v.val) println " $v.val * $v.name" } more = s.nextSolution() }
SimpBlog Case Study...
AgileAust 2011 - 57
© A
SE
RT
2006-2
011
def m = new choco.cp.model.CPModel() def s = new choco.cp.solver.CPSolver() daysOfWeek = ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"] def bart = makeIntVar('Bart', 0, 6) def homer = makeIntVar('Homer', 0, 6) def marge = makeIntVar('Marge', 0, 6) def lisa = makeIntVar('Lisa', 0, 6) def maggie = makeIntVar('Maggie', 0, 6) def simpsons = [bart, homer, marge, lisa, maggie] ...
...SimpBlog Case Study...
AgileAust 2011 - 58
© A
SE
RT
2006-2
011
...
// They never blog on the same day for (i in 0..<simpsons.size()) for (j in 0..<i) m.addConstraint(neq(simpsons[i], simpsons[j]))
// Marge blogs only on a Saturday or Sunday m.addConstraint(or(eq(marge, 0), eq(marge, 6)))
// Maggie blogs only on a Tuesday or Thursday m.addConstraint(or(eq(maggie, 2), eq(maggie, 4)))
// Lisa blogs only on a Monday, Wednesday or Friday m.addConstraint(or(eq(lisa, 1), eq(lisa, 3), eq(lisa, 5)))
// Bart blogs only on the day after Lisa m.addConstraint(eq(plus(lisa, 1), bart))
// Homer only blogs if noone else blogged the previous // day and doesn't allow anyone to blog the next day m.addConstraint(and(distanceNEQ(homer, marge, 1), distanceNEQ(homer, bart, 1), distanceNEQ(homer, maggie, 1), distanceNEQ(homer, lisa, 1))) ...
...SimpBlog Case Study
AgileAust 2011 - 59
© A
SE
RT
2006-2
011
...
s.read(m) def more = s.solve() if (!more) println "No Solutions Found" else println pad("Solutions:") + simpsons.collect{ pad(it.name) }.join() while (more) { print pad("") println simpsons.collect { def v = s.getVar(it) pad(daysOfWeek[v.val]) }.join() more = s.nextSolution() }
def pad(s) { s.padRight(12) }
Solutions: Bart Homer Marge Lisa Maggie Thursday Saturday Sunday Wednesday Tuesday Tuesday Saturday Sunday Monday Thursday Saturday Tuesday Sunday Friday Thursday Thursday Sunday Saturday Wednesday Tuesday
Logic solvers: Going further
AgileAust 2011 - 60
© A
SE
RT
2006-2
011
• Available tools – JSR-331: Java Constraint Programming API
http://jcp.org/en/jsr/detail?id=331
– JaCoP: Java Constraint Programming solver
http://jacop.osolpro.com/
– ChocoSolver
http://www.emn.fr/z-info/choco-solver/
Topics
• Introduction
• Using testing DSLs
• All combinations and all pairs
• Auto generated tests
• Model-based testing
• Business/logic rules
Example driven testing
• Further information AgileAust 2011 - 61
© A
SE
RT
2006-2
011
AgileAust 2011 - 63
© A
SE
RT
2006-2
011
Canoo WebTest • Description – Open source tool for automated testing of web applications
– Declarative approach in XML or testing DSL in Groovy
– Has Test Recorder
– Excellent reporting options
– Ant-based under the covers
<target name="login" >
<testSpec name="normal" >
&config;
<steps>
<invoke stepid="get Login Page"
url="login.jsp" />
<verifytitle stepid="we should see the login title"
text="Login Page" />
<setinputfield stepid="set user name"
name="username"
value="scott" />
<setinputfield stepid="set password"
name="password"
value="tiger" />
<clickbutton stepid="Click the submit button"
label="let me in" />
<verifytitle stepid="Home Page follows if login ok"
text="Home Page" />
</steps>
</testSpec>
</target>
EasyB Example ...
• When run will be marked as pending – perfect for ATDD
AgileAust 2011 - 65
© A
SE
RT
2006-2
011
scenario "Bart posts a new blog entry", { given "we are on the create blog entry page" when "I have entered 'Bart was here' as the title" and "I have entered 'Cowabunga Dude!' into the content" and "I have selected 'Home' as the category" and "I have selected 'Bart' as the author" and "I click the 'Create Post' button" then "I expect the entry to be posted" }
...EasyB Example...
AgileAust 2011 - 66
© A
SE
RT
2006-2
011
description "Post Blog Entry Feature" narrative "for feature", { as_a "Blogger" i_want "to be able to post a blog" so_that "I can keep others informed" } before "posting blog", { given "we are on the create blog entry page", { webClient = new com.gargoylesoftware.htmlunit.WebClient() page = webClient.getPage('http://localhost:8080/postForm') } } scenario "Bart was here blog", { when "I have entered 'Bart was here' as the title", { form = page.getFormByName('post') form.getInputByName('title').setValueAttribute( 'Bart was here (and so was EasyB)') } ...
...EasyB Example...
AgileAust 2011 - 67
© A
SE
RT
2006-2
011
... and "I have entered 'Cowabunga Dude!' into the content", { form.getTextAreaByName('content').setText('Cowabunga Dude!') } and "I have selected 'Home' as the category", { form.getSelectByName('category').getOptions().find { it.text == 'Home' }.setSelected(true) } and "I click the 'Create Post' button", { result = form.getInputByName('btnPost').click() } then "I expect the entry to be posted", { // check blog post details assert result.getElementsByTagName('h1').item(0).textContent.matches('Post.*: Bart was here.*') def h3headings = result.getElementsByTagName('h3') assert h3headings.item(1).textContent == 'Category: Home' // traditional style h3headings.item(2).textContent.shouldBe 'Author: Bart' // BDD style // expecting: <table><tr><td><p>Cowabunga Dude!</p></td></tr></table> def cell = result.getByXPath('//TABLE//TR/TD')[0] def para = cell.firstChild assert para.textContent == 'Cowabunga Dude!' // para.shouldHave textContent: 'Cowabunga Dude!' } }
JBehave Example...
AgileAust 2011 - 69
© A
SE
RT
2006-2
011
Given we are on the create blog entry page When I have entered "Bart was here" as the title And I have entered "Cowabunga Dude!" as the content And I have selected "Home" from the "category" dropdown And I have selected "Bart" from the "author" dropdown And I click the 'Create Post' button Then I should see a heading message matching "Post.*: Bart was here.*"
import org.jbehave.scenario.Scenario import org.jbehave.scenario.steps.Steps class NewPostScenario extends Scenario { NewPostScenario() { super([new CreateBlogSteps()] as Steps[]) } }
Scenario:
Given we are on the create blog entry page
When I have entered "Bart was here" as the title
And I have entered "Cowabunga Dude!" as the content
And I have selected "Home" from the "category" dropdown
And I have selected "Bart" from the "author" dropdown
And I click the 'Create Post' button
Then I should see a heading message matching "Post.*: Bart was here.*"
...JBehave Example...
AgileAust 2011 - 70
© A
SE
RT
2006-2
011
import org.jbehave.scenario.steps.Steps import org.jbehave.scenario.annotations.* import com.gargoylesoftware.htmlunit.WebClient class CreateBlogSteps extends Steps { def page, form, result @Given("we are on the create blog entry page") void gotoEntryPage() { page = new WebClient().getPage('http://localhost:8080/postForm') } @When('I have entered "$title" as the title') void enterTitle(String title) { form = page.getFormByName('post') form.getInputByName('title'). setValueAttribute(title + ' (and so was JBehave)') } @When('I have entered "$content" as the content') void enterContent(String content) { form.getTextAreaByName('content').setText(content) } ...
Interlude: Einstein’s Riddle…
• Wikipedia: The zebra puzzle is a well-
known logic puzzle
–Often called Einstein's Puzzle or
Einstein's Riddle
–Said to have been invented by Albert
Einstein as a boy
–Claim is that Einstein said “Only 2
percent of the world's population can
solve it.”
AgileAust 2011 - 72
© A
SE
RT
2006-2
011
Interlude: …Einstein’s Riddle
AgileAust 2011 - 73
© A
SE
RT
2006-2
011
• Some premises: – The British person lives in the red house
– The Swede keeps dogs as pets
– The Dane drinks tea
– The green house is on the left of the white house
– The green homeowner drinks coffee
– The man who smokes Pall Mall keeps birds
– The owner of the yellow house smokes Dunhill
– The man living in the center house drinks milk
– The Norwegian lives in the first house
– The man who smokes Blend lives next to the one who keeps cats
– The man who keeps the horse lives next to the man who smokes Dunhill
– The man who smokes Bluemaster drinks beer
– The German smokes Prince
– The Norwegian lives next to the blue house
– The man who smokes Blend has a neighbor who drinks water
• And a question: Who owns the fish?
Einstein’s Riddle : Prolog
© A
SE
RT
2006-2
011
% from http://www.baptiste-wicht.com/2011/09/solve-einsteins-riddle-using-prolog % Preliminary definitions persons(0, []) :- !. persons(N, [(_Men,_Color,_Drink,_Smoke,_Animal)|T]) :- N1 is N-1, persons(N1,T). person(1, [H|_], H) :- !. person(N, [_|T], R) :- N1 is N-1, person(N1, T, R). % The Brit lives in a red house hint1([(brit,red,_, _, _)|_]). hint1([_|T]) :- hint1(T). % The Swede keeps dogs as pets hint2([(swede,_,_,_,dog)|_]). hint2([_|T]) :- hint2(T). % The Dane drinks tea hint3([(dane,_,tea,_,_)|_]). hint3([_|T]) :- hint3(T). % The Green house is on the left of the White house hint4([(_,green,_,_,_),(_,white,_,_,_)|_]). hint4([_|T]) :- hint4(T). % The owner of the Green house drinks coffee. hint5([(_,green,coffee,_,_)|_]). hint5([_|T]) :- hint5(T). ...
AgileAust 2011 - 74
Einstein’s Riddle : Groovy testing DSL
© A
SE
RT
2006-2
011
AgileAust 2011 - 75
the Dane drinks tea the German smokes prince the Swede keeps dogs the Brit has a red house the Norwegian owns the first house the man from the centre house drinks milk the owner of the green house drinks coffee the owner of the yellow house smokes dunhill the person known to smoke pallmall rears birds the man known to smoke bluemaster drinks beer the green house is on the left side of the white house the man known to smoke blends lives next to the one who keeps cats the man known to keep horses lives next to the man who smokes dunhill the man known to smoke blends lives next to the one who drinks water the Norwegian lives next to the blue house
ZiBreve…
AgileAust 2011 - 77
© A
SE
RT
2006-2
011
So
urc
e: E
vo
lvin
g W
eb
-Ba
sed
Test A
uto
ma
tion
into
Ag
ile B
usin
ess S
pecific
ation
s,
Mu
grid
ge e
t a
l
SpiderFixture from FitLibrary (FitNesse) ZiBreve
…ZiBreve
AgileAust 2011 - 78
© A
SE
RT
2006-2
011
Source: Evolving Web-Based Test Automation into Agile Business Specifications, Mugridge et al
Example driven testing
• Bridging the communication Gap – Specification by example and agile acceptance testing by
Gojko Adzic
• Specification by Example – How successful teams deliver the right software by Gojko
Adzic: http://manning.com/adzic/
• Exploration Through Example – http://www.exampler.com/blog/ (Brian Marick)
• Evolving Web-Based Test Automation into
Agile Business Specifications
Rick Mugridge et al – http://www.mdpi.com/1999-5903/3/2/159/pdf
AgileAust 2011 - 79
© A
SE
RT
2006-2
011
Topics
• Introduction
• Using testing DSLs
• All combinations and all pairs
• Auto generated tests
• Model-based testing
• Business/logic rules
• Example driven testing
Further information AgileAust 2011 - 80
© A
SE
RT
2006-2
011
Conclusion
• Don’t be afraid to use technology
– For development, testing, deployment
• Embrace continuous learning
• There are many ways to capture business
requirements
– “Deciding what to build”
– Find the best way to represent your
requirements
– Prefer machine understandable,
executable solutions
AgileAust 2011 - 81
© A
SE
RT
2006-2
011
About the Author
83
© A
SE
RT
2006-2
011
Dr Paul King leads ASERT, an organisation based in
Brisbane, Australia which provides software development,
training and mentoring services to customers wanting to
embrace new technologies, harness best practices and
innovate. He has been contributing to open source projects
for nearly 20 years and is an active committer on numerous
projects including Groovy. Paul speaks at international
conferences, publishes in software magazines and journals,
and is a co-author of Manning's best-seller: Groovy in Action.
Paul King Director, ASERT
@paulk_asert
Top Related