Agile Testing Practices

of 82 /82
Paul King Director, ASERT @paulk_asert [email protected] Leveraging Emerging Technologies in Agile Teams … and some testing tools you should try at home

Embed Size (px)

description

A look at different testing approachs for testing applications within Agile teams

Transcript of Agile Testing Practices

  • 1. Leveraging EmergingTechnologies in Agile Teams and some testing tools you should try at home Paul King Director, ASERT @paulk_asert [email protected]
  • 2. Topics Introduction Using testing DSLs All combinations and all pairs Auto generated tests ASERT 2006-2011 Model-based testing Business/logic rules Example driven testing Further information AgileAust 2011 - 2
  • 3. Emerging Technology Areas Development Deployment & Mgmt Continuous integration Continuous deployment Convention over Build optimisation configuration Cloud computing Language evolution Framework Support Agile ASERT 2006-2011 evolution via improved Concurrency Management of Configuration Complexity management A few favourites Testing Grails ATDD/BDD Gpars Testing DSLs Gradle Scripting Tableaux AgileAust 2011 - 3
  • 4. Advances in the Testing Arena Unit Testing Mock/interaction testing Techniques State-based testing Testing DSLs ATDD/BDD Data-driven Logic-driven ASERT 2006-2011 Integration Testing Model-driven Performance testing Acceptance Testing All-pairs & Web drivers combinations Non-web drivers Gpars Test runners AgileAust 2011 - 4
  • 5. Groovy and Testing Tool Spectrum* Utilities Runners AllPairs, Combinations Native Groovy, JUnit, TestNG, Spock, EasyB, Polyglot languages JBehave, Cucumber, Robot Framework, SLIM Logic programming Threads, Parallel / Web Database SOAP / Other Concurrency libraries Drivers Drivers REST Drivers Data-driven libraries Drivers Networking libraries WebTest DbUnit FEST ASERT 2006-2011 XML Processing GroovyWS WebDriver DataSets Email Read/write files / JWebUnit SqlUnit XML-RPC FTP Excel / Word / CSV Reporting, Logging Tellurium groovy.sql CXF AntUnit Selenium JPA Axis2 Telnet HtmlUnit JDO JAX-WS SSH Tools Watij BigTable JAX-RS Exec iTest2, SoapUI, Twist, HttpBuilder JDBC WindowLicker IDEs, JMeter, Text Cyberneko editors, Recorders, Sahi, Build Tools, CI * Tools/libraries/frameworks dont always neatly fall into one category still useful conceptually AgileAust 2011 - 5
  • 6. Application under Test ASERT 2006-2011 AgileAust 2011 - 6
  • 7. Concept Manual HTTP Request / Response Web Server ASERT 2006-2011 Automated Driver Runner Read Script HTTP Request / Response AgileAust 2011 - 7
  • 8. Driver Category Real browser invoker Browser Emulators Runs on platform Can simulate multiple supported by real browsers browser Less platform May need multiple restrictions platforms, e.g. IE6/IE7 Good for CI ASERT 2006-2011 Uses actual JavaScript Easier to not download engine images, resources Can be easier to use Ability to optimise with test recorders JavaScript interactions Automation More extensible capabilities differ Ability to disable across browsers JavaScript Can typically get to all Scope for parallelism aspects of browser AgileAust 2011 - 8
  • 9. What is Groovy? Groovy is like a super version of Java. It can leverage Javas enterprise capabilities but also has cool productivity features like closures, DSL support, builders and dynamic typing. ASERT 2006-2011 Groovy = Java boiler plate code + mostly dynamic typing + closures + domain specific languages + builders + metaprogramming + GDK library AgileAust 2011 - 9
  • 10. Topics Introduction Using testing DSLs All combinations and all pairs Auto generated tests ASERT 2006-2011 Model-based testing Business/logic rules Example driven testing Further information AgileAust 2011 - 10
  • 11. Prefer scripted over tool centric/recorders Scripting Tools/Recorders Choose open Lock-in to vendor tools languages Testing often done Developer friendly after development Test-first friendly Tests separate from Test using whatever code ASERT 2006-2011 levels make sense Proprietary scripting Tests with code languages Reduce testing User-interface documentation focussed Hard-coded details Refactor friendly leads to fragile tests Run with every change No or hard refactoring as part of continuous integration AgileAust 2011 - 11
  • 12. Prefer business talk over tech talk Avoid Prefer import org.openqa.selenium.By import org.openqa.selenium.htmlunit.HtmlUnitDriver def driver = new HtmlUnitDriver() driver.get(http://localhost:8080/postForm) assert driver.title == Welcome to SimpBlog // fill in query form and submit it Source: Evolving Web-Based Test Automation into Agile Business Specifications, Mugridge et al driver.findElement(By.name(title)). sendKeys(Bart was here) driver.findElement(By.name(content)). ASERT 2006-2011 sendKeys(Cowabunga dude!) def select = driver. findElement(By.name(category)) select. findElements(By.tagName("option")). find{ it.text == Home }.setSelected() driver.findElement(By.name(btnPost)).click() ... given we are on the blog entry page when entering Bart was here as the title and entering Home as the category then posting 5 appears AgileAust 2011 - 12
  • 13. Topics Introduction Using testing DSLs All combinations and all pairs Auto generated tests ASERT 2006-2011 Model-based testing Business/logic rules Example driven testing Further information AgileAust 2011 - 13
  • 14. Workshop Our blog application sometimes has errors but only for certain combinations of Author, Category and Content. How should we test it? ASERT 2006-2011 AgileAust 2011 - 14
  • 15. Workshop Our blog application sometimes has errors but only for certain combinations of Author, Category and Content. How should we test it? ASERT 2006-2011 What about: * When a new Category or Author is added in the future? * Tests take too long to run? AgileAust 2011 - 15
  • 16. All Combinations Description test(MacOS, 4G, 250G) Dont have a bunch test(Linux, 4G, 250G) of hard-coded, hard test(Vista, 4G, 250G) to maintain manual test(MacOS, 8G, 500G) test data or even test(Linux, 8G, 500G) manually generated test(Vista, 8G, 500G) // 30 more rows CSV file ASERT 2006-2011 Much better to generate test cases [ from succinct [MacOS, Linux, Vista], [2G, 4G, 6G, 8G], expressions of [250G, 350G, 500G] what you are trying ].combinations().each{ to achieve os, mem, disk -> test(os, mem, disk) } AgileAust 2011 - 16
  • 17. All Combinations Case Study def combos = [ ["Bart", "Homer", "Marge", "Lisa", "Maggie"], ["Work", "School", "Home", "Travel", "Food"], ["foo", "bar", "baz"] ].combinations() println "Found ${combos.size()} combos" combos.each { author, category, content -> ASERT 2006-2011 postAndCheck author, category, content } def postAndCheck(author, category, content) { // ... details not shown ... } Found 75 combos AgileAust 2011 - 17
  • 18. All Pairs Description Sometimes called pairwise testing or orthogonal array testing ASERT 2006-2011 Technique to limit the explosion of test cases by identifying samples of important classes of test cases (equivalence classes) providing maximum coverage with minimum testing Instead of all combinations, systematically use pair- wise combinations of interactions between objects as most faults result from adverse two-way interactions AgileAust 2011 - 18
  • 19. SimpBlog Case Study... def cases = new AllPairs().generate( author: ["Bart", "Homer", "Marge", "Lisa", "Maggie"], category: ["Work", "School", "Home", "Travel", "Food"], content: ["foo", "bar", "baz"]) println "Found ${cases.size()} cases" cases.each { next -> ASERT 2006-2011 println next // just for debugging purposes postAndCheck next.author, next.category, next.content } def postAndCheck(author, category, content) { // ... details not shown ... } AgileAust 2011 - 19
  • 20. ...SimpBlog Case Study Found 18 cases [content:bar, category:Food, author:Bart] [content:bar, category:School, author:Homer] [content:foo, category:Work, author:Bart] [content:baz, category:School, author:Homer] [content:bar, category:Home, author:Maggie] [content:foo, category:School, author:Marge] [content:bar, category:Work, author:Bart] ASERT 2006-2011 [content:baz, category:Travel, author:Bart] [content:foo, category:Home, author:Homer] [content:bar, category:Travel, author:Marge] [content:baz, category:Work, author:Homer] [content:bar, category:Travel, author:Lisa] [content:baz, category:Travel, author:Maggie] [content:baz, category:Home, author:Marge] [content:baz, category:Food, author:Homer] [content:baz, category:Travel, author:Lisa] [content:foo, category:Food, author:Maggie] [content:foo, category:Travel, author:Lisa] AgileAust 2011 - 20
  • 21. All Combinations/Pairs: Going Further Advanced options N-Wise Compulsory combinations Excluding combinations Prioritising combinations Shuffling ASERT 2006-2011 Sites http://en.wikipedia.org/wiki/All-pairs_testing http://www.pairwise.org/ (content starting to age but still useful) Tools SpecExplorer http://code.google.com/p/jwise/ AgileAust 2011 - 21
  • 22. Topics Introduction Using testing DSLs All combinations and all pairs Auto generated tests ASERT 2006-2011 Model-based testing Business/logic rules Example driven testing Further information AgileAust 2011 - 22
  • 23. Workshop Creating good test data is taking a long time Errors seem to occur when the system is used with data that we havent ASERT 2006-2011 tried before What else can we do? AgileAust 2011 - 23
  • 24. Workshop Creating good test data is taking a long time Errors seem to occur when the system is used with data that we havent ASERT 2006-2011 What about: tried before * When fields change in What else can we do? the future? * Measuring coverage? * Repeatable tests? AgileAust 2011 - 24
  • 25. QuickCheck Test Java/Groovy programs more declaratively using automatic random test data generation Replace manually selected scenario-based tests with specification-based testing Generators available: primitive types, ASERT 2006-2011 collections, POJOs value ranges lists, sets and array distinct values distributions determinism (random, deterministic ) generator strategies (composition of generator values (oneOf, frequency, list, array, nullsAnd), transformation, mutation) value frequencies (frequency, oneOf) AgileAust 2011 - 25
  • 26. QuickCheck Samples 682 Ant Bee for (words in someNonEmptyLists(strings())) { 141 assert words*.size().sum() == words.sum().size() Jul } Dog Dog def pets = fixedValues([Ant, Bee, Cat, Dog]) Dog def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) ASERT 2006-2011 801 def months = new Generator() { 177 Generator genDate = dates() String next() { genDate.next().format("MMM") } Ant } Dog def gen = new DefaultFrequencyGenerator(pets, 50) 951 gen.add(nums, 30) Cat gen.add(months, 20) Aug 20.times { Bee def next = gen.next().toString() Bee println next Oct assert next.size() == 3 } Dog 241 AgileAust 2011 - 26
  • 27. QuickCheck Samples 682 Ant Bee for (words in someNonEmptyLists(strings())) { 141 assert words*.size().sum() == words.sum().size() Jul } Dog Dog def pets = fixedValues([Ant, Bee, Cat, Dog]) Dog def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) ASERT 2006-2011 801 def months = new Generator() { 177 Generator genDate = dates() String next() { genDate.next().format("MMM") } Ant } Dog def gen = new DefaultFrequencyGenerator(pets, 50) 951 gen.add(nums, 30) Cat gen.add(months, 20) Aug 20.times { Bee def next = gen.next().toString() Bee println next Oct assert next.size() == 3 } Dog 241 AgileAust 2011 - 27
  • 28. QuickCheck Samples 682 Ant Bee for (words in someNonEmptyLists(strings())) { 141 assert words*.size().sum() == words.sum().size() Jul } Dog Dog def pets = fixedValues([Ant, Bee, Cat, Dog]) Dog def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) ASERT 2006-2011 801 def months = new Generator() { 177 Generator genDate = dates() String next() { genDate.next().format("MMM") } Ant } Dog def gen = new DefaultFrequencyGenerator(pets, 50) 951 gen.add(nums, 30) Cat gen.add(months, 20) Aug 20.times { Bee def next = gen.next().toString() Bee println next Oct assert next.size() == 3 } Dog 241 AgileAust 2011 - 28
  • 29. QuickCheck Samples 682 Ant Bee for (words in someNonEmptyLists(strings())) { 141 assert words*.size().sum() == words.sum().size() Jul } Dog Dog def pets = fixedValues([Ant, Bee, Cat, Dog]) Dog def nums = excludeValues(integers(100, 999, INVERTED_NORMAL), 500..599) ASERT 2006-2011 801 def months = new Generator() { 177 Generator genDate = dates() String next() { genDate.next().format("MMM") } Ant } Dog def gen = new DefaultFrequencyGenerator(pets, 50) 951 gen.add(nums, 30) Cat gen.add(months, 20) Aug 20.times { Bee def next = gen.next().toString() Bee println next Oct assert next.size() == 3 } Dog 241 AgileAust 2011 - 29
  • 30. QuickCheck: SimpBlog Case Study Approach Auto-selected values for author, category Auto-generated values for title, content def authors = ["Bart", "Homer", "Lisa", "Marge", "Maggie"] def categories = ["Home", "Work", "Food", "Travel"] ASERT 2006-2011 10.times { postAndCheck anyString(), anyFixedValue(categories), anyFixedValue(authors), anyString() } def postAndCheck(title, category, author, content) { ... } AgileAust 2011 - 30
  • 31. QuickCheck: SimpBlog Case Study ASERT 2006-2011 Screenshot of fixed titles version AgileAust 2011 - 31
  • 32. Test Generation: Going Further AutoMocks Auto domain classes Assertions http://en.wikipedia.org/wiki/QuickCheck Erlang, Scheme, Common Lisp, Perl, Python, Clojure, ASERT 2006-2011 Scala, Ruby, Java, F#, Standard ML, JavaScript, C++ AgileAust 2011 - 32
  • 33. Topics Introduction Using testing DSLs All combinations and all pairs Auto generated tests ASERT 2006-2011 Model-based testing Business/logic rules Example driven testing Further information AgileAust 2011 - 33
  • 34. Workshop We recently added AJAX to the SimpBlog application and some of our users are reporting issues Depending on the order in which the blog form is filled out there can sometimes be strange error ASERT 2006-2011 messages displayed What testing strategies can we use in this scenario? AgileAust 2011 - 34
  • 35. Workshop We recently added AJAX to the SimpBlog application and some of our users are reporting issues Depending on the order in which the blog form is filled out there can sometimes be strange error ASERT 2006-2011 messages displayed What about: * When a new What testing strategies can we Category or use in this scenario? Author is added in the future? * What does coverage mean? AgileAust 2011 - 35
  • 36. Model-based testing Deriving test suites from source code is usually deemed impractical Instead develop models which describe certain characteristics of the desired system behaviour ASERT 2006-2011 May involve: theorem proving constraint logic programming model checking event-flow models markov chain models AgileAust 2011 - 36
  • 37. ModelJUnit... Description Supports model-based testing Allows you to write simple finite state machine (FSM) models or extended finite state machine (EFSM) models in Java or Groovy ASERT 2006-2011 You can then generate tests from those models and measure various model coverage metrics AgileAust 2011 - 37
  • 38. ...ModelJUnit... // require modeljunit.jar import nz.ac.waikato.modeljunit.coverage.* import nz.ac.waikato.modeljunit.* class VendingMachineModel implements FsmModel { def state = 0 // 0,25,50,75,100 void reset(boolean testing) {state = 0} boolean vendGuard() {state == 100} @Action void vend() {state = 0} boolean coin25Guard() {state = 0 // number ordered * price