Introduction to Automated Acceptance Testing
Transcript of Introduction to Automated Acceptance Testing
copyright (c) 2008 by Micah Martin. All rights reserved.
Introduction to Automated Acceptance
Testing
Micah Martin, 8th Light, [email protected]
copyright (c) 2008 by Micah Martin. All rights reserved.
What are Acceptance Tests?
copyright (c) 2008 by Micah Martin. All rights reserved.
What are Acceptance Tests?
ATs vs UTs
• Written by “Customer”
• Test Features
• Written by developers
• Test Code
Acceptance Tests Unit Tests
copyright (c) 2008 by Micah Martin. All rights reserved.
What are Acceptance Tests?
Criteria for Completion
• Provided by the “Customer”
• Given to developers
• Specify stories
• Criteria for Done-Done
copyright (c) 2008 by Micah Martin. All rights reserved.
Communication
• Forces customers, testers, and developers to work together
• creation
• implementation
• execution
What are Acceptance Tests?
copyright (c) 2008 by Micah Martin. All rights reserved.
FeedbackWhat are Acceptance Tests?
• Are a source of data
• A tool for project management
copyright (c) 2008 by Micah Martin. All rights reserved.
0102030405060708090
100
1 3 5 7 9 11
Total AT'sFailing AT'sPassing AT's
Data From Acceptance
copyright (c) 2008 by Micah Martin. All rights reserved.
Acceptance Tests: A Critical Piece of Agile
copyright (c) 2008 by Micah Martin. All rights reserved.
Analysis
Design
Implementation
DFDERD
DDST
1 May 1 Nov1 Jul 1 Sep
Traditional ApproachAcceptance Tests: A Critical Piece of Agile
Object Mentor
copyright (c) 2008 by Micah Martin. All rights reserved.
The Agile AproachAcceptance Tests: A Critical Piece of Agile
May 1 Jul 1 Sep 1 Nov 1
Analysis
Design
Implementation
copyright (c) 2008 by Micah Martin. All rights reserved.
The Agile ApproachAcceptance Tests: A Critical Piece of Agile
IterationFailing Acceptance
Tests
Passing Acceptance
Tests
copyright (c) 2008 by Micah Martin. All rights reserved.
The Agile ApproachAcceptance Tests: A Critical Piece of Agile
IterationFailing Acceptance
Tests
Passing Acceptance
Tests
TDD
copyright (c) 2008 by Micah Martin. All rights reserved.
• Every story must have at least one Acceptance Test
• A story is not DONE until it passes it’s Acceptance Tests
Criteria for DONEAcceptance Tests: A Critical Piece of Agile
copyright (c) 2008 by Micah Martin. All rights reserved.
Manual Acceptance TestsAcceptance Tests: A Critical Piece of Agile
copyright (c) 2008 by Micah Martin. All rights reserved.
Manual Acceptance TestsAcceptance Tests: A Critical Piece of Agile
Manual Acceptance Tests
copyright (c) 2008 by Micah Martin. All rights reserved.
Manual Acceptance TestsAcceptance Tests: A Critical Piece of Agile
Manual Acceptance Tests
copyright (c) 2008 by Micah Martin. All rights reserved.
Acceptance Tests Are Automated
copyright (c) 2008 by Micah Martin. All rights reserved.
The Button• How often would
you press it?
• When would you press it?
• Who would press it?
• Testers, Developers, Managers, Customers, Spectators, etc.
Acceptance Tests: A Critical Piece of Agile
copyright (c) 2008 by Micah Martin. All rights reserved.
Automated Tests Require:
• High level, yet, precise language
• Unambiguous detail
• “Examples”
• Interaction with the system
Acceptance Tests: A Critical Piece of Agile
copyright (c) 2008 by Micah Martin. All rights reserved.
copyright (c) 2008 by Micah Martin. All rights reserved.
Criteria for DONE
copyright (c) 2008 by Micah Martin. All rights reserved.
Criteria for DONE
+
copyright (c) 2008 by Micah Martin. All rights reserved.
Criteria for DONE
Automated+
copyright (c) 2008 by Micah Martin. All rights reserved.
Criteria for DONE
Automated
Executable Specification
+
copyright (c) 2008 by Micah Martin. All rights reserved.
Executable Specification
• A new paradigm for testing
• Puts quality first
• Removes ambiguity from requirements
Acceptance Tests: A Critical Piece of Agile
copyright (c) 2008 by Micah Martin. All rights reserved.
Who Writes Acceptance Tests?
copyright (c) 2008 by Micah Martin. All rights reserved.
The Customer• Yeah right! Then who?
• The Customer Role
• Stake holder
• Business Analyst
• Quality Assurance
• Product Owner
Who Writes Acceptance Tests?
copyright (c) 2008 by Micah Martin. All rights reserved.
Tests Get Technical
• The “Customer” may need technical help to write tests
• Developers are technical
• Pair test authoring
Who Writes Acceptance Tests?
copyright (c) 2008 by Micah Martin. All rights reserved.
Business Rules Get Fuzzy
• Sometimes developers need help understanding tests
• Customers know business rules
• Pair test implementation
Who Writes Acceptance Tests?
copyright (c) 2008 by Micah Martin. All rights reserved.
Business Decisions are Hard To Make
• Sometimes, the “customer” will be challenged to make decisions
• Stakeholders can help. Developers can help.
• Do you see a pattern of communication?
Who Writes Acceptance Tests?
copyright (c) 2008 by Micah Martin. All rights reserved.
Exercise #1
copyright (c) 2008 by Micah Martin. All rights reserved.
The Login Test
• Write a test plan, in plain text, for the business rules of logging in.
• Web application
• User credentials are stored in relational database
• Successful login redirects to “Welcome” page
Exercise #1
copyright (c) 2008 by Micah Martin. All rights reserved.
Writing Good Acceptance Tests
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Direct browser to URL for login page
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Direct browser to URL for login page
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Direct browser to URL for login page
1. Enter the username ‘wallace’
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Direct browser to URL for login page
1. Enter the username ‘wallace’
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Direct browser to URL for login page
1. Enter the username ‘wallace’
Build a Testable Environment First
copyright (c) 2008 by Micah Martin. All rights reserved.
BOCWriting Good Acceptance Tests
Build
Operate
Check
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Add some users to the system
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Add some users to the system
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Add some users to the system
3. Enter a value into the username field
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Add some users to the system
3. Enter a value into the username field
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Add some users to the system
3. Enter a value into the username field
Be Specific
copyright (c) 2008 by Micah Martin. All rights reserved.
Tests are Examples
• Use concrete examples
• Specify concrete behavior
• No ambiguity allowed
Writing Good Acceptance Tests
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Insert into User table values (’wallace’, ‘ilikecheeze’)
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Insert into User table values (’wallace’, ‘ilikecheeze’)
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Insert into User table values (’wallace’, ‘ilikecheeze’)2. Open a browser to the URL http://localhost/myapp
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Insert into User table values (’wallace’, ‘ilikecheeze’)2. Open a browser to the URL http://localhost/myapp
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test PossibilitiesWriting Good Acceptance Tests
1. Insert into User table values (’wallace’, ‘ilikecheeze’)2. Open a browser to the URL http://localhost/myapp
Avoid Implementation Details
copyright (c) 2008 by Micah Martin. All rights reserved.
Avoid Implementation Details
Writing Good Acceptance Tests
System
Database
Tests
DatabaseDatabase
UI
copyright (c) 2008 by Micah Martin. All rights reserved.
Login Test: Possible Solution
• Add user to system: (’wallace’, ‘ilikecheeze’)
• Process login with username ‘wallace’ and password ‘blah’
• Check login failed
• Process login with username ‘wallace’ and password ‘ilikecheeze’
• Check login succeeded
Writing Good Acceptance Tests
copyright (c) 2008 by Micah Martin. All rights reserved.
Tools
copyright (c) 2008 by Micah Martin. All rights reserved.
Commercial Tools
• WinRunner
• Silk
• Robot
Tools
Are not suitable for Acceptance Testing in an Agile environment
copyright (c) 2008 by Micah Martin. All rights reserved.
Open Source Options
• FIT
• FitNesse
Tools
Among the few tools that support Test Driven Development
copyright (c) 2008 by Micah Martin. All rights reserved.
FIT
• Framework for Integrated Tests
• Created by Ward Cunningham
• Open Source
• The most accepted solution for agile acceptance testing
Tools
copyright (c) 2008 by Micah Martin. All rights reserved.
FitNesse
• Environment build around FIT
• Makes everything easier
• Created by Object Mentor, Inc.
• Open Source
Tools
copyright (c) 2008 by Micah Martin. All rights reserved.
FIT FitNesse
- Tests written in HTML- Tests are executed on the command line- Tables are executed- Non-table markup is ignored- Tables map to Fixtures- Fixtures are code that is aware of the system- Supplies foundational Fixtures- Implementations ported to many languages
- Stand alone web server- Is a wiki- Tests written in wiki text- Tests are executed from within the wiki- Translates tests into HTML- Uses FIT to execute tests- Supports test suites- Supports variables in tests- Supports test refactoring- Written in Java- Supports FIT implementations in any language
copyright (c) 2008 by Micah Martin. All rights reserved.
Thinking in Tables
copyright (c) 2008 by Micah Martin. All rights reserved.
Only Tables ExecuteThinking in Tables
Ignored
Executed
copyright (c) 2008 by Micah Martin. All rights reserved.
Foundational Table Structure
Thinking in Tables
Name of Fixture
Interaction with Application
• Table structure depends on type of Fixture
copyright (c) 2008 by Micah Martin. All rights reserved.
3 Foundation Fixtures
• Column Fixture
• Row Fixture
• Action Fixture
Thinking in Tables
copyright (c) 2008 by Micah Martin. All rights reserved.
Column FixtureThinking in Tables
package eg;// Copyright (c) 2002 Cunningham & Cunningham, Inc.// Released under the terms of the GNU GeneralPublic …
import fit.ColumnFixture;
public class Division extends ColumnFixture { public float numerator; public float denominator; public float quotient() { return numerator / denominator; }}
copyright (c) 2008 by Micah Martin. All rights reserved.
Row FixtureThinking in Tables
package fitnesse.fixtures;
import fit.RowFixture;
public class EmployeePayRecordsRowFixture extendsRowFixture { public Object[] query() throws Exception { EmployeePayRecord[] records = newEmployeePayRecord[2]; records[0] = new EmployeePayRecord(1, 1000); records[1] = new EmployeePayRecord(2,2000); return records; } public Class getTargetClass() { return EmployeePayRecord.class; }}
public class EmployeePayRecord { public int id; private double salary; public EmployeePayRecord(int id,
double salary){ this.id = id; this.salary = salary; } public double pay() { return salary; }}
Analogous to comparing against rows in a database table
copyright (c) 2008 by Micah Martin. All rights reserved.
Action Fixture • Think GUI window
Thinking in Tables
Counter WindowCounter:
Counter: 6 Count
public class CountFixture extends Fixture { private int counter = 0; public void count() { counter++; } public int counter() { return counter; }}
copyright (c) 2008 by Micah Martin. All rights reserved.
Exercise #2
copyright (c) 2008 by Micah Martin. All rights reserved.
Login With Tables
• Write the Login test using tables
• Keep in mind what we learned from the previous exercise
• BOC
• Examples
• Avoid Implementation Details
Exercise #2
copyright (c) 2008 by Micah Martin. All rights reserved.
Possible SolutionExercise #2
copyright (c) 2008 by Micah Martin. All rights reserved.
FitLibrary
• Extension to FIT
• Written by Rick Mugridge
• Adds some handy Fixtures
copyright (c) 2008 by Micah Martin. All rights reserved.
FitLibrary Fixtures• ArrayFixture for ordered lists
• SetFixture for unordered lists
• Supports
• Graphics
• Tree structures
• Nested Tables
FitLibrary
copyright (c) 2008 by Micah Martin. All rights reserved.
Text
DoFixtureFitLibrary
• Broken tables
• Highly readable
• Flexibility
copyright (c) 2008 by Micah Martin. All rights reserved.
Wiki
copyright (c) 2008 by Micah Martin. All rights reserved.
What is it?• A collaborative web site
• Editable by any
• Created by Ward Cunningham
• Every project should have one
• http://c2.com/wiki
• http://en.wikipedia.com
Wiki
copyright (c) 2008 by Micah Martin. All rights reserved.
Creating Tests•Use Wiki syntax to create a page with test tables•Label the page as a Test Page
•Use a page name of the form Test…•Turn on the Test property
•Make sure your Fixtures are in the classpath•Use !path widget
•Mechanics•!path values are concatenated•Java command to start FitServer is executed•Testable HTML is passed to FitServer•FitServer runs the tests•Results are passed back to FitNesse
Wiki
copyright (c) 2008 by Micah Martin. All rights reserved.
Creating Suites
•Set the Suite property•Create a page with the Suite property•Created test pages inside this page•When the suite is executed, all child test pages will
be included in the suite execution
•Use the !see widget•!see <name of test page>•All “included” tests pages will be included in the
suite execution
Wiki
There are 2 ways to make Suites
Run a Suite by clicking the Suite button
copyright (c) 2008 by Micah Martin. All rights reserved.
New Advancements
copyright (c) 2008 by Micah Martin. All rights reserved.
SlimNew Advancements
• New Extension to FitNesse
• Alternative to FIT
• More Flexible Integration
• Written by Uncle Bob
copyright (c) 2008 by Micah Martin. All rights reserved.
RSpec/CucumberNew Advancements
• BDD version of Acceptance Testing
• Pure Text
• Very Readable (Given, When, Then)
• Dan North, David Chelimsky, Aslak Hellesoy
• Works in Java (JRuby), .NET (Iron Ruby), C (Ruby)
copyright (c) 2008 by Micah Martin. All rights reserved.
The End.Micah Martin, 8th Light, Inc.
http://www.fitnesse.org http://www.8thlight.com