Agile Testing PracticesProduct Owner: focused on ROI and owns priorities on the backlog Pigs and...
Transcript of Agile Testing PracticesProduct Owner: focused on ROI and owns priorities on the backlog Pigs and...
Agile Testing Practices
Megan S. SumrellDirector of Transformation ServicesValtech
Introductions
About us…Now about you…
Your nameYour companyYour roleYour experience with Agile or Scrum?
Personal Expectations
Agenda
IntroductionsAgile OverviewTraditional QA TeamsTraditional Automation ApproachesRole of an Agile TesterTesting Activities
Refine Acceptance CriteriaTDDManual / Exploratory TestingDefect ManagementDocumentationPerformance TestingRegression Testing
Agenda Continued
Test Automation on Agile TeamsTesting on a Greenfield ProjectTesting on a Legacy ApplicationEstimation SessionsSprint Planning MeetingsRetrospectivesInfrastructureSkills and TitlesClosing
Agile Overview
Agile Manifesto
"We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more."
Scrum
Terms and Definitions
User Story: high level requirementsProduct Backlog: list of prioritized user storiesSprint : one cycle or iteration (usually 2 or 4 weeks in length)Daily Stand-up: 15 minute meeting every day to review statusScrum Master: owns the Scrum process and removes impedimentsProduct Owner: focused on ROI and owns priorities on the backlogPigs and Chickens
Traditional QA Teams
How are you organized?When do you get involved in the project?What does your “test phase” look like?What testing challenges do you have?
Traditional Test Automation
Automation Challenges
Cost of toolsHard to learnCan’t find timeMaintenanceUI dependentOnly a few people can run them
Traditional Test Pyramid
UNIT TESTS
Business Rules
GUI TESTS
Will these strategies work in an Agile environment?
Food for Thought…….
“Tests play several pivotal roles during the software development process. First, tests unambiguously communicate how things are supposed to work. Second, they provide feedback on whether the system actually works the way it is supposed to work. Third, tests provide the scaffolding that allows developers to make changes throughout the development process…”
Lean Software Development – Mary and Tom Poppendieck
Role of An Agile Tester
Role of “QA” on an Agile Team
Testing starts with the specification, not verificationFocus on acceptance criteriaFocus on automation, not manual testingUser Stories are not done until the automated acceptance tests are passingForces the team to think about testability at the beginning
Testing Activities
Testing Activities
Refine acceptance criteriaTest Driven DevelopmentManual / Exploratory TestingDefect ManagementDocumentationPerformance TestingRegression Testing
Refine Acceptance Criteria
Requirements vs. Tests
Refine Acceptance Tests
• Who?– Analysts, Testers,
Developers, PO• How?
– Collaborate!– Build executable
requirements if possible• Why?
– Avoids misinterpretation of the requirements
– Full transparency into the test cases
– Sets expectations
ATDD: Acceptance Test Driven Development
Development is driven by examplesAcceptance testing happens before development startsBusiness facing testsRepresent workflows and business logic rules
Some Questions to Ask…….
What else do the developers need to know about this user story?What can go wrong during the story?What am I assuming?Are there circumstances when this story may behave differently?
Workshop: Let’s write some acceptance tests….
User Story: As a mathematician, I want to be able to use the calculator to divide numbers and get the answer so that I can solve problems.
Acceptance Tests:Verify that the division is correctVerify that the answer is in ##.## format
Workshop - continued
User Story: As a user, I want to search for a name so that I can find that person’s phone number.Acceptance Tests:
Search by first or last name‘Like’ match searchResults are in first name, last name, phone number order
Test Driven Development
TDD is not unit testingTDD is not writing tests where it is easy to do itTDD is not testing after you have codedTDD is not manually testing your code
TDD IS ABOUT TESTING FIRST
TDD: Test Driven Development
TDD is not magicTo be effective, tests must be:• Accurate• Appropriate• Sufficient
TDD: Test Driven Development
TDD is a discipline that developers must adoptTesters can help educate developers on how to write good tests
Developers tend to test what worksTesters tend to look for what breaks
TDD: Test Driven Development
testDivideByZero extends TestCase {Calculator calc = new Calculator();try {
calc.divide(6, 0);fail(“No exception was thrown”);
} catch(MathException e) {assertEquals(“Cannot divide by zero”, e.message());
}}
Write the Test
Before:
public int divide(int divisor, intdividend) {
return divisor / dividend;}
Write the Code
Write the Code
After:
public int divide(int dividend, int divsor) {if (divisor == 0)
throw new MathException(“Cannot divide by zero”);
return divisor / dividend;}
Refactor
Remove duplicationImprove implementationSimplify
Exploratory Testing
Exploratory Testing
• Who?– Testers
• How?– Paired testing
• Why?– Ensure all negative or
unusual paths are covered
– Paired testing adds value
Session Based Exploratory Testing
James and Jonathan Bach – 2000Consists of the following:
CharterSessionSession ReportDebrief
“Exploratory testing is simultaneous learning, test design, and test execution.”
-James Bach
Workshop - Create Charters
Think about buying something on-line. What charters would you create for exploratory testing on an on-line checkout system
Defect Management
Defect Management
• Who?– Testers (sometimes
Analysts)• How?
– Verify that unit tests were written
– Add automated tests for the future
• Why?– Ensure that the defects
were fixed– Automate tests to ensure it
doesn’t break again
Documentation
Workshop: Documentation
What do you document on your agile teams?What are your test artifacts?
Eliminate Waste
Ensure all artifacts are valuableDon’t create things people won’t useDon’t Repeat Yourself (DRY)
If a test can take the replace of a requirements document, don’t write both
Don’t do more than is necessaryIf a wireframe sketch is sufficient to describe a GUI, don’t spend the time and money to create a photo-realistic mock-up
Performance Testing
Performance Testing
• Who?– Performance Testers
• How?– Map test cases to
performance requirements
• Why?– Uncover performance
issues early!!!
Regression Testing
How can you regression test every iteration?
Automation (lots and lots of it!)Should happen every dayImmediate feedback when code changes are made
Test Automation on Agile Teams
Agile Test Pyramid
GUI TESTS
Business Rules
UNIT TESTS
Automate Acceptance Tests
• Who?– Testers and Developers
• How?– Collaborate!– Work to automate defined
acceptance tests• Why?
– Allows everyone to track progress
– Ensures requirements are met
– Allows for fast, easy regression testing
More Reasons Why We Tackle Automation this Way….
Test FirstStart automation without the UIInvolves entire team in testingInvolves customer in testingVisibility of resultsAvoid gold platingLeads to good architecture and designTests are easy to readCollapse the V modelTesting code is minimalAnd more…….
Executable Requirements
Think back to our requirement vs. test conversation…..Let’s see this in action
FitNesse
Unit vs. GUI vs. FitNesse
Code oriented
GUI TESTING
FitNesse
UNIT TESTING
GUI specific
Customer Oriented
FIT TABLE FIXTURE SYSTEM
FitNesse has several fixture types…
Column - calculationsRow/Array – data setsDo/Action - workflowActionDatabaseAnd more….
Division
Numerator Denominator Quotient?
10 2 5
100 2.5 40
First Row Identifies the Test/Requirement. This is the name of the fixture (the class)
Anatomy of a Column Fixture
Division
Numerator Denominator Quotient?
10 2 5
100 2.5 40
Second Row is the inputs and outputs. Expected outputs have the ? Or () after them, inputs do not
Division
Numerator Denominator Quotient?
10 2 5
100 2.5 40
Each additional row is an instance of the test that explains the requirement we are testing
Workshop: Write a Column Fixture
User Story 1: As a mathematician, I want to be able to use the calculator to divide numbers and get the answer so that I can solve problems.Acceptance Tests:
Verify that the division is correctVerify that the answer is in ##.## format
Anatomy of a Row Fixture
ListofUsers Active
FirstName LastName UserName
John Smith Jsmith
Beth Jones Bjones
Suzy Plus splus
First Row Identifies the Test/Requirement. This is the name of the fixture (and the name of the class)
Other cells on the first row represent inputs passed into the fixture
ListofUsers Active
FirstName LastName UserName
John Smith Jsmith
Beth Jones Bjones
Suzy Plus splus
The second row lists the data elements in the expected output
ListofUsers Active
FirstName LastName UserName
John Smith Jsmith
Beth Jones Bjones
Suzy Plus splus
The 3rd through nth rows list the data you expect the test to return
Workshop: Write a Row Fixture
User Story 3: As a user, I want to search for a name so that I can find their phone number.Acceptance Tests:
Search by first or last name“Like” match searchResults are in first name, last name, phone number
Do Fixture
Do fixtures should read like a storyUsed to test workflowsOften tie to epicsEach step in the do fixture likely has detailed tests elsewhereLet’s look at an example….
Example of a Do Fixture
As a roulette player, I want to place multiple bets and have my winnings automatically calculated so that I can become a millionaire.To do this, a series of steps need to happen (that could each be a separate story)
PLACE BETS
SPIN WHEEL
CALCUALTE WINNINGS
FIXTURE NAME: defines the class
SIMPLE ACTIONS: method names that execute them
�
�
�
�
ROW FIXTURES: method returns a row fixture
�
�
Some Additional Considerations
Your automation should control your dataYour automation should be quickYour automation should be machine agnosticYour automation should let you sleep at nightYour test should run in any order
Some Additional Considerations
Who will write the tests?Fitnesse is designed to permit Customers to write testsHowever, the Customer will probably need help from QA• Fitnesse Wiki syntax• Corner cases and unhappy paths
Some Additional Considerations
Impact on DevelopmentThe fixtures used in Fitnesse tests need internal implementations written in the same language as the codeThese internal fixtures are the “glue” that binds the test tables to the application’s implementation that is being testedThis code must be written and maintained by developers
Testing on a Greenfield Application
Define Acceptance Tests
Exploratory
Automate Acceptance Tests
TDD (Test Driven Development)
Manual Testing
Verify Bug Fixes
Full Regression
Performance
SPRINT START SPRINT END
Testing on a Legacy Application
Legacy Challenges
What challenges exist with Legacy Applications that make agile testing challenging?
Strategy 1: Not recommended
Define Acceptance Tests ExploratoryAutomate Acceptance Tests
TDD (Test Driven Development)
Manual Testing
Verify Bug Fixes
Full Regression
Performance
SPRINT START SPRINT END
Strategy 2: Hardening Sprint
Exploratory
SPRINT ONE SPRINT N(Hardening Sprint)
SPRINT TWO
Full RegressionAutomate Acceptance Tests
TDD
Manual Testing
Verify Bug Fixes
Exploratory
Automate Acceptance Tests
TDD
Manual Testing
Verify Bug Fixes
Exploratory
Performance
Estimation Sessions
Estimation Techniques
• Planning Poker• Rock, Paper, Scissors• Others?
Detail an Accuracy
• Some believe that better estimates require more detailed analysis
– More detailed analysis takes additional time and resources
• Others believe that the initial “gut” reaction is sufficient for Agile planning purposes
– Time required for additional analysis does not result in sufficiently better estimates to justify the added costs
Detail and Accuracy
• Estimates can be like statistics– Out of context, an individual estimate may
have little meaning– In the overall iteration, though, some
estimates will be low and others will be high• In the end, they cancel each other out
What are things testers often think about when estimating?
Sprint Planning Meetings
What are their Purpose?
Define the sprint backlogDefine “DONE”Commit to the sprint
Workshop: Let’s Create Tasks
User Story 1: As a mathematician, I want to be able to use the calculator to divide numbers and get the answer so that I can solve problems.
Task Breakout
• Who creates tasks?
• Who estimates tasks?
• Who owns tasks?
Definition of Done
Retrospectives
Retrospectives
Opportunity to review what worked and what didn’tMust be conducted in a non-threatening environmentTo be effective
All participants must feel comfortable and willing to shareShould be held at each iteration boundaryShould result in tangible action items that are put into place immediately
Workshop: Run a Retrospective
Infrastructure
Continuous Integration
“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily -leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible.”– Martin Fowler
Check-in Code
Run Tests
Build Entire Project
PublishResults
How Does CI Work?
Single Source Code RepositoryBuild is AutomatedBuild is self-testingEveryone commits every dayThe build is fastVisible resultsAutomated deployment
Test Environments?
Do you have an environment you have complete control over?Are builds pushed or pulled to it?How often do you get builds?Does it look like your production environment?Do you “own” it?
Test Data
• Where do you get your test data?
• How well do you know your test data?
• Do you control it?• Can you re-set it?• Can you run the same
script over and over again?
Data Needs
Different testing activities require different data – one database will not suit all your needs.
Manual / ExploratoryAutomationPerformanceAny others?
Skills and Titles
Titles….
QA Analyst
QA Engineer
Software
Tester
QA Architect
Developer
Senior DeveloperTest Engineer
Workshop: Hiring an Agile Tester
What skills would you want from your agile testers?
Closing
Agile Foundations
© 2009 Valtech, Inc. All rights reserved.
For more information,
Visit us on the web at http://www.valtech.us
Email us directly at [email protected]
Call us at (972) 789-1200
Fax us at (972) 789-1340
Write us (or just drop by) atValtech19111 North Dallas Pkwy, Suite 300Dallas, TX 75287
Sources
Mountain Goat Software (www.mountaingoatsoftware.com)“Testing Extreme Programming” by Lisa Crispin and Tip House“Test Driven .NET Development with FitNesse” by Gojko AdzicFitNesse.org“Agile Estimating and Planning” by Mike Cohn“User Stories Applied” by Mike Cohn“Lean Software Development” by Mary and Tom Poppendieck“Collaboration Explained” by Jean Tabaka