Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing...
Transcript of Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing...
![Page 1: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/1.jpg)
Automated Web Service Testing in Agile
Edmund Barton
![Page 2: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/2.jpg)
Contents
1. TEC & Industry Training – the “puzzle”
2. ITR as a solution to “the puzzle”
3. Our Agile Team 3. Our Agile Team
4. Technology & Tools
5. A Test Approach in Agile
6. What went well
7. What didn’t
8. Lessons Learned
![Page 3: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/3.jpg)
Take-homes
• Where modern software development is
• Where functional testing is heading
• How to reduce the testing “bottleneck”
• Insight into a project that used test automation well
![Page 4: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/4.jpg)
1. TEC & Industry Training – the “Puzzle”
The Client
![Page 5: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/5.jpg)
Tertiary Education Commission
Multi-$bn budget across the Tertiary Education Sector
1. TEC & Industry Training – the “Puzzle”
Governmental body
![Page 6: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/6.jpg)
Industry Training
• Funded by TEC
• Workplace training programmes
1. TEC & Industry Training – the “Puzzle”
• Modern Apprenticeships
Industry Training Organisations – ITOs
Question?
• How do we know how well ITOs are performing?
![Page 7: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/7.jpg)
1. TEC & Industry Training – the “Puzzle”2. ITR as a solution to “the Puzzle”
The Answeris to collect data
... like a vac
![Page 8: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/8.jpg)
I.T.R. or Industry Training Register
Collects data about all Industry Training
TEC gets a real-time view of Industry Training
2. ITR as a solution to “the Puzzle”
TEC gets a real-time view of Industry Training sector
• Accurate measures of ITOs performance
• Checks data against TECs policy
Data source for further analysis of Tertiary Sector
![Page 9: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/9.jpg)
I.T.R. as an IT solution
ITR is a Web Service, from scratch
No User Interface
2. ITR as a solution to “the Puzzle”
Application
e.g.
Student
Management
System
Web
Service
XML SOAP Messages
![Page 10: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/10.jpg)
Building ITR
2. ITR as a solution to “the Puzzle3. Our Agile Team
![Page 11: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/11.jpg)
A candidate for Agile?
Requires Flexibility
3. Our Agile Team
![Page 12: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/12.jpg)
ITR’s Agile Team
Requirements
partially
Therefore a
candidate for Agile
3. Our Agile Team
partially
flexible
Resources
3-6 developers
2-3 testers
Timescale
Fixed April ‘10
to Feb ‘11
candidate for Agile
![Page 13: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/13.jpg)
Agile Team
“The Business”
Development / Project Management
3. Our Agile Team
Development / Project Management
Test / QA - Qual IT
![Page 14: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/14.jpg)
ITR Project goes Agile
Adopted SCRUM
Testing at the same time as Development
3. Our Agile Team
Multiple User Stories per iteration
Software “Demo” after each iteration
Question:
• But there’s no UI, how did we demo progress?
![Page 15: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/15.jpg)
What was the role of the tester in this project?
3. Our Agile Team
![Page 16: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/16.jpg)
Role of the Tester in Agile
Test Automation Engineer
Distributor of knowledge
3. Our Agile Team
Project “glue”
Story quality gate
Usual defect wrangling / management
Facilitator of business acceptance during iteration
![Page 17: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/17.jpg)
Example Agile User Story
As a generic example:
As a:Consumer of ITR
3. Our Agile Team
I want to:receive an error message if I submit an invalid message
So that:I know I’ve made a mistake
List of Acceptance Criteria
conditions where this story applies
![Page 18: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/18.jpg)
Agile Project Approach
Fixes
This high-level process applies to each user story
3. Our Agile Team
Development Test
Bugs
Fixes
Application Feature
AutomatedTest
Process Outputs
Plan / Design
![Page 19: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/19.jpg)
Agile Project Approach
Test all the way through the iteration
More efficient use of test resources
3. Our Agile Team
Use “Test Driven Development”
Reliance on Automation technology
![Page 20: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/20.jpg)
What technology was used?
... here comes the science bit
3. Our Agile Team 4. Technology & Tools
![Page 21: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/21.jpg)
Agile Development Technologies
4 . Technology & Tools
We assume that:
• An application will function the same in each environment (if we control data and configuration)
![Page 22: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/22.jpg)
Agile Development Technologies
Each team member (inc Test) has:
• own VM (a Build workstation)
4 . Technology & Tools
• own VM (a Build workstation)
• Visual Studio 2008
• Team Foundation Server
• Common test data
• Build Server status
• Team City
![Page 23: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/23.jpg)
s
Build Server (Team City)
Provides feedback on the current state of the “solution” i.e. code doesn’t compile
4 . Technology & Tools
Team
Foundation
Server
Source code(under source control)
Team City
Build Server
![Page 24: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/24.jpg)
Build Server (Team City)
Nightly Builds (Team City)
• Run ALL automated functional tests (Regression suite)
• Nightly feedback on state of “solution” i.e. some new
4 . Technology & Tools
• Nightly feedback on state of “solution” i.e. some new code regressed existing functionality
Broken builds
• Buy / Make the team coffee
![Page 25: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/25.jpg)
Successful Nightly Builds
Essential for Agile software development
• Tell us we have working software
• Could be deployed right now!
4 . Technology & Tools
• Could be deployed right now!
New code hasn’t regressed other parts of the application
Rely heavily on the automated functional tests
Really important to have good test coverage
![Page 26: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/26.jpg)
4. Technology & Tools5. A Test Approach in Agile
What was the testing approach?
![Page 27: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/27.jpg)
Test Automation approach
Test Engineer has 3 key tasks:
• Create and support the automation framework
5. A Test Approach in Agile
• Create automated tests – validate features
• Add / remove from regression suite as project changes
• Can be a lot of work (ITR has over 2000 test cases)
![Page 28: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/28.jpg)
Example of Tools
Open Source
5. A Test Approach in Agile
Front end for Web Services – the missing UI
Large community of users – easy to find help
Data Driven test cases
![Page 29: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/29.jpg)
Example of framework
ITR written in .NET
Use NUnit to execute SOAP UI test cases
5. A Test Approach in Agile
ITR
Web
Service
SOAP Messages
Test suite project
.NET test
method
ITR
database
![Page 30: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/30.jpg)
Example of testing
Really grey-box testing, not fully open white box testing but more in-depth than black box
Achieved 100% requirements coverage
5. A Test Approach in Agile
Achieved 100% requirements coverage
SOAP UI projects self documenting
Traceable via “Spreadsheet of Doom”
![Page 31: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/31.jpg)
Agile – Approach to change
Agile projects change
Complete manual testing unfeasible
5. A Test Approach in Agile
Maintenance overhead?
Yes and No
• Yes – but don’t worry about throwing tests away
• Yes – but don’t maintain, rebuild
• No – just reflects churn within the project
![Page 32: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/32.jpg)
So what worked well?
5. A Test Approach in Agile 6. What went well
… what’s so flash?What worked well?
![Page 33: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/33.jpg)
1. Houston, we have a problem ...
6. What went well
![Page 34: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/34.jpg)
How to make it faster?
Other options failed to make a difference:
• Database tweaking
• Disabling logging
6. What went well
• Disabling logging
• Better hardware
Key architectural components were too slow
ITR needed more than just refactoring
![Page 35: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/35.jpg)
Improving performance
Developers:
• Rip out the poorly performing frameworks and re-engineered some of the architecture
6. What went well
engineered some of the architecture
• Re-factor poorly performing code
Use the automated functional test suite as the only validator of behaviour
Test First Development
![Page 36: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/36.jpg)
A real improvement
Result
• Massive performance
6. What went well
performance improvement
• 13x increase in message processing speed
![Page 37: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/37.jpg)
2. Better Deployments
Deploying wrong version of code = bad
Build server set up to automatically:
6. What went well
• run the regression suite
• check every test passed
• build a deployment package with correct versions
![Page 38: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/38.jpg)
2. Better Deployments
Greatly simplified the deployment
Over 50 deployments
6. What went well
Guaranteed the functionality of the deployment package:
• All deployment problems were configuration related
• much easier to fix
• Zero deployment functionality problems
![Page 39: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/39.jpg)
3. Using SOAP UI Tests for Business Acceptance
Web service has no UI
• Difficult to demo to the business
6. What went well
Demo the features created during the Agile iteration
Improved Agile acceptance “gates”
![Page 40: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/40.jpg)
So what worked well?
6. What went well7. What could have been improved
• … what’s so flash?What could have been done better?
![Page 41: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/41.jpg)
1. Agile troubles – Web Service
Fixed Time / Fixed Requirements – not enough resources. Requirements
partially
7. What could have been improved
Project struggled to be Agile
partially
flexible
Resources
3-6 developers
2-3 testers
Timescale
Fixed April ‘10
to Feb ‘11
![Page 42: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/42.jpg)
1. Agile troubles – A Web Service
Web Services have set list of operations
SOAP Messages
7. What could have been improved
SOAP Messages
ITR
Web
Service
ITR Client
Cannot just remove them – they are mandatory
operation
![Page 43: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/43.jpg)
2. Agile troubles – No Refactoring
Refactoring was discouraged in favour of more User Stories – “Velocity”
Iteration after iteration of on-the-fly decisions
7. What could have been improved
Iteration after iteration of on-the-fly decisions produces a terrible codebase
Often difficult to accommodate new requirements
Agile progress or “velocity” is not more important than code quality.
Address code quality as you go
![Page 44: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/44.jpg)
So what worked well?
7. What could have been improved8. Lessons Learned
• … what’s so flash?Lessons Learned
![Page 45: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/45.jpg)
1. Lessons about Agile Projects
• If the project isn’t a candidate, don’t bend it to fit
• Needs a strong PM with Agile experience
8. Lessons Learned
• Testing does fit – Testers become part of the development team
• Test Automation pretty much essential on projects of this scale
![Page 46: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/46.jpg)
2. Lessons about Agile Development
• Agile projects are not just about “velocity”, quality must also be a driver
• Consequence of design decisions being made
8. Lessons Learned
• Consequence of design decisions being made on-the-fly is poor code quality
• There’s no getting away from a need to plan / design and architect
![Page 47: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/47.jpg)
3. Lessons about Test Automation
Steep Learning Curve
Highly valuable
8. Lessons Learned
• Time Saved on Project Timelines
• Confidence in development for Business
• Test Driven Development and Regression
![Page 48: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/48.jpg)
A final word from the Developers
From the post-implementation review
• “The testing was awesome”
8. Lessons Learned
![Page 49: Automated Web Service Testing in Agile - ANZTB Barton... · 2013-11-04 · Example of testing Really grey-box testing, not fully open white box testing but more in-depth than black](https://reader034.fdocuments.us/reader034/viewer/2022050208/5f5ac6c5f6f309295c0c4288/html5/thumbnails/49.jpg)
Questions / Comments