Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption...
Transcript of Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption...
![Page 1: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/1.jpg)
Model-Based Testing for the Masses
Harry Robinson
![Page 2: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/2.jpg)
Is the Industry Ready for Model-Based Testing?
![Page 3: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/3.jpg)
The State of Testing Today
$ 59 500 000 000€ 49 400 000 000
“It's not going to work.”Jim Allchin, Senior MS executive,
Wall Street Journal 23 Sept 2005
Windows Vista Release Pushed Back to January 2007
Tuesday, March 21, 2006
![Page 4: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/4.jpg)
But Formality Intimidates Testers
![Page 5: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/5.jpg)
And Big Models Intimidate Testers
Model Based Testing for Real: The Inhouse Card Case StudyPretschner, Slotosch, Lötzbeyer, Aiglstorfer, Kriebel
![Page 6: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/6.jpg)
“Modelling is hard”
- AGEDIS Case Studies: Model-Based Testing in Industry
And …
![Page 7: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/7.jpg)
A Great Industrial Technology should be …
• Cheap
• Easy to apply
• Measurable
![Page 8: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/8.jpg)
CheapTest teams typically have
• Little budget to start
• Less after buying shelfware
• Very little budget for methodologies
![Page 9: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/9.jpg)
Easy to Apply (1)
• This will depend on the tester’s– Background
– Methodology training
![Page 10: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/10.jpg)
Industry Testers: Education
Some college 28%
PhD 1%
Master’s 20%
Bachelor’s 51%2005 Salary Survey
Better Software magazineMarch 2006
![Page 11: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/11.jpg)
Industry Testers: Area of Study
CS/IS 31%
Other 21%
Liberal Arts 10%
Business 14%Engineering 20%
Math 4%
2005 Salary Survey Better Software magazine
March 2006
![Page 12: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/12.jpg)
Industry Testers: Methodologies
vs.
![Page 13: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/13.jpg)
Easy to Apply (2)
• Integration with existing processes– Lightweight
– Incremental approach
pls
test
See “What Do You Mean ‘pls test?’ ”http://www.qualitytree.com/ruminate/041597.htm
![Page 14: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/14.jpg)
Measurable (1)
Number of test cases created
Number of test cases generated?
![Page 15: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/15.jpg)
Measurable (2)
Number of bugs found
Number of bugs prevented?
“Nobody Ever Gets Credit for Fixing Problems that Never Happened”- Repenning and Sterman
![Page 16: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/16.jpg)
Measurable (3)
“Do your testing for nuthin’, find your bugs for free!”- Noel Nyman, Microsoft
![Page 17: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/17.jpg)
Factors in Favor of MBT
• Time• Machines
![Page 18: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/18.jpg)
Time
PM Dev Test
PM Dev TestT
bugs
T
bugs
T
bugs
T
bugs
T
bugs
T
bugs
bugs
bugs
bugs
bugs
![Page 19: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/19.jpg)
Machines
• A typical test machine• Costs € 250 ($300 USD ) per year to buy and operate• Can work 100 hours per week
• A typical test engineer • Earns € 41 500 ($50 000 USD ) per year• Works 40 to 50 hours per week
![Page 20: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/20.jpg)
0100002000030000400005000060000700008000090000
1 tester, 1 cpu
2 testers, 2 cpus
1 tester, 10 cpus
cpu costtester cost
52000hrs/yr
2080hrs/yr
4160hrs/yr
€ Hours of Testing vs Cost
![Page 21: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/21.jpg)
0100002000030000400005000060000700008000090000
1 tester, 1 cpu
2 testers, 2 cpus
1 well-paid tester,10 cpus
cpu costtester cost
52000hrs/yr
2080hrs/yr
4160hrs/yr
€ Hours of Testing vs Cost
![Page 22: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/22.jpg)
Caught in the Middle
MBT
programmers
academics
"Because MBT … requires strong software development skills, when I recruit test engineers, I look for developers and get them excited about testing.”
Jeff Feldstein, Cisco Systems
testers
![Page 23: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/23.jpg)
What’s Happening Now?
• Home-Brew Models– C# / Python– Watir– Graphviz
• Half-hearted, piecemeal adoption– Whiteboard modeling– Hand-traced test sequences
![Page 24: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/24.jpg)
![Page 25: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/25.jpg)
Where Could Models Take the Industry?
• Better specs
• Better metrics
• Integration of testers into the development process
• Continuous testing
• Testers becoming more technical
![Page 26: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/26.jpg)
MBT is Entering the Mainstream“These [High Volume Test Automation] techniques are not widely used in
industry, but we believe they have the potential to help us substantially increase the reliability of software.”
Experiments with High Volume Test AutomationCem Kaner, Pat McGee
“Modeling in general seems to be gaining favor; particularly in domains where quality is essential and less-than-adequate software is not an option.”
Model-Based Software TestingJames Whittaker, Ibrahim El-Far
![Page 27: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/27.jpg)
What Needs to Happen?
Responses:• Easier notation• Incremental approach• Pilots and examples
bool Running = false;bool Scientific = false;
[Action] void SetScientific(bool newScientific)requires Running;requires Scientific != newScientific; {Scientific = newScientific;
}
Spec#
ZIndustry’s issues with modeling:• Too academic• Too process-heavy• Too steep a learning curve
![Page 28: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the](https://reader033.fdocuments.us/reader033/viewer/2022060220/5f06f2497e708231d41a869c/html5/thumbnails/28.jpg)
How Can You Help?• Do case studies of well-known systems
• Publish in QA magazines
• Teach Advanced Software Testing
• Spend time with practitioners
• Encourage your students to study industrial testing
• Share your insights with the industrial community