What Do We Automate First
-
Upload
rrice2000 -
Category
Technology
-
view
406 -
download
0
description
Transcript of What Do We Automate First
![Page 1: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/1.jpg)
THE LOW-HANGING FRUIT OF TEST AUTOMATION - WHAT DO WE AUTOMATE FIRST?
RANDALL RICE, CTAL
WWW.RICECONSULTING.COM
March 19, 2014
![Page 2: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/2.jpg)
2
BIO - RANDALL W. RICE • Over 35 years experience in building and testing
information systems in a variety of industries and technical environments
• ASTQB Certified Tester – Foundation level, Advanced level (Full)
• Director, American Software Testing Qualification Board (ASTQB)
• Chairperson, 1995 - 2000 QAI’s annual software testing conference
• Co-author with William E. Perry, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems
• Principal Consultant and Trainer, Rice Consulting Services, Inc.
• www.riceconsulting.com & randallrice.blogspot.com
![Page 3: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/3.jpg)
3
FEASIBILITY 1. Is it easy (or even feasible) to automate the function/
component? • Problematic issues
• Custom objects • Non-standard objects • Objects with dynamic attributes (session IDs, etc.) • Streaming, dynamic data (think stock tickers and other data
that is changing even while you are testing it) • The tool doesn’t handle a certain type of object or component.
![Page 4: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/4.jpg)
4
CONTROL 2. Who controls/develops the application?
• Are any components (or even the entire application) vendor-developed?
• How stable is the code? • Are there a lot of hotfixes or maintenance releases • If so, do you know when they occur?
• Do you have prior knowledge of changes? • Some vendors have problems in creating and distributing
release notes, even to their own testers.
![Page 5: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/5.jpg)
5
OTHER QUESTIONS 3. Is the function easily repeatable?
• Do you feel like a robot testing it? 4. Is the function easily performed?
• Does it take a lot of set-up? • Are there a lot of steps to perform?
• Long scripts are hard to maintain? 5. How frequently is it used?
• X times per day (or hour) 6. How long does the function take to test manually, with all
test conditions?
![Page 6: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/6.jpg)
6
NUMBER OF TESTS Ti
me
per f
unct
iona
l exe
cutio
n
Number of times executed per test
Long term ROI (not so good)
![Page 7: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/7.jpg)
7
TEST FREQUENCY Ti
me
per t
est s
uite
exe
cutio
n
Number of times tested per day, week, etc.
Good choice
Long term ROI (not so good)
Much depends on the length of the test. 5 test conditions vs. 500 obviously makes a big Difference!
![Page 8: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/8.jpg)
8
REAL-LIFE EXAMPLE • A financial services client had a simple
function that took about 5 minutes per condition to test.
• However, it took the tester all day to test it due to the number of conditions.
• We spent 3 days to automate it, with the result being 15 minutes to test all conditions!
![Page 9: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/9.jpg)
9
HELPFUL TEST AUTOMATION METRICS (2) • Effort saved by test automation (manual vs. automation) • Effort needed to automate new tests • Effort needed to analyze failed automated tests • Effort to maintain automated tests • Test coverage achieved by automated tests
![Page 10: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/10.jpg)
10
EQUIVALENT MANUAL TEST EFFORT • Also known as EMTE • How much time would it take if the automated tests were
performed manually?
![Page 11: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/11.jpg)
11
EMTE EXAMPLE • Automated test takes 1 hour to run and are performed for
10 cycles. • Manual version of test takes 4 days to run. • EMTE = 40 days
• 4 days X 10 cycles
![Page 12: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/12.jpg)
12
RETURN ON INVESTMENT (ROI)
Time
Effort Spent on
Automating Tests
Development Effort on
Production Code
Initial Effort
Increased Effort (Hump)
Reduced Effort
Source: xUnit Patterns by Gerald Meszaros
Saved Effort
![Page 13: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/13.jpg)
13
RETURN ON INVESTMENT (2) - NOT SO GOOD
Time
Effort Spent on
Automating Tests
Development Effort on
Production Code
Initial Effort
Increased Effort (Hump)
Ongoing Effort
Source: xUnit Patterns by Gerald Meszaros
Saved Effort
![Page 14: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/14.jpg)
14
MORE QUESTIONS 7. Does the function require:
• Human judgment for outcomes? • Creative types of testing
• “Off the scripted path?” 8. Are you looking for new defects?
• Test automation is more confirmatory than discovery in nature.
• i.e., the defects found by automation are normally regression-type defects.
![Page 15: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/15.jpg)
15
NEW OR MAINTENANCE? 9. Is this new software or maintenance?
• Great debate: When should the automation be created – during the project or after?
• Since functional changes ripple through automation, great care must be taken not to automate too early.
• However, automation can be helpful during the final phases of a project to speed up regression testing.
• Keep in mind it takes time to create test automation and the distraction can actually turn into a project risk!
![Page 16: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/16.jpg)
16
WHO AND WHEN? 10. Who will create and use the automation?
• Simple capture/playback (when it works) can be created by testers.
• When problems are encountered, technical help at the developer level is often needed.
11. Which phase of testing?
• Maintenance testing is ideal • UAT is not good (you want humans testing) • System testing is a possible win • Unit test automation is ideal and essential
![Page 17: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/17.jpg)
17
NEED FOR PRECISION 12. What is the need for precision?
• Exact regression testing requires automation
• One mistake can invalidate the entire test
• Otherwise, you are doing “pseudo-regression” testing
• What is the risk? • If exact testing is needed, such as
medical devices, avionics, etc., automation can help mitigate risks
• WARNING: test automation is “software testing software”, therefore the test scripts may have bugs
![Page 18: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/18.jpg)
18
KNOWLEDGE AND STABILITY 13. Do you have a sufficient way to compare the test results
for pass/fail? • You must have a source (non-human)
14. How stable is the function to be tested?
• This is an irony • Functional errors (which you want to find) will cause the
scripts to fail • However, they will make test automation difficult because you
will need a way to deal with errors without stopping other tests in the suite.
![Page 19: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/19.jpg)
19
![Page 20: What Do We Automate First](https://reader034.fdocuments.us/reader034/viewer/2022051616/55388eb14a79595b7a8b47a8/html5/thumbnails/20.jpg)
20
CONTACT INFORMATION
Randall W. Rice, CTAL Rice Consulting Services, Inc. P.O. Box 892003 Oklahoma City, OK 73170 Ph: 405-691-8075 Fax: 405-691-1441 Web site: www.riceconsulting.com e-mail: [email protected]