Test Automation: From Record/Playback to Frameworks · 2019-07-25 · 9. Arnold Thomas, R.II:...
Transcript of Test Automation: From Record/Playback to Frameworks · 2019-07-25 · 9. Arnold Thomas, R.II:...
Test Automation: From Record/Playback to
Frameworks John Kent,
Simply Testing, UKTh14
© Simply Testing Ltd 2007 Slide 2
Test Automation:
From Record/Playback
to FrameworksJohn Kent: [email protected]
© Simply Testing Ltd 2007 Slide 3
Test Automation:
From Record/Playback
to Frameworks
1. Introduction
2. Record/Playback
3. Basic Data-Driven
4. Limitations of Basic Approaches
5. Advanced Frameworks
6. Summary
© Simply Testing Ltd 2007 Slide 4
1. Introduction
• What Type of Automation?
• What Type of Tools?
• How the Tools Work
© Simply Testing Ltd 2007
Not:
x Performance Testing
x Test Harnesses/Drivers for Unit Test
x NUnit/JUnit frameworks
But:
Simulating User Interaction via User Interface
Automated Functional Testing via User Interface
What Type of Automation?
© Simply Testing Ltd 2007
Script-based tools for User Interface (UI) testing
Script-based means that they are programming
development environments
E.g.
• WinRunner, QuickTest Pro (Mercury HP)
• QARun, TestPartner (Compuware)
• Robot, Functional Tester (IBM Rational)
• Others
What Type of Tools?
© Simply Testing Ltd 2007 Slide 7
How Automation Tools Work
Actions Played back
Actions Recorded
© Simply Testing Ltd 2007 Slide 8
User Interface Objects
• Called Controls in Windows
• Called Widgets in Unix
• Objects with Standard Window Classes
– Text boxes
– Combo boxes
– Buttons
– Option Buttons
– Check boxes
– Menu items
– List boxes
– Non-Standard
© Simply Testing Ltd 2007 Slide 9
User Interface Object
Functions• Functions in the tool language
• Provide actions on UI objects
E.g.Button: button_press ( buttonID [, mouse_button ] );
E.g. Edit Box object:edit_set ( editboxID, text );
Generally of Form:
function (Object Identifier, Data)
© Simply Testing Ltd 2007 Slide 10
UI Object Maps• GUI Map in WinRunner
• Object Repository in QTP
• Robot does not have one but can build one
• QARun has one
A UI Object Map lists
the objects in each
window/page. Each UI
Object is given a
logical name. The
map contains the
information that the
test tool needs to
identify each object
uniquely. The test tool
script uses the logical
name to perform
actions on UI Objects
© Simply Testing Ltd 2007 Slide 11
2. Record/Playback
• Script is recorded whilst user performs
actions on SUT
• Can modify scripts with programming
(scripting)
• Test : test program ratio = 1:1
© Simply Testing Ltd 2007 Slide 12
Record/PlaybackFor Each Test
There is an Automation program
© Simply Testing Ltd 2007 Slide 13
3. Basic Data-Driven
Test scripts
Reads data file,
navigates to
screen/windows
and inputs data
Test ToolAdd Accounts
Data File
© Simply Testing Ltd 2007 Slide 14
Basic Data-Driven
• Can create more tests by adding data – not code
• Test : test program ratio not 1:1
• More maintainable than Record/Playback method
• Automation code becomes more and more tested
with use
© Simply Testing Ltd 2007 Slide 15
4. Limitations of
Basic Approaches
• Key features of test automation:– Automation is software engineering
– Maintenance of automation code is critical
– Testing of test automation code
• Test scripts must be– Developed
– Maintained
– Tested
© Simply Testing Ltd 2007 Slide 16
Limitations of
Record/Playback
• We are interested in what may happen,
not what has happened
• It produces ‘difficult’ code
• May not re-run due to synchronisation
problems
• The script (code) is not tested rigorously
© Simply Testing Ltd 2007 Slide 17
The Main Limitation of
Record/Playback
Number of Tests
Automated
Number of Automation
Programs
(Lines of Automation
Code)
stsNumberOfTedetomationCoLinesOfAmo
• Test : test program ratio =1:1
• The more the tests the larger the maintenance task
• Reaches limits of maintainability
© Simply Testing Ltd 2007 Slide 18
Limitations of
Data-Driven• It provides repetitive actions not tests
• The data does not look like a test
– Can add lots of (e.g.) accounts
– Good for repetitive actions
• Pumps data into SUT in unnatural way -e.g. all customers added then all accounts added
• Generally used for data load - i.e. not actually a test
© Simply Testing Ltd 2007 Slide 19
5. Advanced Frameworks
• What is Required?
• What is a Framework?
• Keyword Driven Automation– Business Scenario Level
– Window Level
• Object Mapping Approach
• Test Re-use
© Simply Testing Ltd 2007 Slide 20
What is Required?
• A software engineering approach to the
automation code
• Maintainable automation code
• An ‘elegant’ way to create tests
– A way to automate tests easily
© Simply Testing Ltd 2007 Slide 21
What is a Framework?
• Software Architecture
• Testware Architecture (Fewster, Graham)
• Software Structure
“The test framework is like an application architecture” Linda Hayes in The Automated Testing Handbook
“The software architecture of a program or computing system is the structure or structures of the system, which comprise software components, the externally visible properties of those components, and the relationships among them.”
Bass, Clements, and Kazman of Carnegie Mellon’s SEI - Software Architecture in Practice see http://www.sei.cmu.edu/architecture/
© Simply Testing Ltd 2007 Slide 22
Examples of Advanced
Automation
Frameworks/Architectures
• TestFrame from LogicaCMG
• EMOS_GPL open source
• SAFS open source
• ATAA from Simply Testing
© Simply Testing Ltd 2007 Slide 23
Keyword Driven Automation
For Each
Window/screen/page
or Business scenario
There is only one
automation
program/function
- a Wrapper
Two levels of
wrapper/keywords
Business scenario
KeywordsWindow Level
Keywords
Keyword Driven – the wrapper is the function which ‘performs’ the keyword
© Simply Testing Ltd 2007 Slide 24
Driver
WrappersData File Test Tool
Keyword Driven
Keyword Supplier_Add
wrapper
Function Supplier_Add
Function Loc_Add
Function Stock_Item
Driver reads the keyword
and calls the function for
that keyword
© Simply Testing Ltd 2007
Actions in Test Data
Some Navigation in Test Data (Keywords)
Data is abstracted business process
Business Scenario
Level Keywords
#Supplier_Add Sup Name Description Type External Flag
Supplier_Add SC-SUP00 SC-SUP00 Supplier Company OFF
Supplier_Add SC-SUP01 SC-SUP01 Supplier Company ON
# Loc_Add Loc Name Description Corp Corporate Y/N
Loc_Add SC-SXX-1 Test LO Stock Room only ON OFF
# SI Name Item Description Name External Flag
Stock_Item_Add SC-STCK00 CABLE COAX Test SI.
Stock_Item_Add 99 PUBLICATION BOOK Wind in the ...
# SI Name Description Type Quantity
Stock_Loc_Add SC-STCK00 SC-SXX-1 1-01 10
Stock_Loc_Add 99 SC-SXX-1 1-01 10
© Simply Testing Ltd 2007 Slide 26
Business Scenario WrappersFor Each Business
Scenario
There is only one
automation
program/function
© Simply Testing Ltd 2007 Slide 27
Business Scenario
Level Keywords
• Business tasks to be automated are defined
• Wrappers (functions) are programmed manually
• Each business task has test code ‘Wrapper’
– Wrapper is passed test data
– Wrapper performs all the actions
• Tests specified at business process level using keywords
• Test data specifies the keyword to perform
© Simply Testing Ltd 2007 Slide 28
Window Level Keywords
• Window name is the Keyword
• Totally Data Driven – all actions in data
• The excel data is the script
Window User Interface
Object
Action
Data
© Simply Testing Ltd 2007 Slide 29
Window Level WrappersFor Each
Window/screen/page
There is only one
automation program
© Simply Testing Ltd 2007 Slide 30
Window Wrappers
• Windows/screens/pages are ‘wrapped’ by
test programs
• Wrapper may have identification of UI
objects
• Wrapper is passed test data and actions
© Simply Testing Ltd 2007 Slide 31
Wrappers Approach
Number of Tests
AutomatedAll Windows/Business
tasks automated
Number of Automation
Programs
(Lines of Code -LOC)
{Automation
Framework/Architecture
ndowsNumberOfWiLOC
OR
zScenariosNumberOfBiLOC
© Simply Testing Ltd 2007 Slide 32
5. UI Object Map Approach
• At the Window level, scripting is unnecessary
• When automation is running it needs:
– The UI Object it must act on
– The action to perform
– The data required
• This is all contained in the data
• In Object Map based frameworks, wrappers
are built for each UI Object Type – Control
Wrappers
© Simply Testing Ltd 2007 Slide 33
UI Object Map Approach• No system specific code is built
• Only a map
• Framework has Control Wrappers for each class of UI Object
• Only need to maintain the map - not code
© Simply Testing Ltd 2007 Slide 34
UI Object Map ApproachFor Each
Window/screen/page
There is only a map in the
Object repository
© Simply Testing Ltd 2007 Slide 35
UI Object Map Approach
• Data specified at UIObject level
• The excel data is the script
WindowUser Interface
Object
Action
Data
Allowed Actions are
dependent upon UI
Object Type
© Simply Testing Ltd 2007 Slide 36
UI Object Map Approach• UI Object Map built
• Control Wrappers are built for each UI Object type (if they do not already exist)
• Can use same control wrappers in all projects with that type of control
• The Control Wrappers implement the actions
MyAccount is a link UI Object.
This line of data is passed to
the Link Control Wrapper which
performs the CLICK action
firstName is a textbox object.
This line of data is passed to
the TextBox Control Wrapper
which performs the SET action
© Simply Testing Ltd 2007 Slide 37
UI Object Map Approach
No User Interface Specific
Automation Code.
Only UI Object Maps
Number of Tests
Automated
Number of Automation
Programs
(Lines of Code -LOC)
{Automation
Framework/Architecture
• Can create more tests without increasing code maintenance task
© Simply Testing Ltd 2007 Slide 38
Write
Automation
Scripts
Create
Test
Data
Maintain
Automation
Scripts
Maintain
Data
Test Automation
Project Tasks
© Simply Testing Ltd 2007 Slide 39
Object
Mapping
reduces
build
Create
Test
Data
Test Automation
Project Tasks – reducing
costs
Maintain
Data
Automated
Impact
Analysis
Object
Mapping
reduces
maintenance
© Simply Testing Ltd 2007 Slide 40
Automated Mapping• Can generate the UI Object maps automatically from SUT:
– From source code
– From .exe
– .Net Reflection
• Only for Non-Dynamic User Interfaces
© Simply Testing Ltd 2007 Slide 41
Test Re-Use • Can re-use tests and substitute values
• Higher level tests are at Business Language Level – abstracted from system UI
RegisterQUIT RTO – Re-usable Test Object
These tests specify that
the RegisterQUIT test is
run substituting in the
DOBYear values
High Level Tests
© Simply Testing Ltd 2007 Slide 42
4. Summary
© Simply Testing Ltd 2007 Slide 43
The Main Limitation of
Record/Playback
Number of Tests
Automated
Number of Automation
Programs
(Lines of Automation
Code)
stsNumberOfTedetomationCoLinesOfAmo
© Simply Testing Ltd 2007 Slide 44
UI Object Map Approach
No User Interface Specific
Automation Code.
Only UI Object Maps
Number of Tests
Automated
Number of Automation
Programs
(Lines of Code -LOC)
{Automation
Framework/Architecture
© Simply Testing Ltd 2007 Slide 45
Summary
UI Object Map Automation Approach:
• Removes need for scripting
• Reduces automation development time
• Vastly reduced automation maintenance
• Reduces risk: tried and tested automation code
• Independence from tool vendors: For all major automation tools: - WinRunner, QTP, QARun, TestPartner, Robot, Functional Tester
© Simply Testing Ltd 2007 Slide 46
Programming
Object Mapping
2 Basic Data Driven
3 Wrapper Based
4 UI Object Mapping
5 Automated Mapping
1 Record/Playback
Automation Maturity Model
Keyword
Driven
Business Task Level or
Window Level
Wrappers
© Simply Testing Ltd 2007 Slide 47
References1. Fewster, Mark and Graham, Dorothy : Software Test Automation, 1999 Addison-Wesley
2. Kent, John: Ghost in the Machine Part 5, Professional Tester, Oct 2004
3. Nagle, Carl J: Test Automation Frameworks (Web).
4. Zambelich, Keith: Totally Data-Driven Automated Testing, http://www.sqa-test.com/
5. Kent, John: Overcoming the Problems of Automated GUI Testing, presentations to STAR
1994 and British Computer Society SIGST Dec 1993.
6. Kent, John: An Automated Testing Architecture, presentation to the British Computer
Society SIGST July 1997.
7. Kent, John: Generation of Automated Test Programs Using System Models, Quality
Week 1999, EuroSTAR 1999.
8. Hayes G. Linda: The Automated Testing Handbook
9. Arnold Thomas, R.II: Building an Automation Framework with Rational Visual Test, ST
Labs Report 1997
10. Buwalda, Hans Testing with Action Words, From Automating Software Testing (Chapter
22), Addison Wesley Longman 1999
11. Ottensooser, Avner: Data Independent Test Scripts, paper presented at STAR 1997.
12. Michael Kelly: Choosing a Test Automation Framework - (Web)
13. Kaner, Cem: Improving the Maintainability of Automated Test Suites, Quality Week 1997
14. Dwyer, Graham and Graham Freeburn: Business Object Scenarios: a fifth-generation
approach to automated testing, From Automating Software Testing (Chapter 22), Addison
Wesley Longman 1999