Hybrid Keyword Data Driven Automation · PDF fileHybrid Keyword Data Driven Automation...
Transcript of Hybrid Keyword Data Driven Automation · PDF fileHybrid Keyword Data Driven Automation...
ANZTB 2010 – ConferenceHybrid Keyword Data Driven Automation Frameworks by Jonathon Wright
Tuesday, 2nd March 2010
© ANZTB 2010
Hybrid Keyword Data Driven Hybrid Keyword Data Driven
Automation FrameworksAutomation Frameworks
Test Automation as a Service (Test Automation as a Service (TaaaSTaaaS))
Real world case studyReal world case study
Jonathon Lee WrightJonathon Lee Wright
[email protected]@iee.org
www.automation.org.ukwww.automation.org.uk
www.linkedin.comwww.linkedin.com/in/automation /in/automation
11th November 201011th November 2010
• Complete “Automation Lifecycle Management ” (ALM) solution;
• Providing “Automated Testing as a Service ” in the Cloud ;
• “Pay As You Go ” flexible billing (no expensive tool licensing costs);
• Distributed continuous build integration with “Application Under Test ”;
• Shorter market deployment times through quicker Automated Testing ;
• Shorter environment deployment times through Automated Health Check ;
• Better quality testing freeing specialist test resource to concentrate on non-repetitive test tasks;
• Easy interface for non-technical testers to create their own “user stories ”;
• Capability Maturity Model Integration (CMMI) through Business Process Modelling (BPM) comprising of Business Level Keywords (BLK);
• Introduction of “Full ” Agile automation testing and “First ” day automation.
What value did this case study deliver?
What approach?
• Manual Approach;
• Traditional Automated Approach:
• Hybrid Keyword Data Driven Framework Approach.
Vendor Tool(Record & Playback,
Adapt Scripts)Limited requirements Limited output
Framework Approach
Hybridutilising the best technologies and resourcesHybrid
Keyword
Data
Driven (Modularity)
Keywordbusiness level keywords represent user stories
Datadynamic use of sanitised business data
Driven (Modularity)modularity of test procedures to provide reliable processing of the framework components
Hybrid Approach
.NET 4.0 Development (Visual Studio 2010)
Cloud (Windows Azure/SQL/AppFabric)Automation (Ranorex/Coded UI APIs))
●●●●●Hybrid
Vendor Tool (Ranorex IDE)●○○○○Traditional
ToolsModularityDataKeywordBPTBPMApproach
○ - Limited Support
● - Full Support
Single Technology Stack (SPOF)
Tool Vendor Dependency
C# Project incompatible with
Visual Studio
Technology Blocks(Custom Flex Objects/Classes)
Limited non-technical tester
interface
Vendor Support(maintenance subscription)
Object Repository
(object-based Record / Replay)
ConstraintsAdvantages
Traditional (Vendor Tool)
In-house framework development & maintenance effort.
Additional skills / knowledge
resources
Azure Platform Subscription
Object Repository (limited Record / Replay support)
Multiple Technology Stack(VS2010 / C# / VB.NET / ASP.NET / TFS
Windows Azure / (WCF) Web Services / DLL/ XML/ Azure SQL/AppFabric
Coded UI / Ranorex / AIR / Flex / XPATH )
Integration with development (Visual Studio 2010)
Integration with deployment(TeamCity v5.1)
Integration with testing(Quality Center v11)
ConstraintsAdvantages
Hybrid (.NET Development)
Return on Investment
1 2 3 4 5 6 7
Number of Excecutions
Inve
stm
ent i
n E
ffort
MarketConfigurationTesting + HealthCheck
MarketConfigurationTesting
Traditional
Manual
MCT = 3 MCT+ HC = 3.4 Traditional = 3.7
Initial Investment
LeverageHybrid (Framework)
Vendor Tool
Rn = Aa / Am = (Va / n + n*Da) / (Vm / n + n*Dm)
ANZTB 2010 – ConferenceHybrid Keyword Data Driven Automation Frameworks by Jonathon Wright
Tuesday, 2nd March 2010
© ANZTB 2010
Automation Goals
Reusablemodularity of
test procedures & functions
Manageableeffective test design,
execution and traceability
Maintainableto reduce the amount
of test maintenance effort
Robustproviding
object/event/errorhanding & recovery
Flexibleindependent of
test environment and be scalable
Measurablecustomisable reporting
& test execution progress
Business Process Testing (Process)
Business Process Modelling (Example)
Index- Complete- In Complete- Supported - Out of Scope
Business Level Keywords (Testing)
Translates
Automation Lifecycle Management (Process) Technology Stack (TaaaS)
ANZTB 2010 – ConferenceHybrid Keyword Data Driven Automation Frameworks by Jonathon Wright
Tuesday, 2nd March 2010
© ANZTB 2010
Overview (TaaaS) Benefits (TaaaS)
Architecture (TaaaS) High Level (TaaaS)
Automation Lifecycle Management (ALM) Requirements - Business Process Data
AUT Database
Quality Center
Test Automation Database
Mock Live Data (Sanitised)
Team Foundation Server
ANZTB 2010 – ConferenceHybrid Keyword Data Driven Automation Frameworks by Jonathon Wright
Tuesday, 2nd March 2010
© ANZTB 2010
Test Planning - MasterDriver (Excel)
Object Repository
Test Repository
Test Planning - XML Schema
Test Execution - Testing in the Cloud
WebServices (WCF)
<< Realtim
e Test Results >>
<< Realtim
e Error Reporting >>
<< Check Test Schedule >>
Test Execution - Automation Scheduler
Selectable list of user case stories (Test Sets)
Available machines
Global Stop Buttons (Stops all tests on all running machines)
Global Pause button
Global Play button
Web Services (WCF)
XML
Communication - Message Definition (WCF)
ANZTB 2010 – ConferenceHybrid Keyword Data Driven Automation Frameworks by Jonathon Wright
Tuesday, 2nd March 2010
© ANZTB 2010
Windows Service (Lifecycle) Event / Object Handling
Supported events:
� button.click
� text.parmsetcheck
� text.validate
� link.click
� reporttype.selecttreeitem
� reporttype.opentreenode
� list.parmsetcheck
� checkbox.select
� tree.validate
� tree.parmsetcheck
� radio.parmsetcheck
� checkbox.parmsetcheck
� checkbox.validate
Business Level Keywords (Process)
Business Level Keywords represent a number of logical test steps that make up a business process i.e. “Login.Process”
User Stories are represented by a number of Business Level Keywords:
Each User Story represents client Market Configuration Testing mappings:
LoginFrm.Logo.Image
LoginFrm.UserName.Label
LoginFrm.Password.Label
LoginFrm.LoginBoxHeader.Label
LoginFrm.LoginButton.Label
LoginFrm.ChangePWButton.Label
LoginFrm.ForgotPWLink.Label
LoginFrm.ServiceDeskTel.Label
LoginFrm.ServiceDeskMail.Label
LoginFrm.Copyright.Label
ABCD
E
FGHIJ
F
G
H I J
A
B
C
D
E
Market Configuration Testing (Process)
Process the first test step configuration data value located in the “Test Repository” i.e. #dh.MyServices_Login_UserNameLabel
Using the data value to confirm that the “Market Requirement” exists for the target market (<Client> + <WorkStream>)
Once confirmed the configuration value can be extracted from the “Market Specification” by selecting the target market (<Language>)
Test Step 1 – Using the “Configuration” value
to verify that market configuration is correct:
UserNameLabel = “User Name:”Result = FAIL
Object Recognition (Smart Identification)
Get the test step object name from the Test Repository (MasterDriver) i.e. LoginFrm.Password.Input
Using the object name extract the object class & associated value from the Object Repository (MasterParameters) i.e. Html.EditBox & “password”
Using the object class extract the primary & secondary class tag identifiers from the Class Definition (MasterParameters) i.e. “InnerText” & “Type”
Attempt #1 – Using the “WebElement” Class
Describe the Object Physical Description:
WebElement Object= [@InnerText = “password”]
Attempt #2 – Using the “InputTag” Class
Describe the Object Physical Description:
InputTag Object= [@Type = “password”]Physical Object
Physical Description
Object Recognition (Fuzzy Logic)
FlexObject flexObj = @"/dom[@domain='']/frameset/frame/body/flexobject[@id='CrossShop']";FlexObject flexObj = @"/dom[@domain='']/frameset/frame/body/flexobject[@id='CrossShop']";
IList<Ranorex.Button> flexButtons = flexObj.FindDescendants<Ranorex.Button>();IList<Ranorex.Button> flexButtons = flexObj.FindDescendants<Ranorex.Button>();
Get all the screen objects on the flex page.
Retrieve all the Buttons on the page.
For ( int I = 1; >= flexbuttons.Count; i++)
{
// get the xpath for each element.
RxPath me = flexButtons.ElementAt<Ranorex.Button>(i).GetPath();
// convert the xpath to a string
string[] str = me.ToString().split(‘/’);
// convert xpath to a split of strings
string[] str1 = xPath.Split(‘/’);
// compare both string paths until a common tree route is found (this will be your button)
CompareTreePaths( str, str1 )
}
When you have identified a tree path, you can then use it in the way described in the MasterDriver file (e.g. flexButton.Click).
ANZTB 2010 – ConferenceHybrid Keyword Data Driven Automation Frameworks by Jonathon Wright
Tuesday, 2nd March 2010
© ANZTB 2010
Unified Test Reporting (UTR) Unified Test Reporting – Real-time Log
TestRunnerInput (XML) Output (WebService)
.NET Object WCF
Error/Event Handling
Logging (Debug)
Results (XML)
Process Improvement
� Agile Planning – Team Foundation Server linked to Task Board (SfTS v2);
� Coding Standards – Code reviews (internal/external to team), coding tools, unit test standards, agile hints, logging etc etc;
� Testing - Internal peer reviews, unit tests developed, exploratory testing for each story as a task, Integration test story in each sprint;
� Wiki – Knowledge capture including online framework documentation (screenshot)
� Technical Notes /Debt - on project wiki (intranet SharePoint).
Key Achievements
• Centralised - management of test assets to provide open connectivity as a generic source of test scenario data;
• Dynamic - generation of sanitised input test data validated against business rules along with an object repository based on a combination of descriptive programming containing XPATH regular expressions;
• Legacy - support for static data types plus full/partial fall back support for manual test execution;
• Unified reporting - using funnel virtualisation techniques to analyse the quality of builds, assisted by diagnostic data adapters;
• Generic reusable - test components, modules and libraries providedreliable processing of the entire test automation effort.
SSUMMARYUMMARY
•• HybridHybrid –– utilising the best technologies and resources to do the job;utilising the best technologies and resources to do the job;
•• Keyword Keyword –– creating simple and robust test scenarios written in business creating simple and robust test scenarios written in business
level keywords combined with natural language;level keywords combined with natural language;
•• Data Data –– effective use of dynamic business data to provide an input sourceffective use of dynamic business data to provide an input sourcee
•• Driven Driven –– processes the test component actions, objects and events processes the test component actions, objects and events
seamlessly;seamlessly;
•• Automation Automation –– that is collaborative, distributed and scalable;that is collaborative, distributed and scalable;
•• Frameworks Frameworks –– independent of application or environment under test;independent of application or environment under test;