PREDICT Model for Test Automation. Does it sound familiar to you? Organization has procured test...

PREDICT Model for Test Automation

Does it sound familiar to you?Does it sound familiar to you?

• Organization has procured test automation tools

• Management expectations are high

• Multiple test automation project initiatives started

• Enthusiasm, commitment and motivation declined with

time

• Script maintenance had become a significant challenge

• Test automation tools are back on shelf

• Testing and test automation requires same skills

• Capture/Replay is a successful test automation approach

• Test automation is an integral part of the over all test process

• Test automation is effective in solving regression testing issues

• Every test and test type can be automated

• Test automation development is software development

• Test automation projects can be delivered by personnel with strong

project management skills

Spot QuizSpot Quiz

Essential building blocks for successful test automation solutions

Test automation solution approachTest automation solution approach

PREDICT ModelPREDICT Model

RationalizeRationalize

TransitionTransition EstimateEstimate

DesignDesign

ImplementImplement

ConsolidateConsolidate

PlanPlan

PREDICT ModelPREDICT Model

• Structured methodology• Multi stage process consisting of seven components

• Plan• Rationalize• Estimate• Design• Implement• Consolidate• Transition

• Step-wise approach• Easy to apply

PREDICT Model - PlanPREDICT Model - Plan


TransitionTransition

PlanPlan

EstimateEstimate

DesignDesign

ImplementImplement


Need for test automation projectNeed for test automation project

• What do you consider to start a project ?• Test related checks

• Objective• Scope• Documentation• Test ware

• Organization related checks• Investments• Expectations• Resource availability

• Technology related checks• Stability of SUT• Test tool suitability

Need for test automation projectNeed for test automation project

• Are you getting bang for your buck ?• Costs

• Licensing• Framework• Implementation• Maintenance factor

• Benefits• Saved manual effort per test run• Number of test runs• Re-usability factor

• Amortizing benefits• Life cycle of the application/product• Opportunities for test automation

Plan of approachPlan of approach

• Introduction• Objective• Pre-requisites• Set-up pilot project• Activities• Planning• Deliverables• Organization• Resources

PREDICT Model - RationalizePREDICT Model - Rationalize



PlanPlan

EstimateEstimate

DesignDesign

ImplementImplement


How do we rationalize requirements ?How do we rationalize requirements ?

• 5 P factors

– Project

– Process

– Product

– People

– Price

• Test suite attributes

• Building trade-offs based on project context

ProjectProject

• Objective and scope• Clear objectives in terms of time, money and quality

• Strategy• Test automation strategy fitment with manual testing strategy

• Build• Build frequency

• Configuration management• Bad experiences

• User interface• Stability

• Cycle time• Regression possible

ProcessProcess

• Quality documentation• Manual test scripts, documentation

• Development approach• Phases clearly defined, how early can test automation

start

• Test practice• New builds tested before delivering to QA• Regression moments

• Naming conventions• User Interface conventions

ProductProduct

• Stability• Base platform stability

• Testability• Degree of interaction between the tool and product

under test

• Test tool integration• Integration between test management tool and

configuration management

• Test oracles• Interaction with non UI components,database

• Automation framework• Availability of frame work to reuse

PeoplePeople

• Organization readiness• Management expectations

• Resource availability• Machine, Licenses and people with right skills

• Non- programming resources• Framework inclined towards non-programming resources

• QA population• Ratio of development to QA

• Collaboration• Collaboration between QA and Dev

• Technical skill set• Test engineers with technical skills

PricePrice

• Return on Investment• Opportunities of reusing test automation investment

• Reusability of Investment• Are there any products which are tested in a similar

way

• Business Plan• Is there a business plan to realize benefits of Test

Automation

Test suite attributesTest suite attributes

• Maintainability• Reusability• Reliability• Usability• Portability

Which is best test automation regime ?Which is best test automation regime ?

IT DEPENDS ON THE PROJECT CONTEXT

PREDICT Model - EstimatePREDICT Model - Estimate



PlanPlan

EstimateEstimate

DesignDesign

ImplementImplement


Test automation estimation challengesTest automation estimation challenges

• Test organization maturity• Scope of requirements• Framework• Skill level• Test case complexity• Domain complexity• Test tool• Testability

Estimation techniquesEstimation techniques

• Delphi• Gather estimates from each team member• Ask high and low estimator to explain estimates• Repeat twice, then use average

• Three point method• Have team estimate best, worst case and expected case• Use expected case• Variances between best and worst useful to gauge

estimate accuracy

Work Break down StructureWork Break down Structure

Risk categoriesRisk categories

• Technical• Test tool• Application/product• Communication

Risk managementRisk management

PREDICT Model - DesignPREDICT Model - Design



PlanPlan

EstimateEstimate

DesignDesign

ImplementImplement


Test suite design issuesTest suite design issues

• Testability• Test suite design patterns

Testability considerationsTestability considerations

• Tools• Design• Naming Conventions• Exception & Error Handling• Timing issues• Logging• Visual Cue• Processes & Documentation

• Record/Play back

• Functional de-composition

• Key-word Driven

• User Interface-driven

• Model-based

Test automation development approachesTest automation development approaches

• Advantages

• Useful in determining how the tool interacts with the application under test

• Provides initial ideas on how to develop test scripts

• Useful while ‘Playing around’ with the tool

• Disadvantages• Test scripts contains hard-coded values

• Un-reliable scripts : Any small change in the application results in not being able to replay

• Maintenance costs are astronomical

• Not viable & cost-effective

Record & PlaybackRecord & Playback

• Reduces test scripts into their fundamental tasks• Driver Scripts : Initialization, Calling scripts in a desired

order • Generic scripts : General,Application

specific,Navigation,Data verification,Reporting,logging etc…

• Business Function Scripts : ‘Re-usable’ Business functionality,Makes calls to Generic Scripts

• Test Script : Test Logic, Makes calls to Business Function Scripts and Generic Scripts

• Test data and logic are separated• Hierarchical structure is employed :

Modular design• Each test script unique data file

Functional decompositionFunctional decomposition

Test Director

Reports

Test Scripts

Generic Library BusinessLibrary

Test Scripts

EnvironmentConfiguration

Stored Proc ObjectRepository

Technical ArchitectureTechnical Architecture

Functional decomposition: AnalysisFunctional decomposition: Analysis

• Advantages• Modular design : Reduces redundancy and duplication of

efforts • Test data input & output stored in a single data file : Easy

maintenance• Scripts can be developed whilst application development is

in progress : Generic Scripts• Single point maintenance is possible using Business

Function scripts• Scripts can return TRUE (or) FALSE to the calling scripts.

This makes it easy for ‘Error Recovery’ & un-attended test execution

• Assemble tests on demand

• Disadvantages• Proficiency in the TSL• Complex test data file management

• The test script is represented as a set of key-words in a spread sheet• Examples of Key-words are start_test, Initial

window,Url,Input,Verify etc..

• Driver Script Performs the initialization as required • Driver Script Reads and processes the file name(s)

of the test scripts• Driver Script Matches the ‘Key Words’ contained in

the file• Driver Script invokes Business Function & Generic scripts• Report the return values back• The entire process is data-driven

Key-word drivenKey-word driven

• Advantages• Preserves most of the advantages of the ‘Functional

decomposition’ method• Very similar to documenting manual test script• Minimal programming knowledge is required to write and

execute test scripts• Return on investment can be achieved quicker than other

methods in most of the cases• Re-usability significantly increases over a period of time• Test script development is possible with out a functioning

application

• Disadvantages• Investment in initial frame work• Training might become tedious

Key-word driven - AnalysisKey-word driven - Analysis

GenericScripts

Keyword Driven Test Cluster

Business FunctionScript4

Reports

Driver Script




GUI Map


• Abstraction of test structure: Test set Test case Test step Test action

• Key-words are identified based on the application under test

• Test steps are created using user interface• Test set management (Creation/execution)

Test set

Test case

Test step

Test action

User interface drivenUser interface driven

Automation Framework

RecordingEngine

Script Generation

Module

APPLICATION

User Interface

Test

Test Script

XML Object Repository

Object Creation Module

Execution Engine

Win Runner/QT

P/Silk Test/…

Library Structure


• Advantages• Preserves most of the advantages of the previous

methods• Insulates tests both from the changes of the application

and failings of test automation tools• Zero-programming required• Return-on-investment is very high• Re-usability significantly increases over a period of time• Robustness and maintainability is high• Partly application independent• Minimal learning curve

• Challenges• High initial investment in creating framework• Functioning application needed

User interface, Model driven - AnalysisUser interface, Model driven - Analysis

Test Case Generator(In key words)

Keyword Translator engine(Test tool specific)

Automated Test Scripts

Test drivergenerator

Behavior modeler

Requirements (UML)

Win Runne

r

Win Runne

r

Silk Test

Silk Test

Other test tools

Other test tools

Model DrivenModel Driven

PREDICT Model - ImplementPREDICT Model - Implement



PlanPlan

EstimateEstimate

DesignDesign

ImplementImplement


Test automation implementation issuesTest automation implementation issues

• Team structure• Work allocation• Coding guide-lines and check lists• Peer reviews• Test runs• Integration• Configuration management

Work allocationWork allocation

Coding guidelines and checklistCoding guidelines and checklist

ReviewsReviews

Building integrationsBuilding integrations

Test Script1

Test Script2

Test Script3

Call Test Script1()Call Test Script2()Call Test Script3()

Configuration management : VSSConfiguration management : VSS

Test executionTest execution

• Test suite execution check-lists are used• Test Execution

• A separate test environment is prepared • Fresh test data in test DB• Fresh test data in Application DB

• Tests are executed • Test suite execution issues & root cause

analysis is documented• Report Test Results• Update test scripts if needed

Test execution logTest execution log

PREDICT Model - ConsolidatePREDICT Model - Consolidate



PlanPlan

EstimateEstimate

DesignDesign

ImplementImplement


• Update scripts and test data• Complete documentation (User Manual, installation

Guide and maintenance guide)• Freeze & archive test suite• Professionally packaged shippable test suite• Easy distribution mechanism

Activities in consolidationActivities in consolidation

User manualUser manual

Professional packagingProfessional packaging

PREDICT Model - TransitionPREDICT Model - Transition



PlanPlan

EstimateEstimate

DesignDesign

ImplementImplement


• Acceptance test runs performed• Knowledge transfer to the user groups• Gain feedback from the field• Prepare for maintenance and sustenance• Up-gradation plans• Discussion for next project

Transition processTransition process

• Successful test automation projects require mature test automation leadership skills

• Test automation projects should be treated like a development project

• Understanding PREDICT Model• Challenges, issues, tips and tricks in applying

PREDICT model

Wrap upWrap up

PREDICT Model for Test Automation. Does it sound familiar to you? Organization has procured test...

Documents

Transcript of PREDICT Model for Test Automation. Does it sound familiar to you? Organization has procured test...