Test Automation Strategies For Agile

38
Test Automation Strategies Sunil Deshmukh & Mukesh M Agile India 2010 www.zentestlabs.com

description

What are the Key drivers for automation? What are the Challenges in Agile automation and How to deal with them? How to automate? Who will automate? Which tool to select? Commercial or open source? What to automate? Which features? Here is what our experience says

Transcript of Test Automation Strategies For Agile

Page 1: Test Automation Strategies For Agile

Test Automation Strategies Sunil Deshmukh & Mukesh M

Agile India 2010www.zentestlabs.com

Page 2: Test Automation Strategies For Agile

www.zentestlabs.com 2Agile India 2010

Test Automation Strategies

- Important Automation Strategies- Part 1

- Introduction to Open Source Automation tool- Sahi

- Sahi tool Walkthrough- How to use it.

- Important Automation Strategies- Part 2

Page 3: Test Automation Strategies For Agile

www.zentestlabs.com 3Agile India 2010

Test Automation process?

What is your test automation approach?

Page 4: Test Automation Strategies For Agile

www.zentestlabs.com 4Agile India 2010

Ten Questions: Six Choices and Four Trends

Six Choices

1. To automate or not to automate at all?The do or don’t choice

2. To automate now or automate later? The time choice

3. To automate through this or that? The tool choice

4. To automate this or that?The test case choice

5. Vertical Automation or Horizontal Automation?The flow choice

6. Test data hard coded or Test data kept reusable? The data design choice

Four Trends

1. Keyword or functional decomposition? The combined trend2. Whether to automate and execute early? The agile trend3. Whether to run in a sequence or concurrently? The concurrent user trend4. Whether to offer scripts to clients? The selling trend

Page 5: Test Automation Strategies For Agile

www.zentestlabs.com 5Agile India 2010

Q #1 To automate or not to automate?The do or don’t choice

The do or don’t choice

Knowing the key drivers behind test automation is important

Different objectives would require different test automation strategies

Clear understanding of “Why to automate” helps in reducing the expectation gap between management and test automation team

Page 6: Test Automation Strategies For Agile

www.zentestlabs.com 6Agile India 2010

Q #1 : To automate or not to automateCommon objectives of test automation

Saves Time

2 hour regression suite

Time to market increases

Increases accuracy

Reduced person dependency

Reduces Cost

Better ROI

Improves coverage

Localization

Reusability

Page 7: Test Automation Strategies For Agile

www.zentestlabs.com 7Agile India 2010

Q #2 : To automate now or automate later?The time choice

The time choiceKey to successful test automation is knowing when to automate.

Often overlooked and sometimes undervalued.

Even the best of automation approaches could fail to give ROI if the timing of automation is wrong.

This question is critical since test automation isPointless if it is at the fag end of a product lifecycle

Painful if the product is unstable

Possible even if the product is not ready (future trend)

Page 8: Test Automation Strategies For Agile

www.zentestlabs.com 8Agile India 2010

Q #2 : To automate now or later?Factors to be considered before starting automation

Quality of manual testing processDetailed test conditions and pre conditions

Accurate test data and expected results

Stability of module/applicationCore functionality and navigation flow is approved and accepted by end client

No bug fix should impact major functionality

No planned major enhancements in the functionality for minimum next 3 regression rounds

Page 9: Test Automation Strategies For Agile

www.zentestlabs.com 9Agile India 2010

Q #3 : To automate through this or that?The tool choice

The tool choiceFunctionality

Usability

Maintainability

Flexibility

Affordability

Compatibility

Page 10: Test Automation Strategies For Agile

www.zentestlabs.com 10Agile India 2010

Q #3 : To automate through this or thatFactors to be considered for choosing a tool

Compatibility

Is the tool compatible with the application? Does it identify all the objects in your application?

Identify different classes of object in the application. (Standard controls & customized)

Identify different possible events for each object (mouse over, mouse down, type, drag, etc)

Record and Playback for above

Functionality

Rate the application against the features list required for automation team (Evaluation Criterion)

Page 11: Test Automation Strategies For Agile

www.zentestlabs.com 11Agile India 2010

Q #3 : To automate through this or thatFactors to be considered for choosing a tool

Usability

How easy it is to learn and adapt?

Availability of trainings.

Maintainability (Support)

How good is support by the company.

Online user community?

Flexibility

Licensing policy of the company (This points is here thanks to the rigid licensing policies of some companies

Affordability

€ O $ £ : After all.

Page 12: Test Automation Strategies For Agile

www.zentestlabs.com 12Agile India 2010

Q #4 : To automate this or that The test case choice

The test case choice

After knowing when to automate, its critical to know what to automate and what not to automate.

Remember: Its not feasible to automate all the test cases.

Decide a test selection criterion for automation to improve automation effectiveness.

Page 13: Test Automation Strategies For Agile

www.zentestlabs.com 13Agile India 2010

Q #4 : To Automate this or thatTest case selection criterion

Is the test case repeatable?Does this test case require manual intervention?Has the test case passed manual verification?Are all the preconditions for the test case taken

care-of?Are the execution steps very clear?Do we have test data for this test case?Is the expected result clear enough to decide the

test case status (PASS & FAIL)?Will the test case survive the functional changes

around it?Is the test case straightforward for automation?Can I trust this script to really test this part of the

feature?

Page 14: Test Automation Strategies For Agile

www.zentestlabs.com 14Agile India 2010

Q #5 : Vertical or Horizontal Automation?The flow choice

The flow choice

Automating End to End Flow where each test cases is independent of each other such as

Create payment1

Create payment2

Approve payment1

Reject payment2

TC1&2: Login(user1) Logout->Login(user2)

Horizontal Automation

Vert

ical

Au

tom

ati

on

Create payment1

Approve payment1

Logout->Login(user2)TC1: Login(user1)

Create payment2

Reject payment1Logout->Login(user2)TC2: Login(user1)

Vertical automation is executing test cases feature wise to save execution time. e.g.

Page 15: Test Automation Strategies For Agile

www.zentestlabs.com 15Agile India 2010

Q #5 : Vertical or Horizontal Test AutomationAdvantages of each

Advantages of Horizontal Automation

Flexibility in running any test case any where as each test case is independent of other.

Easy to organize and automate. Less administration is required

Test Execution progress can be ascertained at any point since every completed test case gets immediately logged

Advantages of Vertical Automation Faster test execution as navigation is minimized, (such

as login -logout)

Effective when same functional flow is tested with different data sets

Page 16: Test Automation Strategies For Agile

www.zentestlabs.com 16Agile India 2010

Q #5 : Vertical or Horizontal Test AutomationHow to automate whether vertically or otherwise

1. Prioritize functionality Test cases.( Functionality with maximum depth usually gives higher ROI.)

2. Based on Test cases, identify different business processes (PIPE)

- Identify Pass and Fail criteria

- Identify Input criteria (Test Data)

- Identify Exceptions/ user flows.

- Identify Precondition

3. Identify the GUI objects.

4. Identify the global and local parameters.

5. Automate the Business Processes into User Defined Functions (UDFs)

6. Compile UDFs to create test case using Driver script.

7. Test the script.

Page 17: Test Automation Strategies For Agile

www.zentestlabs.com 17Agile India 2010

Q #6 : Test data hard coded or kept reusable The data design choice

The data design choice

A good automation architect ensures that scripts are reusable.

A great automation architect ensures that scripts and test data, both, are reusable.

Page 18: Test Automation Strategies For Agile

www.zentestlabs.com 18Agile India 2010

Q #6 : Test data hard coded or kept reusable Reusing test data

Generally , there is more emphasis on reusing test script than reusing test data.

Generally one test set is used per test case. For e.g. approve payment test case and reject

payment test case may use same user (i.e. user1) but the test data sheet is stored within the test case and gets repeated again and again for each test case.

On the other hand to keep the test data reusable: Store data screen wise and not test case wise

Assign reference id to each test data.

Pass reference id to each test case for accessing the test data.

Page 19: Test Automation Strategies For Agile

www.zentestlabs.com 19Agile India 2010

Q#7 : Whether to automate earlyThe early trendAgile Test Automation Principles by James Bach

Consider thinking of test automation as …Any use of tools to support testing (James Bach)

Test automation means tool support for all aspects of a test project, not just test execution.

Test automation progresses when supported by dedicated programmers (toolsmiths).

Toolsmiths are directed by testers.

Test toolsmiths gather and apply a wide variety of tools to support testing.

Test toolsmiths advocate for testability features and produce tools that exploit those features.

Test automation is organized to fulfill short term goals.

Long term test automation tasks are avoided in the absence of specific approval based on a compelling business case.

Page 20: Test Automation Strategies For Agile

www.zentestlabs.com 20Agile India 2010

Q#7 : Whether to automate earlyThe early trend

The early trend

Ideally automation should start once the application is stable but with good technical QA engineers one can start much before the application is even ready for manual testing. This can be achieved by

Using abstract automation approach for building the automation flow and important components.

Manually scripting application objects and user actions.

There is high dependence on good requirements and screen layout.

Traditional automation is mainly used by QA engineers for regression testing, but latest trends show that the automation suite can yield more returns when developers can use it for their unit testing.

Page 21: Test Automation Strategies For Agile

www.zentestlabs.com 21Agile India 2010

Q#8: Keyword driven or functional decompositionThe combined trend

Keyword driven

In the Keyword drivenapproach, each businessprocess is mapped into actions and further each operation is mapped as a keyword. It is easy for non technical users to create test scenarios without knowing much of the testing tool. Scripts are not modular and major advantages of functional decomposition are lost.

Functional decomposition

In the functional decomposition approach, business processesare created first and whilecreating the test scenarios

andtest scripts, each businessprocess is called in a

sequence.This approach is modular but for every test scenario, a test script is required. In this approach scripts are maintainable to the extent that implementation of the business process is not changed.

Page 22: Test Automation Strategies For Agile

www.zentestlabs.com 22Agile India 2010

Q#8: Keyword driven or functional decompositionThe combined trend

Keyword driven

Keyword calling

EnterText- username, mukeshEnterText- password, helloClick OK

ClickLink create paymentEnterText account_name, 1209892EnterText amount 122$EnterText date 04-Aug-2007. . . so on

Functional decomposition

Library

Login(){Enter usernameEnter passwordClick ok}------------------------------------------Test Script for approvallogin ()create_payment ()logout ()login ()approve_payment()

Page 23: Test Automation Strategies For Agile

www.zentestlabs.com 23Agile India 2010

Q#8: Keyword driven or functional decompositionThe combined trend

The combined trendMoving beyond FD and KD, the trend shall be to combine bothKeywords as a set of business processes, which are packaged as user defined functions. These keywords/user defined functions can be called in a sequence in excel or a database to test the business rules.

This gives users the modularity of functional decomposition and the usability of keyword driven

ZenTEST Labs’ ZenFRAME is one such framework that combines both

Page 24: Test Automation Strategies For Agile

www.zentestlabs.com 24Agile India 2010

Page 25: Test Automation Strategies For Agile

www.zentestlabs.com 25Agile India 2010

Q #8 : Keyword driven or functional decomposition The basic theme to build an extensible framework

“Keep everything, that changes or has chances of changing, separate from the script.”

Object properties change. So keep that separate. This is a default feature in most of the tools.

Test data changes. So keep test data separate from the script

Sequence/ flow of application changes. So separate that into an excel sheet or a database

(Most automation engineers follow the first 2 points as more or less they are default feature in the automation tools, now lets take advantage of the third approach)

Page 26: Test Automation Strategies For Agile

www.zentestlabs.com 26Agile India 2010

Q #9: Whether to run in a sequence or concurrently?The concurrent users trend for functional automation

What if, you have a time frame and you have to finish test execution within that time frame. Possible Option 1: Reduce the number of testing cases

by sampling and execute only the selective ones

Possible Option 2: Execute 100 test cases on one machine, another 100 on second machine and so on.

How about this idea of automating functional test cases using load testing tool ? This will help in executing multiple test cases at the

same time.

You can even execute thousand different test cases on an high end server and finish complete test suite execution in just few hours.

Same set of functional test cases can be further used when doing actual performance testing.

Page 27: Test Automation Strategies For Agile

www.zentestlabs.com 27Agile India 2010

Q #10 : Whether to offer scripts to your end clientThe selling trend

The selling trendRecent Trends indicate that soon test scripts

will also be shipped along with the application.Every patch release will have the modified test

scripts with it.Clients have started demanding automation

scripts for doing their UAT and testing application during bug fixes.

It has also become a new source of income for product companies, wherein additional cost can be billed for automation scripts which are anyways available with QA team.

This also brings in additional budget for future automation effort and automation gets buy-in from senior management.

Page 28: Test Automation Strategies For Agile

www.zentestlabs.com 28Agile India 2010

Ten Questions: Six Choices and Four Trends

Six Choices

1. To automate or not to automate at all?The do or don’t choice

2. To automate now or automate later? The time choice

3. To automate through this or that? The tool choice

4. To automate this or that?The test case choice

5. Vertical Automation or Horizontal Automation?The flow choice

6. Test data hard coded or Test data kept reusable? The data design choice

Four Trends

1. Keyword or functional decomposition? The combined trend2. Whether to automate and execute early? The agile trend3. Whether to run in a sequence or concurrently? The concurrent user trend4. Whether to offer scripts to clients? The selling trend

Page 29: Test Automation Strategies For Agile

www.zentestlabs.com 29Agile India 2010

Sahi – Web automation tool

-Open source web automation tool

-Created by Narayan Venkatraman

-Useful for automating functional testing and configuration testing.

-Official sahi website www.sahi.co.in

-Commercial support available.

Page 30: Test Automation Strategies For Agile

www.zentestlabs.com 30Agile India 2010

Sahi – Benefits

- Install once and use anywhere: Once Sahi server is running, tester can access Sahi Controller (aka IDE) from any supported browsers.

- The browser accessible controller is an all in one utility including a script recorder, object inspector, log browser, debugger, utility to test code without recording etc.

- Easy to learn, simple, powerful APIs.

- Javascript as the scripting language

- Built-in color coded HTML logs, accessible over browser

- Extensible Javascript APIs (Does not need a rebuild)

- Extensible on the server with Java

- Parallel execution of tests (Does not require additional tools).

Page 31: Test Automation Strategies For Agile

www.zentestlabs.com 31Agile India 2010

Sahi – Architecture

-Sahi server – runs as a proxy

-Recording and Playback both is managed by sahi server.

-Uses Rhino

-DB connectivity managed by the proxy server

Page 32: Test Automation Strategies For Agile

www.zentestlabs.com 32Agile India 2010

Sahi – Demo

-Configure the proxy.

-Open IDE (Alt + Ctrl+ Double click)

-Record

-Playback

-Functional decomposition (Fill-Click)

-Test data management strategy.

-Business process using keywords

Page 33: Test Automation Strategies For Agile

www.zentestlabs.com 33Agile India 2010

Uses in Agile

-Good integration with build tools via ant tasks and command line

-Can be used for continuous integration.

-Can create highly reusable scripts

-Easy to modify scripts.

Page 34: Test Automation Strategies For Agile

www.zentestlabs.com 34Agile India 2010

About ZenTEST Labs…

An independent testing company focusing on functionality testing and quality consulting.

World’s number 1 automation provider

World’s largest eLearning solutions provider

Europe’s largest bank

World’s number 2 computer manufacturer

One of the world’s leading cash management solutions provider

One of the world’s leading customer interaction software for payments

Europe’s leading Infrastructure Management software provider.

One of Middle East’s largest banking software providers

To enable client experience ‘Zen’ through our Testing and Quality services

Purpose Focus

Sample List of clients Offshore Testing Centre

Located at Pune, India

55 People Company

Domain ExpertiseFinance and banking

eLearning and education

Document and Project Mgmt

Technology ExpertiseCertified Mercury Product consultants

Proprietary Test Automation Framework, viz. ZenFRAME

Specialized testing services

Highlights

Page 35: Test Automation Strategies For Agile

www.zentestlabs.com 35Agile India 2010

About ZenTEST Labs.

Outsourced Testing

Functionality Testing

Unit Testing

Test Automation Projects

Functional Automation

Performance Testing

Test Maintenance Projects

Maintain Regression Suites

Specialized Testing

eLearning specific testing

Security Testing

Compliance Testing

Localization Testing

Usability Testing

Agile Testing

Mercury WinRunner/ QTP

Mercury Load Runner

Mercury Quality Center

Rational Testing Tools

Test Project Management

Test Estimation

Advanced Test Automation

How to write test cases

The mind of a software tester

Test Process Assessment

Automation Consulting

Testing Project Management

Testing Consulting Testing Projects

Testing Training

Page 36: Test Automation Strategies For Agile

www.zentestlabs.com 36Agile India 2010

Further Help

As a practice from our past STAR conferences talk, we shall be pleased to conduct this presentation at no cost over the phone for your team.

Please email at [email protected] if you are interested in the same.

To view other ZenTEST whitepapers and presentations, please visit www.zentestlabs.com download page.

Page 37: Test Automation Strategies For Agile

www.zentestlabs.com 37Agile India 2010

Bibliography

When should a test be automated?Article by Brian Marick, 1998.

When to automate testingDavid Weiss blog

Agile Test AutomationWhite paper by James Bach

Page 38: Test Automation Strategies For Agile

website: www.zentestlabs.com email: [email protected]

blog: http://www.zentest.typepad.com

Thank You Sunil D

Mukesh M