Continuous testing

Continuous Testing

Dr Ganesh Neelakanta Iyer

QA Architect

GANESHNIYER

[email protected]

http://ganeshniyer.com

© 2016 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved.2

About Me

B.Tech. in CSE from Mahatma Gandhi University, Kerala in 2004 with University First Rank

Seven years of Industry work experience

Masters and Ph.D. from National University of Singapore

Several talks/workshops in India, USA, Europe

Research interests: Cloud computing, Game theory,

Software Quality Analysis, IoT solutions

Personal Interests: Kathakali, Teaching, Traveling,

Photography, Cooking

Website: http://ganeshniyer.com

Currently

• QA Architect at Progress Software, India

• Visiting Faculty at IIIT-H

• Adjunct Professor at BVRIT

GANESHNIYER


Quick Poll

Are you using Version Control?

Safe?

Can you release new version of your software in one day?

Quick

Can you release new, well-tested version of your software

in one day?

Quick & Safe?


Agenda

Software Development today

Continuous Integration and Continuous Testing

24x7 Testing

What are top changes in Software development that were disruptive for

existing test methodologies?


Agile

Credit: Blazemeter; Continuous Testing in Practice


Test Challenges with Agile Methodology

Frequently changing requirements

•Thanks to Agile Changes can happen at any point of time during release cycle

• Requirements and Test Cases developed in parallel or test first strategy

Frequent releases

•Progress DataDirect Cloud released new features every 9 days in 2015*

• Testing needs to happen Early and Often, Everyone needs to Test

Understanding and testing real world behavior

• How customers see the product behavior after a release?

• No SRS-level waterfall documents to drive testing plan

Multiple environments & differences for test and production systems

• 3rd party systems integrated to the product

• Raises a requirement for GoLive Testing But in minimal time


Software Test Challenges With Cloud

Web-driven SaaS applications

Everything in a few mouse clicks

Frequent releases / Frequent changes

No SRS documents

Publically shared resources

Multi-tenant penetration testing

Paradigm Shift

High Availability and Resiliency are key

Seamless upgrades

Everything is Agile

Availability Everything is shared


Continuous Integration & Automation are Key in Agile

Automation

• Nightly / Weekly

Build Integration

• Combination of unit tests and regression tests

Dev-environment

• Unit-tested


Continuous Integration and Continuous Testing


Dev/Test Expectations continue to increaseBetter – Cheaper - Faster

Past

• Balance between speed, cost and risk

Today

• Demand for releasing high quality software faster

Credit: IBM article on Continuous Testing


Applications are becoming more complex to test

Setting up complex environments

Delays caused by lack of system

availability

Third party services can be costly

Testers spend more time setting up

tests than testing

Testing teams spend on setting up test environments, instead of testing

30-50% average

time

Credit: IBM article on Continuous Testing


Continuous Integration and continuous testing

Accelerate software delivery – for faster

time to value

Balance speed, cost, quality and risk

– for increased capacity to innovate

Reduce time to customer feedback

– for improved customer

experience

PC Credit: https://https://goo.gl/A2AaQw

https://https/goo.gl/A2AaQw


Shift left, test early, test often


The Testing Process – From Development to Operations



Continuous Testing in Practice – Requirements Checklist

Tools, Scripting and Overall Test Creation

• The provision of libraries and APIs to create “homegrown” tests

• The ability to create browser-based tests

• Ability for load and scalability testing

Provisioning and Resources

• The option to run the same sets of tests with different provisioning

• The ability to easily adopt a new provisioning scheme when a test configuration is executed in a different environment (e.g. Dev, CI, Pre-Prod, Post-Prod)

• The ability to start testing as part of the CI process

Automation• The ability to run any test configuration, a test, or a set of tests

using simple API calls

• The option to run as many tests as required in parallel

On-Demand Testing and Re-Running

Failed Use Cases

• The ability to run any type of test on-demand. This is critical for test development, debugging, troubleshooting and identifying the root cause of a failure

• The option to automate test executions and run the same test on-demand



Continuous Testing in Practice – Requirements Checklist

Modules, Builds, Release Candidates,

Releases and Production

• The ability to combine different configuration fragments into one test configuration in order to comprehensively test build, release, and production snapshots.

Version Control Friendly Incremental Testing

• The ability to support version control, incremental testing and associate test configurations, sets of tests, and tests with versions (e.g code, build, RC, releases).

Automatic ‘Failure’ Alerts

• Automatic indications of failures, with an alerting scheme per developer, module and project

• The ability to gather all test artifacts and immediately send to relevant people

• The option to run the failed tests again to identify the root cause

Reporting

• A seamless integration with existing reporting solutions (e.g. Jenkins Performance Trend)

• The option to group tests by builds

• The ability to give jobs a ‘pass’ or ‘fail’ status and group accordingly

• Pass/fail trend reports



24x7 Testing; A Case study of what we do at Progress


24 x 7 Testing

Understanding and testing real world behavior

What is 24x7?

• Running a set of test cases continuously 24 x 7

• Simulates real time customer behavior

How 24x7?

• A few parallel threads accessing the product

• Tests consisting of regression (or it’s subset) tests

Why 24 x 7?

• Many things can’t be caught by regular functional testing

• Error handling and retries

• Product can be made more resilient and robust


24 x 7 Testing

VMs / Build Integration Machines

ANT ScriptTests Reports using

Kendo UI controls

Decides

parallelism

HTTP POST reqBatch

Script

Decides

frequency

Logs

SUT

• Progress products such as Rollbase

• Uses any test environments

• Existing test cases can be leveraged

Batch Script

• Decides the frequency

• E.g. run on an hourly basis

• Run in an infinite loop 24x7

ANT Script

• We configure #Threads, #Tests to run in parallel

• Tests can run on VMs / Build integration machines

Reports

• Shown using Keno UI controls / widgets

• Driver scripts sends results to the portal

• Machine stores the logs


Case Study - Example

Cloud platform that let you build SaaS-

based business apps in days/hours

Responsive UI with built-in themes

Typical apps customers build

• CRM

• Employee Management

• Bug Tracking System

What could be some typical tests for this platform?


Case study - Identifying tests; Example

100 parallel treads

• 50 of them access “Bug Tracking System” application

• 50 of them access “CRM Application”

Use cases

1. Create a bug and assign it to a user which will run an email trigger and related record trigger

– 10 users

2. Update the workflow status of some existing bug which will run a trigger – 20 users

3. Filter operation on bug object – 20 users

4. Create a lead from CRM portal and attach the lead to an account – 20 users

5. Look for all leads whose workflow status is “Open - Not Contacted” – 10 users

6. Create a new opportunity, assign it to an owner and attach it to an account – 10 users

7. Change the workflow status of an existing opportunity to “Closed-won” – 10 users

Tests should not be self-cleaning

Why?


Layout of the infrastructure

BATCH script

• Checkout latest test scripts

• Compile and run them

ANT script

• Properties file containing the details of test agents

• Invoke scripts in the agent machine

Central Machine

Test Agent 1

Test Agent 2

…Test

Agent n


Central Server


Central Server – Build.properties


Central Server – <Product name>.properties

testtype=Telerik

testlist="Psexec.exe \\\\172.29.36.205 -u Administrator -p Progress!2009 -i 2 -w c:\\Telerik

c:\\Telerik\\telerik.bat";"Psexec.exe \\\\172.29.36.207 -u Administrator -p Progress!2009 -i 2

-w c:\\Telerik c:\\Telerik\\telerik.bat";"Psexec.exe \\\\172.29.36.208 -u Administrator -p

Progress!2009 -i 2 -w c:\\Telerik c:\\Telerik\\telerik.bat";"Psexec.exe \\\\172.29.36.209 -u

Administrator -p Progress!2009 -i 2 -w c:\\Telerik c:\\Telerik\\telerik.bat";"Psexec.exe

\\\\172.29.36.215 -u Administrator -p Progress!2009 -i 2 -w c:\\Telerik

c:\\Telerik\\telerik.bat";"Psexec.exe \\\\172.29.36.90 -u Administrator -p Progress!2009 -i 2 -

w c:\\Telerik c:\\Telerik\\telerik.bat";"Psexec.exe \\\\172.29.36.227 -u Administrator -p

Progress!2009 -i 2 -w c:\\Telerik c:\\Telerik\\telerik.bat";"Psexec.exe \\\\172.29.36.231 -u

Administrator -p Progress!2009 -i 2 -w c:\\Telerik c:\\Telerik\\telerik.bat";"Psexec.exe

\\\\172.29.36.229 -u Administrator -p Progress!2009 -i 2 -w c:\\Telerik

c:\\Telerik\\telerik.bat";"Psexec.exe \\\\172.29.36.233 -u Administrator -p Progress!2009 -i 2

-w c:\\Telerik c:\\Telerik\\telerik.bat"


24 x 7 Testing – Sample report


Key Advantages of 24x7 testing

Catch issues which are difficult to catch using normal functional

testing

Make products resilient and robust

Catch unexpected integration issues

Test real time customer behavior


Catch issues which are difficult to catch using normal functional testing

https://cdn.tutsplus.com/wp/uploads/2013/12/session-diagram.jpg

http://biobeasts.artix.com/content/images/2015/07/memory-leak.jpg

https://cdn.tutsplus.com/wp/uploads/2013/12/session-diagram.jpg

http://biobeasts.artix.com/content/images/2015/07/memory-leak.jpg


Make products resilient and robust


Catch unexpected integration issues


Test real time customer behavior

http://blog.perfectomobile.com/wp-content/uploads/2015/10/WINDTUNNELGRAPHIC.jpg

http://blog.perfectomobile.com/wp-content/uploads/2015/10/WINDTUNNELGRAPHIC.jpg


Advantages of continuous Testing

Helping to make software development faster

• Continuous testing can make it possible to making software available sooner without compromising quality

Reduced Risk

• Every time a change is made to code or new code is written, there is the potential for error to occur. Identifying the problem is much easier when continuous testing is performed and the issue is caught immediately so problems can be corrected.

Making consistency simpler

• Automation of testing makes it easier to achieve consistency by providing a carefully configured environment and maintaining the same correct configuration for all relevant tests

Faster releases

• The process of regularly using continuous testing means that cleaner code can be released to end users in a timely manner

https://www.etesting.com/2014/02/25/benefits-of-continuous-testing/

https://www.etesting.com/2014/02/25/benefits-of-continuous-testing/

Are you ready to implement Continuous Testing in your

organization?

[email protected]

http://ganeshniyer.com

GANESHNIYER

Continuous testing

Technology

Transcript of Continuous testing