Continuous Testing vs. Test Automation: Three Key Differences · W9 Test Automation Wednesday, May...

W9 Test Automation Wednesday, May 2nd, 2018 1:45 PM

Continuous Testing vs. Test Automation: Three Key Differences

Presented by:

Wayne Ariola

Tricentis

Brought to you by:

350 Corporate Way, Suite 400, Orange Park, FL 32073 888-- -268- - -8770 ·· 904- --278-- -0524 - [email protected] - http://www.stareast.techwell.com/

Wayne Ariola Tricentis Known for his efforts to promote continuous testing, Wayne Ariola has contributed to the design of many innovative technologies and received several patents for his inventions. A recognized leader on topics such as service virtualization, SOA and API quality, software quality governance, and application security, Wayne is a frequent contributor to industry publications and author of Continuous Testing for IT Leaders. He has been a contributor to software testing for fifteen years and in the software industry for more than twenty.

4/25/18

1

© 2016 by © 2018 by

Wayne Ariola

Continuous Testing v. Test Automation

© 2016 by 2

4/25/18

2

© 2016 by

Testing at the Speed of DevOps

What has Changed? 1

2

3

Agenda

The three differences with Continuous Testing

© 2016 by 4

How would you rate the success of test automation…

a success story

so so

a failure

?

4/25/18

3

© 2016 by

Rate the Success of Test Automation…

0

10

20

30

40

50

60

Q3-‐2016 Q1-‐2017 Q3-‐2017 Q1-‐2018

Success So So Failure

© 2016 by

Test automation has not kept its promises.

4/25/18

4

© 2016 by

80% Manual Testing

20% Automated Testing

UI

API

Today

The reliance on manual testing is the

top technical challenge in app development.

World Quality Report, 2017

Test automation requires developers.

© 2016 by

We didn’t do anything wrong, but somehow, we lost.

‹‹ Stephen Elop, CEO Nokia, 2013 ››

4/25/18

5

© 2016 by

What has Changed?

§ People § Process § Technology § Tools

© 2016 by

Offshore Labor

38 kUSD/yr

60%

Traditional Onshore Labor

100 kUSD/yr

Save cost through

labor arbitrage

Satisfied?

People

4/25/18

6

© 2016 by

Process Test as a time-boxed event

Fixed Fixed

Plan Develop Build Test Release Plan Develop

Build Test

Release

Build Test

Build Test

Build Test

Build Test

Build Test

Build Test

Build Test

Test is Continuous

© 2016 by

Technology and Disruption

• Security • Safety CriDcal • Financial • FDA • AutomoDve • Aerospace

Compliance

• DevOps • Agile • Lean • Bi-‐Modal

IniDaDves

• Microservices • Containers • Cloud • IoT

Technology

• ProducDon Outage • ApplicaDon Slow • User Experience • ConnecDvity

Failure

• SaaS • Mobile • APIs • IoT

Switching Cost

4/25/18

7

© 2016 by

Top Tier Vendors

IBM Punts Development to HCL http://www.theregister.co.uk/2016/10/03/ibm_hcl_software_outsourcing_deal/

HPE Strategy Clear As Mud http://it-tna.com/2016/09/28/hpe-strategy-clear-as-mud/

Tricen@s Delivering Con@nuous Tes@ng at the Speed of DevOps

© 2016 by

Continuous Testing

Analysts Align

“Continuous Testing is required for extreme automation”

voke

“Continuous Delivery Needs Continuous Testing”

Forrester

“DevOps initiatives will cause enterprises to implement continuous testing”

Gartner

”Continuous Testing has obvious application in agile environments.”

Bloor

4/25/18

8

© 2016 by

The Perfect Storm – Digital Disruption

Mitigate Business Risk

ConDnuous TesDng • Model Based Test AutomaDon • Risk Based TesDng •  Exploratory TesDng • Mobile TesDng

•  Security •  Safety CriDcal •  FDA • AutomoDve • Aerospace

Compliance • DevOps • Agile •  Lean • Bi-‐Modal

IniDaDves • Microservices • Containers • Cloud •  IoT

Technology • ProducDon Outage • ApplicaDon Slow • User Experience • ConnecDvity

Failure •  SaaS • Mobile • APIs •  IoT

Switching Cost

• BI/DW TesDng • API TesDng •  Test Data Management •  Service VirtualizaDon

© 2016 by © 2018 by

Wayne Ariola

3 Key Differences Continuous Testing v. Test Automation

4/25/18

9

© 2016 by

Continuous testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain feedback on the business risks associated with a software release candidate as rapidly as possible.

What is Continuous Testing

© 2016 by

Continuous Testing – What’s Different?

What’s Different?

RISK A feedback system for beXer trade-‐off decisions between release scope, Dme and quality

BREADTH The boundaries of an applicaDon are eroding. Advanced automaDon is required for end-‐2-‐end tesDng

TIME Its speed because Dme-‐to-‐market maXers not because development can release faster

4/25/18

10

© 2016 by

Survey on Risk 2017 Survey on the Future Role of the Tester hXps://www.surveymonkey.com/r/SWB7F3Q

© 2016 by

Does the release candidate have an acceptable level of risk?

4/25/18

11

© 2016 by

How to Achieve Quality @ Speed

§  Business risks drive quality activities

§  Teams collaborate on risk definition

§  Risk definition drives test case design

§  Execution prioritized per risk definition

§  Acceptance criteria visible and measured

§  Results of quality activities are visible and

translated for all levels of business

© 2016 by Ever-Growing Challenge

Technical Complexity & Rate of Change

Years Months

Months Weeks

Weeks Days

Days Hours

Hours Minutes

Delivery Cycle Time

Perceived Disruption

Today

Introduce Continuous Testing to meet demand

4/25/18

12

© 2016 by 23

18% (official) overall test automation

degree

8% (official) enterprise

test automation degree

Really ? X ?

© 2016 by

Continuous Testing requires Automation

24

80% Manual Testing

20% Automated Testing

UI

API

Today Future

Exploratory Tests

Automated UI Tests

API Tests

Test Driven Service Virtualization

+85%

- key enabler for high automation rates -

4/25/18

13

© 2016 by © 2018 by

Wayne Ariola

Key Testing Practices for Continuous Testing

© 2016 by

Session Based Exploratory Testing

Risk Based Prioritization Test Case Design

UI- and API based test automation

Active Test Data Management Test Driven Service Virtualization

CT Integration (CI, CD)

Automate

!

Explore

!

Manage

!

Integrate

!

Optimize

!

4/25/18

14

© 2016 by

Introduce Session Based Exploratory Testing

Automate

!

Explore

!

Manage

!

Integrate

!

Optimize

!

© 2016 by


All testing starts with a Risk Based Prioritization

Automate

!

Explore

!

Manage

!

Integrate

!

Optimize

!

4/25/18

15

© 2016 by


Risk Based Prioritization Optimize your risk coverage: Test Case Design

Automate

!

Explore

!

Manage

!

Integrate

!

Optimize

!

© 2016 by



UI- and API based test automation Automate

!

Explore

!

Manage

!

Integrate

!

Optimize

!

4/25/18

16

© 2016 by




Prevent false+: Active Test Data Management Automate

!

Explore

!

Manage

!

Integrate

!

Optimize

!

© 2016 by




Active Test Data Management Provide stable infrastructure: Test Driven Service Virtualization

Automate

!

Explore

!

Manage

!

Integrate

!

Optimize

!

4/25/18

17

© 2016 by






Automate

!

Explore

!

Manage

!

Integrate

!

Optimize

!

© 2016 by






Automate

!

Explore

!

Manage

!

Integrate

!

Optimize

!

4/25/18

18

© 2016 by

Key Points of Continuous Testing hXps://www.tricenDs.com/what-‐is-‐conDnuous-‐tesDng/

•  Primary goal is assessing business risk coverage

•  Establishes a safety net that helps the team protect the user

•  Always on, stable test environment

•  Seamlessly integrated into the so`ware delivery pipeline and DevOps toolchain

•  Delivers acDonable feedback appropriate for each stage of the delivery pipeline

© 2016 by

Q&A § Continuous Testing vs Test Automation Paper

https://www.tricentis.com/resource-assets/continuous-testing-vs-test-automation/

§ What is Continuous Testing (w/ 14 Key Points) https://www.tricentis.com/what-is-continuous-testing/

§ Survey – The Role of "Tester" in the New Digital Age https://www.surveymonkey.com/r/SWB7F3Q

Continuous Testing vs. Test Automation: Three Key Differences · W9 Test Automation Wednesday, May...

Documents

Transcript of Continuous Testing vs. Test Automation: Three Key Differences · W9 Test Automation Wednesday, May...