Focusing on the right test tools and industrializing their...

Testing trends… Focusing on the right test tools and industrializing their usage

SASTQB – TEST IT Conference 2015

• This document and the information contained therein shall be treated strictly confidential and remain exclusive property of Accenture.

• Accenture is the exclusive holder of copyrights resulting from the content.

• This document and the information contained therein may not be used for any purpose other than the purposes stated therein and in the context of which the document was made available.

• Any reproduction or distribution, complete or in parts of this document and/or the information contained therein requires Accenture’s prior written approval.

• The trademarks and service marks of the respective companies, including the Accenture marks and logos, are the exclusive property of such companies, and may not be used without prior written permission. Any reproduction approved by Accenture for other than the aforesaid purposes requires the incorporation of a respective copyright note regarding Accenture or the affected third persons.

Disclaimer

2

Agenda

Intro & about the speaker 4

Testing trends – what the analysts say 6

Test Architecture and how TMMi fits into the picture 13

Testing ecosystem, choosing the right tools sets 19

Test tools industrialization 24

Q & A 36

3

Intro & about the speaker

4






Q & A 36

5

Agenda

Testing trends – what the analysts say

Testing Spend

• Worldwide Discrete (Standalone) Enterprise Application Testing Services market was $16.7B in 2013 and growing 15% annually to $33.7B in 2018 (Fig.1)

• U.S. Discrete (Standalone) Enterprise Application Testing Services market was $8B in 2013 and growing 13% annually to $15B in 2018 (see IDC report for graph)

• Budget spend % per projects is continuously increasing (Fig.2)

Fig.1 Source: IDC, Doc #249925 / July 2014 - Worldwide and U.S. Discrete

Testing Services 2014–2018 Forecast

Fig.2 Source: HP/SOGETI, World Quality Report 2014-2015

6


Drivers for using 3rd party services

• While drivers are multi-faceted, 31% of all respondents indicate that the primary driver for using 3rd party testing services centers on driving down costs.

• 37% of public sector organizations indicate the primary driver for using 3rd party testing services is support for complex infrastructure testing, with the second most important being driving down costs.

• The top three characteristics buyers indicated were most significant when selecting a provider for application testing services were: reasonable costs (35%), high level of responsiveness/rapid deployment (20%) and cloud testing experience (20%)

0 5 10 15 20 25 30 35

Other

Support for vertical solution testing (e.g., banking, telecom)

Use turn-key test solutions for performance test or test automation

Support for localization – country specific readiness testing

Leverage fully managed specialized application test frameworks

Leverage fully managed application lifecycle testing services

As insurance against potential liability resulting from insufficiently tested products by in-…

Standardize internal QA environment across projects

Help with test data management services

Automate regression testing (to support complex business processes across multiple …

Leverage third-party tool expertise, talent and skills

Standardize testing tools and processes

Test staff augmentation

Drive productivity gains (e.g. more time for staff to pursue other initiatives)

Support for complex infrastructure testing (compatibility, interoperability)

Drive down costs

Source: IDC, Doc #249925 / July 2014 - Worldwide and U.S. Discrete Testing Services 2014–2018 Forecast 7


Fig.1 Source: IDC, Doc #249925 / July 2014 - Worldwide and U.S. Discrete

Testing Services 2014–2018 Forecast


Testing Spend

• Worldwide Discrete (Standalone) Enterprise Application Testing Services market was $16.7B in 2013 and growing 15% annually to $33.7B in 2018 (Fig.1)

• U.S. Discrete (Standalone) Enterprise Application Testing Services market was $8B in 2013 and growing 13% annually to $15B in 2018 (see IDC report for graph)

• Budget spend % per projects is continuously increasing (Fig.2) 8



Challenges and Outsourcing trends

• There is common trend of outsourcing testing services to specialized vendors, with a drop from 51% of companies using internal resources in 2012 to 30% in 2014 (Fig.1)

• In Agile environments, hybrids persist (i.e. Scrumfall), no real Agile environments, no specific methodologies nor adequate tools utilized, with difficulties in executing automation (Fig.2)


9


Main Themes

• Mobility as a continued focus with various tooling evolving to support specific testing needs • Full lifecycle integration and linking to development apps / Uis • Increased Traction and Maturity with Open Source Tooling • Continued focus on Services layer (API) testing and virtualization • Tools specifically designed for Agile • Analytics and most importantly Predictive Analytics are now a main focus for the test organizations and tool vendors alike

• “Time-to-market issues are driving this market alongside a greater need for governance and regulatory compliance. There is a lot of change going on in the market as an increasing number of development groups adopt agile practices.”

• “…the market is very mature, yet it also continues to evolve as the key trends discussed in this research would strongly indicate. As a result, we predict that, during the next five years, there will be a number of disruptive changes that will effectively reboot many vendor market positions in future Magic Quadrants.”

• “…we also note that open source is beginning to have a strong effect on the market, and we project that Selenium WebDriver will become the standard for functional test execution and this will marginalize vendors that can't provide strong higher-level test functionality.”

©Copyright Gartner. Magic Quadrant for Integrated Quality Suites. August 28, 2014

Gartner Magic Quadrant for Integrated Software Quality Suites 2014 – Selected Quotes:

10


Conclusions – key take away’s

• Digital transformation / business is here. One’s QA methodologies and tools need to cater digital/TaaS/mobile testing. Virtualization (network, services etc.), testing in the cloud are all elements to consider and invest in.

• Applications are more fluid, delivered faster, which implies, leaner development testing. Agile will take even more precedence than before, consequently develop competencies and centralized capabilities.

• More automation (non-traditional) + DevOps are required for faster, organized release to production and for bridging the gab between dev teams and operations.

• With increase in overall testing spend, investing (more) in tools adoption is the next logical inherent step.

• Security testing is still undermined (Sony knows for sure), companies need to invest in proper enterprise IT security governance frameworks and conduct more testing in this space.

• Investments in skills and tools on one side, as well as change management on the other are the crucial elements in winning the testing battle.

• With commoditization of testing, one needs to invest into more creative cost efficient TaaS models, more agile service based, output driven models, cross-shared and cross-skilled resources vs. legacy, traditional waterfall processes, dedicated resources and restrictive licensing models.

11






Q & A 36

12

Agenda

Test architecture and how TMMi fits in the picture What is a Test Architect?

What a test architect IS What a test architect ISN’T

Testing expert: a test architect MUST come from a testing background, with experience in all delivery models Advisor: solution architecture, estimations, test planning, metrics definition (and implementation), sales proposals Fixer: get into challenged projects, produce short term/quick wins and long term improvement plans Testing evangelist: deliver papers, presentations, training, provide technical leadership and strategic direction for their testing organization Implementer: Setup of TCoE / Test Factory, execute Test Assessments (Internal, External & TMMi), Enterprise Test Strategy (Definition/Setup), Test Optimization & Improvement, Non-Functional Testing, Standard and Emerging Technologies / Test Tools

Test Manager “Fire-fighter” for daily operational issues Delivery person (project wise)

Great article on the topic here: https://blogs.oracle.com/johnmorrison/entry/test_architect 13

https://blogs.oracle.com/johnmorrison/entry/test_architect

Test architecture and how TMMi fits in the picture How TMMi fits in the picture of testing tools

Test Tools Maturity Model

(based on TMMi)

Test Capability Models

(based on TMMi)

Test Maturity Model

integration

(TMMi)

14


• TMMi has 5 levels

• A TCoE will aim at a level 2 to 3, while test factories, between level 3 and 5

• Out of experience, we know that the perceived maturity level is higher than the actual one

15


• Accenture based

Test capabilities models

(based on TMMi)

16


Leve

l 3

Leve

l 4

Leve

l 2

Leve

l 5

Leve

l 1

Defect Management

Test Management

Requirement Management

Planning (release management)

Libraries and cross project asset sharing and reuse

Automation, Performance, Mobile, Security

Complete reporting capabilities

Executive Scorecards

Tool QA governance

Seamless integration to 3rd party tools (CI)

Defect Management

Test Management


Planning (release management)

Automation, Performance

Basic Mobile, Security

Advanced reporting (Dashboards across programs)

Scorecards

Tool QA governance

Basic CI

Defect Management

Test Management


Advanced reporting (Project dashboards)

Project templates

KPI Scorecard

Some automation & performance

Defect Management

Test Management

Basic reporting Ad-hoc usage

Basic defect management

Testing Tool Usage- Capability Maturity Level

Low High 17






Q & A 36

18

Agenda

Testing ecosystem, choosing the right tools sets Plan

Analyze Design Build Test Deploy

HP ALM/QC

ALI – Application Lifecycle Intelligence: Dev Integration

XS – Executive Scorecard

IBM RTC

JIRA

SELENIUM WORKSOFT

CERTIFY

IBM GREENHAT

CA LISA

3rd parties: SAP Sol Man; ARIS; MS TFS;

Perforce; Eclipse; Hudson; SVN; Rally; Jira; ITSM; Service Now; Remedy

Data Maker; Perfecto Mobile

HP SR – Storm Runner

Test Mgmt. Tool

Automation

TRICENTIS TOSCA

Performance

Security

Environments

Agile dev.

Mobile PERFECTO MOBILE

ALM Synchronizer

HP SC – Security Center (Fortify)

NEOTYS NEOLOAD


ALM Synchronizer


ALM Synchronizer

• Define (Test Architect) what tools to use in the test organization based on needs and long term requirements

• Ensure that all testing tools are integrated for a streamlined testing process

• Maximize tools ROI via:

a) Tools industrialization

b) Strategic partnerships with tools vendors, for band discounts

19

Testing ecosystem, choosing the right tools sets Agile Test Mgmt. Tools Automation Performance

Perf. Diagnostics

Security Mobile Test Data

Mgmt. Network

Virtualization Service

Virtualization Analytics / Reporting

IBM RTC

(Rational Team

Concert)

CLM (Collaborative Lifecycle

Management)

RQM (Rational Quality Manager)

RFT (Rational Functional

Tester)

Workbench

RPT (Rational Performance

Test)

APD (Application Performance Diagnostics)

AppScan

Workbench MobileFirst

Quality Assurance

Optim Green Hat

DOVE (Distributed Virtual Ethernet)

RST (Rational Service Tester

for SOA)

RTVS (Rational Test

Virtualization Server)

Insight

HP AGM (Agile Manager)

ALM (Application Lifecycle

Management)

QC (Quality Centre)

UFT (Unified Functional

Testing)

BPT* (Business Process Testing)

Sprinter*

LR (Load Runner)

PC (Performance Centre)

SR (Storm Runner)

Diagnostics Webinspect

Fortify

MC (Mobile Center)

UFT mobile -

NV (Network Virtualization - ex

Shunra)

SV (Service Virtualization)

XS (Executive

Scorecard)

Microsoft TFS

(Team Foundation Server)

VSTP (Visual Studio Test

Professional)

VSTP VSU

(Visual Studio

Ultimate)

- - VSTP - - - VSTP TFS

Oracle -

OATS (Oracle Application Testing

Suite)

OTM (Oracle Test Manager)

OFT (Oracle Functional Testing)

OLT (Oracle Load Testing)

- - - - - - -

Borland Agile Silk Central Silk Test

Silk TestPartner

Silk Performer Silk Performer

CloudBurst Silk WebMeter -

Silk Mobile Silk Central

Connect - - -

Silk Central TestBook

Grid Tools - - - - - - - Data Maker - - - 20

Testing ecosystem, choosing the right tools sets

Worksoft Certify – special features:

• Certify Analyze

• Certify Impact

• Certify Data

• Business Process Procedures

Accenture is a preferred alliance partner

Tricentis Tosca – special features:

• Model-Based Test Automation

• Service Virtualization & Orchestration

• Test Data Designer

• Tosca as a Service’ (TaaS)

Accenture is a certified implementation partner

• While automation teams have historically built their own test automation frameworks, there are now a small number of commercially available test frameworks that provide this capability independently from other test automation tools.

21

Testing ecosystem, choosing the right tools sets

22






Q & A 36

23

Agenda

Test tools industrialization

Test tools Industrialization

Test Tool Map

Tools Strategy

Tools Team

Upgrade versions

Upgrade editions

Accelerators

Integrations

Reporting

Quick Wins

Shift Left

24



Test Tool Map

Tools Strategy

Tools Team

Upgrade versions

Upgrade editions

Accelerators

Integrations

Reporting

Quick Wins

Shift Left

• Set up a test tools roadmap vs. your value chain / Software Development Life Cycle model

• Ensure that all testing areas are covered (Manual, Automation, Performance, Mobile, Sec., TDM, TEM

25



Test Tool Map

Tools Strategy

Tools Team

Upgrade versions

Upgrade editions

Accelerators

Integrations

Reporting

Quick Wins

Shift Left

• Define and implement an overall test tool strategy for the next 3-5 years

• The test tool strategy needs to include capital and operating expenses (including software, hardware and services costs) in order to provide the most efficient, cost effective solution for the organization

26



Test Tool Map

Tools Strategy

Tools Team

Upgrade versions

Upgrade editions

Accelerators

Integrations

Reporting

Quick Wins

Shift Left

• Set up a dedicated test tools team to handle all test technologies in the organization

• Establish a test tool service catalogue & use the tools as bridging mechanisms into other business areas

27



Test Tool Map

Tools Strategy

Tools Team

Upgrade versions

Upgrade editions

Accelerators

Integrations

Reporting

Quick Wins

Shift Left

• Upgrade/Procure the latest & best test tools versions and license types to make sure your organization gets the best value out of the tools

• Make sure that your current tools versions are still under support

HP ALM / QC Vendor Support Time-Table

Version 9.x 10.x 11.0 11.5 11.52 12.0 12.01

Support End Date Nov 2010 Aug 2013 Jan 2015 TBA TBA TBA TBA 28



Test Tool Map

Tools Strategy

Tools Team

Upgrade versions

Upgrade editions

Accelerators

Integrations

Reporting

Quick Wins

Shift Left

• Consider upgrade in Editions, in order to maximize the ROI on the test tools

29



Test Tool Map

Tools Strategy

Tools Team

Upgrade versions

Upgrade editions

Accelerators

Integrations

Reporting

Quick Wins

Shift Left

• Utilize accelerators, either tools or other reusable assets in order to achieve maximum efficiency with the organization test tools sets.

• SAP ERP Project Templates, Oracle accelerators, Predictive Analytics, E-approve, Business Intelligence, Automation harnesses and other related accelerators etc.

30



Test Tool Map

Tools Strategy

Tools Team

Upgrade versions

Upgrade editions

Accelerators

Integrations

Reporting

Quick Wins

Shift Left

• Ensure that there are proper integration flows between test tools and that redundant / duplicate work is not executed.

• Examples: Jira <-> ALM, Sol Man <-> Aris <-> SAP TAO <-> ALM, RQM <-> Sourceforge etc.

31



Test Tool Map

Tools Strategy

Tools Team

Upgrade versions

Upgrade editions

Accelerators

Integrations

Reporting

Quick Wins

Shift Left

• Define and implement an adequate test metrics framework

• Use the test tools innate reporting capabilities in conjunction with 3rd parties business intelligence apps; link to KPIs and SLAs.

• Automate and publish all reports into dashboards, Share Point Sites, executive dashboards etc.

32



Test Tool Map

Tools Strategy

Tools Team

Upgrade versions

Upgrade editions

Accelerators

Integrations

Reporting

Quick Wins

Shift Left

• Use the test capability to produce quick wins list that can be used with immediate effect in order to improve the test processes (via the tools)

• Example below: defect management process, consistent project template

• Training: Adoption readiness Tool and other practical hands-on classes included.

33



Test Tool Map

Tools Strategy

Tools Team

Upgrade versions

Upgrade editions

Accelerators

Integrations

Reporting

Quick Wins

Shift Left

• Use the test tools to drive Shift Left transformation into the organization

• Use simplified version of Shift Left (from a defect mgmt. perspective) – improve quality via defect root cause analysis and improved management

34






Q & A 36

35

Agenda

Q & A

Thank you

[email protected]

36

Focusing on the right test tools and industrializing their...

Documents

Transcript of Focusing on the right test tools and industrializing their...