Assurance Leadership Forum (ALF) - Cognitive Adaptive Testing

1

Cognitive Adaptive Testing based on analytics-driven digital experiences

Jonathon WrightAutomation, DevOps & Digital [email protected] | linkedin.com/in/automation

2

Abstract

Jonathon Wright

DevOps, Automation & Digital Assurance

3

The success of our technologies largely depend

on the fact that, while we were speculating about the

possibility of ultra-intelligence, we increasingly

enveloped the world in so many devices, sensors,

application and data that it became an IT-friendly

environment, where technologies could replace us

without having any understanding, intentions,

interpretations, emotional states, semantic skills,

consciousness, self-awareness or flexible intelligence.

Memory as in algorithms and immense datasets now

outperforms intelligence. MegaTech, Technology in 2050, The Economist, Daniel Franklin, 2015

“

”

THE NEED FOR SPEED

94%of executives face increased pressure to release appsmore quickly

“

Journey to the Digital Testing

3

Based on 2014 Vanson Bourne study commissioned by CA

4

Digital Testing, Quality & Assurance

DIGITAL TESTING, QUALITY & ASSURANCE

http://bit.ly/digital-assurance http://a.co/jlxMHWp

5

DIGITAL TESTING, QUALITY & ASSURANCE

Digital Testing, Quality & Assurance

http://www.docjamesw.com https://youtu.be/e3PzRnY-zTg

6

‘Shift Right’ – Performance Lifecycle

7

IdeasBeliefs, Values, Culture

Heuristics

Behaviors

Value

AgileLean

Continuous Testing 101Webcast Series – Value-Driven Delivery – Episode I

Methods and Skills

http://catechnologies.podbean.comRay Arell

8

Shift Left - ResearchModel Driven Delivery (MDD)

SHIFT LEFT

https://www.ca.com/content/dam/ca/us/files/industry-analyst-report/continuous-testing-as-a-digital-business-enabler.pdf

Number of custom AUT

9

Digital TestingUrban 4.0 - Predictive Crime feat. video analytics

SHIFT RIGHT

https://www.slideshare.net/Jonathon_Wright/oracle-digital-evolution-enterprise-of-things

10

Cognitive Adaptive Testing (CAT)

11

Digital Testing

TESTING IN THE DIGITAL WORLD

COGNITIVE ADAPTIVE ADOPTION (DESIGNOPS)

COGNITIVE ADAPTIVE DELIVERY (VALUE-DRIVEN)

COGNITIVE ADAPTIVE LEARNING (ANALYTICS-DRIVEN)

COGNITIVE ADAPTIVE TECHNOLOGY (INSIGHT-DRIVEN)

WHERE NEXT?

1

2

3

4

5

6

12 © 2016 CA. ALL RIGHTS RESERVED.12

Portfolio Management

Unify long term strategy, investment and portfolio planning.

Collaboratively plan, prioritize and track work across the enterprise.

Project ManagementDevelop & Test

Synchronize efforts to dramatically speed app development & increase quality

Manage & Monitor

Make a great customer experience a competitive advantage

Release & Deploy

Control the release process, to continuously advance application quality, improve the customer experience and reduce costs.

ContinuousDelivery

Minimum Viable Experience

Minimum Viable Quality

Value-Driven Delivery

BUSINESS INITIATIVE

Complexity-Informed Organizational Change

BUSINESS VALUE

MinimumViableProduct

CUSTOMER

THINK

MAKE

Digital TestingComplexity-Informed Organizational Change

13

DesignOpsCreate and deliver Ideas to Outcomes

Pervasive Security

Redefine how work is planned, managed and

executed

Agile Management

Provide a flawless app experience optimized for

performance

Agile Operations

Accelerate and streamline development, testing and

release

Continuous Delivery

Continuous Development Continuous Adaptive Testing Continuous Release

Digital TestingValue-Driven Delivery (Shift-Left & Shift-Right)

14

DIGITAL ASSURANCE & TESTING

VALUE-DRIVEN DELIVERY

1Cognitive Adaptive Delivery

2Cognitive Adaptive Learning

4Cognitive Adaptive

Technology

3Cognitive AdaptiveTesting

Digital TestingCognitive Adaptive Adoption (CA-A)

16

The ability to reliably release high-quality solutions at any time

COGNITIVE ADAPTIVE TESTING

Accelerate Deliveries

Reduce Errors

Manage Complexity

Increase Visibility

Drive Collaboration

Continually Improve

Digital TestingContinuous Adaptive Testing (CAT)

17

> 50%large organizations globally will compete using advanced analytics & proprietary algorithms

40%enterprises' net-new analytics investment will go to predictive / prescriptive

75%technology-oriented business intelligence competency centers will evolve to focus on information value generation

Source: Gartner

Digital TestingIntelligent Analytics is the New Battleground

18

SOI

Visualization

(Big) Data

AnalyticsDriven

Inference (Machine

Learning & AI)

Intelligent Process

Automation

Digital Technology Integrations

Digital TestingSystems of Intelligence

18

Inspired by “Intelligent DevOps – Driving Continuous Delivery by Harnessing the Power of Analytics & Autonomics”, Shamim Ahmed, 11/16/2016, CA World 2016, Las Vegas

19

Digital TestingWhat is analytics-driven quality?

Model ProgressiveRefinement

Analytics-Driven

ActionsPredictionsMeta Data


20

Portfolio backlog prioritization and

value analysis

Improved scheduling and

capacity

Validated requirements in

backlog

Improved estimates

Predict failures based on code

commits

Dynamic code promotion

criteria

Predict defects/failures

Optimized Tests

Scenario identification

based on trends

DX Scores

Anomaly detection

PLAN DEFINE DEVELOP TEST RELEASE OPERATEPhase

Examples of Cognitive Adaptive

Intelligence

Examples of Outcomes

Investment decision analytics

Code quality

analytics

TestOptimization

& Defectprediction

A/B Testing

& Canary

Log & DX Analytics

DX-basedrequirements


Digital Testing

Cognitive Adaptive Intelligence (CAi)

Systems of Intelligence

21

TEST MAINTENANCE STATIC RULES BIG (un-structured) DATA

IOT, IOE

CX / DX SECURITY

STRUCTUREDAUTOMATION

Digital TestingKey Challenges Posed by Adaptive Digitalization


22

▪ Failure detection and prevention

▪ Scripted

▪ Limited process automation based on static rules

▪ Individual metrics based on structured data in lifecycle silo

▪ Limited insight and perspectives

▪ Reactive: deals with “Past”

▪ Failure prediction

▪ Automate the automation

▪ Intelligent process automation based on dynamic inference

▪ Analytics based on large scale structured and unstructured data across the lifecycle

▪ Predictive: from “What did happen?” to “What would happen?

TRADITIONAL TESTING

COGNITIVETESTING

Digital TestingIntelligent Analytics - Characteristics


23

AUTOMATED & SELF-HEALING

DYNAMIC/ADAPTIVE RULES

BIG DATA ANALYTICS

IOT-Ops

DX-Ops SEC-Ops

INTELIGENT / ADAPTIVE / COGNITIVE

+ +

+

+

+

+

+

Digital TestingChallenges addressed by Cognitive Adaptive Delivery (CA-D)


24 © 2017 CA. ALL RIGHTS RESERVED.

Digital Testing – Cognitive Adaptive Insight

25

Deep reinforcement learning

Digital TestingCognitive Adaptive Learning (CA-L)

Machine learning with neural dictionaries

https://www.luis.ai

“We are not at the centre of the universe (Copernicus), of the biological kingdom (Darwin) or of the realm of rationality (Freud). After Turing, we are no longer the centre of the infosphere, we share the infosphere with digital technologies”. The ethics of artificial intelligence, Luciano Floridi

26

Test Data Set

Test Algorithms Set

Test Meta Data

Source - IEEE Transactions on Pattern Analysis and Machine Intelligence, 21st September 2016, http://arxiv.org/abs/1609.06647Source - Google AI APIs - https://cloud.google.com/prediction/docs/apis

Digital TestingPattern analysis and machine intelligence

27

REAL TIME, BIG DATA ANALYTICS EMBEDDED THROUGHOUT OUR PORTFOLIO

AGILE MANAGEMENT AGILE OPERATIONS DEVELOPER PRODUCTS

CONTINUOUS DELIVERY MAINFRAME SECURITY

27

API analyticsRelease management dashboard Access incident responseReal-time app performance

Digital TestingContinuous Adaptive Delivery (CA-D)

Cognitive Adaptive Insight


31

Digital Testing – Cognitive Adaptive Learning

32

Emergent embedded AIfor neural networks

Digital TechnologyCognitive Adaptive Intelligence (iCA)

Singularity Ultra-Intelligence (2030)

“Let an ultra-intelligent machine be defined as a machine that far surpass all the intellectual activates of any person however clever” 1960s

33

Mode 0 - Descriptive

Cognitive Adaptive Testing (CAT):

1. “How long will it take to test these new features?” --- How long historically (test cases / execution time)2. “What is the associated risk rating of the new platform?” --- Failure rate (cyclomatic complexity > 13)3. “How much technical debt do we have?” 4. “What is the defect density and prediction for this solution?”5. “What is the current conference rating of the solution?”6. “What is the value of this capability?”

How will Intelligent Analytics help?

Digital TestingExample use cases for Intelligent Analytics

34

What were the past defect trends?

What were the root cause of the past defects?

What defects are we likely to find?

What preventive measures should we take?

Have we overlooked anything for defect prevention?

Predictive analytics

Descriptive Analytics

Diagnostic Analytics

Prescriptive Analytics

Deep Learning

Diagnosis Analytics EngineExamples of Analytics use cases

Cognitive Adaptive Intelligence (CAi)


35

How do you pin point the failures within the release

pipeline?

Model-Driven Design of Tests from Use Cases 1

2

3

Execute ‘Automation as Code’ from the Model

Pinpoint Failure and Diagnose Issue from Results

“Easy, soon as you check in the new features ready for test, the automation will run then the results are analysed automatically and if we have any failures then it identified them and raise them as bugs!

Cognitive Adaptive LearningTest in DevOps (TiD)

36

Mode 2 - Predictive

Cognitive Adaptive Testing (CAT):

1. “Can you create me all the tests I need to fully test this system?”2. “How do I successfully test this new feature?”3. “How much testing has been done already?”4. “Have you detected any anomalies within the system?”5. “How much testing will be enough?”

How will Intelligent Analytics help?

Digital TestingExample use cases for Intelligent Analytics

51

Where next?

54

Cognitive Adaptive Technology (CA-T)Next Generation of Digital Experiences (DX)

1. Digital Mobile, Chapter 2, Digital Experiences (DX), April 2017, Jonathon Wright

55

Q & A

58

Blogs #ExcuseFreeTesting – Legacy is your Legacy!

https://dzone.com/articles/legacy-is-our-legacy-1

https://dzone.com/articles/legacy-is-our-legacy-1

Assurance Leadership Forum (ALF) - Cognitive Adaptive Testing

Technology

Transcript of Assurance Leadership Forum (ALF) - Cognitive Adaptive Testing