Gartner - Ha(API)y testing in the hybrid-cloud & beyond

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Ha(API)y testing in the hybrid-cloud & beyondHybrid Automation (API) Testing

Jonathon Wright / 19th May, 2014

Emerging Technologies, Innovation & Automation

Gartner ADDI (SPS2):

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2

non-Ha(API)y testing

Currently we have over 5,000 end

users of HP ALM across the company

In a single ALM project we have 19,646 tests

scheduled for manual execution

Part of the current initiative we have

targeted 2,700 applications across the

company for automation

The current process is that we walk through

each test first before it can be automated


Each wave delivers automation for 250

projects across 100s of applications

Once delivered automation executes against

each build / release and can take over a week

non-Ha(API)y testing

The current initiative touches over

10,000 resources & stakeholders

All have reporting requirements ranging from

executive board level to individual teams

practicing agile and traditional test

methodologies


Adopting Global Agile Portfolio Management

Board of Directors

Exec Board

VP (Portfolio)

VP-1 (Domains)

VP-2 (Apps)

VP-3 (Projects)

VP-4 (Teams)

CIO

ITPO

ITDO

ITAO

ITPM

ITTM

CIO-1

ITPO-1

ITDO-1

ITAO-1

ITPM-1

ITTM-1

CIO-2

ITPO-2

ITDO-2

ITAO-2

ITPM-2

ITTM-2

CIO-3

ITPO-3

ITDO-3

ITAO-3

ITPM-3

ITTM-3

CIO-4

ITPO-4

ITDO-4

ITAO-4

ITPM-4

ITTM-4

Tier 0 - Programme – Executive Level (x4)

Tier 1 - Workstreams – Portfolio Level (x8)

Tier 2 - Domains – Solution Level (x330)

Tier 3 - Sub-Domains – Application Level (x2,738)

Tier 4 - Project-ID – Project Level (x7,324)

Tier 5 - Staff-ID – Team Level (x10,000+)


Adopting Ha(API)y testing

83%

6%

11%Automation (API)

Unsuitable

Unsupported

41%

33%

6%

20%Automation (UI)

No UI

Unsuitable

Unsupported

Native Support x2

Testing B(UI)ilds Ha(API)y Testing

Sample – 2,738 applications under test (AUTs)


TESTING

AS A SERVICE(TaaS)

INFRASTRUCTURE

AS A SERVICE(IaaS)

SOFTWARE

AS A SERVICE(SaaS)

REAL-TIME VISIBILITY

AND CONTROL

INSTANT SCALABILITYSCALE UP AND DOWN

HIGH AVAILABILITY

CMMI

PLATFORM

AS A SERVICE(PaaS)

ISO 29119PAY AS YOU USE(PAYU)

SLA’s

Adopting Cloud - Challenges & Opportunities

“We’ve all heard the claims that cloud computing will, without any up-front investment, provide instant

scalability, flexibility, and availability for testing-on-demand. But how well does this work in practice?”


Adopting Cloud – FrAgile (UI) Testing

Focus shift right

from physical (UI)

Identity

Services

Connectivity

(HTTPS)

Workflows

(Logic)

DOM

(HTML5)

BIG Data

(Database)

External

Services

Workflows

(Logic)

• OCR / Image

• Object Models

• XPATH

• Regular

Expressions

• Descriptive

Programming

• Fuzzy Logic

• Social

Intelligence

• Artificial

Intelligence

www.slideshare.net/jonathon_wright


Adopting Cloud – Testing Opportunities (API)

Focus shift right from physical (UI)

to logical (API) / messaging layer /

database layer

SOA

(API)

AJAX

CORBA

AMF

HTTP

SOAP

REST

Database(s)

ODBC

Ole DB

ADO .NET

JDBC

Java

Native

Messaging

ESB

AMQP

MSMQ

MQ

JMS

Tibco


Cloud Discovery – Learn physical endpoint(s)

SOA

Traditional functional testing focused on the physical (UI) endpoint

Traditional non-functional testing focused on the messaging layer endpoint(s)

Messaging Layer API SOA DBAPI

Physical Server Endpoint

Physical Client Endpoint

Functional Non-Functional

UI

Ha(API)y Testing


Cloud Discovery – Model logical endpoint(s)

Endpoint discovery Cloud Map modelling

“Globally, mobile phone users generate 5 petabytes of mobile data”

petabytes[1pb equals 1,073,741,824mb]

Source – Harvard School of MT Lab (2014)

• Should we be adapting Network Fabric

Models to map our Private Cloud(s)?

• Imagine been asked as your first task to

index 14,000 database endpoints?


Embracing Hybrid Cloud – Testing Challenges

Introducing emerging complex

cloud ecosystems

(commercial or community)

• Third-party public / private cloud endpoint(s)

– Access to live public endpoints for testing?

(test.paypal.com)

– Access to private endpoints for testing?

(VPN tunneling)

• Security challenges around cloud platforms

• Big Data (masking / protection / regulations)

• Cloud 2.0 (moving away from legacy cloud)

– BlackOps “data-in, data-out’ (DIDO) model


Hybrid Cloud demands Hybrid Testing

Manu’mation“Manu’mation is an approach to automated testing that leverages the manual effort

involved in traditional testing to encapsulate both user and system under test interactions”Source – Test Automation Patterns - Wikispace (2014)

Source - http://en.wikipedia.org/wiki/Hybrid_testing

Source - Experiences of Test Automation – Book (2012)

‘First Day’ Testing“The key to ‘first day’ test automation is to create a dynamic test repository cube based on

a combination of both fuzzy logic and descriptive languages (DSLs) to support the design

and development of tests before the actual delivery of the Solution Under Test (SUT)”

http://en.wikipedia.org/wiki/Hybrid_testing


Manu'mation

Adopting the ‘Test First Delivery’ (TFD) approach

Manu’mation

• Execute the manual tests via

HP Sprinter

• Capture test assets including

associated injected data set

• ‘Export to UFT’ as test artefacts that

can be extracted in meta data (XML)

Learn physical endpoint(s)

• Define physical endpoint(s)

• Learn (record) physical endpoint(s)

• Simulate (playback) virtual endpoint(s)

• Define (checkpoints) verification points

• Validate (replay) virtual endpoint(s)

Ha(API)y Testing

• Leverage HP UFT API Testing

• Enable Service Virtualisation

• Execute tests in minutes not hours

• Accurately report test results back into

HP ALM against each build / release

“Manu’mation is an approach to automated testing that leverages the manual effort involved

in traditional testing to encapsulate both user and system under test interactions”

Source – Test Automation Patterns - Wikispace (2014)

http://testautomationpatterns.wikispaces.com/MANU%27MATION

http://testautomationpatterns.wikispaces.com/MANU'MATION


Manu'mation powered by HP Sprinter

‘Export to UFT’ plus associated data injection

set

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. http://msdn.microsoft.com/en-us/bb906065.aspx

Cloud Testing – Testing ChallengesFocus shift from the private cloud network accessible with direct access via the

messaging layer

“An enterprise service bus (ESB) is a software architecture model used for designing and

implementing the interaction and communication between mutually interacting software

applications in Service Oriented Architecture (SOA)”

To secure (https) public cloud with limited access to the enterprise (internet)

service bus


Cloud Testing – Learn physical endpoint(s)HP Service Virtualisation

• Learns from the real

endpoint(s)

• Simulate response using

recorded or user-defined

response

• Emulate endpoint(s)

performance


PublicVNet-to-VNet

Private

SUT SVNV NVSV

Community

Clouds

Private

Clouds

Public

Clouds

VNet Endpoint VNet Endpoint

Hybrid Cloud


Hybrid Cloud

Public

Clouds

Private

Clouds

Hybrid Cloud

VNet-to-VNet

Public

VNet Endpoint

Private

VNet Endpoint

RRAS/VPN/NAT connections

OSI 3 - Network - IPSec

OSI 5 - Session - PPTP

OSI 7 - Application - SSL VPN

RRAS/VPN/NAT communication

FTP - 21

HTTP - 80

RPC - 135

CIFS - 139/445

Source - http://msdn.microsoft.com/en-us/library/azure/jj156007.aspx

SUT SVNV NVSV

http://msdn.microsoft.com/en-us/library/azure/jj156007.aspx


Syn

c

Systemic Failure

(DNS or Network)

Hybrid Cloud – Testing Challenges

How do you provide realistic DR / resilience testing in the public cloud?

“Don’t assume the ecosystems (commercial or community) and

other fulfilment systems to be tested in anything but isolation.” - Julie Gardiner (2012)


Application

ServerRules

Engine

MQ API

Small

Data

Cloud Testing – Define virtual endpoint(s)

Application

ServerRules

Engine

MQ

Discover

Big

Data

XML XML

APIMQXML

Test Execution (API)

Service Virtualization

Learn physical endpoint(s)

Define virtual endpoint(s)

Stub component(s)

Shim method(s)

Stub database(s)

API


‘First Day’ Testing

Adopting the ‘Test First Delivery’ (TFD) approach

TFD >= ATDD/TDD/BDD

• Create test definition(s)

• Model the sequence of test event(s)

• Build the Control Flow within UFT

• Define (checkpoints) verification

points

Define virtual endpoint(s)

• Define virtual endpoint(s) in SV

• Define (mock) the rule(s) in SV

• Generate (emulate) data-driven in SV

• Simulate (playback) virtual endpoint(s)

• Validate (replay) virtual endpoint(s)

Ha(API)y Testing

• Execute UFT BPT components

• Enable Service Virtualisation

• Execute tests without dependency of

the Solution Under Test (SUT)

• Leverage SV with HP LR

“The key to ‘first day’ test automation is to create a dynamic test repository cube based on a

combination of both fuzzy logic and descriptive languages to support the design and

development of tests before the actual delivery of the Solution Under Test (SUT)”

Source – Experiences of Test Automation (2012)

http://www.dorothygraham.co.uk/automationExperiences

http://www.dorothygraham.co.uk/automationExperiences


‘First Day’ - API Testing – Test Designer

Drag-and-drop test

elements to the canvas

DBDB

SOA

DBDB

Enterprise

DBDB

Middleware

DBDBSystem

DBDB

Flow Control

DBDBCustom


Endpoint(s) Service Modelling

‘First Day’ powered by HP Service Virtualization

Endpoint(s) Performance Modelling Endpoint(s) Data Modelling


‘First Day’ powered by HP Network Virtualization

Virtualize network conditions


‘First Day’ powered by HP UFT (API Testing)

vs. PCAP packet capture (promiscuous mode)Define virtual endpoint(s) Learn remote physical endpoint(s)


Leveraging the ‘First Day’ approach of

‘Test First Delivery’ (TFD) utilising

HP Service Virtualisation & UFT (API)

Enabled us to execute all of the tests without

the dependency of the Solution Under Test

(SUT) from day-zero.

Ha(API)y testing - Summary

Adopting the ‘Manu’mation’ approach of

‘Test First Delivery’ (TFD) utilising HP Sprinter

Enabled us to capture 19,646 tests over 4 major

releases across 12 sprints.


Ha(API)y testing – Takeaways

• UI Automation is Costly & FrAgile

• API Automation is Repeatable, Reliable and Fast

• Create Bridges (VNet to VNet) “If you build it, they will test”

• Use Cloud Maps to Learn, Discover and Model endpoint(s)

• Enable Day-zero Performance, Penetration & Security Testing

• Continuous Build, Integration & Delivery in YourTestCloud

• Embrace Community Test Clouds (Collaborate/Share Recipes) Testing Recipes (IFTTT)If <solution under test> exists Then <Learn Physical Endpoints>Else If <solution under test> does not exist Then <Define Virtual Endpoints>


select <IaaS>

select <PaaS>

select <SaaS>

select <TaaS>

select <TaaaS>

Cloud Recipe for Solution Under Test

Cloud Recipe (IFTTT)

Cloud 2.0 – Moving beyond the legacy cloud

“IBM are working with other cloud suppliers to establish the standards for cloud service provider compliance.”

- Murat Gunenc, IBM Cloud Representative @Rhodes W1 (2011)

CloudSystem Automation

YourTestCloud.com

“Heterogeneous test cloud that are open, flexible and extensible”

Testing Cloud

If <Solution Under Test>

Then <Build Cloud 2.0>

Testing Cloud


Final Thoughts

Thank you

“The best aspects of these proven approaches demonstrate

how they have evolved over the past decade, and this echoes

some of the changes towards more lean and agile business

methodologies. They are in a constant state of evolution –

just as the underpinning technology evolves over time.”

My section in the ‘Experiences in Test Automation’ book

(Dorothy Graham & Mark Fewster) 2012.

Gartner - Ha(API)y testing in the hybrid-cloud & beyond

Technology

Transcript of Gartner - Ha(API)y testing in the hybrid-cloud & beyond