Better Software East 2016: Evolving Automated to Continuous

Parasoft Copyright © 2016 1

2016-11-23

Evolving from Automated to Continuous Testing

Better Software 2016 - Orlando


Intro

Arthur Hicken is Chief Evangelist at Parasoft where he has been involved in automating various software development and testing practices for over 20 years. He has worked on projects including cybersecurity, database development, the software development lifecycle, web publishing and monitoring, and integration with legacy systems and maintains the IoT Hall-of-Shame http://bit.ly/iotshame

Follow him @codecurmudgeon

Blog: http://codecurmudgeon.com

Web: http://parasoft.com

http://bit.ly/iotshame

https://twitter.com/zeichick

http://codecurmudgeon.com/

http://parasoft.com/


Agenda

Today’s Challenges

What IS Continuous?

Barriers

Best practices

Role of automation

Role of static analysis!?


Normal Situation

Pressure to deliver software quickly

Pressure to reduce defect rate

Late stage “bug-hunting”

No time for error prevention

Quality can be rushed

Security is forgotten


How to you know a build is successful?

When it compiles?

When all unit-tests have run?

When the right coverage goal is achieved?

When the right % of test failures occurs?

When it’s deployed?


Continuous

Build, Integration, Testing, Release, Delivery, Deployment

Builds on and enables DevOps

High dependency on automation:

processes

assessment

decisions


Feature Release

Business Stakeholder Developer Customer Support

Continuous Testing is Beyond Automation

SoftwareFeature

Continuous measurements mean continuous refinement of the process

Real-time feedback from

Objective assessment and go/no go Defects are eliminated at the point

that they are easiest to fix

Business Stakeholder Developer Customer Support

Quality gates: Organizations can automatically promote software through quality gates when business expectations have been met


Challenges of Continuous

Tests must produce binary decision go/no-go

Reuse unit test and functional tests from devto QA

High level of automation

Requires disciplined mature process

Testing must automatically answer

Is it stable

Will it do what it’s supposed to do


Importance of Testing

How do you know when you're done?

How do you know if a fix for a minor bug broke a major function of the system?

How can the system evolve into something more than is currently envisioned?

Testing, both unit and functional, needs to be an integrated part of the development process.


Continuous Delivery

Continuous Integration

Delivery Team

Version Control

Build & Unit Test

Automated Acceptance

Tests

User Acceptance

TestsRelease

Check In Trigger

Trigger Trigger

TriggerTriggerApproval Approval

Check In

Check In

Feedback

Feedback

Continuous Deployment

Production

Business

Continuous *


Elements of Continuous Testing

Continuous Testing re-positions the question from “are you done testing?” to “is the level of risk understood and accepted?”

ContinuousTesting

RiskAssessment

PolicyAnalysis

RequirementsTraceability

TestEnvironment

Access

TestOptimization

AdvancedAnalysis

Ensure access to complete test environments

Automate defect prevention andPolicy measurement

Expand test coverage and measure test effectiveness

Define actionablepractices

Connect functional with non-functional requirements

Process improvementopportunities


Prerequisites

Tools

• Version control

• Build server

• Deployment server

• CI tools

• Automation tools (test, etc)

Process

• Commit/update often (each change)

• Always create tests (pass and fail)

• Test regularly

• Run regularly


Continuous Testing

Tests are logically componentized

Tests correlated with business requirements

Tests are incremental

Tests are repeatable

Tests are deterministic

Tests are maintainable in a process

A process that isprescriptive based on test results


Penetration of Automated Testing Practices

Aggregated from analyst research: Gartner, Forrester, voke, IDC, Ovum for enterprise IT (embedded market eliminated)

Practice Penetration Opportunities

Static Analysis 16 –38% • Business/Risk driven models• Process enabled

Unit Testing 15 – 32% • Workflow • Test optimization• Deterministic tests

Peer Review ~ 26% • Integrated results• Contextual review• Compliance triggers

API Testing 12 – 21% • 2nd lowest penetration• Regression models

Load Testing 32 – 49% • Smoke testing • Continuous

Service Virtualization 9 - 16% • Green field• Process enabler• ROI is a NO-Brainer!


Continuous needs

Proper infrastructure

A binary definition of “done”

Reliable quality gates

Extreme automation


Infrastructure

Real requirements system

Flexible developer level project/scrum management system

Automated build/testing system

Data collection that can answer

Is it done?

When will it BE done?

Is it good enough to release


Defining Done

Are requirements coded?

Do tests exist for the requirements and code?

Are the tests passing

To your satisfaction (often not 100%)

Will the tests be enough?


Delivery is Part of Software

Delivery is an important part of software too

Development and Test need access to production-like systems

Deployment has to be repeatable and reliable

Quality must be validated (measured)


Defining Quality

• Consistent

• Repeatable

• Meaningful

Automated objective measure

• Unit tests

• Functional tests

• Regressions tests

• Static analysis

Made up of


Effective Continuous Testing

Avoid manual end-to-end testing

Invest more into automated tests at component

Follow testing pyramid rules

Measure: test results

test effectiveness

Determine: Risk


How to do it better ?

Give absolute priority to automated tests

Invest time into designing interfaces (API)

Use API (service) testing tools to cover interfaces

Measure the quality of the test


How Can Static Analysis Help?

Coding standards

Quality gate (bug detection)

Prevention


Bug Gates

Binary Decisions

No “bugs”

No “critical” static analysis findings


Quotable Quotes

An ounce of prevention is worth a pound of cure. – Benjamin Franklin

Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place. – Deming

Simply finding bugs with static analysis isn’t enough.

Avoid risky behavior


Fix or Prevent


Static Analysis Continuous Feedback Loop

Identify Error

Isolate Root Cause

FixNew Rule to Prevent

Monitor

Code review

Regression

QA

Field bugs


Choosing rules

Things happening in

the field

Things you worry will

happen

Things happening in

the news

Standards you must comply

with


Main Points for Today

We must define “all” quality expectations upfront

We must change the primary goal of testing

We must accept that testing is only a task in the

quality process

We must be able to measure trust of the process

We must automate breaking the application


Re-Inventing the SDLC

The evolution of the software development lifecycle over the past 5 years is forcing the transformation of software testing

Transformation Business Impact

• “Digital Industrial Revolution” makes all a software company

• Trend toward Dev/Test insourcing as competitive advantage is via software

• Strong adoption of “agile” development practices

• Forces more technical testing, early. Agile is forcing “shift-left”

• DevOps is applying lean practices to SDLC

• DevOps is exposing 20 years of processbarriers protected by silos and culture

• Cloud has disrupted traditional SDLC and is driving down costs

• Switching costs associated with applications at an all time low

• Business risk associated with software failure at an all time high

• Public companies lose an average of $2.5 Bn in market cap on the day of the announcement


DevOps is Continuous Improvement

Speeding up the conveyor belt does not yield better results…

Modern DevOps must embrace systematic process improvement—Focused on testing


How to Achieve Quality @ Speed

Business risks drive quality activities

Teams collaborate on risk definition

All team members trained on risks

Acceptance criteria visible and measured

DevTest team activities prioritized per risk definition

Results of quality activities are visible and translated for all levels of business


Survey on Non-Functional Requirements

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Waterfall

Hybrid

Iterative

Agile

Agile-ish

“Yes” My Team Monitors Compliance to NON-Functional Requirements (NFRs)

34%

58%


From Automated to Continuous


Continuous Testing Mitigates Business Risk

Why ContinuousTesting?

Systematic, centralized decision making to factor

business risk into SDLC

Safety net to allow developers to bring innovations to market faster

…and automation to break the application

A feedback system for better trade-off decisions between release scope,

time and quality


Enabling Technologies

Stubs

Service Virtualization

API testing

Test data management

Environment management

Self-service test environments


How to do it better ?

Use service virtualization to improve testing automation

Isolate at the message layer

Simulate functional scenarios and performance conditions


The Test Environment Challenge

Test environment access is outside the control of development and test leaving gaps in the process

IT OperationsParallel development delays… Need simple, realistic access to dependent components…

Too much time waiting for access…Need reliable test data

Need a realistic testenvironment easy to maintain

Scheduling

Configuration

Access Limits

DependentApplications

Staged Assets

3rd Party Assets Virtual Environments

Hyper Visor

App App App


The Test Environment Challenge

Test environment access is outside the control of development and test leaving gaps in the process

IT Operations

Scheduling

Configuration

Access Limits

DependentApplications

Staged Assets

3rd Party Assets Virtual Environments

Hyper Visor

App App App

Create, Manage, Provision


The Challenge Multiple teams using the same test database

Teams not respecting data integrity & others test data records

Regression tests consistently failing. Takes >1 hour to determine that it was due to “data changes”.

“Real problems” were getting lost in the noise

SharedDatabase

Eliminated 83% of configuration time for a

major telecom company

Test data management for complex transactions

✔

✖

✖


The Solution Setup Virtual Assets to model the SQL queries and use API testing tool

to manage automated nightly regressions against both virtual assets and live systems

The Business Benefit Test teams able to focus on ‘real regressions’ and separate out data

integrity issues from functional test failure

Virtual Asset

Virtual Asset

Virtual Asset

✔

✔

✔

Eliminated 83% of configuration time for a

major telecom company


Reduced wait time for test team by 60% for a major media conglomerate

The Challenge

Large agile development effort to adopt Service Oriented Architecture (SOA)

High risk project but the Test team “stuck waiting for the first build”

Development of functionality was not easy to coordinate as different teams had different schedules; not all finished at the same time

Agile/Parallel development limited by system dependencies

Iteration

Iteration

Iteration

Team A

Team B

Team C

Iteration Iteration

Iteration Iteration Iteration

Iteration Iteration

Current Development/Testing

dependencies


The Solution

Use descriptions of the new services (WSDL, XSD, example JSON payloads) to build Virtual assets prototyping the new functionality.

Test team builds tests with against the prototypes with API testing tool and the independent development tests use the prototypes to perform early stage Integration Testing

The Business Benefits

Met business goals and timelines, were able to test functionality “as soon as” it was available. Practiced TDD against prototypes to get a head-start on ‘full system testing’

Iteration

Iteration

Iteration

Team A

Team B

Team C

Iteration Iteration

Iteration Iteration Iteration

Iteration Iteration

✔✖✔

✖✔✖

Reduced wait time for test team by 60% for a major media conglomerate


Summary

Continuous * requires mature infrastructure and process

Quality gates must produce a binary answer

Automation is everything

Policy is critical, because automation is critical

Static analysis gets you to the gate


Web http://www.parasoft.com

IoT Hall-of-Shame http://bit.ly/iotshame

Blog http://alm.parasoft.com

http://codecurmudgeon.com

Book: http://alm.parasoft.com/continuoustestingbook

http://www.parasoft.com/

http://bit.ly/iotshame

http://alm.parasoft.com/

http://codecurmudgeon.com/

http://alm.parasoft.com/continuoustestingbook

Better Software East 2016: Evolving Automated to Continuous

Software

Transcript of Better Software East 2016: Evolving Automated to Continuous