TechTalk: Going Full Circle - Linking Code to Tests to Requirements and Back Around Again

World®’16

TechTalk:GoingFullCircle- LinkingCodetoTeststoRequirementsandBackAroundAgainJamesWalker– PrincipalSoftwareEngineer– CATechnologies

DO5T05

DEVOPS

2 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLD

©2016CA.Allrightsreserved.Alltrademarksreferencedhereinbelongtotheirrespectivecompanies.

Thecontentprovidedinthis CAWorld2016presentationisintendedforinformationalpurposesonlyanddoesnotformanytypeofwarranty. The informationprovidedbyaCApartnerand/orCAcustomerhasnotbeenreviewedforaccuracybyCA.

ForInformationalPurposesOnlyTermsofthisPresentation


Abstract

Dependencymappinglinksrequirements,codeandtests,andiscentraltogooddevelopmentpractice.

Thissessionwillshow:

§ HowCAAgileRequirementsDesignercanbeusedtocreateathreewaybindbetweentherequirements,codeandtests.

§ Howformalmodeling canbeusedtocreateafulldependencymapofeventhemostcomplexsystems,whileintroducingtraceabilitybetweenthedesign,codeandtests.

Withfulldependencymapping,whenachangeismadetoagivencomponentintherequirements,itsimpactacrosstheentiresystemcanbeidentified,informingthedevelopersofexactlywhatneedschangingandwhere.Testscanbeautomaticallygeneratedfromtheformalmodel,andcanbeupdatedautomaticallytovalidatethatthischangehasbeensuccessfullyimplementedinthecode.

Whenanotherchangeismade,rounditgoesagain…

James

WalkerCATechnologiesPrincipalSoftwareEngineer


SOFTWAREDEVELOPMENTCHALLENGES

MODELBASEDTESTING(MBT)

FEEDBACKDRIVENTESTING

CONCLUSION

1

2

3

4

Agenda


AboutMe– JamesWalker

§ BSc,MRes,PhD– SwanseaUniversity,Wales

§ ResearchinDataVisualisation/BigDataproblems

§ Grid-Tools– SoftwareEngineer(2012– 2015)

§ CATechnologies– LeadEngineeronCAARD(2015)


Maybeyouwon’tquitegettoenjoyyourweekendyet!


TypicalProject

Requirements

Design

Development

QA

Release

Analyst&Customer

Architect

Developer

Tester

Release


Waterfall


Problematic...§ Inflexible(linear)

§ Fewbusinesssystemshavestablerequirements

§ Longreleasecycle– Expensive– Faillate


WaterfallProjectRequirements

Design

Development

QA

Release

Analyst&Customer

Architect

Developer

Tester

Release

Months/Years

WinstonRoyce– demonstratedthewaterfallasacriticismforhownotto

developsoftware!


ContinuousDelivery


ContinuousDeliveryProjectRequirements

Design

Development

QA

Release(MVP)

Months/Years

Requirements

Design

Development

QA

Release(MVP)

Requirements

Design

Development

QA

Release(MVP)

Weeks


InTheoryThisisGreat…§ Eachtimewetestthedeliverable

§ Highlyadaptivetochange

§ Deliverfaster

§ Earlierfeedback(failfast)


InReality…Heroculture— Fullthrottletosavetheproject


TheProblem:Alackofclarityandvisionduringeachiteration

BusinessAnalyst Programmer TesterUser

TheUserKnowswhattheywantTheAnalystspecifieswhatthatis

TheProgrammerwritesthecode

TheTesterteststheprogram

Thefurtherthevisionsdiverge


Traditionaltestingmethodsaretooslow,manual,andlinear forContinuesdelivery,compromisingspeedandquality

0% 10% 20% 30% 40% 50% 60% 70%

Time/resourcesintestdatacompliance(PII)

Defectsstemmingfromambiguousrequirements

Testinginnefficienciesleadingtohighercost

Lackoftestcoveragecreatingdefects/rework

Difficultyfindingtherightdataforaparticulartest

Manualtestingleadingtoprojectdelays

What'sthebiggestproblemyouface?


1– PoorRequirements

§ Aplethoraoftechniquesexist,mostarewritteninambiguousnaturallanguage

§ Therequirementsare“static”- theyoffernowaytoderivetestsdirectlyfromthem…

§ …nowaytoupdatetestswhentherequirementschange– thishastobedonemanually

Bottlenecks


2– ManualTestCaseDesign§ Currentlymanual– atimeconsuming,error-prone process

§ Isunsystematic,adhoc,andhasnorealnotionof“coverage”

§ Over-testing andunder-testing – 10-20%coveragewith4timesover-testing

§ Poorrequirementsleadtopooroveralltesting,withtestershavingtofillinthegaps

§ Nolinkagetotestdata – processismanual,painstakingandverytime-consuming

§ Noflexibilityforchangerequests:acriticalweaknessinanagileorcontinuousdeliveryenvironment.Changestakelongerthantheoriginalrequirement.


3– PoorTesting->AutomatedtestingAutomatedtesting:Manualscriptgeneration

§ Automatedtestingframeworksareheavilyscripted

§ Scriptgenerationisusuallydonemanually

§ Aswellasthemaintenanceofscripts

§ Alternativesolutionsuse:

§ Recordplayback

§ Script-lessautomationframeworks(keyword)

Butyou’restilldoingmanualtestcasedesign!


TheVisionClarityandVisionDuringDevelopment

BusinessAnalyst Programmer TesterUser

Therearelessbugsandtheproductisdeliveredfaster

Thecloserthevisionmeanstheusergetsaqualityproduct


ModelBasedTesting


Modelbasedtestinglet’syoudefinethebehaviorofasystemundertest– inotherwordswhatissupposedtohappen.

WhatisModelBasedTesting?

Model

System

Themodelisthenusedtotesttheactualsystemtoseeifitdoeswhatitismeantto.


ModelRequirementsasan“Active”Flowchart

§ AformalmodelthatisaccessibletothebusinesswhoalreadyuseVISIO,BPM,etc.

§ Whichisalsoamathematicallyprecisemodelofasystem,sothatiteliminatesambiguityandincompleteness

§ Itcanbeusedbytestersanddevelopers– itbringstheend-user,businessandITintoclosealignment


NewDeliveryParadigm:FromIdeationtoDesign,DevelopmentandTesting


TestingWiththe“Active”Flowchart

§ Testerscanoverlaytheflowchartwithallthefunctionallogicanddatainvolvedinasystem

§ Testscanthereforebeautomaticallyderivedfromit


§ Exhaustivelytestamodel– extractallpossibleroutesfromstart->end

§ Eachroute/pathbecomesatestcase

§ Numberofroutesgrowsexponentiallywitheveryaddeddecision

32nodes+62edges=1,073,741,824 possibleroutes

GeneratingTestCases

2145Paths


§ Coverage,inmostcases,isrelatedtohowmuchfunctionalityisbeingcoveredinatestcase.

§ Traversethemodelbysatisfyingcoverageconstraints:

Optimization/Coverage

19PathsEdgePairs

9PathsInOutEdges

5PathsEdges

3PathsNodes


OverlayingAutomation

§ Testautomaters overlaycodesnippetsontoprocesseswhichcorrespondtothecodewhichperformseachaction

§ Automationscriptscanthereforebeautomaticallyderivedfromit


ProvideFullTraceabilityWithRequirements

§ Knowwhatneedstobere-testedandwhentheintegrityofasystemisatrisk…“IfIchangethis,whatwillIbreak?”

§ Theimpactofachangemadetoanindividualcomponentisidentifiedsystemwide

§ Theimpactontestcasesanduserstoriesupanddownasystemcanalsobeidentifiedautomatically


DEMO

Visitourboothformore!


FeedbackLoops– TheFuture

32 ©2016CA.ALLRIGHTSRESERVED.@CAWORLD#CAWORLDRunResults

TypicalBuildProcess

ExecutionEngine

BuildEngineDevelopment/

Code

DifferentvisualrepresentationsofPass/Failresults.

TrafficLights Dashboarding Email(Jenkins)


Scenario

Testing– Grayarea


NeedleinaHaystackWithouttheRightInfrastructure


TestingBermudaTriangle

Code

RequirementsTestCases

Thefurtherapartthesearethelessquality

• Modelrepresentstherequirements• Codeisimplementedusingthemodel(canaddlinkstospecificobjects/methods).

• Testcasesaregeneratedfromthemodel

Usingtheseweobtainaclosebindingbetweenallthree.

Fulltraceabilityenablessolvingthepreviousproblemsefficiently


TheFutureisFeedbackLoops(NewWorldOrder)

ExecutionEngine

BuildEngine

Development/Code

Model

Runresults


ScenarioRevisit– Whatdoesthepass/failrepresent?Testresultsbounddirectlytotheirroutesthrough

thesystem.


ScenarioRevisit– ExploratorytestingAfterafailurethemodelcanbeusedtogeneratememoretest

casesbypinpointtesting– testcasesfocusedondiscoveringmoreknowledgeaboutthefailure.


ScenarioRevisit– RiskAnalysisUsemeta-deta assignedtothemodeltoperformriskanalysis.Aretheeffectedroutesthroughthesystemaboveasetthreshold?

=63%ofUserActivity

Can’tRelease


ScenarioRevisit– GettingitFixed!We’vefoundtheprocess(es)responsible– nowwecantraceback

totheresponsibleteams!

DevelopmentPinpointedFailure

Defectcreated


Conclusion


WhydoIneedthecode,requirements,andtestsstronglybound?§ AshiftingITlandscapehastransformedtheroleoftesters–

testingisnowcross-functionalanditerative,withanemphasisonfeedbackthroughout

§ Newtestanddevelopmentemerged,butthe“heroculture”remained…

§ …Alongwiththechallengesofslow,manualtesting,ambiguousrequirementsand“toolshops”


TheVision:ShiftLeft,ShiftRight– OverlappingFeedbackLoops

§ ShiftLeft:workiterativelywithuserstogainupfrontknowledgeoftherequirementsandusethesetodriveactivitythroughoutthedevelopmentcycle

§ ShiftRight:connect-uptoolstacks anduse(test)analyticstoharnesstheinsightsoftesting,productionandoperations

§ Feedtheseinsightsbackintothemodel,andthewholeprocessgoesbackroundagainsoyoucanenjoyyourweekend

WhydoIneedthecode,requirements,andtestsstronglybound?


RecommendedSessions

SESSION# TITLE DATE/TIME

DO5X27S What’sNewinCAAgileRequirementsDesigner 11/17/2016at12:45pm

DO5X29SModel-BasedTestingforAchievingMaximumCDPipelineAcceleration 11/17/2016at3:00pm

DO5X40S CaseStudy:WhyRabobankWasInspired 11/17/2016at3:45pm


Stayconnectedatcommunities.ca.com

Thankyou.


DevOps– ContinuousDelivery

FormoreinformationonDevOps– ContinuousDelivery,pleasevisit:http://cainc.to/PiTFpu

TechTalk: Going Full Circle - Linking Code to Tests to Requirements and Back Around Again

Technology

Transcript of TechTalk: Going Full Circle - Linking Code to Tests to Requirements and Back Around Again