Risk and Impacts of Inadequate infrastructure for Service...

MIDAS•ProjectNumber318786•WhitePaper•Page1of23

ProjectNumber:318786

ModelandInferenceDrivenAutomatedtestingofServicesarchitectures

WhitePaper

RiskandImpactsofInadequateinfrastructureforServiceOrientedArchitectureTesting

WhitePaper


RevisionChart

Version Date Author (Partner) Description

1 2 December 2015

Andrea Nicolai Final Version

WhitePaper


1 MARKETANALYSISANDMIDASMARKETNICHESoftwaretestingrepresentsperhapsoneofthelastmajoropportunitiesforbusinessestoreduceIToverheadsandimproveefficiencythroughautomatedtestingtools.Manyof the traditional barriers associated with software testing (ill-defined testing andrequirements processes, limited understanding of actual testing costs) can beovercomeby selectinganautomated testing, indeedautomationof software testingcanreducethecostofthetestingfunction(by25%ormore,aswellasimprovingtimetomarketandoverallsoftwarequality).

Figure6Businessexpectations

ThisisvitalbecausesomanybusinessesnowheavilyrelyonITapplicationsfunctioningfaultlessly, timeafter time.Testingactivitiesareclosely linkedto thechallenges thatcompanies currently face: flexible reaction to changes on themarket and customerside,highspeedofintroductionofnewproductsthroughoptimized“time-to-market”,andgreaterefficiencyindeliveringservices.Software-testingandqualityassuranceisone of the most important IT topics for companies, and there is a high interest incollaboration with professional service providers for software-testing and qualitymanagementandalargesetofcompaniesalreadyintegrateexternalserviceproviders

WhitePaper


into their testingactivities, in themajorityofcases regularlyoron thebasisof long-termwithmanagedtestservicesagreements.

Service Virtualisation Testing (SVT) particular sweet spot is when applicationdevelopment and testing teamsareunderpressure todevelopand test fasterwhilealsodealingwithcomplexapplicationscenarios.

Figure7IntegrationTestingComplexArchitectures

Thepressureincreasesasthedemandformorespeedincreases.FirmsadoptSVTto:

• Create and provision complex test environments. A hallmark of SVT tools istheir ability to simulate hard-to-duplicate production environments.Applications are increasingly composed of a plethora of services that run ondiverse application infrastructure; these services are not always easilyaccessiblefortestingandcanbeexpensivetoaccessrepeatedly.Forexample,amobileinsuranceclaimappmayaccesscustomerprofilesfromaneCommerceplatform, claims data from amainframe, and third-party services likeGoogleMaps. It can be a challenge to create a test environment that represents orduplicates all of these interdependent services. SVT enables testing throughservice virtualization and test scenario simulation to recreate a testenvironmentthat’sascloseaspossibletotheproductionenvironment.

• Test earlier andmoreoften.Multiple development teamsworking in paralleloften face service dependencies that hold up full integration testing. Forexample,teamAmayuseaservicedevelopedbyteamBbutdelaytestinguntilteamB finishes an initial versionof the service. TeamA can create a servicestub tocontinue itswork,but that requiresadditionaldevelopment timeandmaynotrepresentthefullfunctionalityrequiredtoperformapropertest.SVT

WhitePaper


toolsallowdeveloperstocreatevirtualservicesthatmorecloselysimulatethefunctionality of the service. This allows developers and QA professionals tostart testing much earlier, leading to a shorter SDLC and fewer bugs inproduction. Ifwe look at results from surveyswe canunderstandhow todaydevelopersareapproachingtesting,morethanhalfdoesnottestatall.

Figure8DevelopersUseoftestingtools

• Automate regression testing. Many applications integrate with old legacysystemsforwhichnoonehasthesourcecodeortheknowledgetoautomateregression tests. These legacy apps might carry large batteries of manualregression tests that consume a lot of time and energy. SVT enables theautomationofmanual regression testsby recordingpayloadsexchangedwithlegacyappsonthewireandthenvirtualizingservicesbasedonthepayloadssothattheteamcanrunregressiontestsautomatically.Thiseliminatestheneedtoknowthecodeorappbehaviortocreatevirtualassetstotestagainst,savingtime and improving delivery speed even in the presence of complex legacysystems.

• Safelyenablecontinuousintegration.Development,testing,andoperationsareunderincreasedpressuretodelivernewapplicationfeaturesandupdatesmorefrequently.Asa result,many firmsaremoving toanAgiledevelopmentSDLCand continuous integration of new versions in production. The danger of

WhitePaper


releasingapplicationupdatesmorefrequentlyisthatnewbugswillappear.SVTtools combine simulated production and automated regression testing toprovide continuous testing for the SDLC, resulting in safer, more successfulcontinuous integration. The integration of SVT toolswith release automationandDevOps tools enables continuous testing support for broader applicationlife-cyclemanagement(ALM).

Figure9SVTandDevelopmentapproach

• Lower stub development costs. If developers want to test the integration ofservices thataren’tavailable, theyhavetocreateandtest theirownstubsoftheseservices.Whilethismightbeconvenientinsomecases,itoftengetsquiteexpensive,asdevelopersspendtimebuildingcodethatdoesn’tgotowardnewfeaturesorcreatevalueforthebusiness.SVTreducesthesecostsbyprovidingsupportingtoolstocreatestubsmorequicklyandallowingthemtobesharedacrossmultiple teams efficiently and for later testing purposes. Applying SVTincreasesteamproductivityandlowersdevelopmentcosts.

1.1 MARKETOVERVIEWANDSIZESoftware failures are everyday more expensive. On the day of the

announcementofasoftwarefailure,organizationlostanaverageof-2.3Billiondollarsofshareholdervalue.Thisequatestoabout-3.75%ofshareholdersvalue,Withsocialmediaandnewsfeedsonmobiledevices–newsoutletsarereadytopounceonissuesraising.Newsarticlesaboutanorganization’s secondoffense increaseonaverageof167%.

WhitePaper


Just as an example a series of software failures and security breaches leftSony’sgamingservicesdownforweeks,severalanalystscalledfortheoustingoftheSonyCEO.EstimatedcumulativelossforSonywasaround$18Billion

Figure10Sonyestimatedlossforsoftwareerrorsingamingerrors

Also,notableisthatthemarketsdon’tforget.Organizationsthathadasecondoffensewerepunishedharderwithanaverageof-5.68%declineinstockprice.

WhitePaper


Figure11Lossofsoftwareerrors

Withsocialmediaandnewsfeedsonmobiledevices–newsoutletsarereadytopounce.Newsarticlesaboutanorganization’ssecondoffenseincreaseonaverageof167%.

Figure12Companiesaffectedbyserioussoftwareerrors

821.000

1.080.000

451.000 1.400.000

896.000

1.550.000

5.500.000

650.000

3.490.000

2.240.000

CLOUDSERVICESAIRLINE BANK SOFTWARE RETAIL

1stEvent 2ndEvent

WhitePaper


Assessment of automated testing market is quite a difficult tasks given thesegmentation of the market and of the different testing approaches. For thisdocumentwewillfocusonwhatisdefinedasservicevirtualisationoftesting(SVT).

GlobalTestingoutsourcingmarkethas increasedthreetimeseveryfouryears. Itwasevaluatedat6.1B$in2005andisestimatedtobeabout60B$today.

Figure13Testingglobalmarketsizeestimation

Severalresearchorganizationsawthenahighdouble-digitgrowthin2011at33.2%onlow numbers in the cloud testing and Automated Software Quality SaaS withcontinued reinvestment. User engagement is driving adoption and uptake in thiscompetitiveareaforautomatedsoftwarequality.Althoughthemarketrevenuesizeisstillslightandsomewhatearlyinitsevolution,highgrowthnumbersonasmallbasein2011 (andalso in2010)aresignificant.Amongothers IDCassessedthecloudtestingandASQ SaaSmarket at $261.4million for 2011, up from$196.2million and28.6%growth in2010,withahighergrowthof37.8%expectedfor2012.CloudtestingandASQSaaScombinedrepresentedaround11.8%oftheoverallASQ$2.2billionmarketfor2011

WhitePaper


Figure14Cloudtestingmarkettrends

1.2 MARKETTRENDSThetestingtoollandscapeischangingunderseveralpullsofuserneeds.Someanalystsdoidentifythefollowingtrends:

1. Testing and development are getting more and more integrated and becoming anintegralpartof theSDLCfrom itsvery initialphases.Developersareadjustingtothistrend;

2. Developers love open source tools and choose them as a first option.Becausedevelopers are gettingmore involved in testing, specifically in unit and automation(functionalandnonfunctionaltesting),performanceandintegrationtesting,theyalsohave a strong say in the tools used for testing. Youwill see in the testing tools docresearch that open source tools are flourishing in all main testing categories: testmanagement, test data management, automation, and performance. In tools,categories like test-driven development, unit testing, code quality, and bug trackingareopensourceanddevelopersareanentrenchedbinomial.

3. Service virtualization tools.With software becoming more strategic to enterprises,software quality and testing are becoming first-class citizens aiming at primarybudgets. So simulation in the form of service virtualization (a higher level ofabstractioncomparedwiththevirtualizedtestenvironmentsandlabsyouareprobablymore familiar with) becomes an important enabler for providing a stable testingenvironmentthatsimulatesarealcomplexsoftwareintegrationlandscape.

4. CompanieslikeIBM,HP,andCAareallheatinguptheirmarketingenginestopositionthemselves in thisnascentmarket.Parasofthasa solution in this space too.Weareseeing an interesting evolution of this whole area into testing optimization, whereintegration of service virtualization with downstream automation tools provides acomplete end-to-end testing and continuous deployment solution (both IBM’sacquisitionofUrbanCodeandCA’sacquisitionofNoliogiveanindicationofthis).

Market,2010,196,2

Market,2011,261,4

Market,2012,360,2092

%Increase,2011,28,6

%Increase,2012,37,8

%Increase

Market

WhitePaper


The huge change of testing is undergoing has an impact on all of the testing toolscategories: from testmanagement tools to functionalandnonfunctionalautomationtesting toperformanceand load testing to securityand testdatamanagement toolsand,finally,tothenewcomers'servicesvirtualizationforsoftwaretestingtools.Morethaneverthere isaneedforaconstant flowof informationamongBAs,developers,andtesters.Astestingshiftsmoreinthehandsofdevelopers,vendorsneedtoadapttools that easily plug into the developers' integrated development environments(IDEs),whileQAandothersoftwareprofessionalsprefertoolsthatofferahigherlevelof abstraction and are easy to use. The pressure on tool vendors is to emphasizequalities like collaboration/communication, continuous automation, simplicity, andseamlessintegration.

1.3 COMPETITIVEENVIRONMENT:SOATESTINGINDUSTRY

There is a number of SVT testing tools available in the software market.Although the core functions of these tools are similar, they differ in functionality,features, usability and interoperability. CA Technologies, HP, IBM, Parasoft, andSmartBearSoftwaredoconcentrateonservice.

CA Technologies and IBM are among the leaders with rich features and acompelling strategy. CA and IBM lead with rich, comprehensive, in-depth SVTcapabilitiescoupledwithastrongstrategyandrichpost-salesservicesandtraining.CAfocuses on the downstream part of the life cycle by integrating the LISA (formerlyITKO) SVT solution with its release automation (acquired from Nolio) and APImanagement(acquiredfromLayer7Technologies).IBMleadswithitsRTVS(formerlyGreenHat)SVT,buthasamorecomprehensiveend-to-endDevOps(viaitsacquisitionof UrbanCode) and ALM testing integration tool strategy. CA has more accuratesimulation capabilities,while IBMhas stronger test lab provisioning. CAhas a largeruserbase;IBMhasstrongermarketmomentum.

HPandParasoftarestrongperformers.Parasoft’sfeaturesareonparwiththeleaders,butitsstrategywasnotgoodenoughtoqualifyforthattier.Parasoftisasolidchoice for customers that prefer broad capabilities, ease of use, and quick startup.Parasoft supportscontinuous testingduring theSDLCandaclear separationof rolesbetween testers who design, develop, and consume virtual assets and those whomanageenvironments.

HParrivedlatetotheSVTmarket;until2011,HPofferedITKOuntilCAacquiredthe product, at which time HP decided to go with its own product, HP SV. HP isespeciallyattractivetoitshugecaptivemarket.Itsproductprovideseaseofuseandanenjoyableuserexperiencewhentestingscenariosneedtoleverageintegrationwithitsown ALM and testing tools like LoadRunner. HP’s strategic product road map isparticularlyinterestingforimplementingself-servicetestingfactories.

SmartBear Software satisfies techies but lacks key SVT features. SmartBearfocusesnarrowlyontheSOAPandRESTwebservicesmarket.SmartBearSoapUIPro

WhitePaper


provides developers and technical testers with scripting and wizard-based featuresthatneedtocreatetheirownstubsormockingservicesfortesting.SoapUIProisforclients that takea very lightweightapproach toSVT. SmartBearalsoprovidesa verysuccessful and open source mocking tool, SoapUI, which acts as a gateway to theSoapUI Pro commercial solution. Both products have thousands of users and bettermarketpenetrationandmomentumthantheotherSVTplayers. Inaddition to thesegeneral-purpose solutions, firms that wish to benefit from SVT solutions may alsoconsiderthesealternativeorcomplementarytools.

Figure15SVTComplementarytools

Stubbing or mocking servers. These tools integrate with developmentenvironmentsandofferfeatures,scriptinglanguages,orprogrammingframeworkstospeedstubdevelopment.They’rethelittlebrothersofSVTtools,originallycreatedtotestservice-orientedarchitecture;now,firmsoftenusetheminconjunctionwithmorecomprehensive SVT suites. Clients usemocking servers as an alternative to broaderSVT suites when they don’t have a complex application scenario with multipleplatforms, protocols, and message formats or aren’t looking to scale SVT as anenterprise service for continuous testing and development. Parasoft offers a littlebrother to Virtualize called SOAtest, while SmartBear offers open source SoapUI inadditiontoSoapUIPro.OtheropensourceoptionsareMockitoandjMock.

Testdataprovisioningandvirtualization tools. Theseare comprehensivedata

WhitePaper


managementandtestdatamanagement (TDM)tools thatarehighlysynergisticwithSVT.TDMtoolsallowfirmstodiscover,subset,mask,refresh,andanalyzetestdatatoimprove quality and testing. Integrating TDM and SVT tools enhances testing andquality by making hard-to-reach “proper” test data virtually accessible. TDM toolsallowformaskinganddesensitizingvirtualizedtestdataandsubsettingthequalityoftest data to be used in simulated scenarios. TDM players active in the SVT spaceincludeGrid-ToolswithIntelligentVirtualServicesanddynamicmessagemaskingandIBMwithInfoSphereOptimTestDataManagement.ItdoesnotappearthatotherTDMplayers—likeInformaticaorCamouflageSoftware—areactiveinSVT.

Network virtualization tools (NVT) for software testing. NVT allows firms toemulatenetworkconditions like latency, limitedbandwidth,packet loss,andjittertotestapplicationperformanceandotherparameters.NVT isespeciallyeffectivewhenmultiplenetworkconnectionsarerequiredtosupportthird-partyservicesorexternalresources, and when unexpected loads or concurrency happen in any geography.ShunraSoftwareisthemostsignificant—andonly—playerwithanNVTsolutionthatweknowofandisakeypartnerofmostoftheSVTtoolproviders.TheintegrationofShunra and SVT tools enables the testing of complex end-to-end application andservice performance scenarios while taking lower-level network variations intoconsideration.

Microsofttakesanalternateapproach,withjust-in-timeproductiontestingandmonitoring.MicrosoftdoesnotofferSVTtools.Instead,ittakesanalternateapproachwithinthehostedVisualStudioOnlineserviceandtheVisualStudioIDEthatleveragesproductionapplicationdataanalytics.MicrosofthasbeenofferingIntelliTraceforquitesometime,whichtakestheproblemawayfromproductionandgivesittodevelopersto solve, helping them to immediately analyze the root cause. Today, ApplicationInsightsisthenewcollectorofIntelliTrace..

TheaimoftheMIDASprojectismodel-basedextremeautomationofSOA/APItesting tasks. TheMIDAS TAAS user builds a collection of structural, functional andbehaviouralmodelsoftheservicesarchitectureundertestthatallowtheMIDAStestmethodstoautomate:(i)theproductionoftestcasesandoraclesand(ii)theplanning,scheduling,execution,arbitrationandreportingofunitandintegrationtestcampaigns.

Themeasure of the impact of theMIDAS framework and technology on thesoftwareandserviceindustrypracticeisanimportantissue.ItisacknowledgedintheMIDASDOWasanobjectivetobeattainedandhasbeenputunderconsiderationfromthebeginningoftheproject.Theproblemcanbeformulatedintermsofthefollowingquestion:what is the goodmethod that allows estimating how the SOA/API testingpracticeswillbeimpacted-qualitativelyandquantitatively-bytheavailabilityoftheMIDAStechnology?Thesearchforananswertothisquestionisnoteasyperseandismadeevenmorecomplexbecauseoftwofacts:

1. Servicetestingisnotonlyanactivityofserviceproviders,butalsoofserviceusers. Service providers and service users are roles of developers whosesoftwareusesservices inorderto implementtheservicesthat itprovides.Hence,servicedevelopershavetotesttheservicesthattheirsoftwareuses,

WhitePaper


inadditiontotesttheservicesthatitprovides,andthisfromthebeginningof theSDLC (testdrivendevelopment).Thisneed,which isbroughtby thesoaroftheSOA/APIeconomy,pushesforarealchangewithrespecttotheactualdevelopmentandtestingrulesofgame.

2. The MIDAS SAAS approach - and the underlying cloud technology -downsizes of one ormore order ofmagnitudesall the testing equipmentcosts, making these equipment accessible to people that traditionally donotpracticeorpracticeverylittletesting(suchasSMEs).Moreover,onthebasis of pay-per-use policy, the MIDAS SAAS approach downsizes thelicensing costs of testmethods and tools too. In other terms, theMIDAStechnology disrupts the SOA testing tool market. The objective is thatpeople(suchasSMEs)practicinglittletestingornotestingatallenterthe“new”SOA/API testing-as-a-servicemarket.Asaconsequence, inorder toestimate correctly the impact, we should measure the actual cost of notestingtoo.

Afteralmosttwoyearsofresearch,weshouldagreeonsomefacts:

• There are very little publicly available data and studies on the testingcostsingeneraland,afortiori,onSOAtestingcosts.Theonlyinterestingsourceofdataandarguments is thefamousNISTstudy1. Inparticular,the European Commission hasn’t issued anything comparable for theEuropean Union. Moreover, reports of market research organisationssuchasGartner areabout the (SOA) testing toolmarket, not theSOAtestingpractice.

• There are no data and no studies at all about the consequences andcostsofnotesting(exceptintheaforementionedNISTstudy).

• Interviewed professionals - managers, developers and consultants –have limited knowledge and awareness of the testing and no testingpracticesandcostsandarereluctanttotalkfranklyaboutthisissue.

Putting aside theequipment and licensing costs, it is really difficult toobtainreliable data of thehuman effort needed to perform the testing activity and of theeventual downsizing of this effort brought by theMIDAS technology. This proposal,which is firstly directed to the MIDAS UMC, aims to collect this kind of data. Thegeneral idea developed in the remainder of this document is a sort of “reverseengineering” of the problem that also satisfies the principle of “comparing what iscomparable”.Thisdocumentisnotaworkplan;itsimplyexposesandargumentstheideaandcallsforremarks,suggestionsandadiscussionabouttheissue.

1NIST-02-3(2002).TheEconomicImpactofInadequateInfrastructureforSoftwareTesting.PlanningReport02-3.NationalInstituteOfStandards&Technology.URLhttp://www.nist.gov/director/planning/upload/report02-3.pdf

WhitePaper


1.4 THEMAINCONTRIBUTIONSOFMIDASINNOVATIONMIDAS is a Service/API testing facility accessible as a service on the Internet.

TheMIDASfourkeyinnovationpointsare:

1. Low-costTestingAsAServiceoncloud.

2. ProgrammabletestingfacilitythroughAPIs.

3. Evolutionaryportfolioofenhancedtestmethods.

4. Extremetestingautomation.

Thesepointsareillustratedintheparagraphsbelow.

1.4.1 Low-costTestingAsAServiceoncloudTheMIDAStestingfacilityisdeliveredasaserviceimplementedoncloud.This

delivery mode ensures: (i) a reduction in equipment costs of at least one order ofmagnitudeand(ii)theshiftfromcapitalexpendituretooperationalexpenditure.

Theserviceisself-provisioned,andthelogicalandphysicalresourceallocationisnoonlyscalable,but,aboveall,highlyelastic.Thepay-per-usepolicycanbeappliedinaradicalwaybecauseit iscoupledwithtransparentaccounting,pricingandbillingbasedonfairmetering.

The“radical”elasticityofresourceallocationandradical“payperuse”pricingpolicy are themost important disruptive innovations in terms of costs. There is nocomparableofferinthemarkettoday,evenformorebasictestingfunctionalities2.

FairmeteringmeansthattheMIDASutilisationpricecouldbecalculatedonthebasisoftheunderlyingcloudplatformpriceofthecomputationalresourceseffectivelyusedby the customer, plus amargin. This policy canbeappliedonlybyusing cloudproviders that practice radical pay per use policy and sophisticated accountingfeatures3. Furthermore, in a dynamic perspective, the public cloud price will lower4rapidly. The choice that cloud-based service providers, such as the MIDAS facilityprovider,areconfrontedwithwillbebetween(i)followingstrictlythecloudproviderloweringpricepolicyand (ii)practicingextra-margins.Probably therealisticstance isin-between. In any case, the prices of the testing as a service delivery mode willdecreasefastinthefuture,asamechanicaleffectoftheloweringofcloudcosts.Thedisruptive cost cut (making the same thingwith costs that are at least oneorderofmagnitudelower),aswellasdecreasingcosts,areessentialtraitsoftheMIDASoffer.

The usage of a public cloud provider for the underlying infrastructure couldraise a security concern. The conclusions of an objective analysis are that securityconcernisstillpresentasaperceptionbutisfarfromrealityforaMIDAScustomer.In

2ItcanbecomparedwithfreeproductssuchasSoapUI,thattheuserrunsonher/hispremises.3ThisisthecaseforAWS,thecurrentMIDAScloudprovider.4InthepasteightyearsAWShaslowereditsprices42times(onceevery2/3months).

WhitePaper


fact,what is runningonMIDAScloud is thetestsystem,not thesystemsunder test.Thecustomerexecutablecode isneveruploadedontheMIDAScloud.Thecustomercriticaldataareneveruploadedon theMIDAS facility.Thecustomeruploadson theMIDAS cloud only models for testing and testing data. We discuss in the nextparagraphsprivacyconcernsaboutthesemodelsanddata.

Models for testingdescribe theexternally accessible functions, the interfacesand the external behaviour (at the interfaces) of the SAUT components and theirmutualservicedependencies.

In general, services/API are classified as public (available on the Internet),partner (available in infranets of business partners) and private (accessible on theintranet of the organisation). Services architectures can be classified in the samecategories.Aservicesarchitectureis:(i)publicifallitsservicesarepublic;(ii)partnerifatleastoneserviceispartnerandtherearenoprivateservices;(iii)privateifatleastoneserviceisprivate.

For public architectures there is no privacy concern by definition. Partnerarchitectures are multi-owner and deployed in partner networks, whereas privatearchitecturesaremono-owneranddeployedontheownerprivatenetwork.Forboth,models(serviceinterfacesandservicesarchitecturetopology)couldbeconfidential.

Generally speaking, service interfaces can be private5 but are seldomconfidential. The trend, in the most important business domains is towardsstandardisation. This is the case for theMIDAS pilot domains (Health and Logistics)thatcanbeextremelysensitivetoprivacy.Thedataexchangedinthefieldwithinthesearchitecturesareclassified (especially forHealth),but thedata formatsare standardandpublic.TheMIDASpilotpartnershaveput inplace servicesarchitectureson thebasisofbusinessstandardsofinterfacesanddataformats.

Testingdataareconfidentialonlyifalsotheinterfacesareconfidential,becausefrominstantiateddataandconcreteinteractionsitispossibletoreverseengineerthedata formats, interfaces, exchange protocols. But testing data per se are notconfidentialbecausetheyarefictitious.

Testing data must not be customer’s real data. Using real data for testcases/oraclesisverybadpracticeand,insomebusinessdomains(suchasHealth),itistotally forbidden by current regulations. A customer can utilise as testing data theresultofsometransformationsofrealdata(dataobfuscation,anonymization).MIDAScouldsupplydownloadabledataobfuscationsoftwarecomponentstobeexecutedonthe customerpremises.Of course,MIDAS cannot give anynon-disclosure guaranteeaboutrealdatathatareusedastestingdataanduploadedontheMIDASplatform.

The lastsecurityconcern isabout the interactionof theMIDASplatformwiththecustomers’servicesarchitectures,whosecomponentsareinstalledbytheirownersonpremisesorontheirpublicorprivateclouds.MIDAS interactswiththeSAUT,not

5Interfaces,asthesourcecode,canbecopyrighted.Thismeansthatyoucannotrunyourserviceusinginterfacescopyrightedbysomeoneelse.Ascopyrighted,theyaredisclosed.Theonlyprivateinterfacesthatarealsoconfidentialarecoveredbythetradesecret(andtheyarenotcopyrighted).

WhitePaper


withtheSAIF(ServicesArchitectureIntheField).TheSAUTmustnotbetheSAIF.Thesame considerations made about real data apply. Testing directly the SAIF,concurrently with business operations, is very bad practice and, in many businessdomains (such as Health), it is forbidden by current regulations. Of course, MIDAScannotgiveanyguaranteeaboutfunctionalandoperationalreliabilityofaSAIFthatistested concurrently with the flow of real business operations. The obvious goodpracticeistoputinplaceaSAUTthatistotallyseparatedfromtheSAIF.

The conclusions about security concerns, such as access control, privacy andintegrityofdataarethat:(i)averylargepartoftheMIDASpotentialaudiencedoesnothavespecialstrongsecurityconcernsaboutmodelsfortestingandtestingdataand(ii)amoregeneraltrendisthatCloudFUD(fear,uncertaintyanddoubt)isstillpresentasaperceptionbutfar fromreality6.Hence,thecurrentsecurity featuresoftheMIDASplatform(accesscontrol,usersandbox,etc.)basedon theunderlyingcloudplatformsecurity services seems to be sufficient for a large part of the potential market.Moreover,thedeliverymode(TAAS)andlowcostsoftheMIDAStestingservicesallowdecentralised fruition by enterprise departments in BYOC (bring your own cloud)mode7.

1.4.2 ProgrammabletestingfacilitythroughAPIsThe MIDAS functionalities are fully accessible through public APIs that are

documentedon theMIDASportal.Thiskey innovation (in theworldof testing tools)allowsMIDASenteringthedigitalserviceeconomy.

Four customer/partner populations are concerned with this key innovationfeature:(i)endusers(businessdevelopers);(ii)softwareengineeringtooleditors,(iii)businesspackageandverticalservicesvendors;(iv)consultancy,includingprofessionaltesters.

The MIDAS APIs allow the business service developer (the end user) to“integrate”theMIDAStestingserviceswithher/hishomemadeCAST(Computeraidedsoftwaretesting)environment.Theintegrationprocess,supportedbypayperuseandradicalelasticity,issmooth.ThebusinessdevelopercanstartconsumingonlysomeoftheMIDAStestingfunctionalityandendreplacingher/hisenvironmentwithMIDAS.Inanycase,usersareconfidentbecausetheycaneasilymodifythetypeandfrequencyofMIDASutilisationwithout paying extra-fees (if youdonothing, youpaynothing).Ofcourse,s/hecanstoptheMIDASutilisationatanymoment.

TheMIDAS APIs allow software engineering tool developers to add value totheirofferbyproposingtoaccessMIDAStestmethodsandfunctionalitiesdirectlyfromtheirtoolinanintegratedmanner.

The tools thatarecandidate for integrationwithMIDAScouldbeclassified inthefollowingcategories:

6AccordingtoaPonemonInstituteLLCstudy(http://www.ponemon.org/)focusedoncloudsecurity,66%ofbusinessesthatarefocusedoncloudsecurityputsensitiveinformationinthecloud,whileonly40%ofthebusinessesconsideredtobelessconcernedwithsecuritydothesame.7SuchasDropbox.Acreditcard(andaminimalbudget)sufficestouseMIDASservices.

WhitePaper


1. Modellingtools,

2. IntegratedDevelopmentEnvironments(IDEs),

3. ApplicationLifecycleManagement(ALM)tools,

4. Computer-AidedSoftwareTesting(CAST)tools,

5. SOAgovernancetools,

6. APImanagementtools.

Globallyspeaking,actorsofthiscategorycouldberesellersofMIDASservices.TheycanbealsoMIDAScompetitors(especiallyCASTtooleditors).Possibly,theMIDASservicescanbeconsumedin“OEM”mode,asiftheyweredirectlyofferedbythetool.Of course, a specific marketing strategy with detailed contractual policies must bedesigned and put in place towards this population. Partnerships with actors of thiscategorycreatevaluechains.

The third category of customers/partners includes vendors of businesspackagesandbusinessservicesinanybusinessdomain(starting,forinstance,withtheHealthandLogisticsdomains).ThiscategoryisrepresentedintheprojectconsortiumbyDedalus.

Today, in the digital service economy: (i) business packagesmust be able toprovide and to consume services through APIs, and (ii) business services (API) areconsumed inunexpectedmanners.The focusofMIDAS isservice integrationtesting,hencetheseactors:(i)mayutiliseMIDASforinternaltestingneeds,(ii)mayproposetotheircustomerstheMIDASfacility,packagedMIDASartefactsthatfacilitateMIDASusein a specific context and an accompanying support to integrate (and test theintegration of) their package or service within more general services architecture.These actors can be MIDAS resellers. Of course, also for this category, a specificmarketing strategy with detailed contractual policies must be designed and put inplace.Partnershipswithactorsofthiscategorycreatevaluechains.

Lastbutnotleast,theconsultancypartnerscanfostertheMIDASdiffusionandarekeyactorsoftheMIDASsuccess.ManybusinessescanbeinterestedintheMIDASservices,butdonothave the timeand thepersonnel forentering theMIDASworld.Two important issues must be managed: (i) the model-based approach to testautomationand(ii)theinteractionthroughAPI8.Onbothpoints,consultantscanhelpcustomers to put in place quickly effective testing with MIDAS. Consultants arefacilitators:itmustbemadeclearthatfirstlevelconsultancyontestingwithMIDASisnot thebusinessof theMIDASCompany9,whose job isbuildinge-learning toolsand

8AprobableevolutionoftheMIDASfacility,similarwiththeevolutionofAWS,isthatthemostcurrentutilisationsoftheMIDASfacilitywillbedriventhroughascriptinglanguagealongsidetheinvocationsoftheMIDASAPIsbytheclientsoftware.9 MIDAS Company is a generic and conceptual term to cover the who of “who will carry out theexploitation”. It could be an individual partner, sub-consortium, full consortium, joint venture, newcompany,thirdpartyoranyothertypeofentity.

WhitePaper


easeofuse,andthatcreatinganindependentMIDASexperts’ecosystemaroundtheMIDASfacilityisaimportantfactoroftheMIDASsuccess.

The MIDAS business maintains an ambivalent relationship with professionaltesters. On one side, MIDAS extreme automation makes obsolete many of theactivities that are today performed “by hand” by professional testers. On the otherside, professional testers could enter very quickly the MIDAS consultancy market,certainlyquickerthan“generic”ITconsultancies(thatareknownforhavinglowtestingcultureand competence).MIDASboosts theproductivityof theprofessional testers,and the smartest of them can take advantage of this situation and use MIDAS toenlargetheirmarketfootprint.

Of course, also for this category, a specific marketing strategy with detailedcontractualpoliciesmustbedesignedandputinplace.Partnershipswithactorsofthiscategorycreatevaluechains.

1.4.3 EvolutionaryportfolioofenhancedtestmethodsThefruitionmodeoftheMIDASfunctionalities(TAASoncloud,APIsaccessible

on the Internet) requires continuous evolution: MIDAS is a living service that shallcontinuouslyoffernew,uptodate,enhancedtestmethodsandfunctionalities.

The thirdMIDASkey innovationpoint is that theMIDAS facility isbuiltonanopen platform that is loosely coupled with the test methods that it hosts. Theresearchers and experts on testing and the developers of testing tools (testmethoddevelopers intheMIDASterminology)upload,registerand install theirtestmethods,developedontheirpremises,onasandboxoftheMIDASplatformthatallowssafelychecking, verifying and testing themethods.Once a rigorous certification process isachievedsuccessfully,thesetestmethodsaremadeavailabletotheMIDASusers.

MIDAS can become an attracting point for research and development abouttesting(forinstance,forcurrentandfutureEuropeanprojects10).Aspecificstrategyforcapturingnewadvances in the researchandpracticeof SOA/API testingandmakingthemavailableasnewenhancedtestmethodsthatfostertheautomationofeffectivetest generation and test run cyclesmust be put in place. It could be interesting tocreate a community of MIDAS test method developers, to organise test methodcontests, to develop consensus about how to measure test method efficacy andefficiency, etc..Apart from important technical problemsof testmethodverificationandvalidation,thisstrategymustclarifyissuessuchas:

• Whatabouttheintellectualpropertyofthesetestmethods?

• Howtorewardtheauthors?

• Whoisinchargeofthemaintenanceandevolution?

10ThereareongoingtalkswiththeFP7Prowessproject(http://www.prowessproject.eu/),asiblingofMIDAS.Thegeneralideaisthatsometestmethodsdevelopedwithinthisproject(thatdoesn’ttargetexclusivelyservicetestingasMIDASdoes)canbeuploadedontheMIDASplatformandbecomepartoftheMIDAStestmethodportfolio.Therearenointellectualpropertyconcernsaboutthisoperation,becausethesemethodsareopensource.

WhitePaper


Partnershipwithresearchersandexpertsontestingandwiththedevelopersoftestingtoolscreatesvaluechains.

1.4.4 ExtremetestingautomationLastbutnotleast,extremeSOAtestingautomationisthefinalMIDASobjective

and key innovation point. The other innovation points / objectives detailed in theprecedingparagraphsareinstrumentalwithrespecttothisone.

Services/APIs are the building blocks of the digital economy. Modernapplicationsarecompositeaggregatingnotonlycomponentsbutalsoprivate,partnerand public Services/APIs. We witness a progressive shift from ‘specify, design andimplement’componentstowards‘select,evaluateandintegrate’servicesthroughAPIs.

Services/APIsaretheenablersofcriticalbusinesstransactionsand,potentially,theweakestlinksinthesetransactions.ThequalityoftheAPIsthatabusinessproduceandconsumeisnowmoreimportantthanever-ifyouconsumeit,youownit.

The business impact of any application failure is the same regardless ofwhetherthefaultlieswithinthecomponentsdirectlydevelopedbythebusinessortheAPIs that the business consumes. Finger pointing does little to foster customersatisfactionandbrandloyalty.

Whenthereisnoalternativetodigitaleconomy,i.e.whenallrelevantbusinessprocesses become digital, and there are no more backup manual processes, allprocessesandtheservicesthatsupportthembecomecritical.

Testing the APIs that a business provide and consume, and the servicesarchitectures the business is involved in is the only means for increasing theconfidenceof the stakeholders. Test drivendevelopmentof services and test drivenincrementalintegrationofservicesarchitecturesarenomoreoptions.

With the rising adoption of agile development, incremental integration andcontinuousdelivery,changeisacontinuousprocess.SOA/APItestingshouldnotbeasingulareventanymore,butaSDLCcontinuousactivity.

Hereweare.Thequestion is:Why is service testso littlepracticedwith,asaconsequence, enormous costs of no testing for everybody, including the generalpublic?

Afterall,SOA/APInotesting isariskybusinessforbothserviceprovidersandconsumers: the business risk of the delivery of non-dependable services is high,becausewhentheserviceisyourbusiness,nondependableserviceisnondependablebusiness.

Theanswerhasbeengivenby thewell-knownNIST report [1]more than tenyearsago:becauseSOA/APItestingisexpensiveandhardandaneffectiveandcheap“infrastructuretechnology”11thatsupportstestingislacking.

11Theterm‘infrastructure’isutilisedhereinageneralsense.Inthissense,wecansaythattheMIDASfacilityisaninfrastructuretechnology.

WhitePaper


SOA/APItestingisexpensivebecause:

• it is a labour very intensive activity - manual test case/oracle design,manualconfigurationofthetestenvironment,eyeballarbitrationoftestoutcome, handmade test reporting, human-based scheduling of testruns,human-basedplanningoftestcycles;

• theneededequipment(hardware,licences,maintenance)issteep;

• withthecurrentlyavailable technologies, thedurationof thestandardqualitytestingphasesgoesfarbeyondthetimetomarket.

SOA/APItestingishardbecause:

• Thetestingactivityisbothlow-rewardingandknowledgeintensive.Theinvolved skills (deep functional knowledgeof the services architectureto be tested, competence of effective testingmethods and tools) arescarceresources.

• Human cognitive abilities are not well-adapted to an activity that, infrontofcomplexservicesarchitecture,requiressustainedattentionforlong periods and the ability to cope with an enormous quantity ofdetailedinformation.Theimmediateconsequenceisthatthisactivityiserror prone (useless test cases, wrong oracles, false positives, falsenegatives).

• Commercially available tools offer limited functionalities, i.e. themechanisation of few clerical tasks applied only to single-service unittesting.

• Testisdifficulttoplan,toscheduleandtomanage.

TheMIDASfacilityoffersthecompleteautomationofallthetestingtasksinthedomains of functional and security/vulnerability testing12. It automates all the basictestingtasksofthegenerationcycle(testcaseproduction,testoracleproduction)andof the run cycle (test execution, test arbitration, test reporting). Furthermore, itautomatesalso themiddle-management tasks (theworkflowof thegenerationcycleandtheschedulingofthetestruncycle).Finally,aMIDASobjectiveistheautomationofthecompletetestcycle.Moreover,thecomponentsthatdrivethehighleveltasks(theschedulerandtheplanner)areimplementedasBayesianagentsthatareabletocopewiththefundamentalcharacteristicsofthetestasanempiricalactivitybasedonheuristicsandperformedinanuncertainenvironment.

ExtremeSOA/API testing automation, coupledwith the key innovationpointspresented in theprecedingparagraphs (low-cost testing as a service, programmabletestingfacilityandevolutionarytestmethodportfolio)enablesaservicedevelopmentlife cycle that is model based, test driven and adapted to incremental serviceintegrationandtest.

12Automatedperformancetestingshouldcompletethecommercialoffer.PerformancetestingisnotinthescopeoftheMIDASproject.

WhitePaper


Extreme automation means that once the end user has built the SAUT structural,functional and behavioural models, the testing activity runs automatically,asynchronously and in background. The end user can focus on her/his job which isdesigninggreatservicesandMIDAStakescareofthecontinuoustestoftheseservices.

WhitePaper


CONCLUSION

Testingactivitiesareclosely linkedtothechallengesthatcompaniescurrentlyface:flexiblereactiontochangesonthemarketandcustomerside,fastintroductionofnewproductsthroughoptimized“time-to-market”,andgreaterefficiencyindeliveringservices.

TheMIDASSaaSapproach-andtheunderlyingcloudtechnology-downsizesofoneormoreorderofmagnitudesallthetestingequipmentcosts,makingtheMIDASsolution accessible to all the relevant stakeholders (users) in testing domain thattraditionally are excluded or perform very little testing activity(such as SMEs).Moreover,onthebasisofatypicalpricingpolicyforcloudcomputing,theMIDASSAASapproachdownsizesthelicensingcostsoftestmethodsandtoolstoo.Inotherterms,theMIDAStechnologydisruptstheSOAtestingtoolmarket,allowing“all”toenterthe“new”SOA/APItesting-as-a-servicemarket.

Risk and Impacts of Inadequate infrastructure for Service...

Documents

Transcript of Risk and Impacts of Inadequate infrastructure for Service...