Best Practices for Testing In-the-Cloud Security Products ... · cloud-based technologies, testing...

BestPracticesforTestingin-the-CloudSecurityProducts

Copyright©2016Anti-MalwareTestingStandardsOrganization,Inc.Allrightsreserved.Nopartofthisdocumentmaybereproducedinanyform,inanelectronicretrievalsystemorotherwise,withouttheprior

writtenconsentofthepublisher.

2

NoticeandDisclaimerofLiabilityConcerningtheUseofAMTSODocuments

ThisdocumentispublishedwiththeunderstandingthatAMTSOmembersaresupplyingthisinformationforgeneraleducationalpurposesonly.Noprofessionalengineeringoranyotherprofessionalservicesoradvice isbeingofferedhereby. Therefore,youmustuseyourownskillandjudgmentwhenreviewingthisdocumentandnotsolelyrelyontheinformationprovidedherein.

AMTSObelievesthattheinformationinthisdocumentisaccurateasofthedateofpublicationalthoughithasnotverifieditsaccuracyordeterminedifthereareanyerrors.Further,suchinformationissubjecttochangewithoutnoticeandAMTSOisundernoobligationtoprovideanyupdatesorcorrections.

Youunderstandandagreethat thisdocument isprovidedtoyouexclusivelyonanas-isbasiswithoutanyrepresentationsorwarrantiesofanykindwhetherexpress, impliedorstatutory. Without limitingthe foregoing, AMTSO expressly disclaims all warranties of merchantability, non-infringement,continuousoperation,completeness,quality,accuracyandfitnessforaparticularpurpose.

InnoeventshallAMTSObeliableforanydamagesorlossesofanykind(including,withoutlimitation,any lost profits, lost data or business interruption) arising directly or indirectly out of any use of thisdocument including, without limitation, any direct, indirect, special, incidental, consequential,exemplary and punitive damages regardless of whether any person or entity was advised of thepossibilityofsuchdamages.

Thisdocument isprotectedbyAMTSO’s intellectualpropertyrightsandmaybeadditionallyprotectedbytheintellectualpropertyrightsofothers.



3

BestPracticesforTestingin-the-CloudSecurityProducts

Introduction

Thisdocumentprovidesguidelinesfortestinganti-malwaresolutionswhichmakeuseof ‘in-the-cloud’technology. Its aim is to give an overview of the issues involved in the accurate testing of suchtechnologies, and how tests may be designed so as to produce valid and useful test results. Theseguidelinesarenotacomprehensivelistingofallsuchissues.Unlessotherwisedefinedherein,alltermsincluded in this document areusedwith their commonmeaning. The followingdocument shouldbereadinconjunctionwithAMTSO’sFundamentalPrinciplesofTesting,BestPracticesforDynamicTesting,andotherinformationavailableonwww.amtso.org.

In-the-CloudTechnologiesinAnti-MalwareSolutions

Throughout thisdocument, the terms ‘cloud’ and ‘in-the-cloud’ refer, respectively, to the internet (orotherresourcesexternaltoaprotectedsystem),andtoresourcesandtechnologiesrunorservedfromthere-onlinedetectiondatabases,reputationsystems,black-andwhitelists,managedservicesandsoon.

Such technologies present significant difficulties to testers. Tests of standalone products havetraditionallybeen run ina sealedenvironment– thisgives testersabsolutecontroloverallaspectsoftheir tests,allowingthemtobecompletelyrepeatableandreproduciblebyanyonewithaccesstothefull range of resources used in the original test. However, as solutionsmakemore andmore use ofcloud-based technologies, testing such solutions fully and fairly requires that they be given access totheseexternalresources,forcingthetestertocedecontrolofthetestenvironment.Essentially,runningsuch tests requires allowing external parties andmechanisms to influence the test environment, andtests can thus no longer be accurately or reliably reproduced.When running comprehensive tests ofmulti-layered solutions, tests can include a wide range of scenarios, referred to throughout thisdocumentas‘testcases’,andinmanysituationssomeaspectsofthetestcaseitselfmaynotbeunderthetester’scontrol.

Beyondthesimplelossofcontroloverthetestenvironment,anumberofotherfactorscomeintoplaywhen running tests connected to the live internet, many of which will impact the design andimplementation of appropriate tests. Here some of these issues are addressed, along with somesuggestedtechniquesforovercomingthem,orat leastminimizingtheir impactontheaccuracyoftestfindings.

TestEnvironmentsAreNoLongerControlledandReproducible

With the implementationof in-the-cloudtechnologies, it isno longerpossible for testers toprepareastandardtestenvironmentandtoruneachproductundertestinexactlythesameconditions,oneafteranother,with the option of rerunning a test in those same conditions at any time. Solutions utilizingweb-basedresourcescannotbe‘frozen’,astheseremoteresourcesoperatebeyondthetester’scontrol.



4

Tocomparesuchsolutionsinafairandbalancedway,thetraditionalmethodologyofdirectcomparisonofdetectionratesinstaticon-demandscansbecomesdifficulttoimplement,andadifferentapproachbasedonstatisticalmeasurementofperformanceagainstnon-identicalsetsoftestcasesmaybemoreappropriate.

DirectComparatives

In thedirectcomparisonmodel,allproductsunder testareexposed to thesameselectionof threats,usuallylargesetsofmalicioussamplesforstatictestingandsmallersetsfordynamictesting.Inbothofthese test types, when applied to solutions with resources hosted online, test cases should besynchronizedas closely aspossible, inorder to avoidbiasing the test. Suchabiasmaybe introducedwhere products are tested sequentially rather than in parallel, since products tested later in thesequencehavemoreopportunitytodiscover,analyzeandaddprotectionagainstagiventhreat.

Scanninglargesamplesetsmayintroduceafurtherbiasevenwhenscansarestartedsimultaneously,asslowerproductsagaingetmoretimetoupdatetheirprotection.Itwouldbepreferabletorunmultiplesmallerscans,orideallytotesteachsampleasaseparatescanortestcase.Thiswillofcoursebemoretime-consuming,butwillproducemoreaccurateresultswithlessriskofbias.

StatisticalComparatives

Evenwhen testsare synchronizedas closelyaspossible,anduse themostup-to-the-minute samples,there will inevitably be differences in the response time of solutions and the behavior of test casesthemselves,especiallywhentestingdynamicallyagainstactivemalwarewithitsownremoteresourcesand influences. For example, when a test includes a malicious sample served from a given URL, thenature of that sample at any given time is subject to changes outside the control of the tester. Theimpactoftheseanomaliesontestresultscanbereducedbyrunninglargenumbersoftestcasesoveranextendedperiod,andusingstatisticaldatatoprovideanaccuratereflectionofeachsolution’srelativeperformance.

When such statistical data is collected in large enough quantities, the need for direct comparison isreduced.Bymeasuringa solution'sperformanceagainst significantquantitiesof threatsovera longerperiodoftime,performancequalitycanbeaccuratelycomparedevenwhensolutionshavebeentestedagainst different sets of threats at different times, as the effect of anomalous test cases should beleveled out. This approach avoids many of the synchronicity issues raised by testing against largernumbers of test cases in a shorter time frame, and also enables the tester to performmore realisticdynamictesting.Suchtestingcanberuncontinuously,withperiodicreportingofresults,providinganindepthpictureofperformanceovertime.

Virtualization

Testingmultiple products simultaneously and over long periods is inevitablymore resource-intensivethanrunningmultipletestsinseriesonasmallnumberoftestsystemsinashortperiod.Insomecasesvirtualization may be an appropriate means of reducing the costs of such testing, but it may alsointroducefurtherproblemsasnotallsolutionsarefullysupportedinvirtualizedenvironments.Runningmultiple solutions in parallel in virtual environments on a single host also presents issues for speedmeasurement,asVMprioritizationmaymeanthatnotallguestsystemsarerunatthesamespeed.This



5

approachmay also impact tests in otherways, as competition for network and other resourcesmayaffect detection levels. It may bemore appropriate to take a statistical approach here also, runningmultiplespeedtestsatdifferenttimesandrecordingaveragespeedsforeachsolutionundertest.

In the case of dynamic testing, malware samples may also exhibit anomalous or uncharacteristicbehaviorsinvirtualizedenvironments,asdiscussedinthe‘AMTSOBestPracticesforDynamicTesting'.

TestEnvironmentsCannotBeSealedOffforSecurity

Oneoftheprimetenetsofanti-malwaretesting,‘firstdonoharm’,isoftenbestmaintainedbykeepingtestenvironmentssealedofffromexternalnetworks.Whentestingsolutionswithcomponentshostedin the cloud, it is no longer possible to maintain complete separation from the internet, and soalternative methods must be implemented to minimize the risk of unintended leakage of maliciousactivity.

ConnectionFiltering

The simplest way to allow solutions access to remote resources is to filter traffic at the gateway,allowing complete connectivity only to the specific URLs, domains, ports and protocols required bysolutions. Other traffic, including malicious test cases, can then be blocked, redirected, throttled orotherwise filtered according to the specificmalware-handlingpolicies of the individual tester and therequirementsofthetest.

Whendesigningsuchfiltering,it is importanttoconsiderthepropertiesoftheprotectiontechnologiesbeing tested, and thepossible impactof any variations from real-worldbehavior imposedby the testenvironment on the protection provided. Protection technologiesmay analyze a range of behavioralinformation includingURLs, domains and IP addresses communicatedwith, communicationprotocols,formats and patterns, and much else besides, and any manipulation of test case behavior, such asemulationofservicesorspoofingofonlineresources,mayimpacttheperformanceofthesolutionifnotimplementedwithcomplete transparencyandaccuracy.At thevery leasthowever, testenvironmentsshould not allow test cases to perform actions which could present a danger to the public, such asspamming,denial-of-serviceattacksorwormpropagation.

Solution developers are encouraged to share with testers any necessary information about onlineresources and communication protocols used by their solutions, and the types of behavioral dataanalyzed.

DataLeakage

Anothereffectofallowingsolutionstotransferdataoutsideoftestinglabsisapotentiallossofprivacy.Itmaynotbeappropriateforspecificdataheldintestlabstobeaccessibleexternally,includingdetailsof sample collections andotherpotentially confidential information, dependingon thepolicies of theindividualtestingbody.

Tominimize the problems posed by such potential data leakage, testersmay wish to obtain furtherinformationontheoperationofsolutionsundertest,suchaswhatkindofdataistransmittedexternally,andinwhatmanner;itmayalsobenecessarytodecryptorotherwiseanalyzetrafficbetweensolutions



6

andremoteresources,toensurethatnoprivatedataisleaked.Again,developersmaywishtoprovidetesterswithdetailsoftheinternalworkingsofsolutions,theirremoteresourcesandthecommunicationbetweenthetwo.

Testersmayfinditusefultomonitoralltrafficbetweentheirtestenvironmentsandremotelocations,togathertheirownindependentverificationofrequestsmadeandresponsesprovided.Suchdataonthetraffic producedby the communicationsof solutionsunder testmayprovide a usefulmeasureof thecomparativebandwidthloadimposedbydifferenttechnologies.

OtherIssues

TheInternetasPartofaTestEnvironment

Solutionswhichmake use of online resourcesmust access those resources via the internet,which isoutsidethecontrolofthetester.Variationsinconnectivitymayhaveanimpactonboththeprotectionprovidedand thespeedof solutions. Locationof the testnetwork in relation toonline resourcesmayaffectresponsetimes,soitmaybeusefulfortesterstopublishthelocationoftheirtestfacilities.Webconnectivityserviceprovidersmayemploymeasureswhich filteror throttleconnectionsbetweentestlabsandonlineresources,andappropriateserviceagreementsshouldbemadeensuringprovidersareawareoftheprobabilityofunusualtrafficpatternsfromtestlabs.

Changingprovidersandtest locationsduringtesting,whileperhapsdesirable,maybe impractical,andtheuseofin-the-cloudhostingservicestoruntestsmaybeawayofavoidinglocationandconnectivityissues, but such an approach brings its own issues. Testing on virtualized systems may affect thebehaviorofbothsolutionsandtestsamples,asdiscussedabove,andthepoliciesofhostingprovidersregarding the introduction of malware into their networks should always be consulted and carefullyobserved,particularlyindynamictesting.

ReliabilityofProtection

As well as speed implications, network connectivity can impose problems of resilience on solutionsunder test. Some solutions may offer dual modes of operation, with local detection technologiessupplemented by additional remote resources. Some test scenarios might usefully test resilience tonetwork problems, and compare the levels of protectionofferedwhen access to remote resources isremoved,asmayoccurduetointermittentnetworkingorintentionalblockingofconnectiontoasystemforsecuritypurposes.

Testing for full coverage of all malware types is also important when testing solutions using onlineresources. Online databases of known bad files allow amuch broader range of files to be identifiedspecifically, but such technologies may not provide full protection against some malware, such aspolymorphic viruses. When such samples are used in tests, they should be replicated in significantnumbers by testers, to properly test the capability of solutions to handle these kinds of malwareeffectively.



7

SampleSelection

Sampleselection isasignificant issue inall testmethodologies,andtofullytesttheresponsivenessofreal-time systems samples should be as ‘fresh’ as possible. Inmost tests,maximum freshness can beachievedbytestingsolutionsagainstallavailablesamplesandperformingsamplevalidationlater,withonlysuccessorfailureagainstprovenvalidsamplestakenintoconsiderationwhenreportingresults.

VersionInformation

Testreportsshould,wherepossible,providedetailedinformationonthespecificproductsandsolutionsincluded intests, includingversiondetails,sothattheiraudiencecantellexactlywhat isbeingtested.Whenrunningsolutionsoverlongperiods,withsomeportionsofthemmanagedremotely,suchversioninformationmaynotbeavailable,ormaychangefrequentlyasvariouscomponentsareupdated.Insuchcases,testersmaywishtoprovideversioninformationonlyfromthebeginningand/orendofthetest,alongwith details of the timeframeof the test. A clear policy on how thiswill be handled should bedefinedaspartoftestmethodology.

TrustRelationshipswithDevelopers

Itmaybeusefulfortestersandsolutionproviderstoreachaleveloftrust.Asdiscussedabove,detailsofthetechnologyandresourcesused inagivensolutionwillhelptestersmeasure itsperformancemoreaccurately, while safely controlling malicious samples. In some cases, solutions may respondanomalouslytotheunusualactivitiesofatestenvironment,particularlyduringtherunningoflargescalestatictests.Developersmaybeabletocorrectforthisifprovidedwithdetailsofatest,toensuretheirsolutionsbehaveastheywouldinanormalenvironment.

Itmayalsobeusefulfortesterstohaveaccesstologgingdatagatheredattheserversideofsolutions,andindeedsometestformatssuchasauditingrelyheavilyonsuchdata.Toenablesuchexchangesofsensitive information,trustrelationships,possibly includingformalnon-disclosureagreements,mayberequired.Trustbetweentestersandsolutionprovidersisbestfosteredbyopencommunication.OneofthecoreaimsofAMTSOistoencourageandfacilitatesuchopenness.

AnExampleMethodologyforTestingin-the-CloudSolutions

A sensible comparative testmethodology for host-based anti-malware solutions utilizing in-the-cloudtechnologymayincludethefollowingsteps:

• Thetestenvironmentisconfiguredtoallowalltrafficbetweentestsystemsandonlineresourcesprovided by their developers, using the specific domain/port/protocol requirements of theproduct.Thistrafficismonitoredandloggedatthetestenvironment’sgateway,andifsufficientinformationisavailabletotesters,maybedecryptedandparsed.

• Solutionsselectedfortestingneednotnecessarilyallmakeuseofin-the-cloudtechnologies,butthosewhichdonotshouldbegivenequalaccesstoexternalnetworksfortheirmoretraditionalupdatingetc.Allsolutionsundertestare installedon identicaltestsystems,usingaconsistentpolicyof setupandconfiguration.Amatchingcontrol system isusedasa reference for threatandsystembehavior.



8

• Thetrafficproducedbytestcasesiscontrolledandfilteredtoproperlymanagemalwarerisks,inaccordancewiththepoliciesoftheindividualtestbodyandtherequirementsofthespecifictesttype. This filtering is designed to minimize impact on threat behavior and solution dataexchange,includingsuchdetailsasIPaddressesandURLsusedbymalicioustestcases.

• Arrangements are made with providers of internet connectivity to ensure testing is notinterfered with, and ISPs may be changed periodically during the test. Where necessary toensureaccuratetesting,developersaremadeawareofthedesign,methodologyand/ortimingofthetest,sotheycanensurethatfulllogsarekeptandthatanomalyfilteringdoesnotbiastheperformanceofproducts.

• Eachsystem is thenexposed toa setof testcasescomparable inquantity, type, severity,andsignificance. These may be identical and synchronized as closely as possible, but mayalternativelyachievestatisticalequivalencythroughtherunningof largenumbersoftestcasesoveranextendedperiod,possiblyevencontinuously.

• Bothmaliciousandfalsepositivetestcasesareperformed.Themostrecentmalicioussamplesareused, forbestmeasurementofresponsivenessandhandlingofnewthreatsnotpreviouslyseenbysolutiondevelopers,andwherenecessarypost-testvalidationofsamplesisperformedtoensureonlydataonvalidthreatsisincludedintestreports.

• Dataontestresultsisgatheredfromtestsystemsandtestenvironmentgateways,andpossiblyalso provided by solution developers. After appropriate parsing and interpretation, data isstored for traceability and,where appropriate, provided to solution developers for their ownanalysis.Testresultsarepublished,makingclearthelocationofthetestenvironmentandotherfactorsaffectingnetworkedsolutions.

______________________________________________________________________________

ThisdocumentwasadoptedbyAMTSOonMay7,2009

Best Practices for Testing In-the-Cloud Security Products ... · cloud-based technologies, testing...

Documents

Transcript of Best Practices for Testing In-the-Cloud Security Products ... · cloud-based technologies, testing...