Chapters18‐20

Highlights&Summary

Tes9ng,Reliability,&Quality•  Tes9ngcannotproveaproducthasnoerrors

–  Itcanonlyfinderrorsand/orshowtheyhavebeenfixed–  …Orfailtofinderrors–  Itcannotproveaproductiserrorfree

•  Reliability–  probabilityoffailurefreeopera9onofacomputerprograminaspecified

environmentforaspecified9meperiod–  SoKwarereliabilityproblemscanusuallybetracedbacktoerrorsindesignor

implementa9on•  Sta9s9calQualityAssurance

–  Collect,categorizeandtraceeacherror–  Isolatethemaincauses

•  80%errorsfrom20%source•  Alsocalled“viralfewcauses”

–  Focusonfixingthoseerrorsources–  Result:80%reduc9oninerror

CS‐584/Emory2

SoKwareTes9ngObjec9ves

•  Tes9ngistheprocessofexecu9ngaprogramwiththeintentoffindingerrors.

•  Agoodtestcaseisonewithahighprobabilityoffindinganas‐yetundiscoverederror.

•  Asuccessfultestisonethatdiscoversanas‐yet‐undiscoverederror.

CS‐584/Emory3

WhiteBoxvsBlackBoxTes9ng

•  WhiteBox–  Knowledgeofinnerworkingsoftheprogram–  UnitTes9ngisanexample–  EVERYstatement/condi9ontestedatleastonce  Errorlikelihoodisinverselypropor9onaltopath’sexecu9onprobability

•  BlackBox–  Noimplicitknowledgeofinnerworkingsoftheprogram–  Independenttestersrun“blackbox”tests(theydon’tknowthecode)

•  Moderntes9nghasdroppedtheseterms–  Func9onal:Doesitdowhatitsaysitwilldo?–  Opera9onal:Doesitbehaveasexpected?–  Performance:Doesitscale?–  Generaluserinterfaceandsystemresponse–  BoundaryValueAnalysis(ifnormallimits=a,bthentesta±1andb±1)

CS‐584/Emory 4

GoodTestAkributes

•  Agoodtesthasahighprobabilityoffindinganerror.

•  Agoodtestisnotredundant.•  Agoodtestshouldbebestofbreed.•  Agoodtestshouldnotbetoosimpleortoocomplex.

CS‐584/Emory 5

ObjectOrientedSoKwareTestStrategies

•  the definition of testing must be broadened to include error discovery techniques applied to object-oriented analysis and design models

•  the strategy for unit and integration testing must change significantly, and

•  the design of test cases must account for the unique characteristics of OO software.

CS‐584/Emory 6

Tes9ngObject‐BasedSoKware

•  Thread-based testing –  integrates the set of classes required to respond to one

input or event for the system •  Use-based testing

–  begins the construction of the system by testing those classes (called independent classes) that use very few (if any) of server classes.

–  After the independent classes are tested, the next layer of classes, called dependent classes

•  Cluster testing [McG94] –  defines a cluster of collaborating classes (determined

by examining the CRC and object-relationship model) is exercised by designing test cases that attempt to uncover errors in the collaborations.

CS‐584/Emory 7

Object‐OrientedTes9ngApproach

•  Eachtestcaseshouldbeuniquelyiden9fiedandshouldbeexplicitlyassociatedwiththeclasstobetested,

•  Thepurposeofthetestshouldbestated,•  Alistoftes9ngstepsshouldbedevelopedforeachtestandshould

contain[BER93]:–  alistofspecifiedstatesfortheobjectthatistobetested–  alistofmessagesandopera9onsthatwillbeexercisedasa

consequenceofthetest–  alistofexcep9onsthatmayoccurastheobjectistested–  alistofexternalcondi9ons(i.e.,changesintheenvironmentexternal

tothesoKwarethatmustexistinordertoproperlyconductthetest)–  supplementaryinforma9onthatwillaidinunderstandingor

implemen9ngthetest.

CS‐584/Emory 8

Object‐OrientedTestMethods

•  Fault‐basedtes9ng–  Thetesterlooksforplausiblefaults(i.e.,aspectsofthe

implementa9onofthesystemthatmayresultindefects).Todeterminewhetherthesefaultsexist,testcasesaredesignedtoexercisethedesignorcode.

•  ClassTes9ngandtheClassHierarchy–  Inheritancedoesnotobviatetheneedforthoroughtes9ngofall

derivedclasses.Infact,itcanactuallycomplicatethetes9ngprocess.

•  Scenario‐BasedTestDesign–  Scenario‐basedtes9ngconcentratesonwhattheuserdoes,notwhat

theproductdoes.Thismeanscapturingthetasks(viause‐cases)thattheuserhastoperform,thenapplyingthemandtheirvariantsastests.

CS‐584/Emory 9

OOTMethods:BehaviorTes9ng

CS‐584/Emory 10

emptyacctopen setup Accnt

set upacct

deposit(initial)

workingacct

withdrawal(final)

deadacct close

nonworkingacct

deposit

withdrawbalance

creditaccntInfo

Figure 14.3 State diagram for Account class (adapted from [KIR94])

WebAppErrors•  Because many types of WebApp tests uncover problems that are first

evidenced on the client side, you often see a symptom of the error, not the error itself.

•  Because a WebApp is implemented in a number of different configurations and within different environments, it may be difficult or impossible to reproduce an error outside the environment in which the error was originally encountered.

•  Although some errors are the result of incorrect design or improper HTML (or other programming language) coding, many errors can be traced to the WebApp configuration.

•  Because WebApps reside within a client/server architecture, errors can be difficult to trace across three architectural layers: the client, the server, or the network itself.

•  Some errors are due to the static operating environment (i.e., the specific configuration in which testing is conducted), while others are attributable to the dynamic operating environment (i.e., instantaneous resource loading or time-related errors).

CS‐584/Emory 11

WebAppTes9ng‐1

•  Contentisevaluatedatbothasyntac9candseman9clevel.–  syntac9clevel—spelling,punctua9onandgrammarareassessed

fortext‐baseddocuments.–  seman9clevel—correctness(ofinforma9onpresented),

consistency(acrosstheen9recontentobjectandrelatedobjects)andlackofambiguityareallassessed.

•  Func)onistestedforcorrectness,instability,andgeneralconformancetoappropriateimplementa9onstandards(e.g.,JavaorXMLlanguagestandards).

•  Structureisassessedtoensurethatit–  properlydeliversWebAppcontentandfunc9on–  isextensible–  canbesupportedasnewcontentorfunc9onalityisadded.

CS‐584/Emory 12

WebAppTes9ng‐2

•  Usabilityistestedtoensurethateachcategoryofuser–  issupportedbytheinterface–  canlearnandapplyallrequirednaviga9onsyntaxandseman9cs

•  Navigabilityistestedtoensurethat–  allnaviga9onsyntaxandseman9csareexercisedtouncoverany

naviga9onerrors(e.g.,deadlinks,improperlinks,erroneouslinks).

•  Performanceistestedunderavarietyofopera9ngcondi9ons,configura9ons,andloadingtoensurethat–  thesystemisresponsivetouserinterac9on–  thesystemhandlesextremeloadingwithoutunacceptableopera9onal

degrada9on

CS‐584/Emory 13

WebAppTes9ng‐3•  Compa)bility

–  testedbyexecu9ngtheWebAppinavarietyofdifferenthostconfigura9onsonboththeclientandserversides.

–  Theintentistofinderrorsthatarespecifictoauniquehostconfigura9on.

•  Interoperability–  testedtoensurethattheWebAppproperlyinterfaceswithotherapplica9onsand/ordatabases.

•  Security–  testedbyassessingpoten9alvulnerabili9esandakemp9ngtoexploiteach.

–  Anysuccessfulpenetra9onakemptisdeemedasecurityfailure.

CS‐584/Emory 14

AssessingContentSeman9cs •  Istheinforma9onfactuallyaccurate?•  Istheinforma9onconciseandtothepoint?•  Isthelayoutofthecontentobjecteasyfortheusertounderstand?•  Caninforma9onembeddedwithinacontentobjectbefoundeasily?•  Haveproperreferencesbeenprovidedforallinforma9onderivedfrom

othersources?•  Istheinforma9onpresentedconsistentinternallyandconsistentwith

informa9onpresentedinothercontentobjects?•  Isthecontentoffensive,misleading,ordoesitopenthedoortoli9ga9on?•  Doesthecontentinfringeonexis9ngcopyrightsortrademarks?•  Doesthecontentcontaininternallinksthatsupplementexis9ngcontent?

Arethelinkscorrect?•  Doestheaesthe9cstyleofthecontentconflictwiththeaesthe9cstyleof

theinterface?

15CS‐584/Emory

DatabaseTes9ng

16CS‐584/Emory

client layer - user interface

server layer - WebApp

server layer - data transformation

database layer - data access

server layer - data management

database

HTML scripts

user data SQL

user data

SQLraw data

UserInterfaceTes9ng •  Interfacefeaturesaretestedtoensurethatdesignrules,

aesthe9cs,andrelatedvisualcontentisavailablefortheuserwithouterror.

•  Individualinterfacemechanismsaretestedinamannerthatisanalogoustounittes9ng.

•  Eachinterfacemechanismistestedwithinthecontextofause‐caseorNSUforaspecificusercategory.

•  Thecompleteinterfaceistestedagainstselecteduse‐casesandNSUstouncovererrorsintheseman9csoftheinterface.

•  Theinterfaceistestedwithinavarietyofenvironments(e.g.,browsers)toensurethatitwillbecompa9ble.

17CS‐584/Emory

Tes9ngInterfaceMechanisms‐I •  Links—naviga9onmechanismsthatlinktheusertosome

othercontentobjectorfunc9on.•  Forms—astructureddocumentcontainingblankfieldsthat

arefilledinbytheuser.ThedatacontainedinthefieldsareusedasinputtooneormoreWebAppfunc9ons.

•  Client‐sidescrip)ng—alistofprogrammedcommandsinascrip9nglanguage(e.g.,Javascript)thathandleinforma9oninputviaformsorotheruserinterac9ons

•  DynamicHTML—leadstocontentobjectsthataremanipulatedontheclientsideusingscrip9ngorcascadingstylesheets(CSS).

•  Client‐sidepop‐upwindows—smallwindowsthatpop‐upwithoutuserinterac9on.Thesewindowscanbecontent‐orientedandmayrequiresomeformofuserinterac9on.

18CS‐584/Emory

Tes9ngInterfaceMechanisms‐II•  CGIscripts—acommongatewayinterface(CGI)scriptimplementsa

standardmethodthatallowsaWebservertointeractdynamicallywithusers(e.g.,aWebAppthatcontainsformsmayuseaCGIscripttoprocessthedatacontainedintheformonceitissubmikedbytheuser).

•  Streamingcontent—ratherthanwai9ngforarequestfromtheclient‐side,contentobjectsaredownloadedautoma9callyfromtheserverside.Thisapproachissome9mescalled“push”technologybecausetheserverpushesdatatotheclient.

•  Cookies—ablockofdatasentbytheserverandstoredbyabrowserasaconsequenceofaspecificuserinterac9on.ThecontentofthedataisWebApp‐specific(e.g.,useriden9fica9ondataoralistofitemsthathavebeenselectedforpurchasebytheuser).

•  Applica)onspecificinterfacemechanisms—includeoneormore“macro”interfacemechanismssuchasashoppingcart,creditcardprocessing,orashippingcostcalculator.

19CS‐584/Emory

UsabilityTests •  DesignbyWebEteam…executedbyend‐users•  Tes9ngsequence…

–  Defineasetofusabilitytes9ngcategoriesandiden9fygoalsforeach.–  Designteststhatwillenableeachgoaltobeevaluated.–  Selectpar9cipantswhowillconductthetests.–  Instrumentpar9cipants’interac9onwiththeWebAppwhiletes9ngis

conducted.–  DevelopamechanismforassessingtheusabilityoftheWebApp

•  Differentlevelsofabstrac9on:–  theusabilityofaspecificinterfacemechanism(e.g.,aform)canbe

assessed–  theusabilityofacompleteWebpage(encompassinginterface

mechanisms,dataobjectsandrelatedfunc9ons)canbeevaluated–  theusabilityofthecompleteWebAppcanbeconsidered.

20CS‐584/Emory

Compa9bilityTes9ng •  Compa9bilitytes9ngistodefineasetof“commonlyencountered”client

sidecompu9ngconfigura9onsandtheirvariants•  Createatreestructureiden9fying

–  eachcompu9ngplaxorm–  typicaldisplaydevices–  theopera9ngsystemssupportedontheplaxorm–  thebrowsersavailable–  likelyInternetconnec9onspeeds–  similarinforma9on.

•  Deriveaseriesofcompa9bilityvalida9ontests–  derivedfromexis9nginterfacetests,naviga9ontests,performancetests,and

securitytests.–  intentofthesetestsistouncovererrorsorexecu9onproblemsthatcanbe

tracedtoconfigura9ondifferences.

21CS‐584/Emory

Component‐LevelTes9ng

•  FocusesonasetofteststhatakempttouncovererrorsinWebAppfunc9ons

•  Conven9onalblack‐boxandwhite‐boxtestcasedesignmethodscanbeused

•  Databasetes9ngisoKenanintegralpartofthecomponent‐tes9ngregime

22CS‐584/Emory

Naviga9onTes9ng •  Thefollowingnaviga9onmechanismsshouldbetested:

–  Naviga)onlinks—thesemechanismsincludeinternallinkswithintheWebApp,externallinkstootherWebApps,andanchorswithinaspecificWebpage.

–  Redirects—theselinkscomeintoplaywhenauserrequestsanon‐existentURLorselectsalinkwhosedes9na9onhasbeenremovedorwhosenamehaschanged.

–  Bookmarks—althoughbookmarksareabrowserfunc9on,theWebAppshouldbetestedtoensurethatameaningfulpage9tlecanbeextractedasthebookmarkiscreated.

–  Framesandframesets—testedforcorrectcontent,properlayoutandsizing,downloadperformance,andbrowsercompa9bility

–  Sitemaps—Eachsitemapentryshouldbetestedtoensurethatthelinktakestheusertothepropercontentorfunc9onality.

–  Internalsearchengines—Searchenginetes9ngvalidatestheaccuracyandcompletenessofthesearch,theerror‐handlingproper9esofthesearchengine,andadvancedsearchfeatures

23CS‐584/Emory

Tes9ngNaviga9onSeman9cs‐I •  IstheNSUachievedinitsen9retywithouterror?•  Iseverynaviga9onnode(definedforaNSU)reachablewithinthecontext

ofthenaviga9onpathsdefinedfortheNSU?

•  IftheNSUcanbeachievedusingmorethanonenaviga9onpath,haseveryrelevantpathbeentested?

•  Ifguidanceisprovidedbytheuserinterfacetoassistinnaviga9on,aredirec9onscorrectandunderstandableasnaviga9onproceeds?

•  Isthereamechanism(otherthanthebrowser‘back’arrow)forreturningtotheprecedingnaviga9onnodeandtothebeginningofthenaviga9onpath.

•  Domechanismsfornaviga9onwithinalargenaviga9onnode(i.e.,alongwebpage)workproperly?

•  Ifafunc9onistobeexecutedatanodeandtheuserchoosesnottoprovideinput,cantheremainderoftheNSUbecompleted?

24CS‐584/Emory

Tes9ngNaviga9onSeman9cs‐II •  Ifafunc9onisexecutedatanodeandanerrorinfunc9on

processingoccurs,cantheNSUbecompleted?

•  Isthereawaytodiscon9nuethenaviga9onbeforeallnodeshavebeenreached,butthenreturntowherethenaviga9onwasdiscon9nuedandproceedfromthere?

•  Iseverynodereachablefromthesitemap?Arenodenamesmeaningfultoend‐users?

•  IfanodewithinanNSUisreachedfromsomeexternalsource,isitpossibletoprocesstothenextnodeonthenaviga9onpath.Isitpossibletoreturntothepreviousnodeonthenaviga9onpath?

•  Doestheuserunderstandhisloca9onwithinthecontentarchitectureastheNSUisexecuted?

25CS‐584/Emory

Configura9onTes9ng •  Server‐side

–  IstheWebAppfullycompa9blewiththeserverOS?–  Aresystemfiles,directories,andrelatedsystemdatacreatedcorrectly

whentheWebAppisopera9onal?–  Dosystemsecuritymeasures(e.g.,firewallsorencryp9on)allowthe

WebApptoexecuteandserviceuserswithoutinterferenceorperformancedegrada9on?

–  HastheWebAppbeentestedwiththedistributedserverconfigura9on(ifoneexists)thathasbeenchosen?

–  IstheWebAppproperlyintegratedwithdatabasesoKware?IstheWebAppsensi9vetodifferentversionsofdatabasesoKware?

–  Doserver‐sideWebAppscriptsexecuteproperly?–  Havesystemadministratorerrorsbeenexaminedfortheiraffecton

WebAppopera9ons?–  Ifproxyserversareused,havedifferencesintheirconfigura9onbeen

addressedwithon‐sitetes9ng?

26CS‐584/Emory

Configura9onTes9ng

•  Client‐side–  Hardware—CPU,memory,storageandprin9ngdevices–  Opera)ngsystems—Linux,MacintoshOS,MicrosoKWindows,a

mobile‐basedOS–  BrowsersoMware—InternetExplorer,Mozilla/Netscape,Opera,Safari,

andothers–  Userinterfacecomponents—Ac9veX,Javaappletsandothers–  Plug‐ins—QuickTime,RealPlayer,andmanyothers–  Connec)vity—cable,DSL,regularmodem,T1

•  Thenumberofconfigura9onvariablesmustbereducedtoamanageablenumber

27CS‐584/Emory

SecurityTes9ng

•  Designedtoprobevulnerabili9esoftheclient‐sideenvironment,thenetworkcommunica9onsthatoccurasdataarepassedfromclienttoserverandbackagain,andtheserver‐sideenvironment

•  Ontheclient‐side,vulnerabili9escanoKenbetracedtopre‐exis9ngbugsinbrowsers,e‐mailprograms,orcommunica9onsoKware.

•  Ontheserver‐side,vulnerabili9esincludedenial‐of‐serviceakacksandmaliciousscriptsthatcanbepassedalongtotheclient‐sideorusedtodisableserveropera9ons

28CS‐584/Emory

PerformanceTes9ng•  Doestheserverresponse9medegradetoapointwhereitisno9ceable

andunacceptable?•  Atwhatpoint(intermsofusers,transac9onsordataloading)does

performancebecomeunacceptable?•  Whatsystemcomponentsareresponsibleforperformancedegrada9on?•  Whatistheaverageresponse9meforusersunderavarietyofloading

condi9ons?•  Doesperformancedegrada9onhaveanimpactonsystemsecurity?•  IsWebAppreliabilityoraccuracyaffectedastheloadonthesystem

grows?•  Whathappenswhenloadsthataregreaterthanmaximumservercapacity

areapplied?

29CS‐584/Emory

LoadTes9ng•  TheintentistodeterminehowtheWebAppanditsserver‐sideenvironmentwillrespondtovariousloadingcondi9ons–  N,thenumberofconcurrentusers–  T,thenumberofon‐linetransac9onsperunitof9me–  D,thedataloadprocessedbytheserverpertransac9on

•  Overallthroughput,P,iscomputedinthefollowingmanner:

•  P=NxTxD

30CS‐584/Emory

StressTes9ng•  Doesthesystemdegrade‘gently’ordoestheservershutdownascapacity

isexceeded?•  DoesserversoKwaregenerate“servernotavailable”messages?More

generally,areusersawarethattheycannotreachtheserver?•  Doestheserverqueuerequestsforresourcesandemptythequeueonce

capacitydemandsdiminish?•  Aretransac9onslostascapacityisexceeded?•  Isdataintegrityaffectedascapacityisexceeded?•  WhatvaluesofN,T,andDforcetheserverenvironmenttofail?Howdoesfailure

manifestitself?Areautomatedno9fica9onssenttotechnicalsupportstaffattheserversite?

•  Ifthesystemdoesfail,howlongwillittaketocomebackon‐line?•  ArecertainWebAppfunc9ons(e.g.,computeintensivefunc9onality,data

streamingcapabili9es)discon9nuedascapacityreachesthe80or90percentlevel?

31CS‐584/Emory