Data-Driven and Keyword-Driven Test[1]
-
Upload
aabha-khandekar-madhugiri -
Category
Documents
-
view
219 -
download
0
Transcript of Data-Driven and Keyword-Driven Test[1]
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
1/602
HELSINKIUNIVERSITYOFTECHNOLOGYDepartmentofComputerScienceandEngineeringSoftwareBusinessandEngineeringInstitute
PekkaLaukkanen
Data-DrivenandKeyword-Driven
TestAutomationFrameworks
Mastersthesissubmittedinpartialfulfillmentofthe
requirementsforthedegreeofMasterofScienceinTechnology.
Espoo,
February24,2006
Supervisor:ProfessorReijoSulonen
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
2/602
Instructor:HarriTohonen,M.Sc.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
3/602
HELSINKIUNIVERSITYABSTRACTOFTHEOFTECHNOLOGYMASTERSTHESIS
Author:PekkaLaukkanenNameofthethesis:Data-DrivenandKeyword-DrivenTestAutomation
FrameworksDate:February24,2006Numberofpages:98+0Department:Department
ofComputerProfessorship:T-76ScienceandEngineeringSupervisor:Prof.ReijoSulonenInstructor:Harri
Tohonen,M.Sc.Thegrowingimportanceandstringentqualityrequirementsof
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
4/602
softwaresystemsareincreasingdemandforefficientsoftwaretesting.Hiringmoretestengineersorlengtheningthetestingtimearenotviablelong-termsolutions,rather
thereisaneedtodecreasetheamountofresourcesneeded.Oneattractive
solutiontothisproblemistestautomation,i.e.allocatingcertaintestingtasksto
computers.Therearecountlessapproachestotestautomation,andthey
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
5/602
workdifferentlyindifferentcontexts.Thismastersthesisfocusesononlyoneofthem,large-scaleframeworksforautomatedtestexecutionandreporting,butother
keyapproachesarealsobrieflyintroduced.Thethesisopensitsdiscussionoftest
automationframeworksbydefiningtheirhigh-levelrequirements.Themostimportantrequirementsareidentified
asease-of-use,maintainabilityand,ofcourse,theabilitytoautomatically
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
6/602
executetestsandreportresults.Moredetailedrequirementsarederivedfromthesehigh-levelrequirements:data-drivenandkeyword-driventestingtechniques,forexample,areessentialprerequisites
forbothease-of-useandmaintainability.Thenextstepinthethesisis
constructingandpresentingaframeworkconceptfulfillingthedefinedrequirements.Theconceptand
itsunderlyingrequirementsweretestedinapilotwherea
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
7/602
prototypeoftheframeworkandsomeautomatedtestsfordifferentsystemswereimplemented.Basedonthepilotresults,theoverallframeworkconceptwasfound
tobefeasible.Certainchangestotheframeworkandoriginalrequirementsarepresented,
however.Themostinterestingfindingisthatitispossibletocoverall
thedata-driventestingneedswiththekeyword-drivenapproachalone.Keywords:
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
8/602
testautomation,testautomationframework,data-driventesting,keyword-driventesting
ii
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
9/602
TEKNILLINENKORKEAKOULUDIPLOMITYONTIIVISTELMA
Tekija:PekkaLaukkanenTyonnimi:Aineisto-jaavainsanaohjatuttestiautomaatiojarjestelmatPaivamaara:24.2.2006Sivuja:98+
0Osasto:TietotekniikanosastoProfessuuri:T-76Tyonvalvoja:Prof.ReijoSulonenTyonohjaaja:
DIHarriTohonenOhjelmistojarjestelmienmerkityksensekalaatuvaatimustenkasvaminenaiheuttaapaineitaohjelmistojentestaukselle.Testaajien
maaranlisaaminentaitestausajanpi-dentamineneioleloputtomastimahdollista,
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
10/602
pikemminkinresurssejahalutaanvahentaa.Yksihoukuttelevaratkaisuontestauksenautomatisointieliosantestaustyonantaminentietokoneidenhoidettavaksi.Erilaisiatapojatestauksenautomatisointiinonlukuisiajane
toimivateritavoinerilaisissatilanteissajaymparistoissa.Tamadiplomityokasitteleetarkemminvainyhta
lahestymistapaa,laajojaautomaatiojarjestelmiatestienautomaattiseensuorittamiseenjaraportoimiseen,muttamyosmuuttavatovat
tarkeita.Testiautomaatiojarjestelmienkasittelyaloitetaanmaarittelemallaniillekorkeantasonvaatimukset.Tarkeimmiksi
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
11/602
vaatimuksiksitodetaanhelppokayttoisyys,yllapidettavyyssekatietenkinkykyautomaattisestisuorittaatestejajaraportoidaniidentulokset.Naidenvaatimustenkauttapaastaantarkempiinvaatimuksiinjatodetaanmm.etta
aineisto-ohjattu(data-driven)jaavainsanaohjattu(keyword-driven)testaustekniikkaovatedellytyksiasekahelppokayttoisyydelleettayllapidettavyydelle.Seuraavaksi
tyossasuunnitellaanmaaritellytvaatimuksettoteuttavatestiautomaatiojarjestelma.Jarjestelmantoimintaasekasenpohjanaolleita
vaatimuksiatestataanpilotissa,jossatoteutetaansekaprototyyppiitsejarjestelmastaetta
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
12/602
automatisoitujatestejaerilaisilleohjelmistoille.Pilotintuloksienperusteellasuunnitellunautomaatiojarjestelmanvoidaantodetaolevanpaaperiaatteiltaantoimiva.Lisaksikokemustenperusteellajarjestelmaansekaalkuperaisiinvaatimuksiinesitetaanjoitain
muutoksia.Mielenkiintoisinloydosonseettakaikkiaineisto-ohjatuttestitvoidaantoteuttaakayttaenainoastaan
avainsanaohjattualahestymistapaa.Avainsanat:testiautomaatio,testiautomaatiojarjestelma,aineisto-ohjattutestaus,avainsanaohjattutestaus
iii
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
13/602
Acknowledgements
ThismastersthesishasbeendoneforaFinnishsoftwaretestingconsultancycompanyQentinelmainlyduringtheyear2005.I
wishtothankmyinstructorHarriTohonen,M.Sc.andallotherQentineliansfor
comments,feedbackandpatience.
FromtheDepartmentofComputerScienceand
EngineeringIfirstofallwanttothankmyinstructor
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
14/602
ProfessorReijoSulonen.AdditionallyIamgratefulforJuhaItkonenandCasperLasseniusfortheirsupportandvaluablecomments.
Ialsowant
toexpressmygratitudetoMarkFewsterwhowaskindenoughtoreview
thethesisinitsearlyform.Markscommentsandpositivefeedbackmademe
believethattheideasIpresentarevalidandthe
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
15/602
remaininghardworkisworththeeffort.AlsoIwanttothankPetriHaapiowhohasbeeninchargeofboththeautomationproject
whereIgottheoriginalideaforthisthesisandanewone
wheretheautomationframeworkpresentedinthisthesishasbeensuccessfullytakeninto
realuse.
Finally,Iwouldliketothank
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
16/602
myfamilyandmywonderfulgirlfriendSonjaforeverything.Kiitos.
Espoo,February24,2006
PekkaLaukkanen
iv
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
17/602
Contents
Termsix
1Introduction1
1.1PromisesandProblemsofTestAutomation.........
..
.
.
.2
1.2DifferentTestAutomationApproaches...
..
.
.
.
.
.
.
.
.
.
.
.
.2
1.2.1Dynamicvs.StaticTesting...................5
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
18/602
1.2.2Functionalvs.Non-FunctionalTesting.............5
1.2.3GranularityoftheTestedSystem................6
1.2.4TestingActivities.........................8
1.2.5SmallScalevs.LargeScale
TestAutomation.........9
1.3Scope...................................11
1.4Methodology...............................
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
19/602
12
1.5Goals...................................12
1.6Structure.................................13
2RequirementsforTestAutomationFrameworks14
2.1HighLevelRequirements........................14
2.2FrameworkCapabilities.........................14
2.2.1
ExecutingTestsUnattended...................15
2.2.2StartingandStoppingTestExecution...
.
.
.
.
.
.
.
.
.
.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
20/602
15
2.2.3HandlingErrors.........................15
2.2.4VerifyingTestResults......................16
2.2.5AssigningTestStatus......................16
2.2.6HandlingExpectedFailures...................
16
2.2.7DetailedLogging.........................17
v
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
21/602
2.2.8AutomaticReporting......................20
2.3Modularity................................21
2.3.1LinearTestScripts........................21
2.3.2TestLibrariesandDriverScripts...
..
.
.
.
.
.
.
.
.
.
.
.
22
2.3.3PromisesandProblems.....................23
2.4Data-DrivenTesting...........................23
2.4.1Introduction...........................23
2.4.2EditingandStoringTestData
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
22/602
.................24
2.4.3ProcessingTestData.......................26
2.4.4PromisesandProblems.....................26
2.5Keyword-DrivenTesting.........................27
2.5.1Introduction...........................
27
2.5.2EditingandStoringTestData.................27
2.5.3ProcessingTestData.......................29
2.5.4KeywordsinDifferentLevels..................29
2.5.5PromisesandProblems.....................31
2.6OtherImplementationIssues......................31
2.6.1ImplementationLanguage....................32
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
23/602
2.6.2ImplementationTechnique....................34
2.6.3TestwareArchitecture......................34
2.7Testability.................................36
2.7.1Control..............................36
2.7.2Visibility.............................38
2.8Roles....................................38
2.8.1TestAutomationManager....................
39
2.8.2TestAutomationArchitect...................39
2.8.3TestAutomator.........................40
2.8.4TestDesigner...........................40
2.9DetailedRequirements
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
24/602
..........................40
2.10ChapterSummary............................42
3ConceptforLargeScaleTestAutomationFrameworks43
3.1FrameworkStructure...........................43
vi
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
25/602
3.1.1TestDesignSystem.......................43
3.1.2TestMonitoringSystem.....................44
3.1.3TestExecutionSystem......................44
3.2PresentingandProcessingData-DrivenTest
Data..........47
3.2.1PresentingTestCases......................48
3.2.2UsingTestData.........................50
3.2.3Example..............................51
3.3PresentingandProcessingKeyword-Driven
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
26/602
TestData........55
3.3.1PresentingTestCases......................55
3.3.2PresentingUserKeywords....................57
3.3.3UsingTestData.........................58
3.4ChapterSummary............................61
4
ImplementationandPilot62
4.1ImplementationDecisions........................62
4.1.1
TechnicalDecisions........................62
4.1.2DecisionsRegardingthePilot..................63
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
27/602
4.2ImplementingReusableFrameworkComponents...........64
4.2.1TestDataParser.........................64
4.2.2Logger...............................66
4.2.3Summary.............................67
4.3
Data-DrivenWindowsApplicationTesting.........
.
.
.
.
.
.68
4.3.1
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
28/602
TestData.............................68
4.3.2DriverScript...........................68
4.3.3TestLibrary...........................71
4.3.4TestLog..............................71
4.3.5Summary
.............................71
4.4Keyword-DrivenWindowsApplicationTesting....
..
.
.
.
.
.
.72
4.4.1TestData
.............................72
4.4.2DriverScript...........................72
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
29/602
4.4.3TestLibrary...........................76
4.4.4TestLog..............................76
4.4.5Summary.............................76
vii
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
30/602
4.5Keyword-DrivenWebTesting......................79
4.5.1TestData.............................79
4.5.2DriverScript...........................79
4.5.3TestLibrary...........................81
4.5.4TestLog..............................81
4.5.5Summary.............................81
4.6
ChapterSummary............................85
5Results86
5.1FeasibilityoftheFrameworkConcept.
.
.
.
.
.
.
.
.
.
.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
31/602
.
.
.
.
.
.86
5.2ChangestotheFrameworkandRequirements........
..
.
.
.87
5.2.1UsingOnlyKeyword-DrivenApproach
..
.
.
.
.
.
.
.
.
.
.
.
87
5.2.2SetUpandTearDown.....................89
5.2.3
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
32/602
TestSuites............................90
5.2.4GenericDriverScript......................90
5.3RevisedListofRequirements......................91
5.4ChapterSummary............................91
6
Conclusions93
Bibliography95
viii
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
33/602
Terms
AcceptanceTesting
ActionWordActualOutcome
AutomationAutomationFrameworkBaseKeyword
Black-BoxTesting
BugCaptureandReplay
Component
ComponentTesting
Aleveloftestingconductedfromtheviewpointofthecustomer,used
toestablishthecriteriaforacceptanceofasystem.Typically
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
34/602
basedupontherequirementsofthesystem.(CraigandJaskiel,2002)
Seekeyword.
Outputsanddatastatesthatthesystem
undertestproducesfromtestinputs.Seealsoexpectedoutcome.(FewsterandGraham,
1999)
Seetestautomation.
Seetestautomationframework.
Atermdefinedinthisthesisforkeywordsimplemented
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
35/602
inatestlibraryofakeyword-driventestautomationframework.Seealsouserkeyword.
Atypeoftestingwheretheinternalworkings
ofthesystemareunknownorignored.Testingtoseeifthesystem
doeswhatitissupposedtodo.(CraigandJaskiel,2002)
Seedefect.
Ascriptingapproachwhereatest
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
36/602
toolrecordstestinputasitissenttothesoftwareundertest.Theinputcasesstoredcanthenbeusedtoreproducethe
testatalatertime.Oftenalsocalledrecordandplayback.(BS7925-1)
Oneofthepartsthatmakeupasystem.Acollection
ofunitswithadefinedinterfacetowardsothercomponents.(IEEE
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
37/602
Std610.12-1990)
Testingofindividualcomponentsorgroupsofrelatedcomponents.(IEEEStd610.12-1990)
ix
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
38/602
Context-DrivenTestingAtestingmethodologythatunderlinestheimportanceofthecontextwheredifferenttestingpracticesareusedoverthepracticesthemselves.The
mainmessageisthattherearegoodpracticesinacontextbutthere
arenogeneralbestpractices.(Kaneretal.,2001)ControlScriptSeedriver
script.Data-DrivenTestingAscriptingtechniquethatstorestestinputs
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
39/602
andexpectedoutcomesasdata,normallyinatabularformat,sothatasingledriverscriptcanexecuteallofthedesignedtestcases.
(FewsterandGraham,1999)DefectIntroducedintosoftwareastheresultofan
error.Aflawinthesoftwarewithpotentialtocauseafailure.Also
calledfaultor,informally,bug.(CraigandJaskiel,2002;Burnstein,
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
40/602
2003)DomaincodePartoftheapplicationcodewhichcontainssystemfunctionality.Seealsopresentationcode.(Fowler,2001)DynamicTestingTheprocessofevaluating
asystemorcomponentbasedonitsbehaviorduringexecution.Seealsostatic
testing.(IEEEStd610.12-1990)DriverAsoftwaremodulethatinvokesand,perhaps,controls
andmonitorstheexecutionofoneormoreothersoftware
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
41/602
modules.(IEEEStd610.12-1990)DriverScriptAtestscriptthatdrivesthetestexecutionprocessusingtestingfunctionalityprovidedbytestlibrariesandmay
alsoreadtestdatafromexternalsources.CalledacontrolscriptbyFewster
andGraham(1999).ErrorAmistake,misconception,ormisunderstandingonthepartof
asoftwaredeveloper.(Burnstein,2003)ExpectedFailureOccurswhena
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
42/602
testcasewhichhasfailedpreviouslyfailsagainsimilarly.DerivedfromFewsterandGraham(1999).ExpectedOutcomeOutputsanddatastatesthatshouldresult
fromexecutingatest.Seealsoactualoutcome.(FewsterandGraham,1999)Failure
Inabilityofasoftwaresystemorcomponenttoperformitsrequiredfunctionwithin
specifiedperformancecriteria.Themanifestationofadefect.(IEEEStd
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
43/602
610.12-1990;CraigandJaskiel,2002)
x
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
44/602
FaultSeedefect.FunctionalTestingTestingconductedtoevaluatethecomplianceofasystemorcomponentwithspecifiedfunctionalrequirements.(IEEEStd610.12-1990)
FeatureAsoftwarecharacteristicspecifiedorimpliedbyrequirementsdocumentation.(IEEEStd610.12-1990)
FrameworkAnabstractdesignwhichcanbeextendedbyaddingmoreorbetter
componentstoit.Animportantcharacteristicofaframeworkthat
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
45/602
differentiatesitfromlibrariesisthatthemethodsdefinedbytheusertotailortheframeworkarecalledfromwithintheframeworkitself.The
frameworkoftenplaystheroleofthemainprogramincoordinatingandsequencing
applicationactivity.(JohnsonandFoote,1988)IntegrationTestingAleveloftestundertaken
tovalidatetheinterfacebetweeninternalcomponentsofasystem.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
46/602
Typicallybaseduponthesystemarchitecture.(CraigandJaskiel,2002)KeywordAdirectivethatrepresentsasingleactioninkeyword-driventesting.Calledactionswords
byBuwaldaetal.(2002).Keyword-DrivenTestingAtestautomationapproachwheretest
dataandalsokeywordsinstructinghowtousethedataarereadfrom
anexternaldatasource.Whentestcasesareexecutedkeywords
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
47/602
areinterpretedbyatestlibrarywhichiscalledbyatestautomationframework.Seealsodata-driventesting.(FewsterandGraham,1999;Kaneret
al.,2001;Buwaldaetal.,2002;MosleyandPosey,2002)LibraryAcontrolled
collectionofsoftwareandrelateddocumentationdesignedtoaidinsoftwaredevelopment,use,
ormaintenance.Seealsoframework.(IEEEStd610.12-1990)Non-FunctionalTesting
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
48/602
Testingofthoserequirementsthatdonotrelatetofunctionality.Forexampleperformanceandusability.(BS7925-1)ManualTestingManuallyconductedsoftwaretesting.See
alsotestautomation.OracleAdocumentorpieceofsoftwarethatallowstest
engineersorautomatedteststodeterminewhetheratesthasbeenpassedor
not.(Burnstein,2003)PreconditionEnvironmentalandstateconditionswhichmust
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
49/602
befulfilledbeforeatestcasecanbeexecuted.(BS7925-1)
xi
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
50/602
PredictedOutcomeSeeexpectedoutcome.PresentationCodePartoftheapplicationcodewhichmakesuptheuserinterfaceofthesystem.Seealso
domaincode.(Fowler,2001)Quality(1)Thedegreetowhichasystem,component,
orprocessmeetsspecifiedrequirements.(2)Thedegreetowhichasystem,component,
orprocessmeetscustomeroruserneedsorexpectations.(IEEE
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
51/602
Std610.12-1990)RecordandPlaybackSeecaptureandreplay.RegressionTestingRetestingpreviouslytestedfeaturestoensurethatachangeoradefectfix
hasnotaffectedthem.(CraigandJaskiel,2002)RequirementAconditionorcapability
thatmustbemetorpossessedbyasystemorsystemcomponentto
satisfyacontract,standard,specification,orotherformallyimposeddocuments.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
52/602
Canbeeitherfunctionalornon-functional.(IEEEStd610.12-1990)SetUpCodethatisexecutedbeforeeachautomatedtestcaseinoneparticulartest
suite.Arelatedtermusedinmanualtestingisprecondition.Seealsotear
down.SmokeTestingAtestruntodemonstratethatthebasicfunctionalityof
asystemexistsandthatacertainlevelofstability
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
53/602
hasbeenachieved.(CraigandJaskiel,2002)SoftwareTestAutomationSeetestautomation.SoftwareTestingTheprocessofoperatingasystemorcomponentunder
specifiedconditions,observingorrecordingtheresults,andmakinganevaluationofsome
aspectofthesystemorcomponent.(IEEEStd610.12-1990)StaticTestingTheprocess
ofevaluatingasystemorcomponentbasedonitsform,
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
54/602
structure,content,ordocumentation.Seealsodynamictesting.(IEEEStd610.12-1990)SystemTestingAcomprehensivetestundertakentovalidateanentiresystemandits
characteristics.Typicallybasedupontherequirementsanddesignofthesystem.(Craigand
Jaskiel,2002)SystemUnderTest(SUT)Theentiresystemorproducttobe
tested.(CraigandJaskiel,2002)
xii
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
55/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
56/602
TearDownCodethatisexecutedaftereachautomatedtestcaseinoneparticulartestsuite.Testautomationframeworksrunthemregardlessthe
teststatussoactionsthatmustalwaysbedone(e.g.releasingresources)should
bedonethere.Seealsosetup.TestAutomationTheuseofsoftware
tocontroltheexecutionoftests,thecomparisonofactual
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
57/602
outcomestopredictedoutcomes,thesettingupoftestpreconditions,andothertestcontrolandtestreportingfunctions.(BS7925-1)TestAutomationAframework
usedfortestautomation.Providessomecorefunc-Frameworktionality(e.g.loggingand
reporting)andallowsitstestingcapabilitiestobeextendedbyaddingnewtest
libraries.TestCaseAsetofinputs,executionpreconditionsand
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
58/602
expectedoutcomesdevelopedforaparticularobjective,suchastoexerciseaparticularprogrampathortoverifycompliancewithaspecificrequirement.(BS
7925-1)TestOracleSeeoracle.TestOutcomeSeeactualoutcome.TestRunnerA
genericdriverscriptcapabletoexecutedifferentkindsoftestcasesandnot
onlyvariationswithslightlydifferenttestdata.TestSuiteA
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
59/602
collectionofoneormoretestcasesforthesoftwareundertest.(BS7925-1)Test-DrivenDevelopment(TDD)Developmenttechniquewhereautomatedunittestsare
writtenbeforethesystemcode.Testsdrivethedesignanddevelopmentofthe
systemandacomprehensiveregressiontestsuiteisgotasaby-product.(Beck,
2003)TestabilityAcharacteristicofsystemundertestthatdefines
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
60/602
howeasilyitcanbetested.Consistsofvisibilityandcontrol.(Pettichord,2002)TestingSeesoftwaretesting.TestwareTheartifactsrequiredtoplan,design
andexecutetestcases,suchasdocumentation,scripts,inputs,expectedoutcomes,setup
andteardownprocedures,files,databases,environmentsandanyadditionalsoftwareorutilities
usedintesting.(FewsterandGraham,1999)
xiii
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
61/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
62/602
UnitApieceofcodethatperformsafunction,typicallywrittenbyasingleprogrammer.(CraigandJaskiel,2002)UnitTestingAlevel
oftestundertakentovalidateasingleunitofcode.Unittestsare
typicallyautomatedandwrittenbytheprogrammerwhohaswrittenthecodeunder
test.(CraigandJaskiel,2002)UserKeywordAtermdefined
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
63/602
inthisthesisforkeywordsconstructedfrombasekeywordsandotheruserkeywordsinatestdesignsystem.Userkeywordscanbecreatedeasily
evenwithoutprogrammingskills.WhiteBoxTestingTestingbaseduponknowledgeofthe
internalstructureofthesystem.Testingnotonlywhatthesystemdoes,but
alsohowitdoesit.(CraigandJaskiel,2002)xUnit
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
64/602
FrameworksFrameworksthateasewritingandexecutingautomatedunittests,providesetupandteardownfunctionalitiesforthemandallowconstructingtestsuites.
ThemostfamousxUnitframeworkisJUnitforJavabutimplementationsexistfor
mostprogramminglanguages.(Hamill,2004)
xiv
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
65/602
Chapter1Introduction
Softwaresystemsaregettingmoreandmoreimportantfororganizationsandindividualsalikeandatthesametime
theyaregrowingbiggerandmorecomplex.Itisthusonlylogicalthat
importanceofsoftwarequality1isalsorising.Softwarefaultshavecausedlossof
hugesumsofmoneyandevenhumanlives.Ifquality
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
66/602
doesnotgetbetterassystemsgrowinsize,complexityandimportance,theselossesareonlygettingbigger.(Burnstein,2003)
Theneed
forbetterqualitymeansmorepressureforsoftwaretestingandfortestengineers
takingcareofit.Testautomation,i.e.givingsometestingtaskstocomputers,
isanobviouswaytoeasetheirworkload.Computersare
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
67/602
relativelycheap,theyarefasterthanhumans,theydonotgettiredorbored,andtheyworkoverweekendswithoutextrapay.Theyare
notidealworkhorses,however,astheyonlyfinddefectsfromplaceswherethey
areexplicitlytoldtosearchthemandtheyeasilygetlostifsomething
inthesystemundertest(SUT)changes.Givingcomputersall
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
68/602
theneededdetailsisnoteasyandtakestime.(FewsterandGraham,1999)
Testautomationcanbeusedinmultipleways.It
canandshouldbeuseddifferentlyindifferentcontextsandnosingleautomation
approachworkseverywhere.Testautomationisnosilverbulleteitherbutithas
alotofpotentialandwhendonewellitcan
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
69/602
significantlyhelptestengineerstogettheirworkdone.(FewsterandGraham,1999)
Thisthesisconcentratesonlargertestautomationframeworksdesigned
fortestexecutionandreporting.Beforethescopecanbedefinedinmore
detailedmannersome
1Newtermsareemphasizedwhenusedforthe
firsttimeandtheirexplanationscanbefoundfromthe
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
70/602
listoftermsonpagesixxiv.
1
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
71/602
CHAPTER1.INTRODUCTION
backgroundinformationaboutdifferentautomationapproachesisneeded,however,andthatispresentedinSection1.2.Evenbefore
thatitistimetoinvestigateabitmorethoroughlywhytestautomation
isneededandwhatarethemainchallengesintakingitintouse.
1.1PromisesandProblemsofTestAutomationA
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
72/602
comprehensivelistoftestautomationpromises,aspresentedbyFewsterandGraham(1999),isshowninTable1.1.Similarpromiseshavebeenreportedalso
byotherauthorslikePettichord(1999),Nagle(2000)andKaneretal.(2001).
MostofthebenefitsinTable1.1canbesummarizedwith
wordsefficiencyandreuse.Testautomationisexpectedtohelp
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
73/602
runlotsoftestcasesconsistentlyagainandagainondifferentversionsofthesystemundertest.Automationcanalsoeasetestengineersworkload
andreleasethemfromrepeatingtasks.Allthishasthepotentialtoincrease
softwarequalityandshortentestingtime.
Allthesepromisesmaketest
automationlookreallyattractivebutachievingtheminreallife
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
74/602
requiresplentyofhardwork.Ifautomationisnotdonewellitwillbeabandonedandpromiseswillneverberealized.Alistof
commontestautomationproblems,againbyFewsterandGraham(1999),canbefound
fromTable1.2.
Thegeneralproblemwithtestautomationseemsto
beforgettingthatanylargertestautomationprojectisa
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
75/602
softwareprojectonitsownright.Softwareprojectsfailiftheydonotfollowprocessesandarenotmanagedadequately,andtestautomationprojects
arenotdifferent.Ofallpeople,testengineersoughttorealizehowimportant
itistohaveadisciplinedapproachtosoftwaredevelopment.(Kaner,1997;Zambelich,
1998;FewsterandGraham,1999;Pettichord,1999;Kaneretal.,
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
76/602
2001;Zallar,2001;Rice,2003)
1.2DifferentTestAutomationApproachesThissectionbrieflyintroducesmaintestautomationcategoriesasanintroductionand
backgroundfortherestofthisthesis.Thefocusedscopeandtargetfor
thisthesisaredefinedinnextsection.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
77/602
CHAPTER1.INTRODUCTION
RunexistingregressiontestsonanewversionofaprogramBeingabletorunpreviouslycreatedtests
withoutextraeffortclearlymakestestingmoreefficient.Runmoretestsmoreoften
Automationmeansfastertestexecutionwhichmeansmoretestrounds.Automationshouldalso
makecreatingnewtestcaseseasyandfast.Performtests
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
78/602
whichwouldbedifficultorimpossibletodomanuallyForexampleperformanceandstresstestsarenearlyimpossibletoconductwithoutautomation.Betteruse
ofresourcesAutomatingrepeatingandboringtasksreleasestestengineersformoredemanding
andrewardingwork.ConsistencyandrepeatabilityoftestsTestsarealwaysrunthe
samewaysotestresultscanbeconsistentlycomparedto
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
79/602
previousresultsfromprevioustestingrounds.Testscanalsobeeasilyrepeatedondifferentenvironments.ReuseoftestsReusingtestsfromearlierprojectsgives
starttoanewproject.akickEarliertimetomarketReusingtests
andshorteningtestexecutiontimefastensfeedbackcycletodevelopers.Intheend
thatshortensthetimetomarket.IncreasedconfidenceRunningan
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
80/602
extensivesetoftestsoften,consistentlyandondifferentenvironmentssuccessfullyincreasestheconfidencethattheproductreallyisreadytobereleased.
Table1.1:Commontestautomationpromises(FewsterandGraham,1999)
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
81/602
CHAPTER1.INTRODUCTION
UnrealisticexpectationsManagersmaybelievetestautomationwillsolvealltheirtestingproblemsandmagicallymakethesoftware
qualitybetter.Automationexpertsshouldhelpmanagerssettingtheirexpectationsright.
PoortestingpracticeIftestingpracticesandprocessesareinadequateitisbetter
tostartimprovingthemthanbringingintestautomation.Automating
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
82/602
chaosjustgivesfasterchaos.
ExpectationthatautomatedAfterautomatedtesthasbeenrunsuccessfullytestswillfindalotofnew
onceitisnotverylikelytofindnewbugsunlessdefectsthe
testedfunctionalitychanges.Automatorsnor
mallyfindmoredefectswhilethey
aredevelopingteststhanwhentestsarere-executed.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
83/602
FalsesenseofsecurityJustseeingatestreportwithnofailuresdoesnotmeanthattheSUTdidnothaveany.Testsmay
beincomplete,eithernottestingallfeaturesornotabletoseefailures
whentheyoccur.Testsmayalsohavedefectsandshowwrongresults.
MaintenanceWhentheSUTchangesalsoitstestschange.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
84/602
Humantestengineersareabletohandleevenmajorchangeswithoutproblemsbutautomatedtestscanfailafteraslightestchange.Ifmaintainingtest
automationsystemtakesmoretimethantestingmanuallyitwillsurelybeabandoned.
Thesamewillhappenalsoifaddingnewfeaturestotheautomationsystem
istoocumbersome.
TechnicalproblemsBuildingandtaking
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
85/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
86/602
inmanyprocesses.
Table1.2:Commontestautomationproblems(FewsterandGraham,1999)
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
87/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
88/602
documentandcodereviewsandstaticcodeanalysis
(e.g.syntaxcorrectness,codecomplexity).Reviewsarebytheirnaturedonemainlybyhumans
whilestaticanalysisismostlyleftforcomputersandthereareplentyof
goodtoolsavailable.(IEEEStd610.12-1990;Burnstein,2003)Ibelievestatictestingis
importantandcangreatlyhelpinfindingdefectsevenin
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
89/602
earlystagesofprojects.Inthisthesis,however,interestisonlyondynamictesting.
1.2.2Functionalvs.Non-FunctionalTestingDynamictestingcan
befurtherdividedintofunctionalandnon-functionaltesting.Aimoftheformer,as
thenameimplies,isensuringthatthefunctionalityofthesystemadheresto
requirements.Thelatterisconductedtoverifythatother,non-functional,
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
90/602
aspectsofthetestedsystemworkaswell.(IEEEStd610.12-1990;Burnstein,2003)
Generallyallfunctionaltestingisdoneintwosteps
asillustratedFigure1.1.Firstsomethingisdonetothetestedsystem.Next
thetestoutcome,outputsproducedbythesystemandchangesinsystemsstate,
isverifiedagainstpredefinedexpectedoutcome.Inmostcasesboth
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
91/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
92/602
CHAPTER1.INTRODUCTION
liabilityandmemorymanagement.Thesetestingareasarereallyimportantandfailuresinanyofthemmayruin
otherwisegoodandfunctionallycorrectproduct.Non-functionaltestingdiffersquitealotfrom
functionaltestinganditisalsoratherheterogeneousitself.Someareas(e.g.usability)
aremainlytestedmanuallybyexpertswhileothers(e.g.performance)
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
93/602
arenearlyimpossiblewithoutadequateautomationtools.
Inthisthesisthescopeisonfunctionaltestingbutmanyideaspresentedarerelevant
alsowhenautomatingnon-functionaltesting.
1.2.3GranularityoftheTestedSystem
Traditionallytestingisdividedintodifferentlevelssuchasunit,integration,systemand
acceptancetesting(Dustinetal.,1999;CraigandJaskiel,2002;
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
94/602
Burnstein,2003).Inmyopinionthisdivisionisnotverypracticalintestautomationcontext,however,becausesameautomationapproachcanbeusedin
manylevels.Forexamplesystemandacceptancetestingdonothavemuchdifference
fromthisperspective.ThatiswhyIpreferclassificationbasedonthegranularity
ofthetestedsystemintounit,componentandsystemtesting,
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
95/602
assuggestedbyMeszaros(2003).
UnitTesting
Thesmallestbuildingblockofanysystemisaunit.Unitshavean
applicationprogramminginterface(API)whichisusedwheninteractingwithotherunitsand
canalsobeusedtotestthem.Unittestingisinmostcases
besthandledbydeveloperswhoknowthecodeundertest
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
96/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
97/602
unitinterfacescanbedrivenwithoutanyspecialtools.Inthislevelinterfacesalsotendtobemorestablethaninhigherlevels,especially
inuserinterfacelevel,andchangesinotherunitshavelittleorno
effectunlessunitsarecloselyconnected.Findingandfixingdefectsisalsocheapest
inunitlevel.
Defactotesttoolsin
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
98/602
unitlevelaremultiplexUnitframeworks.ThefirstofthemanywasSmalltalkUnit(Beck,1994)whichwassoonportedtoJavaasJUnitby
ErichGamma(BeckandGamma,1998).AfterJUnitsbigsuccessimplementations
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
99/602
CHAPTER1.INTRODUCTION
forotherlanguagessuchasPyUnitforPythonandNUnitforMicrosofts.NETfollowedshortly.(Hamill,2004)
Unittestingcanbetakentonewalevelwithtest-drivendevelopment(TDD).
InTDDtestsare,asthenameimplies,createdbeforeactualproductioncode
whichforcesdeveloperstothinkaboutthedesignandtestability
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
100/602
ofthesystem.Asaresultthedesignisclearandnumberofdefectslow(Beck,2003).Fromtestingpointofviewthenice
thingaboutTDDisthatacomprehensiveregressiontestsetisgotas
abonus.Verypromisingresultsfromusingtest-drivendevelopmentarereportedforexample
byGerasetal.(2004)andMaximilienandWilliams(2003),
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
101/602
andmypersonalexperiencesarealsoextremelypositive.
Thescopeofthisthesisisonhighertestautomationlevelsthanunittesting.
ThatdoesnotmeanthatIwouldconsiderunittestingandunittest
automationlessimportant.Onthecontrary,Istronglybelievethatinmostcases
automationeffortsshouldbefirstdirectedintounittestinglevel
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
102/602
whereneededinvestmentsaresmallandlikelytopaybackfast.
ComponentTesting
Componentsvaryalotfromonesituation
toanotherbutroughlyspeakingacomponentisacollectionofrelatedunits
whichhaveacommoninterfacetowardsothercomponents.Theseinterfacesmaybesimilar
programminginterfacesasinunitlevelbuttheycanalso
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
103/602
beimplementedwithmanyhigherleveltechniqueslikeCOM,XMLandHTTP.
Automationincomponentlevelisnormallynottoodifficult.Test
toolscanbehookedintothesameinterfacewhichisusedbyother
components.Inmanycasestherealsoexistsgoodandfreetoolsfordriving
theinterface(e.g.HTTPUnit)andevenifnoready-madetool
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
104/602
isavailabledeveloperscaneasilywriteone.(Pettichord,2002)
SystemTesting
Thedifferencebetweenacomponentandasystemis
thatsystemsfunctionstandalonewhilecomponentsareonlyusedaspartofa
system.Inreallifethisdifferencecanberathervagueassystemsare
oftenintegratedtogethertoformevenbiggersystems.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
105/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
106/602
CHAPTER1.INTRODUCTION
Systemshavedifferentinterfacestowardstherestoftheworlddependingonwhethertheyaregoingtobe
usedbyhumansorothersystems.Systemsusedbyhumanshavesomekind
ofuserinterface,eithergraphicalornon-graphical,andsystemsusedbyothersystems
havesimilarinterfacesascomponents.Graphicaluserinterfacesarenotoriously
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
107/602
hardtoautomatebutotherinterfacesaregenerallyrathereasy.(Pettichord,2002)
1.2.4TestingActivitiesSoftwaretestingismuchmorethatjust
executingtestcasesandsimilarlytestautomationisnotlimitedtoautomatingonly
testexecution.
DesigningTestCases
Functionalsystemandacceptance
testsaredesignedbytestengineersusingsystemrequirementsand
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
108/602
formaltestdesigntechniqueslikeequivalencepartitioningandboundaryvalueanalysis.Designinggoodtestcasesisnoteasyanditisoneofthe
mainskillsaprofessionaltestengineermustpossess.(FewsterandGraham,1999;Craig
andJaskiel,2002;Burnstein,2003)
Therearealsowaystoautomate
thetestdesignprocess.Expectedresultsfortestscansometimes
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
109/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
110/602
basedtestingthesystemismodeledinsuchadetailthattestcasescanbederivedautomaticallyfromthemodel(Dalaletal.,1999;
FewsterandGraham,1999;Robinson,2000).Plentyofothermethodslikeadata
miningapproach(Lastetal.,2003),agoal-drivenapproach(Memonetal.,1999)
andarequirement-basedtestgeneration(Tahatetal.,2001)are
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
111/602
alsopossible.
Thesekindofmethodsareveryinterestingbutscopeofthisthesisissolelyontestexecutionandtaskrelated
toit.Itshould,however,bepossibletointegrateatestdesignautomation
tooltoatestexecutionframeworkpresentedinlaterchapters.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
112/602
CHAPTER1.INTRODUCTION
ExecutingTestCasesandAnalyzingResults
Aftertestcaseshavebeendesignedandcreatedtheycan
beexecutedandresultsverifiedasillustratedinFigure1.1.Testsareoften
re-executedwhenanewversionofthetestedsystemisavailable.Automatingthese
regressiontestsoratleastanexemplarysubsetofthem,
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
113/602
oftencalledsmoketests,canmaketestexecutionandresultanalysisconsiderablyfaster.(Zambelich,1998;Pettichord,1999)
ReportingTestResults
Aftertestengineershaveruntheirteststheyreportfindingstotheproject
teamandmanagement.Iftestexecutionisfullyautomateditmakessenseto
automatealsoreporting.Itprobablyisnotagoodidea
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
114/602
tosendalltestreportsautomaticallytomanagersmailboxesbutifatestreportiscreatedautomaticallytestengineersdonotneedto
spendtimegatheringinformationfromtestlogsandelsewhere.
TestManagement
Manytestmanagementtaskslikeplanning,monitoring,schedulinganddefecttracking
canbesupportedbetools.Thesetoolsandautomationpossibilities
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
115/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
116/602
results.(Zallar,2001)
ToolAidedTesting
Smallscaletestautomationjusthelpsmanualtesting.Testautomationtoolsareusedin
areaswherecomputersarebetterthanhumansandwheretoolsareavailableor
easilyimplemented.Forexampletaskslikecheckingthataninstallationcopiedallfiles
torightplacesandverifyingthattwofileshavesame
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
117/602
datataketimeandareerrorpronetohumansbuteasilyautomated.ThisautomationapproachisstronglyadvocatedbyBach(2003).Henotesthat
insmallscaleautomationthetestcodeis
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
118/602
CHAPTER1.INTRODUCTION
generallysosimpleandinexpensivethatitcaneitherbeeasilyfixedortrashedifitisbroken
bychangesinthetestedsystem.
Bach(2003)proposesanidea
thatsocalledtoolsmithsworkwithtestengineersandprovidethemwithtools
andutilitiesbasedontheirneeds.Toolsmithsshouldhavegood
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
119/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
120/602
toolsbuiltbydevelopersfortheirdevelopmenttimetesting.Bach(2003)andKaneretal.(2001)listmoretoolsandsourceswheretofind
them.AsKaneretal.(2001)putsit,youmayalreadyhavemore
testtoolsthanyourealize.
Eventhoughthescopeofthis
thesisisinlargescaletestautomationIbelievesmall
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
121/602
scaletestautomationcanoftenbeabetterstrategybecauseitdoesnotrequirebiginvestmentsandprovideshelpfast.Especiallyifriskwith
largerframeworksfeeltoobigitisbettertostartsmall,gainmore
experienceandthenlaterinvesttolargerframeworks(Zallar,2001;Bach,2003).
TestAutomationFrameworks
Whentestautomationis
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
122/602
takentohighestleveltestscanbestartedwithapushofabutton,theycanbeleftrunningover-nightunattendedandnextmorning
testresultsarepublished.Thiskindofautomationclearlyrequiressomekindof
asystemwhichmakescreating,executingandmaintainingtestcaseseasy.Thesystem
shouldprovidesomecorefunctionality(e.g.monitoringandreporting)and
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
123/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
124/602
areunstructured,havetestdataembeddedintothescriptsandthereisnormallyonescriptperonetestcase.Scriptsaremainlygeneratedusing
captureandreplaytoolsbutmayalsobemanuallycoded.Thiskindof
scriptisvirtuallynon-maintainableandwhenthetestedsystemchangestheyneedto
becapturedorwrittenagain.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
125/602
CHAPTER1.INTRODUCTION
2.Insecondgenerationscriptsarewell-designed,modular,robust,documentedandthusmaintainable.Scriptsdonotonlyhandle
testexecutionbutalsoforexamplesetupandcleanupanderrordetectionand
recovery.Testdataisstillembeddedintothescripts,though,andthereis
onedriverscriptsperonetestcase.Codeismostly
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
126/602
writtenmanuallyandbothimplementationandmaintenancerequireprogrammingskillswhichtestengineersmaynothave.3.Thirdgenerationframeworkshaveallthesame
goodcharacteristicsalreadyfoundfromsecondgeneration.Theygoforwardbytakingtest
dataoutofthescriptswhichhastwosignificantbenefits.Firstbenefitis
thatonedriverscriptmayexecutemultiplesimilartestcases
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
127/602
byjustalteringthedataandaddingnewtestsistrivial.Secondbenefitisthattestdesignandframeworkimplementationareseparatetasksformercan
begivensomeonewiththedomainknowledgeandlattertosomeonewithprogramming
skills.Thisconceptiscalleddata-driventesting.Keyword-driventestingtakestheconcepteven
furtherbyaddingkeywordsdrivingthetestexecutingintothe
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
128/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
129/602
isfollowing.
Designingandimplementingalargescaletestautomationframeworkforfunctionaltestingincomponentandsystemlevel.Theframeworkmust
beabletoexecutetests,verifyresults,recoverfromexpectederrorsandreport
results.Itmustalsobebotheasytouseandmaintain.Itdoes
notneedtoprovidesupportforautomationoftestplanning
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
130/602
anddesign.
Knowinghowtoautomatesomethingisclearlynotenoughtomakeanautomationprojectasuccessandsomeofthe
otherimportantthingstotakeintoaccountarelistedinTable1.3.These
issuesareexcludedfromthescopeofthisthesisbuttheyarelisted
hereforcompletenesssake.Theyalsohelptounderstandhow
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
131/602
manyissuestherearetohandlewhenconductingtestautomationprojectsandhowbigandchallengingtheseprojectscanbe.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
132/602
CHAPTER1.INTRODUCTION12Whentoautomate.Toolselectionprocess.Whattoautomate.Takingtoolsandframeworksintouse.Whattoautomatefirst.
Testplanninganddesign.Whatcanbeautomated.Effecttodevelopmentandtesting
processes.Howmuchtoautomate.Otherorganizationalissues.Buildorbuy.Costsand
savings.Buildin-houseoruseconsultants.Metrics.
Table
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
133/602
1.3:Someimportantissuesexcludedfromthescope
1.4MethodologyHevneretal.(2004)assertthatsoftwareengineeringresearchischaracterizedby
twocomplementaryparadigms:behavioralscienceanddesignscience.Behavioralscienceistheoreticaland
hasitsoriginsinnaturalsciencewhiledesignscienceoriginatesfromengineeringand
triestofindpracticalsolutionsforresearchproblemsfromthe
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
134/602
field.Theimportanceofpracticalresearchandconcrete,evaluatedresultsisemphasizedbothbyHevneretal.(2004)andShaw(2002)andthemethodology
ofthisthesisisbasedonmodelspresentedbythem.
Much
oftheknowledgeofthisthesiscomesfrommultipletestingandtestautomation
booksandarticleslistedinthebibliography.Thisisnot
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
135/602
onlyaliteraturestudy,however,asIhavesixyearsexperiencefromtestingandtestautomationusingdifferenttoolsandtechniques.Thisexperiencegives
firsthandknowledgeaboutthesubjectandalsogreatlyhelpsprocessingtheinformation
readfromtheliterature.
1.5GoalsTheobjectiveofthethesis
ispresentingaconceptforalargescaletestautomation
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
136/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
137/602
feasibilityoftheframeworkbasedonthem.Adaptrequirementsbasedonthenewinformationifnecessary.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
138/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
139/602
howtomeettheserequirements.Chapter3isthecoreofthethesis.InthatchapteraframeworkmeetingtherequirementssetinChapter
2isintroducedandexplainedindetail.
InChapter4the
presentedframeworkistestedagainstdefinedrequirements.Thisisdonebyimplementinga
prototypeoftheframeworkandusingitintwosimple
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
140/602
butnon-trivialpilotstestingaWindowsapplicationandawebpage.
ResultsandexperiencesfromthepilotarecollectedtogetherinChapter
5.Basedontheresultsthegeneralfeasibilityofthesuggestedframeworkis
evaluatedandpossiblychangesmadetotherequirementsetdefinedearlier.
Chapter6isleftforconclusions.Thereitisevaluated
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
141/602
howwellrequirementsforthisthesis,definedinSection1.5,aremet.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
142/602
Chapter2
RequirementsforTestAutomationFrameworks
Chapter1introduceddifferentviewstotestautomationandstatedthatthe
scopeofthisthesisislargescaletestautomationframeworks.Theaimof
thechapteristodefinerequirementsfortheseframeworksanddiscussmethodologiesand
techniquesneededtomeettherequirements.
2.1High
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
143/602
LevelRequirementsAshortlistofhighlevelrequirementsapplicableforalllargescaletestautomationframeworks,derivedfromKit(1999)andothersources,
isshowninTable2.1.
Automatictestexecutionisahigh
levelfunctionalrequirementfortestautomationframeworks.Howitcanbeaccomplishedand
howitbreaksintomoredetailedrequirementsisdiscussedin
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
144/602
Section2.2.Easeofuseandmaintainabilityarenonfunctionalintheirnatureandingeneralachievingthemisharderthansimplyimplementingthefunctional
requirements.Sections2.32.8presentmethodsandtechniqueshowtheycanbefulfilledand
howtheyturnintomoredetailedrequirements.Detailedrequirementsarefinallysummarizedin
Section2.9.
2.2FrameworkCapabilitiesThissectionexplains
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
145/602
whatcapabilitiesatestautomationframeworkmusthave.Discussedfeaturescanbeseenasfunctionalrequirementsforautomationframeworks.
14
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
146/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
147/602
tousebytestengineersoritisverylikelytobeabandoned.Frameworkusersmustbeabletodesignandedittests,run
themandmonitortheirstatuseasilywithoutanyprogrammingskills.(Kaner,1997;Kit,
1999;Nagle,2000)
MaintainabilityItmustbeeasyandfastto
maintainbothtestdataandframeworkcodewhenthetested
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
148/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
149/602
pushofabuttonandruntestsonitsown.Thismeansthatframeworkmustbeabletosetupthetestenvironmentand
preferablyalsocheckthatallpreconditionsaremet.(FewsterandGraham,1999;Pettichord,
1999)
2.2.2StartingandStoppingTestExecutionItmustbepossible
tostarttestexecutionmanually.Itisalsoconvenientif
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
150/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
151/602
canbeimplementedsimilarlyusingexternaltools.
2.2.3HandlingErrorsPartofrunningtestsunattendedisrecoveringfromerrorscausedbythe
testedsystemorthetestenvironmentnotworkingasexpected.Testframeworkought
tonoticeerrorsituationsandcontinuetestingwithoutmanualintervention.Handling
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
152/602
CHAPTER2.REQUIREMENTS
allpossiblebutrareerrorsisseldomworththeeffort,though,andover-engineeringshouldbeavoided.(Pettichord,1999)
2.2.4VerifyingTestResultsAnintegralpartoftestexecutionis
verifyingtestresults.FewsterandGraham(1999)defineverificationasoneormore
comparisonsbetweenactualoutcomeofatestandpredefinedexpected
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
153/602
outcome.Theyalsoexplaindifferentcomparisontechniqueswhichareoutofthescopeofthisthesis.
2.2.5AssigningTestStatusAftera
testisexecutedanditsresultsverifieditoughttobegivena
status.Ifthetestwasexecutedwithoutanyproblemsandallcomparisonsbetween
actualandexpectedoutcomesmatchthetestgetsapass
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
154/602
status.Ineveryothercasethestatusisfail.Besidesthestatuseverytestcaseshouldalsogetashortbutdescriptivestatusmessage.
Forpassedteststhismessageisnormallynotthatimportantbutwithfailed
testsitcangivedetailsaboutthecauseoftheproblem(e.g.Calculation
failed:expected3,got2orInitializingtestenvironmentfailed:
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
155/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
156/602
HandlingExpectedFailuresThemostfrustratingpartoffailureanalysisisgoingthroughtestcaseswhichareknowntofailandcheckingwhetherthey
havefailedsimilarlyasbeforeorhavenewdefectsappeared.Thatiswasteful
andclearlysomethingthatshouldbeautomatedasFewsterandGraham(1999)recommends.
Tobeabletodifferentiateexpectedfailuresfrom
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
157/602
newonestheframeworkmustknowwhatistheexpectedoutcomewhentestfails.Thismeansthattheframeworkmuststoreboththeexpected
outcomeandtheexpectedfailedoutcomeofthetest.Whenatestfails,
andisexpectedtodoso,theexpectedfailedoutcomecanbe
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
158/602
CHAPTER2.REQUIREMENTS
Figure2.1:Handlingexpectedfailures
comparedagainsttheactualoutcomeasseenintheFigure
2.1.Ifoutcomesmatchanexpectedfailureisdetected,otherwiseanewdefect
hasbeenfound.Irecommendcommunicatingexpectedfailuresusingstatusmessagesbut,as
alreadydiscussed,itcanbearguedthataspecifictest
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
159/602
statusshouldbeusedinstead.IneithercaseitsureisfasttoanalyzefailedtestwhichrightawaytellssomethinglikeExpectedfailure.
Defectid:#2712.
FewsterandGraham(1999)alsorecommendnotifyingtest
engineerswhenatestwhichisexpectedtofailpasses(i.e.defecthas
beenfixed).Thiswaytheycanmanuallyverifythatthe
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
160/602
testreallyhaspassedanddefectisfixedwithoutsideeffects.Aftereverypassedtesttheframeworkshouldcheckwhetherthetestwasexpected
tofailbycheckingdoesithavetheexpectedfailoutcome.Iftest
wasexpectedtofailitshouldbelabeledaccordinglyIrecommendsettingthestatus
messagetosomethinglikeDefect#2712fixed.Thewholeprocess
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
161/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
162/602
CHAPTER2.REQUIREMENTS
FailUsedtologthestatusofafailedtestcase.PassUsedtologthestatusof
apassedtestcase.FatalUsedwhenanunrecoverablefailurepreventingfurthertest
executionoccurs.Thiskindofproblemismostofteninthetestenvironment
(e.g.diskfullornetworkdown).ErrorUsedwhenan
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
163/602
unexpectederrorpreventingcurrenttestsexecutionoccurs.Forexamplethebuttontopushinatestdoesnotexists.FailureUsedwhenthetest
outcomedoesnotmatchtheexpectedoutcome.WarningUsedwhenarecoverableerror
notaffectingtestexecutionoccurs.Forexampledeletingsometemporaryfilefails.Info
Usedtolognormalpassedverifications.testexecutionlikestarting
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
164/602
testandDebugGivesdetailedinformationaboutwhatthetestautomationframeworkoraparticulartestcaseisdoing.Usedmainlyfordebuggingthe
frameworkortestcases.TraceSimilartodebugbutcontainsevenmoredetailed
information.
Table2.2:Suggestedlogginglevels
testexecutionprocess
andhowthetestedsystembehavedareneeded.Ontop
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
165/602
ofthattheframeworkmustlogwhatitisdoinginternallytomakeiteasiertodebugproblemsintheframeworkitself.
Themaindilemmawithanykindoflogging,asstatedbyPettichord(2002),
ishowmuchtolog.Loggingtoomuchmakesfindingrelevantinformationhard
andhugelogfilescauseastorageproblem.Thenagain
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
166/602
loggingtoolittlecanmakelogstotallyuseless.Thereisnoperfectsolutionforthisproblembutforexampleusingdifferentlogginglevelsmake
theproblemsmaller.
Multiplelogginglevelsprovideamechanismforcontrolling
howmuchdetailiscaptured.Forexamplelowestlevels,thosethatproducemost
output,caninnormalcasesbefilteredoutbeforethey
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
167/602
areevenlogged.Ifdetailedinformationisneededthefiltercanbeadjusted.Laterwhenlogsareviewedentriescanbefilteredevenfurther
sothatonlyinterestinglevelsareviewed.Whatlevelsareneededdependsultimately
onthecontextbutIhavefoundthelistinTable2.2pretty
goodformostsituations.Twoexampleshowtheselevelscan
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
168/602
beusedarefoundinFigure2.2.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
169/602
CHAPTER2.REQUIREMENTS
TimestampLevelMessage2005050215:14:01InfoTestexecutionstarted.SUT:Calculatorv.0.6.1,Testsuite:Calculatorbasic20050502
15:14:04InfoTestenvironmentsetupwithoutproblems2005050215:14:04InfoStartingtestcase
Add012005050215:14:05InfoTest1+2returned32005050215:14:05
PassAdd012005050215:14:06InfoStartingtestcaseAdd
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
170/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
171/602
+0.5returned12005050215:14:09PassAdd042005050215:14:10InfoTestexecutionended.4/4testcasesexecuted(100%),3/4passed(75
%),1/4failed(25%).
TimestampLevelMessage2005050216:23:31Info
Testexecutionstarted.SUT:Calculatorv.0.6.2,Testsuite:Calculatorbasic2005050216:23:43
FatalSettinguptestenvironmentfailed2005050216:23:43InfoTest
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
172/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
173/602
thethresholdlevelarenottobelogged.Adjustingthelevelmustbeeasyforanyoneexecutingtestsanditshouldpreferablybepossible
alsowhiletestsarerunning.Thedefaultlogginglevelshouldnormallybeinfo.
Infolevelgivesrequiredinformationabouttestexecutionbutnodebuggingdetailswhich
shouldnotbeneedednormally.ForexamplelogsinFigure
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
174/602
2.2wouldbetoouninformativeifhigherlevelwasusedandunnecessarilylongiflevelwaslower.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
175/602
CHAPTER2.REQUIREMENTS
2.2.8AutomaticReportingTestlogshavealltheinformationfromtestexecutionbut,especiallyiftheyarelong,
theyarenotgoodforseeingteststatusesataglance.Thiskind
ofviewisprovidedbyconcisetestreports.Testreportsprovidestatisticalinformation
aboutthequalityofthetestedsystemandtheyare
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
176/602
notonlyimportantfortestengineersbutalsofordevelopers,managersandbasicallyeveryoneinvolvedwiththeproject.(FewsterandGraham,1999)
Reportsshouldnotbetoolongordetailed.Havingashortsummaryand
listofexecutedtestcaseswiththeirstatusesisprobablyenough.Listof
itemseverytestreportshouldhaveispresentedbelowadaptedfrom
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
177/602
Buwaldaetal.(2002)andFigure2.3showsasimpleexample.
Nameandversionofthetestedsystem.Identificationofthe
executedtestsuite.Thetotalnumberofpassedtestsandexecutedtests.
Listoferrorsfound.Optionallyalistofallexecutedtests
alongwiththeirstatuses.Testreportscanbeeithercreated
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
178/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
179/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
180/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
181/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
182/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
183/602
functionsand,asillustratedinFigure2.4,theyinteractdirectlywiththetestedsystem.Simplelinearscriptsarefasttowriteinitiallyandthey
arethuswellsuitedforsmalltasks.Theproblemwithlinearscriptsis
thatwhentasksgetbiggerandmorecomplicatedalsotheygrowlonger,more
complicatedandgenerallyhardtomaintain.Theyalsotendto
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
184/602
haverepeatingcodewhichcouldbetakenoutandreused.Ifmanysimilartestscriptsarecreatedthemaintenanceproblemjustgrowsbiggerbecause
onechangeinthetestedsystemcanrequirechangesineveryscript.(Fewster
andGraham,1999)
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
185/602
CHAPTER2.REQUIREMENTS
2.3.2TestLibrariesandDriverScriptsStructuredcodingisacommonprogrammingparadigmanditisofcourse
applicablewithtestcodetoo(Kelly,2003).Inlowlevelitcanmean
simplyusingsmallfunctionsinthesamefileasthemaintestscript.
Inhigherlevelusetestfunctionsareplacedtoexternal,
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
186/602
sharedtestlibrariesfromwheretheycanbeusedbyanytestscript.
Whentestlibrariesdomostoftheactualtesting
worktestscriptscanbeshortandsimple.Becausescriptsareonlydriving
thetestexecutiontheyarecustomarilycalleddriverscripts.Figure2.5showshow
twodriverscriptsusessametestlibrarytointeractwith
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
187/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
188/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
189/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
190/602
of,documenting,naming,andstoringofcreatedscriptsisabigtask.Theyalsonotethatifreusablefunctionsarenoteasytofind
andwelldocumentedautomatorsarelikelytowritetheirownversions.Marick(1997)
notesthatwritingwell-designedAPItoatestlibraryisashardas
writinganygoodAPIanditshouldnotbeexpected
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
191/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
192/602
frameworks.Icannotseehowmaintainabilitycouldbeachievedwithoutthem.Librariessureareanoverkillforsimplecasesbutinthosecasesthe
wholeideaofbuildingabigframeworkisquestionable.
2.4Data-Driven
Testing2.4.1IntroductionSimpletestscriptshavetestdataembeddedintothem.This
leadstoaproblemthatwhentestdataneedsto
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
193/602
beupdatedactualscriptcodemustbechanged.Thismightnotbeabigdealforthepersonwhooriginallyimplementedthescriptbut
foratestengineernothavingmuchprogrammingexperiencethetaskisnot
soeasy.Ifthescriptislongandnon-structuredthetaskishard
foreveryone.Anotherproblemwithhavingthetestdatainside
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
194/602
testscriptsisthatcreatingsimilartestswithslightlydifferenttestdataalwaysrequiresprogramming.Thetaskmaybeeasyoriginalscriptcanbecopied
andtestdataeditedbutatleastsomeprogrammingknowledgeisstillrequired.This
kindofreuseisalsoproblematicbecauseoneparticularchange
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
195/602
CHAPTER2.REQUIREMENTS
inthetestedsystemmayrequireupdatingallscripts.(Strang,1996;Nagle,2000)
Becauseofthese
problemsembeddingtestdataintoscriptsisclearlynotaviablesolutionwhen
buildinglargertestautomationframeworks.Abetterapproachisreadingthetestdata
fromexternaldatasourcesandexecutingtestbasedonit.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
196/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
197/602
2005)
2.4.2EditingandStoringTestDataBecausedata-driventestdataistabularitisnaturaltousespreadsheetprogramstoedit
it.Testengineersandbusinesspeopleare,asMugridgeandCunningham(2005)point
out,alsolikelytobefamiliarwithspreadsheetprogramsandalreadyhavethem
available.Anotherbenefitisthatspreadsheetprogramsareoftenused
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
198/602
forsimpletestmanagementtasksandinthatcasethereislessneedforstoringsamedataindifferentplacesorcopyingitback-and-forth.
Spreadsheetfilescanbesavedincomma-separated-values(CSV),tab-separatedvalues(TSV)or
spreadsheetprogramsnativeformats.CSVandTSVfilesarehandybecausetheyare
veryeasytoparsebutunfortunatelymostspreadsheetprogramsseem
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
199/602
toalterthedatawheytheyopenthesefiles.ForexampleifIstorephonenumber+358912345andversionnumber1.5.1toaCSV
fileandopenittomyExceltheyareauto-correctedto358912345and
1.5.2001,respectively.Theeasiestsolutionforthisproblemisstoringdatausingprograms
nativeformatandonlyexportingitintoCSVorTSV.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
200/602
Anotherpossibilityisprocessingthenativeformatdirectlybutitrequiressomemoreinitialeffort.
HTMLtablesprovidedanothereasy-to-parseformatfor
presentingtestdata.EditingHTMLwithadecentgraphicaleditorisnearlyas
convenientasusingaspreadsheetprogrambutunliketheirnativeformatsHTMLcan
alsobeeditedwithanytexteditorifnothingelse
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
201/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
202/602
theymaynotbeenough.Therealsolutionforscalabilityproblemsisstoring
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
203/602
CHAPTER2.REQUIREMENTS
Figure2.6:Data-drivenapproach
Figure2.7:Data-driventestdatafile
data=
open(testdata.tsv).read()lines=data.splitlines()[1:]#[1:]excludestheheaderrow
forlineinlines:testId,number1,operator,number2,expected=line.split(
\t)#Actualtestfunctionalityexcluded
Listing
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
204/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
205/602
CHAPTER2.REQUIREMENTS
testdataintoacentraldatabase.Thatwaythetestdatacanbeeditedusingforexamplea
web-basedinterfaceanddriverscriptscanquerythedatadirectlyfromthedatabase.
Buildingsuchasystemisabigprojectitselfandcanberecommended
onlyifscalabilityisarealproblem.
Bird
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
206/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
207/602
wouldthenagainratherselectadatabasewithaweb-baseduserinterfacebecauseitwouldalsoprovideacentralizeddatastorage.
2.4.3
ProcessingTestDataImplementingascriptforparsingdata-driventestdatacanbe
surprisinglyeasywithmodernscriptinglanguages.ForexampledatainFigure2.7can
beexportedintoaTSVfileandparsedwithfour
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
208/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
209/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
210/602
isrequiredfromtheparseritshouldbeimplementedasacommonmodulewhichalldriverscriptscanusethiswasalreadyseeninthe
Figure2.6.Howtheparsershouldbeimplementedandwhatkindoffunctionality
itneedsisfurtherdiscussedinSection3.2andChapter3ingeneral.
2.4.4PromisesandProblemsMainbenefitofdata-driven
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
211/602
testautomationisthatitmakescreatingandrunninglotsoftestvariantsveryeasy(Strang,1996;FewsterandGraham,1999;Kaneretal.,
2001).Editingtestsandaddingnewsimilaronesiseasyandrequiresno
programmingskills(Kaner,1997).Anotherbenefitisthattestdatacanbedesigned
andcreatedbeforetestimplementationorevenbeforethetested
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
212/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
213/602
CHAPTER2.REQUIREMENTS
(Strang,1996;Pettichord,2003)anditisusableinmanualtestingevenifautomationimplementationisneverfinished.
Overall,thedata-drivenapproachisthekeyforease-of-useinlargescaletest
automation.
Thedata-drivenapproachalsohelpswithmaintenance.Whenthetested
systemchangesitisoftenpossibletochangeonlyeither
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
214/602
thetestdataorthetestcodeandtheirmaintenanceresponsibilitiescanalsobedividedbetweendifferentpeople.(Strang,1996;Kaner,1997;Marick,1997;
Kaneretal.,2001;Rice,2003)
Thebiggestlimitationofthe
data-drivenapproachisthatalltestcasesaresimilarandcreatingnewkinds
oftestsrequiresimplementingnewdriverscriptsthatunderstanddifferent
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
215/602
testdata.ForexampletestdatainFigure2.7anditsparsingscriptinListing2.1aredesignedfortestingcalculationswithonlytwo
numbersandwouldrequiremajorchangestohandlelongertestslike5*
8+2=42.Ingeneraltestdataanddriverscriptsare
stronglycoupledandneedtobesynchronizedifeitherchanges.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
216/602
Anotherdisadvantageofdata-driventestingistheinitialset-upeffortwhichrequiresprogrammingskillsandmanagement.(FewsterandGraham,1999)
2.5Keyword-Driven
Testing2.5.1IntroductionPrevioussectionintroduceddata-driventestingandstatedthatithas
multiplepromises.Italsomentionedthatitsbiggestlimitationisthatalltest
casesaresimilarandcreatingtotallynewtestsrequiresprogramming
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
217/602
effort.Asolutionforthislimitation,offeredbyFewsterandGraham(1999)andKaneretal.(2001)amongothers,isthekeyword-drivenapproachwhere
notonlythetestdatabutalsodirectivestellingwhattodowith
thedataaretakenfromtestscriptsandputintoexternalinputfiles.
Thesedirectivesarecalledkeywordsandtestengineerscanuse
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
218/602
themtoconstructtestcasesfreely.Thebasicideareadingtestdatafromexternalfilesandrunningtestsbasedonitstaysthesameasin
data-driventesting.AsFewsterandGraham(1999)putit,keyword-driventestingisa
logicalextensiontodata-driventesting.Exampleofakeyword-driventestdatafileis
seeninFigure2.8.
2.5.2EditingandStoring
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
219/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
220/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
221/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
222/602
beimplementedonlyonceandplacedintoacommon,reusablemodule.
Afterthetestdataisparsedthedriverscriptmustbe
abletointerpretkeywordsandexecutethespecifiedactionusingassignedarguments(Fewster
andGraham,1999).Tokeeptheframeworkmodularitisagoodidea
tohaveahandler(i.e.atestfunction)foreach
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
223/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
224/602
theinterfacelevel.Whentestinghigherlevelfunctionalitylikebusinesslogiclowlevelkeywordstendtomaketestcasesverylongandhigherlevel
keywords(e.g.Addinourcalculatorexample)aremuchmoreusable.Sometimesit
ispossibletouseonlyeitherloworhighlevelkeywordsandbuild
keywordhandlersaccordingly.Oftenbothlevelsareneededandthen
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
225/602
itisagoodideatoconstructhigherlevelkeywordsfromlowlevelkeywords.ForexamplelowlevelkeywordsInputandPushcouldbe
usedtocreatehigherlevelkeywordsAdd,Subtract,MultiplyandDivideandsimilarly
EqualscouldbecreatedusingPushandCheck.Figure2.10showshowthese
higherlevelkeywordsmaketestcasesshorter.
A
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
226/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
227/602
createnewones.
Amoreflexiblesolutionisallowingtestengineerstoconstructhigherlevelkeywordsusingthesameinterfacetheyuse
fordesigningtestcases.Thesekindoftechniquesarepresentedwithdifferentimplementation
bothbyBuwaldaetal.(2002)andNagle(2000).Figure2.12showsone
optionhownewkeywordscouldbeconstructed.To
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
228/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
229/602
CHAPTER2.REQUIREMENTS
Figure2.10:Keyword-driventestdatafilewithhigherlevelkeywords
Figure2.11:Creatinghigher
levelkeywordsintestlibrary
Figure2.12:Creatinghigherlevel
keywordsintestdesignsystem
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
230/602
CHAPTER2.REQUIREMENTS
differentiatethesenewkeywordsfromthoseimplementeddirectlyinthetestlibraryIliketocallthemuser
keywordsandbasekeywords,respectively.Testlibrariesdonothavehandlersforuser
keywords,insteadtheyareexecutedbyexecutingbasekeywords,orotheruserkeywords,
theyareconstructedfrom.
Themainbenefitin
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
231/602
makingitpossibletoconstructnewkeywordsusingthetestdesignsystemisthatnewkeywordscanbecreatedandmaintainedeasilywithoutprogramming
skills.Ofcourseitcanbearguedthatcreatinguserkeywordsisalready
programmingbutatleastitissoeasythateveryoneinvolvedwithtest
automationcanlearnit.Thebiggestproblemwiththeapproach
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
232/602
isthatprocessingtestdatagetsmorecomplicatedbut,onceagain,thesmallincreaseininitialeffortisinsignificantwhenbuildingalargescale
automationframework.ImplementationissuesarediscussedingreaterdetaillaterinChapter3
andespeciallyinSection3.3.
2.5.5PromisesandProblemsKeyword-driventesting
hasallthesamebenefitsasdata-driventesting.Asalready
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
233/602
mentionedthemainadvantageoverdata-driventestingisthatthekeyword-driventechniquemakesiteasyevenfornon-programmerstocreatenewkindsoftests.
ComparingFigures2.7and2.8provethispointprettyexplicitly.Keyword-driventestingis
abigstepforwardfrompuredata-driventestingwherealltestsaresimilar
andcreatingnewtestsrequiresnewcodeintheframework.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
234/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
235/602
keywords.Themainproblemofkeyword-driventestingisthatitrequiresamorecomplicatedframeworkthandata-driventesting.
2.6OtherImplementationIssues
Sameprocesses,methodologiesandtechniqueswhichareimportantwhendesigningandimplementingany
non-trivialsoftwaresystemareimportantalsowhenbuildinglargertestautomationframeworks(Zallar,
2001).ThoseimplementationissueswhichIfeelarethemost
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
236/602
importantarediscussedinthissection.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
237/602
CHAPTER2.REQUIREMENTS
2.6.1ImplementationLanguageProgramminglanguagesareneededwhenbuildingtestautomationtoolsandframeworks.Oftenautomationtoolsare
alsocontrolledusinginputscriptswhichhavetheirownlanguage.Requirementsforlanguage
usedwhenbuildingatoolandforlanguageusedwiththetoolcan
beverydifferent.Withmultipleprogramminglanguagesavailablepickingthe
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
238/602
rightchoiceindifferentsituationsisnotaneasytask.Selectingalanguagealreadyusedintheorganizationorknownotherwiseisalwaysa
safebetbutitisnottheonlycriteria.
SystemProgramming
Languages
SystemprogramminglanguagessuchasC++andJavaaredesigned
forbuildingdatastructures,algorithmsandlargesystemsfromscratch
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
239/602
(Ousterhout,1998).Theyareoptimizedforperformanceandscalabilityratherthanforfastprogrammingspeed.Theselanguagesmaywellbeusedwhenbuildingtest
automationtoolsandframeworksespeciallywhendevelopersknowthemwellalreadyanddo
notknowsuitablehigherlevelscriptinglanguages.Foralanguagetobeused
asaninputscriptforatooltheyaretoo
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
240/602
lowlevel.
ScriptingLanguages
ScriptinglanguagessuchasPerl,Python,Ruby,TclandVisualBasicarehigherlevellanguagesthan
systemprogramminglanguages.Ousterhout(1998)statesthatinsteadofbuildingthingsfromscratch
theyaredesignedforgluingcomponentsandsystemstogether.Ousterhout(1998)suggeststhat
programmingsomethingtakesfivetotentimeslesscodein
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
241/602
scriptinglanguagethaninsystemprogramminglanguage.Becauseprogrammerswriteapproximatelythesamenumberoflinesinsametimeregardlessthelanguage,scriptinglanguages
mayfastendevelopmenttimewiththesamemagnitude.Shorterprogramsarealsoeasier
tomaintain.
Ousterhout(1998)mentionsthatscriptinglanguagesaresomewhatslower
thansystemprogramminglanguages,butcurrentcomputersaresofast
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
242/602
thatitisnormallynotimportant.Pettichord(1999)notesthatintestautomationthisslightperformancedifferenceishardlyeveraproblembecausemost
ofthetimeisspentwaitingforthetestedsystemtorespondfor
inputsorsettinguptestenvironment.
Ihaveexperienceusingboth
systemprogrammingandscriptinglanguagesfortestautomationandI
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
243/602
cansecondallthepositivethingsOusterhout(1998)saysabout
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
244/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
245/602
files.Shellscriptsarenotasflexibleashigherlevelscriptinglanguagesbuttheyareveryhandyforsmalltasks.Iuseshellscripts
regularlyforsmallautomationtasksbutIconsiderthemtoocumbersomeforimplementing
largerframeworks.
VendorScripts
Vendorscriptsareproprietarylanguages
implementedbytoolvendorstobeusedwiththesetools.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
246/602
Testtoolsusingvendorscriptshouldbeapproachedwithacertaincautionforreasonslistedbelow.
Nobodyknowsthem.Developersand
sometestengineersarelikelytoknowmanyprogramminglanguagesbutnotlikely
toknowsomespecialvendorscript.Hiringpeoplewithpreviousknowledgeishard.
(Pettichord,2001)Notmatureenough.Vendorscriptsdonot
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
247/602
havethosefeaturesmatureprogramminglanguagesandtheirsupportlibrarieshave.Ifvendorscriptdoesnothaveforexamplegoodenoughstringhandlingcapabilities
youhavetoimplementthemyourself.(Pettichord,2001)Vendorscriptshavecompatibility
problemswithothertoolsandlanguages.(Rice,2003)Becauseofthesereasonsmy
adviceisavoidingvendorscriptsaltogetherifpossible.Luckilythey
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
248/602
aregettingrarenowadaysasvendorshavestartedembeddingexistingmaturescriptinglanguagestotheirproducts.ExamplesofthisareMercurysQuickTestProwhich
usesVisualBasicandPushToTestsTestMakerusingJython(aJavaimplementationofthe
Pythonlanguage).
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
249/602
CHAPTER2.REQUIREMENTS
2.6.2ImplementationTechniqueManualProgramming
Manualscriptwritingisanormalprogrammingtaskandifdone
properlyscriptsareeasytouseandmaintain.Ithastwoobviousdrawbacks:
writingscriptstakessometimeandrequiresprogrammingskillswhichtestengineersmay
nothave.(FewsterandGraham,1999;MosleyandPosey,2002)
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
250/602
CaptureandReplay
Captureandreplaytoolsaresupposedtoovercomeproblemswithmanualscriptwriting.Theideaisthat
youpusharecordbuttonandletthetoolrecordeveryactionkeystroke,mouse
movementorclickwhileexecutingtestmanually.Thetoolstorescapturedactionsina
scriptforlaterplayback.Thisapproachsoundsattractiveand,as
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
251/602
Meszaros(2003)note,itmaybesuitableforsituationswherescriptsareneededfast,theycanbetrashedinthefutureandprogrammingskills
requiredformanualscriptingarenotavailable.
Thebiggestproblemwith
captureandreplayisthatmaintainingrecordedscriptsisnearlyimpossible.Moreproblems
associatedwithitarelistedforexamplebyKaner(1997),
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
252/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
253/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
254/602
isoutofthescopeofthisthesisbutthissectiongoesthroughfewaspectsIfeelmostimportantwhentalkingaboutlargertest
automationframeworks.
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
255/602
CHAPTER2.REQUIREMENTS
VersionControl
Versioncontrolandconfigurationmanagementarerequiredforanylargersoftwareprojectandtest
automationframeworksarenoexception.Maintainingalltheneededtestwareanditsdifferent
versionsisanightmarewithoutanadequateversioncontrolsystem.(FewsterandGraham,
1999;Zallar,2001;Buwaldaetal.,2002)
Coding
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
256/602
andNamingConventions
Codingandnamingconventionscovereverythingfromvariable,functionandfilenamingtocodeindentation.Theideaofcoding
conventionsisthatwheneveryonewritessimilarcodeitistheneasyto
understandandmaintainbyanyone.Consistentnamingofmodules,functionsandfilesgreatly
helpswithreusewhenmoduleandfunctionnamesareeasy
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
257/602
torememberandevenguess.(FewsterandGraham,1999)
Thesekindsofconventionareveryimportantwhenanythinglargerisprogrammedand
shouldnotbeforgottenwhenbuildinglargetestautomationsystems.Conventionsshouldbe
agreedanddocumentedwhenstartingatestautomationproject.Oftenthisiseasy
asconventionsusedwhenimplementingtestedsystemscanbeused
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
258/602
straightawayorwithslightmodifications.
Documentation
Lackofadequatedocumentationmakesusingtestlibrarieshardbecausepeoplenewto
themneedtospenttimefindinghowandunderwhatcircumstancestouse
them.Documentationisalsoneededinmaintenance.Needforqualitydocumentationiscommonly
understoodbuttherearetwoproblems:firstthedocumentationmust
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
259/602
bewrittenandthenitmustbekeptupdated.(FewsterandGraham,1999)
Forcodedocumentationpurposessemi-automaticdocumentationsystemssuchas
Javadocaregettingmoreandmorepopular.Withthesetoolstheoriginaldocumentation
iskeptwithinthecodesothatitisavailablewhencodeis
readandeditedanditcaneasilybekeptupdated
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
260/602
withthecode.Documentationcanbeextractedwithspecialtoolssothatitisreadablewithawebbrowserorsomeotherviewer.Tools
canalsoreadfunctionandvariablenamesfromthecodesothatthey
donotneedtobedocumentedmanually.
Codedocumentationisvery
importantbutnotenoughonitsown.Otherneeded
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
261/602
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
262/602
CHAPTER2.REQUIREMENTS
documentsincludeinstructionsforsettingup,usingandmaintainingtheframework.Itisthequalityofthedocumentation,
notitsquantity,thatisimportant.(FewsterandGraham,1999)
2.7
TestabilityMostoftenthehardestpartofatestautomationeffortisintegrating
theselectedtoolwiththetestedsystem.Pettichord(2002)and
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
263/602
Kaneretal.(2001)statethatinsteadoffinetuningthetooltohandleallpossiblequirksofthetestedsystemitmightbe
bettertoinvestonincreasingthetestabilityofthesystem.Pettichord(2002)has
noticedthatthesuccessofthewholeautomationprojectmaywelldependon
testabilityissuesandalsoothers,e.g.FewsterandGraham(1999),
-
7/31/2019 Data-Driven and Keyword-Driven Test[1]
264/602
Kaneretal.(2001)andBach(2003),underlineimportanceoftestability.Testabilityproblemsaresimilarinsmallandlargescaletestautomationprojectsbut
inlargerscaleitspositiveeffecttomaintenancegetsmoreandmoreimportant.
Pettichord(2002)definestestabilityasvisib