Granules: A Lightweight Runtime for Scalable Computing...

201

Chapter 9

Granules: A Lightweight Runtime for Scalable Computing with Support for Map-Reduce

Shrideep Pallickara, Jaliya Ekanayake, and Geoffrey Fox

Contents9.1 Introduction.............................................................................................2029.2 NaradaBrokering......................................................................................2039.3 Granules...................................................................................................205

9.3.1 ComputationalTask.....................................................................2059.3.2 DatasetsandCollections...............................................................2069.3.3 SpecifyingaSchedulingStrategy...................................................2069.3.4 Finite-StateMachineforaComputationalTask...........................2079.3.5 InterleavingExecutionofComputationalTasks............................208

9.3.5.1 SizingThread-Pools........................................................2089.3.6 Diagnostics...................................................................................208

K10175_C009.indd 201 3/2/2010 12:52:27 PM

202 ◾ Cloud Computing and Software Services: Theory and Practice

9.1 IntroductionCloud computing has gained significant traction in recent years. By facilitatingaccesstoanelastic(meaningtheavailableresourcepoolthatcanexpandorcon-tractovertime)setofresources,cloudcomputinghasdemonstrableapplicabilitytoawiderangeofproblemsinseveraldomains.

Appealing featureswithin cloud computing include access to a vastnumberof computational resources and inherent resilience to failures.The latter featurearises,because incloudcomputingthefocusofexecutionisnotaspecific,well-knownresourcebutratherthebestavailableone.Anothercharacteristicofalotofprogramsthathavebeenwrittenforcloudcomputingisthattheytendtobestate-less.Thus,when failuresdo takeplace, theappropriatecomputationsare simplyrelaunchedwiththecorrespondingdatasets.

Amongthe forces thathavedriven theneed forcloudcomputingare fallinghardwarecostsandburgeoningdatavolumes.Theabilitytoprocurecheaper,morepowerfulCPUscoupledwithimprovementsinthequalityandcapacityofnetworkshavemadeitpossibletoassembleclustersatincreasinglyattractiveprices.Thepro-liferationofnetworkeddevices,Internetservices,andsimulationshasresultedin

9.4 SupportforMap-ReduceinGranules......................................................2099.4.1 TwoSidesoftheSameCoin.........................................................2109.4.2 SettingUpGraphs........................................................................ 2119.4.3 CreatingComputationalPipelines................................................ 2119.4.4 ObservingtheLifeCycleofaPipeline..........................................212

9.5 DevelopingandDeployingApplicationsUsingGranules.........................2129.5.1 DevelopingApplications...............................................................212

9.5.1.1 Initialization...................................................................2139.5.1.2 ProcessingLogic.............................................................2139.5.1.3 SchedulingStrategy........................................................213

9.5.2 DeployingApplicationsUsingGranules.......................................2149.5.2.1 InitializingCommunicationsandResourceDiscovery.....2149.5.2.2 InitializingandDeployingComputationalTasks...........2149.5.2.3 Tracking/SteeringaDeployedApplication..................... 215

9.6 RelatedWork........................................................................................... 2159.7 Benchmarks.............................................................................................217

9.7.1 StreamingSubstrate......................................................................2179.7.2 InformationRetrieval:Exactly-Once............................................2189.7.3 K-Means:Iterative.........................................................................2199.7.4 PeriodicScheduling......................................................................2209.7.5 DataDriven..................................................................................2219.7.6 AssemblingmRNASequences.......................................................222

9.8 Conclusions..............................................................................................224References.........................................................................................................224

K10175_C009.indd 202 3/2/2010 12:52:27 PM

Granules ◾ 203

largevolumesofdatabeingproduced.This,inturn,hasfueledtheneedtoprocessandstorevastamountsofdata.Thesedatavolumescannotbeprocessedbyasinglecomputerorasmallclusterofcomputers.Furthermore,inmostcases,thisdatacanbeprocessedinapleasinglyparallelfashion.Theresulthasbeentheaggregationofalargenumberofcommodityhardwarecomponentsinvastdatacenters.

Map-Reduce[1], introducedbyDeanandGhemawatatGoogle, is themostdominantprogrammingmodelfordevelopingapplicationsincloudsettings.Here,largedatasetsaresplitintosmaller,moremanageablesizes,whicharethenprocessedbymultiplemapinstances.Theresultsproducedbyindividualmapfunctionsarethensenttoreducers,whichcollatethesepartialresultstoproducethefinaloutput.A clear benefit of such concurrent processing is a speed-up that is proportionaltothenumberofcomputationalresources.Map-ReducecanbethoughtofasaninstanceoftheSPMD[2]programmingmodelforparallelcomputingintroducedbyFedericaDarema.ApplicationsthatcanbenefitfromMap-Reduceincludedataand/ortask-parallelalgorithmsindomainssuchasinformationretrieval,machinelearning,graphtheory,andvisualization,amongothers.

In this chapter,which is an extendedversionofourpaper [21],wedescribeGranules [3], a lightweight streaming-based runtime for cloud computing.Granulesallowsprocessing tasks tobedeployedona single resourceora setofresources.BesidesthebasicsupportforMap-Reduce,wehaveincorporatedsup-portforvariantsoftheMap-Reduceframeworkthatareparticularlysuitableforscientificapplications.UnlikemostMap-Reduceimplementations,Granulesusesstreaming fordisseminating intermediate results, asopposed tousingfile-basedcommunications.Thisleadstodemonstrablybetterperformance(seebenchmarksinSection9.7).

Thischapterisorganizedasfollows.InSection9.2,weprovideabriefoverviewoftheNaradaBrokeringsubstratethatweuseforstreaming.WediscusssomeofthecoreelementsofGranulesinSection9.3.Section9.4outlinesoursupportforMap-Reduceandforthecreationofcomplexcomputationalpipelines.Section9.5describestheprocessofdevelopinganddeployingapplicationsusingGranules.InSection9.6,wedescriberelatedworkinthisarea.InSection9.7,weprofileseveralaspectsoftheGranulesruntime,andwherepossible,contrastitsperformancewithcomparablesystems,suchasHadoop,Dryad,andMPI(MessagePassingInterface).InSection9.8,wepresentourconclusions.

9.2 NaradaBrokeringGranulesuses theNaradaBrokering[4–6]streamingsubstrate(developedbyus)forallitsstreamsdisseminations.TheNaradaBrokeringcontentdistributionnet-work(depictedinFigure9.1)comprisesasetofcooperatingrouternodesknownasbrokers.Producersandconsumersdonotdirectlyinteractwitheachother.Entities,whichareconnected tooneof thebrokerswithin thebrokernetwork,use their

AQ2

AQ3

AQ4

AQ5

AQ1

K10175_C009.indd 203 3/2/2010 12:52:27 PM


hostingbroker to funnel streams into thebrokernetworkand, fromthereon, tootherregisteredconsumersofthosestreams.

NaradaBrokeringisapplicationindependentandincorporatesseveralservicestomitigatenetwork-inducedproblemsasstreamstraversedomainsduringdissemi-nations.Thissystemprovisionseasy-to-useguaranteeswhiledeliveringconsistentandpredictableperformancethatisadequateforuseinreal-timesettings.

Consumersofagivendatastreamcanspecify,veryprecisely,theportionsofthedatastreamthattheyareinterestedinconsuming.Bypreferentiallydeployinglinksduringdisseminations,theroutingalgorithm[4]inNaradaBrokeringensuresthattheunderlyingnetworkisoptimallyutilized.Thispreferentialroutingensuresthatconsumersreceiveonlythoseportionsofstreamsthatareofinteresttothem.Sinceagivenconsumeristypicallyinterestedinonlyafractionofthestreamspresentinthesystem,preferentialroutingensuresthataconsumerisnotdelugedbystreamsthatitwillsubsequentlydiscard.

Thesystemincorporatessupportforreliablestreamingandsecurestreaming.Inreliablestreaming,thesubstratecopeswithdisconnectsandprocess/linkfailuresofdifferentcomponentswithinthesystemwiththeabilitytofine-tuneredundancies[5]foraspecificstream.Securestreaming[6]enforcestheauthorizationandcon-fidentialityconstraintsassociatedwiththegenerationandconsumptionofsecurestreamswhilecopingwithdenial-of-serviceattacks.

Some of the domains that NaradaBrokering has been deployed in includeearthquakescience,particlephysics,environmentalmonitoring,geosciences,GISsystems,anddefenseapplications.AQ6

Producers or consumers

Broker node

Figure 9.1 NaradaBrokering broker network.

K10175_C009.indd 204 3/2/2010 12:52:29 PM

Granules ◾ 205

9.3 GranulesGranulesorchestratestheconcurrentexecutionofprocessingtasksonadistributedsetofmachines.Granules is itselfdistributed,and its componentspermeatenotonlythecomputationalresourcesonwhichitinterleavesprocessing,butalsothedesktopfromwheretheapplicationsarebeingdeployedinthefirstplace.Therun-timemanagestheexecutionofasetofprocessingtasksthroughvariousstagesoftheirlifecycle:deployment,initialization,execution,andtermination.Figure9.2depictsthevariouscomponentsthatcompriseGranules.

9.3.1 Computational TaskThemostfundamentalunitinGranulesisthenotionofacomputational task.Thiscomputational taskencapsulatesprocessing functionality, specifies its schedulingstrategy, and operates on different types of datasets. These computational taskscantakeonadditionalinterchangeableroles(suchasmapandreduce)and,whencascaded,canformcomplexexecutionpipelines.

Computationaltasksrequirethedomainspecialiststospecifyprocessingfunc-tionality.Thisprocessingtypicallyoperatesuponacollectionofdatasetsencapsu-latedwithinthecomputationaltask.

Thecomputational task encapsulates functionality forprocessing for a givenfine-grainedunitofdata.Thisdatagranularitycouldbeapacket,afile,asetoffiles,oradatabaserecord.Forexample,acomputationaltaskcanbewrittentoevaluatearegularexpressionquery(grep)onasetofcharacters,afile,orasetoffiles.Insome

R1

GR2

GRN

G

C CG

AG

Discovery services

Life-cycle management

Diagnostics

Deployment manager

Concurrent execution

queueDataset

initializer

Content distribution network

A A

GranulesGC Client Computational resourcesRX

Granules

Figure 9.2 Overview of the Granules runtime.

K10175_C009.indd 205 3/2/2010 12:52:29 PM


cases,therewillnotbeaspecificdataset;rather,eachcomputationaltaskinstanceinitializesitselfusingarandom-seedgenerator.

Computationaltasksincludeseveralmetadata,suchasversioninginformation,timestamps,domainidentifiers,andcomputationidentifiers.Individualinstancesofthecomputationaltasksincludeinstanceidentifiersandtaskidentifiers,whichinturnallowsustogroupseveralrelatedcomputationaltaskstogether.

9.3.2 Datasets and CollectionsIn Granules, datasets are used to simplify access to the underlying data type.Datasetscurrently supportedwithinGranules include streamsandfiles; supportfordatabasesisbeingincorporated.Foragivendatatype,besidesmanagingtheallocationandreclamationofassortedresources,Granulesalsomediatesaccesstoit.Forexample,Granulesperformsactionsrelatedtosimplifyingtheproductionandconsumptionofstreams,readingandwritingoffiles,andtransactionalaccesstodatabases.

Adatacollectionisassociatedwitheverycomputationaltask.Adatacollectionrepresentsacollectionofdatasets,andmaintainsinformationaboutthetype,num-ber,andidentifiersassociatedwitheveryencapsulateddataset.

Allthatthedomainspecialistneedstospecifyisthenumberandtypeofdatas-etsinvolved.Thesystemimposesnolimitsonthenumberofdatasetswithinadata-set collection.During initializationsof thedataset collection,dependingon thetypeoftheconstituentdatasets,Granulessubscribestotherelevantstreams,con-figuresaccesstofilesonnetworkedfilesystems,andsetsupconnections(JDBC)tothedatabases.

Datasetcollectionsallowobservers tobe registered to trackdataavailability,datasetinitializations,andclosure.Thissimplifiesdataprocessing,sinceitobviatestheneedtoperformpolling.

9.3.3 Specifying a Scheduling StrategyComputational tasks specify a scheduling strategy, which in turn governs theirlifetimes.Computationaltaskscanspecifytheirsched-ulingstrategyalongthreedimensions(seeFigure9.3).Thecountsaxisspecifiesthenumberoftimesacompu-tationaltaskneedstobeexecuted.Thedata drivenaxisspecifiesthatthecomputationaltaskneedstobesched-uled for execution whenever data is available on anyoneofitsconstituentdatasets.Theperiodicityaxisspec-ifiesthatcomputationaltasksbeperiodicallyscheduledforexecutionatpredefinedintervals(specifiedinms).

Eachoftheseaxescanextendtoinfinity,inwhichcase, it constitutes a stay-alive primitive. A domain

AQ7

Counts

Data driven

Periodicity

Figure 9.3 Dimensions for scheduling strategy.

K10175_C009.indd 206 3/2/2010 12:52:30 PM

Granules ◾ 207

specialistcanalsospecifyacustomschedulingstrategythatpermutesalongthesethreedimensions.Thus,onecanspecifyaschedulingstrategythatlimitsacompu-tationaltasktobeexecutedamaximumof500timeseitherwhendataisavailableoratregularintervals.

Acomputationaltaskcanchangeitsschedulingstrategyduringexecution,andGranuleswill enforce thenewly established scheduling strategyduring thenextround of execution (Section 9.3.5). This scheduling change can be a significantone—fromdatadriventoperiodic.Theschedulingchangecouldalsobeaminoronewithchangestothenumberoftimesthecomputationneedstobeexecuted,orwithanupdatetotheperiodicityinterval.

In addition to the aforementioned primitives, another primitive—stay alive until termination condition reached—canbespecified.Inthiscase,thecomputa-tionaltaskcontinuestobe“stayalive”untilthecomputationaltaskassertsthatitsterminationconditionhasbeenreached.Theterminationconditionoverridesanyotherprimitivesthatmayhavebeenspecifiedandresultsinthegarbagecollectionofthecomputationaltask.

9.3.4 Finite-State Machine for a Computational TaskAt a given computational resource, Granules maintains a finite-state machine(FSM)foreverycomputationaltask.ThisFSM,depictedinFigure9.4,hasfourstates:initialize,activated,dormant,andterminate.

The transition triggers for this FSM include external requests, elapsed timeintervals,dataavailability,resetcounters,andassertionsoftheterminationcondi-tionbeingreached.

Whenacomputationaltaskisfirstreceivedinadeploymentrequest,Granulesproceedstoinitializethecomputationaltask.TheFSMcreatedforthiscomputa-tionaltaskstartsoffintheinitializestate.

If, for some reason, the computational task cannotproceed in its execution,eitherbecausethedatasetsarenotavailableorthestart-uptimehasnotyetelapsed,thecomputationaltasktransitionsintothedormantstate.Iftherewereproblemsininitialization,thecomputationaltasktransitionsintotheterminatestate.

Terminate

Initialize

Dormant

Activated

Figure 9.4 FSM for a computational task.

K10175_C009.indd 207 3/2/2010 12:52:30 PM


If,ontheotherhand,thecomputationaltaskwasinitializedsuccessfully,andisreadyforexecutionwithaccessibledatasets,ittransitionsintotheactivatedstate.

9.3.5 Interleaving Execution of Computational TasksAteachcomputationalresource,Granulesmaintainsapoolofworkerthreadstomanageandinterleavetheconcurrentexecutionofmultiplecomputationaltasks.

Whenacomputational task isactivatedandreadyforexecution, it ismovedintotheactivatedqueue.Asandwhenworkerthreadsbecomeavailable,thecom-putationaltasksarepulledfromtheFIFOqueueandexecutedinaseparatethread.Uponcompletionofthecomputationaltask,theworkerthreadisreturnedbacktothethread-pool,tobeusedtoexecuteotherpendingcomputationaltaskswithintheactivatedqueue.ThecomputationaltaskisplacedeitherinthedormantqueueorscheduledforgarbagecollectiondependingonthestateofitsFSM.

Afteracomputationaltaskhasfinisheditslatest(orthefirst)roundofexecu-tion,checksaremadetoseeifitshouldbeterminated.Todoso,theschedulingstrategy associated with the computational task is retrieved. If a computationaltaskneedstoexecuteafixednumberoftimes,acheckismadetoseeifthecounterhasreset.Ifthecomputationaltaskspecifiesastay-aliveprimitivebasedeitherondataavailabilityorperiodicity,checksaremadetoseeifthedatasetscontinuetobeavailableoriftheperiodicityintervalhaselapsed.Acheckisalsomadetoseeifthecomputationaltaskhasassertedthatitsterminationconditionhasbeenreached.

Ifnoneofthesechecksindicatethatthecomputationaltaskshouldbetermi-nated,itisscheduledforanotherroundofexecutionorittransitionsintothedor-mantstate.Acomputationaltaskcancontinuallytogglebetweenthedormantandtheactivatedstatetillaterminationconditionhasbeenreached.

9.3.5.1 Sizing Thread-Pools

Thenumberofworkerthreadswithinthethread-poolisconfigurable.Ingeneral,thenumberofthreadsneedstobebalancedsothattheaccruedconcurrencygainsarenotoffsetbycontext-switchingoverheadsamongthethreads.Asageneralrule,it isagoodideatoset thisnumbertobeapproximatelyequal tothenumberofexecutionpipelinesavailableonagivenmachine.Thus,foraquad-coreCPUwithtwoexecutionpipelinespercore,thethread-poolwillbesetuptohaveapproxi-matelyeightthreads.

9.3.6 DiagnosticsInGranules,ausercantrackthestatusofaspecificcomputationaltaskorcollec-tions(job)ofcomputationaltasks.Thesystemmaintainsdiagnostic informationabouteverycomputational task.This includes informationabout thenumberoftimesacomputationaltaskwasscheduledforexecution,itsqueuingoverheads,its

AQ8

K10175_C009.indd 208 3/2/2010 12:52:31 PM

Granules ◾ 209

CPU-boundtime,thetimeitwasmemory-resident,andthetotalexecutiontime.Acomputationaltaskcanalsoassertthatdiagnosticmessagesbesentbacktothecli-entduringany(orsome)ofitsstatetransitions.Ontheclientside,anobservercanberegisteredforcollectionsofcomputationaltaskstotracktheirprogresswithouttheneedtoactivelypollindividualcomputationaltasks.

9.4 Support for Map-Reduce in GranulesMap-Reduce is the dominant framework used in cloud computing settings. InMap-Reduce,alargedatasetisbrokenupintosmallerchunksthatareconcurrentlyoperateduponbymapfunctioninstances.Theresultsfromthesemapfunctions(usually,<key,value>pairs)arecombinedinthereducers,whichcollatethevaluesfor individualkeys.Typically, therearemultiple reducers, and theoutputs fromthesereducersconstitutethefinalresult.ThisisdepictedinFigure9.5.

TheMap-Reduceframeworkhasseveraladvantages.First,thedomainscientistonlyneedstoprovidetheMap-Reducefunctionalityandthedatasets.Second,itistheresponsibilityoftheframeworktotransparentlyscaleasthenumberofavailableresources,andtheproblemsize,increases.Finally,theorchestrationoftheconcur-rentdata-parallelexecutionismanagedbytheframework.

IntraditionalMap-Reduce,intermediatestagesexchangeresultsusingasetof<key,value>pairs.Wehaveincorporatedsupportforthisbasicresulttype.Butwehavealsoincorporatedsupportforexchangeofprimitivedatatypes,suchasint,short,boolean,char,long,float,anddouble.Wehavealsoincorpo-ratedsupportforexchangingarrays([])and2Darrays([][])oftheseprimitivedatatypes.ThereisalsosupportforexchangingObjectsthatencapsulatedcompounddatatypes,alongwitharraysand2DarraysoftheseObjects.

TheintermediateresultsinmostMap-ReduceimplementationsutilizefileIOfor managing results produced by the intermediate stages. The framework thennotifiesappropriatereducerstopullorretrievetheseresultsforfurtherprocessing.

d1

d2

dN

Map1

Map2

MapN

Reduce1

ReduceK

o1

oK

Dataset

Figure 9.5 Basic Map-Reduce framework.

K10175_C009.indd 209 3/2/2010 12:52:31 PM


Dependingontheapplication,theoverheadsintroducedbyperformingsuchdisk-IO can be quite high. In Granules, we use streaming to push these resultsontoappropriatereducers.Streaming,asvalidatedbyourbenchmarks(describedinSection9.7),issignificantlyfaster,andwethinkthatthereareseveralclassesofapplicationsthatcanbenefitfromthis.

Additionally,sincetheresultsarebeingstreamedasandwhentheyhavebeencomputed, successive stages have access to partial results from preceding stagesinsteadofwaitingfortheentirecomputationtocomplete.Thisisparticularlyuse-fulinsituationswhereoneisinterestedingettingasmanyresultsaspossiblewithinafixedamountoftime.

9.4.1 Two Sides of the Same CoinInGranules,mapandreducearetworolesassociatedwiththecomputationaltask.Theserolesinheritallthecomputationaltaskfunctionality,whileaddingfunction-alityspecifictotheirroles.

The map role adds functionality related to adding, removing, tracking, andenumeratingthereducersassociatedwiththemapfunction.Typically,amapfunc-tionhasonereducerassociatedwithit.InGranules,wedonotlimitthenumberofreducersassociatedwithamapfunction.Thisfeaturecanbeusedtofine-tuneredundancieswithinacomputationalpipeline.

Thereduceroleaddsfunctionalityrelatedtoadding,removing,tracking,andenumeratingmapsassociatedwith it.Thereducerhas facilities totrackoutputgeneratedbytheconstituentmaps.Specifically,areducercandetermineifpar-tialorcompleteoutputshavebeenreceivedfromthemaps.Thereducerolealsoincorporates support to detect and discard any duplicate outputs that may bereceived.

Themapand reduce roleshave facilities to create andpublish results.Thepayloadsfortheseresultscanbeprimitivedatatypesthatwediscussedearlier,Objects encapsulatingcompounddata types,<key,value>pairs,arrays,and2Darraysofthesame.InGranules,generatedresultsincludesequencinginfor-mationandmetadataspecifictothegenerator.Additionally,anentityisallowedto assert if these results are partial results and/or if the processing has beencompleted.

SincemapandreducearetworolesofthecomputationaltaskinGranules,theyinherit functionality related to scheduling strategy (and life-cyclemanagement),diagnosticstrategy,anddatasetmanagement.

Individual map and reduce instances toggle between the activated and dor-mant states (Section9.3.5) till such time that theyare ready toassert that theirterminationconditionhasbeenreached.Forexample,areducermayassertthatithasreacheditsterminationconditiononlyafterithasreceived,andprocessed,theoutputsofitsconstituentmaps.

K10175_C009.indd 210 3/2/2010 12:52:31 PM

Granules ◾ 211

9.4.2 Setting Up GraphsGranulessupportsasetofoperationsthatallowgraphstobesetup.Individualmaps can add/remove reducers. Similarly, reducers are allowed to add/removemaps. The functions are functionally equivalent. Granules also allows the mapandreducerolestobeinterchangable:amapcanactasareducer,andviceversa.Figure9.6depictshowsupportforaddition/removalofrolescombinedwithroleinterchangeability can be used to create a graph with a feedback loop. In ourbenchmarks, involving thek-meansmachine learning algorithm,wehave threestageswithafeedbackloopfromtheoutputofstage2toitsinput.Granulesman-agesoverheadsrelatedtoensuringthattheoutputsfromthemapareroutedtothecorrectreducers.

Additionally,Granulescancreateexecutiongraphsoncethenumbersofmapandreduceinstancesinapipelinehavebeenspecified.Granulesensurestheappro-priatelinkageoftheMap-Reduceinstances.

9.4.3 Creating Computational PipelinesTypically, inMap-Reduce,theinstancesthatcompriseanexecutionpipelineareorganized in a directed acyclic graph (DAG), with the execution proceeding insequencethroughmonotonicallyincreasingstages.

InGranules,wehaveincorporatedsupportforcyclestobepresent.ThisallowsGranulestofeedbacktheoutputsofsomestage,withinapipeline,toanyofitspre-cedingstages.Thesystemplacesnorestrictionsonthespanlength,orthenumber,ofthefeedbackinthepipeline.InasenseitcanbearguedthatGranulessupportsbothdata-andcontrol-flowgraphs.AnexampleofsuchacomputationalgraphinGranulesisdepictedinFigure9.7.

One feature of the computational task plays a role in allowing these loops:the notion of the stay-alive computation. Furthermore, since this is available atthemicrolevel(computationaltask),individualstages,collectionofstages,orthe

AQ9

M1

M2

M3

M4

R1

R2

RMRM.addreduce (M1)RM.addreduce (M2)RM.addreduce (M3)RM.addreduce (M4)

Figure 9.6 Creating a simple feedback loop.

K10175_C009.indd 211 3/2/2010 12:52:32 PM


computationalpipelineitselfcanbedependentoniterative,periodic,data-driven,orterminationconditions.

Granulesmanagesthepipelinecomplexity.Thedomainscientistdoesnotneedtocopewithfan-incomplexity,whichcorrespondstothenumberofunitsthatfeedresultsintoagiveninstance.Onceapipelinehasbeencreated,adomainspecialistdoesnothavetocopewithIO,synchronization,ornetworking-relatedissues.Theruntimeincludesfacilitiestotrackoutputsfromprecedingstages.

9.4.4 Observing the Life Cycle of a PipelineAt the client side, during the deployment process, Granules allows a life-cycleobserver toberegistered foranexecutionpipeline.Thisobserverprocessesdiag-nosticmessagesreceivedfromdifferentcomputationalresourcesrunningGranules.Thesediagnosticmessagesrelate tostate transitionsassociatedwiththedifferentcomputational task instances (and the map and reduce roles) and the pertinentmetricsassociatedwiththecomputationtask.Thelife-cycleobserverreportstotheclientuponcompletionofanexecutionpipeline.Theobserveralsoreportserrorsintheexecutionofanyoftheunitsthatcomprisethepipeline.

9.5 Developing and Deploying Applications Using Granules

Inthissection,wedescribetheprocessofdevelopinganddeployingapplicationsusingGranules. Inboth cases,Granules incorporates support forutility classes,whosebehaviormaybeextendedtosuitspecificneeds.

9.5.1 Developing ApplicationsGranules simplifies the process of developing applications. Developers sim-ply extend theMapReduceBase class.This class implements functionality thatencompassesboththemapandreducerolesofacomputation.Onerequirementisthatthederivedclasshasexactlyoneconstructor,whichdoesnottakeanyargu-ments.Developersof thederived classonlyneed to implement theexecute()

Stage 1 Stage 2 Stage 3

Figure 9.7 Creating pipelines with cycles.

K10175_C009.indd 212 3/2/2010 12:52:33 PM

Granules ◾ 213

method.Typicalstepsinvolvedinimplementingthismethodincludeinitializationofthedatasetsanddatastructures,processinglogic,andspecificationofaschedul-ingstrategy.

9.5.1.1 Initialization

Typically, depending on the type of the dataset, initialization of the datas-ets involved in the processing is performed automatically. The designer sim-plyspecifiestheidentifiersforthedataset.Initializationsofthedatastructuresneededbythecomputationcanbeperformedeitherinthenullconstructororintheexecute()method.Inthelattercase,caremustbetakentoensurethattheinitializationsareperformedonlyonceacrosssuccessiveinvocationsoftheexecute()method.

9.5.1.2 Processing Logic

Theprocessinglogicwithintheexecute()methodisdomainspecific.Thispro-cessingwouldinvolveeitherthegenerationofresults,orthemanagementandcol-lationofpreviouslyproducedresults.Inthereducerole,itisalsopossibletocheckifoutputshavebeenreceivedfromalltheprecedingmapsinadditiontodiscardinganyduplicateresultsthatweregenerated.

Thegenerationofresultsiseasy,andthesystemallowsentitiestoattachdiffer-entpayloadstotheseresults.Thesystemcurrentlyallowsforthepayloadsfortheseresultstobe<key,value>pairs,wheretheelementsofthesetuplescouldbeobjectsthatencapsulatecompounddatatypes.Thesystemallowsinstances,arrays([]),and2Darrays ([][]) of primitive data types such asint,short,long,double,float,andchartobeattachedaspayloadsoftheseresults.Thesystemhandlesthemarshallingandun-marshallingofthesepayloadsautomatically.

Theprocessing logic alsoneeds to copewith exceptions thatwillbe thrownasresultsoftheprocessing.Theseexceptionscouldresultfromproblemswiththedatasets,marshallingissues,andnetworkingproblems.

9.5.1.3 Scheduling Strategy

Acomputational task can change its scheduling strategyduring execution.Thischangeisreflectedduringthenextiterationoftheexecute()method.Thesys-temenforcesthenewlycreatedschedulingstrategyassoonasthecurrentiterationoftheexecute()methodterminates.Computationaltasksthathavespecifiedaschedulingstrategythatconstituteseitherastay-aliveprimitive,orimpliesacertainnumberofiterations,canassertthattheirterminationconditionhasbeenreached.Atthistime,thecomputationaltaskisscheduledforgarbagecollectionassoonascontrolreturnsfromtheexecute()method.

K10175_C009.indd 213 3/2/2010 12:52:33 PM


9.5.2 Deploying Applications Using GranulesGranules provides a helper class, the InstanceDeployer, to enable applications,andthecomputationaltasksthatcompriseit,tobedeployedonasetofresources.This class performs several operations related to initializing communications,resource discovery, and deployment of computations. It is recommended that adeployerbecreatedforeachapplication.ThiscanbedonebysimplyextendingtheInstanceDeployer.

9.5.2.1 Initializing Communications and Resource Discovery

Thefirst stepthatanapplicationdeployerneeds toperformis to initializecom-municationswiththecontentdistributionnetwork(NaradaBrokering).Thiscanbeperformedbyinvokingtheconstructorforthebaseclass(InstanceDeployer),whichtakesasetofpropertiesasitsargument.Thisistypicallydonebyinvokingthesuper(streamingProperties)inthederivedclass’sconstructor.Someoftheelementsthataretypicallypartofthissetofpropertiesincludethehostname,theport, and the transport type foroneof the routernodeswithin the contentdisseminationnetwork.Dependingonthetransportoverwhichcommunicationstakeplace,therewouldbeadditionalelementsthatmayneedtobespecified.Forexample,iftheSSLcommunicationsareused,additionalelementsthatneedtobespecifiedincludethelocationsofthetruststoreandthekeystorethatwouldbeusedforsecurecommunications.

Oncecommunicationshavebeenestablished,Granulesautomaticallydiscov-ersresourcesthatarecurrentlyavailable.Thislistcouldbeperiodicallyrefreshedshouldtheneedarise.

9.5.2.2 Initializing and Deploying Computational Tasks

Thedeveloperthenneedstoprovideamethodthatinitializesthecomputationaltasks.Thisinvolvesoneormoreofthefollowing:

1.Initializing the Processing Directives associated with an instance:Thesedirec-tivesareusedtoencodeinstance-specificinformationthatisaccessibleonlytotheinstanceinquestion.

2.Specification of the datasets and collection associated with the computation:Granulesisresponsibleforconfiguringaccesstothesedatasets.

3.Linking of the Map-Reduce roles:Granules ensures that once-linked resultsproducedbythemapsareautomaticallyroutedtotheappropriatereducers.

4.Specifying the scheduling strategy for the computational tasks:Bydefault, theexactly-onceschedulingstrategyisused.

5.Distribution of datasets across these instances:Granules incorporatesutilitiesthatallowthisdistributiontobeperformedefficiently.

AQ10

K10175_C009.indd 214 3/2/2010 12:52:33 PM

Granules ◾ 215

Todeployanapplication,thedeveloperonlyneedstoinvokethedeploy()methodintheInstanceDeployer.Thismethoddeploysthecomputationaltasksonthesetofresourcesthatwerediscoveredduringtheinitializationphase.

9.5.2.3 Tracking/Steering a Deployed Application

The InstanceDeployer implements the JobLifecycleObserver interface, whichallowsonetotrackthestatusofmultiple jobs,andthecomputationaltasksthatcomprisethem.ClassesthatextendtheInstanceDeployerhavetheoptiontoover-ride methods specific to the JobLifecycleObserver interface. Specifically, for agiven Job,Granulesmaintains its registered JobLifecycleObserver and invokesmethodsonthisobserverwheneverthereisanupdatetothedeploymentorexecu-tionstatusofthecomputationaltasksthatcompriseit.

Associatedwitheach Job,GranulesmaintainsaProgressTracker thatmain-tainsinformationabouttheexecutionstateofeachofthecomputationaltasksthatcomprise the application. TheLifecycleMetrics associated with every computa-tionaltaskincludesinformationabout

1.Thearrivaltimeforthecomputationaltask 2.Thequeuingoverheadforthecomputationaltask 3.ThetotalCPU-boundtimeforthecomputationaltaskacrossmultipleitera-

tions(ifthereareany) 4.Theprocessingtimeforthecomputationaltask 5.The current status of the computational task {Awaiting Data,Queued for

Execution,Executing,Terminated,Successful,FAILED}

ThestatusofaJobisthecumulativestatusofthecomputationaltasksthatcom-priseit.

TheInstanceDeployeralsoincorporatesmethodsfortracking/steeringacom-putation.TherearemethodstorefreshthestatusofaspecificcomputationaltaskortheentireJob.Thesemethodsresultinupdatestothelife-cyclemetricsoftherelevant computational tasks. Additionally, Granules also allows computationaltaskstobeabortedwhentheyareinexecution.ThesystemallowseitheraspecificcomputationaltasktobesuspendedortheentireJob.

9.6 Related WorkTheoriginalMap-Reducepaper[1]byGhemawatandDeandescribedhowtheirprogrammingabstractionwasbeingusedintheGooglesearchengineandotherdata-intensiveapplications.Thisworkwasitselfinspiredbymapandreduceprimi-tivespresentinLispandotherfunctionalprogramminglanguages.GoogleMap-ReduceiswritteninC++withextensionsforJavaandPython.Sawzall[7]isan

K10175_C009.indd 215 3/2/2010 12:52:33 PM


interpreted,proceduralprogramminglanguageusedbyGoogletodevelopMap-Reduceapplications.

Hadoop[8]wasoriginallydevelopedatYahoo,andisnowanApacheproject.ItisbyfarthemostwidelyusedimplementationoftheMap-Reduceframework.InadditiontothevastnumberofapplicationsatYahoo,itisalsopartoftheGoogle/IBM initiative to support university courses in distributed computing. Hadoopis also hosted as a framework over Amazon’s EC2 [9] cloud. Unlike Granules,Hadoopsupportsonlyexactly-oncesemantics,meaningthatthereisdirectsupportwithintheframeworkformapandreducefunctionstomaintainstate.

HadoopusestheHadoopDistributedFileSystem(HDFS)filesforcommuni-catingintermediateresultsbetweenthemapandreducefunctions,whileGranulesusesstreamingforthesedisseminations,thusallowingaccesstopartialresults.

HDFS allows for replicated, robust access to files. During the data-stagingphase,Hadoopallowsthecreationofreplicasonthe localfilesystem;computa-tionsarethenspawnedtoexploitdatalocality.Hadoopsupportsautomatedrecov-eryfromfailures.Currently,Granulesdoesnotincorporatesupportforautomatedrecoveryfromfailures;thiswillbethefocusofourfutureworkinthisarea.Here,weplantoharnessthereliablestreamingcapabilitiesavailableinNaradaBrokering.

The most dominant model for developing parallel applications in the HPCcommunityistheSPMD[2]model(firstproposedbyFedericaDarema)intandemwiththeMPI[10]library.TheSPMDmodelisapowerfulone,andMap-ReducecaninfactbethoughtofasaninstanceoftheSPMDmodel.TheuseofMPIhas,however,notbeenaswidespreadoutsidethescientificcommunity.

MicrosoftResearch’sDryad[11]isasystemdesignedasaprogrammingmodelfor developing scalable parallel and distributed applications. Dryad is based onDAGs.Inthismodel,sequentialprogramsareconnectedusingone-waychannels.It is intended tobea super-setof thecoreMap-Reduce framework.Dryadpro-videsjobmanagementandautonomiccapabilities,andmakesuseoftheMicrosoftSharedDirectoryService.However,sinceDryadisdevelopedbasedonDAGs,itisnotpossible todevelopsystemsthathavecycles inthem.Forexample, inourbenchmarks,wewerenotabletoimplementthek-meansmachinelearningalgo-rithm[12]usingthebasicDryadframework.

Phoenix[13]isanimplementationofMap-Reduceformulti-coreandmultipro-cessorsystems.ArelatedeffortisQtConcurrent[14],whichprovidesasimplifiedimplementationoftheMap-ReduceframeworkinC++.QtConcurrentautomati-cally optimizes thread utilizations on multi-core machines depending on coreavailability.Disco[15],fromNokia,isanopen-sourceMap-Reduceruntimedevel-opedusingtheErlangfunctionalprogramminglanguage.SimilartotheHadooparchitecture,DiscostorestheintermediateresultsinlocalfilesandaccessesthemusingHTTPconnectionsfromtheappropriatereducetasks.

Holumbus [16] includes an implementation of the Map-Reduce framework,developed in the Haskell functional programming language at the FH WedelUniversityofAppliedSciences,Germany.

AQ11

K10175_C009.indd 216 3/2/2010 12:52:33 PM

Granules ◾ 217

Skynet[17]isanopen-sourceRuby-basedimplementationoftheMap-Reduceframework. Skynet utilizes a peer-recovery system for tracking the constituenttasks.Peerstrackeachotherand,oncefailureisdetected,canspawnareplicaofthefailedpeer.

We had originally developed a prototype implementation of Map-Reduce,CGL-MapReduce [18], which implemented Map-Reduce using streaming (onceagain, using NaradaBrokering) with the ability to “keep alive” map instances.Granulesrepresentsanoverhaul,andincorporatesseveralnewcapabilities,suchasbuilt-insupportforsophisticatedlife-cyclemanagement(periodicity,datadriven,andterminationconditions),powerfulcreationandduplicatedetectionofresults,anddiagnosticsinadditiontotheabilitytocreatecomplexcomputationalpipelineswithfeedbackloopsinmultiplestages.ThecodebasefortheGranules(availablefordownload)runtimehasalsobeendevelopedfromscratch.

9.7 BenchmarksIn our benchmarks, we profile several aspects of the Granules’ performance.Wearespecifically interested indeterminingsystemperformancefordifferentlifecyclesassociatedwiththecomputationaltasks.Thedifferentlifecycleswebenchmarkincludeexactly-once,iterative,periodic,anddata-drivenprimitives.Where possible, we contrast the performance of Granules with comparablesystems, such as Hadoop, Dryad, and MPI. It is expected that these bench-markswouldbeindicativeoftheperformancethatcanbeexpectedindifferentdeployments.

All machines involved in these benchmarks have four dual-core CPUs, a2.4GHzclock,andan8GBRAM.Thesemachineswerehostedona100MbpsLAN.TheOperatingSystemonthesemachinesisRedHatEnterpriseLinuxver-sion4.AllJavaprocessesexecutedwithinversion1.6ofSun’sJVM.Weusedver-sion3.4.6ofthegcccomplierforC++,andforMPIweusedversion7.1.4oftheLocalAreaMulticomputer(LAM)MPI[19].

9.7.1 Streaming SubstrateSince we use the NaradaBrokering streaming substrate for all communicationsbetweenentities,wepresentasimplebenchmarktogivethereaderanideaofthecostsinvolvedinstreaming.Ourresultsoutlinethecommunicationlatenciesinasimplifiedsetting involvingoneproducer,oneconsumer,andonebroker.Thecommunication latencies are reported for stream fragments with different pay-loadsizes.AdditionalNaradaBrokeringbenchmarksindistributedsettingscanbefoundin[4,5].

Twoclustermachineswereinvolvedinthisbenchmark.Theproducerandcon-sumerwerehostedonthesamemachinetoobviatetheneedtoaccountforclock

AQ12AQ13

AQ14

K10175_C009.indd 217 3/2/2010 12:52:33 PM


driftswhilemeasuringlatenciesforstreamsissuedbytheproducer,androutedbythebroker(hostedonthesecondmachine)totheconsumer.

Thereporteddelay, intheresultsdepictedinFigure9.8,istheaverageof50samplesforagivenpayloadsize,thestandarddeviationforthesesamplesalsobeingreported.TheY-axisforthestandarddeviationistheaxisontherightside(blue)ofthegraph.Streaminglatenciesvaryfrom750μs/hopfor100bytesto1.5ms/hopforastreamfragmentof10KBinclustersettings.

9.7.2 Information Retrieval: Exactly-OnceIn this section, we present results from a simple information retrieval example.Givenasetof textfiles, theobjective is tohistogramthecountsassociatedwithvariouswordsinthesefiles.TheperformanceofGranulesiscontrastedwiththatofHadoopandDryad.TheDryadversiontowhichwehaveaccessusesC#,LINQ,andfile-basedcommunicationsusingtheMicrosoftSharedDirectoryService.TheOSinvolvedintheDryadbenchmarksisWindowsXP.

Forthisbenchmark,wevarythecumulativesizeofthedatasetsthatneedtobeprocessed.Thetotalamountofdatathatisprocessedisvariedfrom20GBto100GB.Therewereatotalof128mapinstancesthatweredeployedonthefivemachinesinvolvedinthebenchmark.

The results depicted inFigure9.9demonstrate thebenefits ofusing stream-ingasopposedtofile-basedcommunications.Asthesizeofthedatasetsincreases,thereisaconcomitantincreaseinthenumberandsizeoftheintermediateresults(filebased).ThiscontributestotheslowerperformanceofHadoopandDryad.WeexpecttheperformanceofDryad’ssocket-basedversiontobefasterthantheirfile-basedversion.

0

1

2

3

4

5

100 1,000 10,0000

1

2

3

4

5M

ean

tran

sit d

elay

(ms)

Stan

dard

dev

iatio

n (m

s)

Content payload size (bytes)

Round-trip delays for different payload sizes (100 B–10 kB)

DelayStandard deviation

Figure 9.8 Streaming overheads in cluster settings.

K10175_C009.indd 218 3/2/2010 12:52:34 PM

Granules ◾ 219

9.7.3 K-Means: IterativeMachinelearningprovidesafertilegroundforiterativealgorithms.Inourbench-nmarks,we considered a simple algorithm in the areaofunsupervisedmachinelearning:k-means.Givenasetofndatapoints,theobjectiveistoorganizethesepointsintokclusters.

Thealgorithmstartsoffbyselectingkcentroids,andthenassociatesdifferentdatapointswithinthedatasettooneoftheclustersbasedontheirproximitytothecentroids.Foreachoftheclusters,newcentroidsarethencomputed.ThealgorithmissaidtoconvergewhenthecumulativeEuclideandistancebetweenthecentroidsinsuccessiveiterationsislessthanapredefinedthreshold.

Ink-means,thenumberofiterationsdependsontheinitialchoiceofthecen-troids,thenumberofdatapoints,andthespecifiederrorrate(signifyingthatthecentroidmovementsareacceptable).Theinitialsetofdatapointsisloadedateachofthemapfunctions.Eachmapisresponsibleforprocessingaportionoftheentiredataset.Whatchangesfromiterationtoiterationarethecentroids.Theoutputofeachmapfunctionisasetofcentroids.

The benchmarks, which were run on five machines, also contrast the per-formance of Granules with MPI using a C++ implementation of the k-meansalgorithm.

ThegraphsdepictedinFigure9.10havebeenplottedonalog-loggraphsothatthetrendscanbevisualizeda littlebetter.Wevariedthenumberofdatapoints in the dataset from 105 to 4×107. The results indicate that Hadoop’sperformanceisordersofmagnitudeslowerthanGranulesandMPI.InHadoop,

0

500

1000

1500

2000

2500

3000

20 30 40 50 60 70 80 90 100

Ove

rhea

d fo

r hist

ogra

mm

ing

wor

ds (s

)

Total size of dataset (GB)

HadoopDryad

Granules

Figure 9.9 Processing time for histogramming words.

K10175_C009.indd 219 3/2/2010 12:52:35 PM


these centroids are transferred using files, while Granules uses streaming.Furthermore, since Hadoop does not support iterative semantics, map func-tionsneedtobeinitializedandthedatasetsneedtobereloadedusingHDFS.Though these file-system reads are being performed locally (thanks to HDFSanddata collocation), these costs can still beprohibitive, as evidenced inourbenchmarks.Additionally,asthesizeofthedatasetincreases,theperformancesoftheMPI/C++implementationofk-meansandtheGranules/Javaimplementa-tionofk-meansstarttoconverge.

9.7.4 Periodic SchedulingInthissection,webenchmarktheabilityofGranulestoperiodicallyscheduletasksforexecution.Forthisparticularbenchmark,weinitialized10,000mapfunctionsthatneededtobescheduledforexecutionevery4s.

TheobjectiveofthisbenchmarkistoshowthatasingleGranulesinstancecanindeedenforceperiodicityforareasonablenumberofmapinstances.

Figure9.11depictstheresultsofperiodicexecutionsof10,000mapsfor17iter-ations.Thegraphdepictsthespacinginthetimesatwhichthesemapsaresched-uledforexecution.TheX-axisrepresentsaspecificmapinstance(assignedIDsfrom1to10,000),andtheY-axisrepresentsthespacingbetweenthetimesatwhichagiveninstancewasscheduled.Eachmapinstancereports17values.

Thefirsttimeacomputationaltaskisscheduledforexecution,abasetime,tb,isrecorded.Subsequentiterationsreportthedifferencebetweenthebasetime,tb,andthecurrent time, tc. Inalmostallcases, thespacingbetweenthesuccessiveexecutions foranygiven instancewasbetween3.9–4.1s. Insomecases, there isasmallnotch;thisreflectscaseswherethefirstexecutionwasdelayedbyasmall

Ove

rhea

d (s

)

0.10.1

10000

1000

100

10

1

1 10 100Number of 2D data points (millions)

HadoopGranules

MPI

Figure 9.10 Performance of the k-means algorithm.

K10175_C009.indd 220 3/2/2010 12:52:36 PM

Granules ◾ 221

amount, the (constant) impactofwhich is reflected in subsequent iterations forthatmapinstance.

9.7.5 Data DrivenInthissection,wedescribetheperformanceofmatrixmultiplicationusingGranules.In this case, theobject is tomeasure theproductof twodense16,000×16,000matrices,thatis,eachmatrixhas256millionelementswithpredominantlynon-zerovalues.

Thematrixmultiplicationexampledemonstrateshowcomputationaltaskscanbe“stayalive,”andbescheduledforexecutionwhendataisavailable.Themapsarescheduledforexecutionasandwhenthedataisavailableforthecomputationstoproceed.

Forthisbenchmark,wevarythenumberofmachinesinvolvedintheexperi-ment from 1 to 8. There are a total of 16,000 map instances. At a given time,eachofthesemapsprocessesportionsoftherowsandcolumnsthatcomprisethematrix.EachGranulesinstancecopeswithafragmentofmorethan2000concur-rentstreams.Intotal,everyGranulesinstancecopeswith32,000distinctstreams.

The results for theprocessing times (plottedon a log-log scale) canbe seeninFigure9.12. Ingeneral, as thenumberof availablemachines increases, thereis a proportional improvement in the processing time. Our plots of the speed-up(Figure9.13)inprocessingtimeswiththeavailabilityofadditionalmachinesreflectthis.

Ingeneral,thesegraphsdemonstratethatGranulescanbringsubstantialben-efitstodata-drivenapplicationsbyamortizingthecomputationalloadonasetofmachines.Domainscientistsdonotneedtowriteasinglelineofnetworkingcode;Granulesmanagesthisinatransparentfashionfortheapplications.

0

10

20

30

40

50

60

70

0 2,000 4,000 6,000 8,000 10,000

Tim

e tas

k sc

hedu

led

at (s

)

Computational tasks

Figure 9.11 Periodic scheduling of 10,000 computational tasks.

K10175_C009.indd 221 3/2/2010 12:52:37 PM


9.7.6 Assembling mRNA SequencesThissectiondescribestheperformanceofGranulesinorchestratingtheexecutionofapplicationsdevelopedinlanguagesotherthanJava.TheapplicationweconsideristheCAP3[20]messengerRibonucleicacid(mRNA)sequenceassemblyapplica-tion(C++)developedatMichiganTech.

AsExpressedSequenceTag (EST)corresponds tomRNAs transcribed fromgenes residingonchromosomes, individualEST sequences represent a fragmentofmRNA.CAP3allowsus toperformESTassembly to reconstruct full-lengthmRNAsequencesforeachexpressedgene.

OurobjectiveaspartofthisbenchmarkwasalsotoseehowGranulescanbeusedtomaximizecoreutilizationsonamachine.CAP3takesasinputasetoffiles.Inourbenchmark,weneedtoprocess256filesduringtheassembly.

AQ15

1024

2048

4096

8192

16384

1 2 4 8

Proc

essin

g tim

e (s)

Number of machines

Processing time

Figure 9.12 Processing time for matrix multiplication on different machines.

1

2

4

8

1 2 4 8

Spee

d-up

Number of machines

Speed-up

Figure 9.13 Speed-up for matrix multiplication.

K10175_C009.indd 222 3/2/2010 12:52:38 PM

Granules ◾ 223

Onagivenmachine,wefine-tunedtheconcurrencybysettingthenumberofworkerthreadswithinthethread-pooltodifferentvalues.Byrestrictingthenum-berofthreads,wealsorestrictedtheamountofconcurrencyandtheunderlyingcoreutilizations.Westartedoffbysettingtheworker-poolsizeto1,2,4,and8on1machine,andthenused8workerthreadson2,4,and8machines.Thisallowedustoreportresultsfor1,2,4,8,16,32,and64cores.

Theresultsofourbenchmarkintermsofprocessingcostsandthespeed-upsachievedaredepictedinFigures9.14and9.15,respectively.Ingeneral,asthenum-berofavailablecoresincreases,thereisacorrespondingimprovementinexecutiontimes.

8

16

32

64

128

256

512

1024

1 2 4 8 16 32 64

Proc

essin

g tim

e (s

)

Number of cores

Processing time

Figure 9.14 Processing time for EST assembly on different cores using Granules and CAP3.

1

2

4

8

16

32

64

1 2 4 8 16 32 64

Spee

d-up

Number of cores

Speed-up

Figure 9.15 Speed-up for EST assembly using Granules and CAP3.

K10175_C009.indd 223 3/2/2010 12:52:39 PM


Theresultsdemonstratethat,whenconfiguredcorrectly,Granulescanmaxi-mizecoreutilizationsonagivenmachine.Thegraphsplottedona log-logscaleindicate that for every doubling of the available cores, the processing time forassemblingthemRNAsequencesreducesbyhalf(approximately).Currently,theGranulesruntimereadythethread-poolsizinginformationfromaconfigurationfile;wewill be investigatingmechanisms thatwill allowus todynamically sizethesethread-pools.

9.8 ConclusionsInthischapter,wedescribedtheGranulesruntime.Supportforrichlife-cyclesup-portwithinGranulesallowscomputationstoretainstate,whichinturnisparticu-larlyapplicableforseveralclassesforscientificapplications.

Granules allows complex computational graphs to be created. As discussed,thesegraphscanencapsulatebothcontrolflowanddataflow.Granulesenforcesthesemanticsofcomplexdistributedcomputationalgraphsthathaveoneormorefeedback loops.Thedomainscientistdoesnothave tocopewithIO, threading,synchronization,ornetworking librarieswhiledevelopingapplications that spanmultiplestages,withmultipledistributedinstancescomprisingeachstage.Thesecomputationalpipelines canbedependenton iterative,periodic,data-driven,orterminationconditions.

DemonstrableperformancebenefitshavebeenaccruedbyGranulesasaresultofusingstreamingfordisseminatingintermediateresults.

Granules’ rich life-cycle support, and its performance when contrasted withcomparable systems,underscores the feasibilityofusingGranules in several set-tings.Aspartofourfuturework,wewillbeinvestigatingsupportforautonomicerrordetectionandrecoverywithinGranules.

References 1. J.DeanandS.Ghemawat,Mapreduce:Simplifieddataprocessingonlargeclusters,

Communications of the ACM,51,107–113,January2008. 2. F.Darema,SPMDmodel:Past,presentandfuture,Recent Advances in Parallel Virtual

Machine and Message Passing Interface: Eighth European PVM/MPI Users’ Group Meeting,Santorini/Thera,Greece,2001.

3. S.Pallickara,J.Ekanayake,andG.Fox,Anoverviewofthegranulesruntimeforcloudcomputing(ShortPaper),Proceedings of the IEEE International Conference on e-Science,Indianapolis,IN,2008.

4. S.Pallickara andG.Fox,Naradabrokering:Amiddleware framework and architec-ture for enabling durable peer-to-peer grids, Proceedings of the ACM/IFIP/USENIX International Middleware Conference Middleware-2003,RiodeJaneiro,Brazil,2003,pp.41–61.

AQ16

AQ17

K10175_C009.indd 224 3/2/2010 12:52:39 PM

Granules ◾ 225

5. S.Pallickaraetal.,Aframeworkforsecureend-to-enddeliveryofmessagesinpublish/subscribe systems,Proceedings of the Seventh IEEE/ACM International Conference on Grid Computing(GRID 2006),Barcelona,Spain,2006.

6. S.Pallickara,H.Bulut,andG.Fox,Fault-tolerantreliabledeliveryofmessagesindis-tributedpublish/subscribesystems,Fourth IEEE International Conference on Autonomic Computing,Jacksonville,FL,June2007,p.19.

7. R.Pike,S.Dorward,R.Griesemer,andS.Quinlan,Interpretingthedata:Parallelanal-ysiswithSawzall,Scientific Programming Journal,SpecialIssueonGridsandWorldwideComputingProgrammingModelsandInfrastructure,13(4),227–298,2005.

8. ApacheHadoop,http://hadoop.apache.org/core/ 9. S.Garfinkel,Anevaluationofamazon’sgridcomputingservices:EC2,S3andSQS,

TechnicalReportTR-08-07,HarvardUniversity,Cambridge,MA,August2007. 10. Message Passing Interface Forum, MPI: A message passing interface, Proceedings of

Supercomputing’93,Portland,OR, IEEEComputerSocietyPress,Washington,DC,November1993,pp.878–883.

11. M.Isard,M.Budiu,Y.Yu,A.Birrell,andD.Fetterly,Dryad:Distributeddata-parallelprogramsfromsequentialbuildingblocks,European Conference on Computer Systems,Lisbon,Portugal,March2007.

12. J.B.MacQueen,Somemethodsforclassificationandanalysisofmultivariateobserva-tions,Proceedings of Fifth Berkeley Symposium on Mathematical Statistics and Probability,Vol.1,Berkeley,CA,UniversityofCaliforniaPress,Berkeley,CA,1967,pp.281–297.

13. C.Ranger,R.Raghuraman,A.Penmetsa,G.R.Bradski,andC.Kozyrakis,Evaluatingmapreduceformulti-coreandmultiprocessorsystems,Proceedings of the International Symposium on High-Performance Computer Architecture(HPCA),Phoenix,AZ,2007,pp.13–24.

14. QtConcurrent,SimplifiedmapreduceinC++withsupportformulticores,April2009,http://labs.trolltech.com/page/Projects/Threads/QtConcurrent

15. Discoproject,http://discoproject.org/ 16. S.Schlatt,T.Hübel,S.Schmidt,andU.Schmidt,TheHolumbusdistributedcomput-

ingframeworkandmapreduceinHaskell,2009,http://holumbus.fh-wedel.de/trac 17. A.Pisoni,Skynet:Arubymapreduceframework,April2009,http://skynet.rubyforge.org/ 18. J. Ekanayake, S. Pallickara, and G. Fox, Map-reduce for scientific applications,

Proceedings of the IEEE International Conference on e-Science,Indianapolis,IN,2008. 19. J.M.SquyresandA.Lumsdaine,AcomponentarchitectureforLAM/MPI,Proceedings

of Euro PVM/MPI,Venice,Itlay,October2003. 20. X. Huang and A. Madan, CAP3: A DNA sequence assembly program, Genome

Research,9,868–877,1999. 21. S.Pallickara,J.Ekanayake,andG.Fox,Granules:Alightweight,streamingruntimefor

cloudcomputingwithsupportformap-reduce,Proceedings of the IEEE International Conference on Cluster Computing(CLUSTER 2009),NewOrleans,LA,2009.

AQ18

K10175_C009.indd 225 3/2/2010 12:52:39 PM

AUTHORQUERIES[AQ1] Pleasecheckifthefixedrunningheadisok.[AQ2] Inthetext,“italics”hasbeenusedforemphasis.Pleasecheckwhetherthis

canbechangedtobold.[AQ3] Pleaseexpand“SPMD,”ifappropriate.[AQ4] Pleasecheckcrossreferencetosectionhereandinotherinstances.[AQ5] SinceanintroductiontoSection9.5wasmissing,onehasbeenaddedhere.

Pleasecheckifthisisfine.[AQ6] Pleaseexpand“GIS,”ifappropriate.[AQ7] Pleaseexpand“JDBC,”ifappropriate.[AQ8] Pleaseexpand“FIFO,”ifappropriate.[AQ9] Pleasecheckifthechangemadetothesentencestarting“Granulessup-

ports…”isokay.[AQ10] Pleaseexpand“SSL,”ifappropriate.[AQ11] Pleaseexpand“HPC,”ifappropriate.[AQ12] Pleasecheckiftheeditofthesentencestarting“Wehadoriginally…”is

correct.[AQ13] Pleasecheckwhether“CGL-MapReduce”shouldbechangedto“CGL-

Map-Reduce”asperotheroccurrencesofMap-Reduceinthischapter.[AQ14] Pleaseexpand“JVM,”ifappropriate.[AQ15] Please check if the edit of the sentence: “As expressed… fragment of

mRNA”iscorrect.[AQ16] Pleasecheckif“ready”shouldbechangedto“reads”inthesentencestart-

ing“Currently,theGranules…”[AQ17] Pleasecheckwhetherthephrase“Supportforrichlife-cyclesupport”can

bechangedto“Richlife-cyclesupport”inthesentencestarting“Supportfor…”

[AQ18] Pleasechecktheinsertedlocationofthepublisherforcorrectness.

K10175_C009.indd 226 3/2/2010 12:52:39 PM

Granules: A Lightweight Runtime for Scalable Computing...

Documents

Transcript of Granules: A Lightweight Runtime for Scalable Computing...