MapReduce SimplifiedDataProcessingonLargeClusters

(WithouttheAgonizingPain)

PresentedbyAaronNathan

TheProblem

•  Massiveamountsofdata– >100TB(theinternet)– Needssimpleprocessing

•  Computersaren’tperfect– Slow– Unreliable– Misconfigured

•  Requirescomplex(i.e.bugprone)code

MapReducetotheRescue!

•  CommonFuncKonalProgrammingModel– MapStep

map (in_key, in_value) -> list(out_key, intermediate_value) •  Splitaproblemintoalotofsmallersubproblems

– ReduceStepreduce (out_key, list(intermediate_value)) -> list(out_value) •  Combinetheoutputsofthesubproblemstogivetheoriginalproblem’sanswer

•  Each“funcKon”isindependent•  HighlyParallelizable

Answer!Answer!

AlgorithmPicture

MAP

REDUCE

Answer!

MAPMAP MAP MAP

DATA

K1:vK1:vK2:v K2:v K1:v K2:vK3:v K3:v

K1:v,v,v K2:v,v,v K3:v,v,v

REDUCE REDUCE

aggregator

SomeExampleCodemap(String input_key, String input_value): // input_key: document name

// input_value: document contents

for each word w in input_value: EmitIntermediate(w, "1");

reduce(String output_key, Iterator intermediate_values): // output_key: a word

// output_values: a list of counts int result = 0; for each v in intermediate_values:

result += ParseInt(v);

Emit(AsString(result));

SomeExampleApplicaKons

•  DistributedGrep•  URLAccessFrequencyCounter•  ReverseWebLinkGraph

•  Term‐VectorperHost

•  DistributedSort•  InvertedIndex

TheImplementaKon

•  GoogleClusters– 100s‐1000sDualCorex86CommodityMachines

– CommodityNetworking(100mbps/1Gbps)– GFS

•  GoogleJobScheduler•  Librarylinkedinc++

ExecuKon

TheMaster

•  MaintainsthestateandidenKfyofallworkers•  Managesintermediatevalues

•  ReceivessignalsfromMapworkersuponcompleKon

•  BroadcastssignalstoReduceworkersastheywork

•  CanretaskcompletedMapworkerstoReduceworkers.

InCaseofFailure

•  PeriodicPingsfromMaster‐>Workers– Onfailureresetsstateofassignedtaskofdeadworker

•  Simplesystemprovesresilient– Worksincaseofa80simultaneousmachinefailures!

•  Masterfailureisunhandled.•  WorkerFailuredoesn’teffectoutput

(outputidenKcalwhetherfailureoccursornot)–  Eachmapwritestolocaldiskonly–  Ifamapperislost,thedataisjustreprocessed– Non‐determinisKcmapfuncKonsaren’tguaranteed

PreservingBandwidth

•  Machinesareinrackswithsmallinterconnects– UselocaKoninformaKonfromGFS

– Ahemptstoputtasksforworkersandinputslicesonthesamerack

– UsuallyresultsinLOCALreads!

BackupExecuKonTasks

•  Whatifonemachineisslow?•  CandelaythecompleKonoftheenKreMROperaKon!

•  Answer:Backup(Redundant)ExecuKons– Whoeverfinishesfirstcompletesthetask!

– Enabledtowardstheendofprocessing

ParKKoning

•  M=numberofMapTasks(thenumberofinputsplits)

•  R=numberofReduceTasks(thenumberofintermediatekeysplits)

•  W=numberofworkercomputers•  InGeneral:

– M=sizeof(Input)/64MB–  R=W*n(wherenisasmallnumber)

•  TypicalScenario:InputSize=12TB,M=200,000,R=5000W=2000

CustomParKKoning

•  DefaultParKKonedonintermediatekey– Hash(intermediate_key)modR

•  Whatifuserhasaprioriknowledgeaboutthekey?– Allowforuser‐definedhashingfuncKon– Ex.Hash(Hostname(url_key))

TheCombiner

•  IfreducerisassociaKveandcommuniviKve–  (2+5)+4=11or2+(5+4)=11–  (15+x)+2=2+(15+x)

•  Repeatedintermediatekeyscanbemerged– Savesnetworkbandwidth– EssenKallylikealocalreducetask

I/OAbstracKons

•  HowtogetiniKalkeyvaluepairstomap?– Defineaninput“format”

– Makesuresplitsoccurinreasonableplaces– Ex:Text

•  Eachlineisakey/pair•  CancomefromGFS,bigTable,oranywherereally!

– Outputworksanalogously

SkippingBadRecords

•  Whatifausermakesamistakeinmap/reduce•  Andonlyapparentonfewjobs..

•  WorkersendsmessagetoMaster

•  Skiprecordon>1workerfailureandtellotherstoignorethisrecord

RemovingUnnecessaryDevelopmentPain

•  LocalMapReduceImplementaKonthatrunsondevelopmentmachine

•  MasterhasHTTPpagewithstatusofenKreoperaKon– Shows“badrecords”

•  ProvideaCounterFacility– Masteraggregates“counts”anddisplayedonMasterHTTPpage

AlookattheUI(in1994)

h6p://labs.google.com/papers/mapreduce‐osdi04‐slides/index‐auto‐0013.html

PerformanceBenchmarks

SorBng AND Searching

Search(Grep)

•  Scanthrough1010100byterecords(1TB)

•  M=15000,R=1

•  StartupKme– GFSLocalizaKon– ProgramPropagaKon

•  Peak‐>30GB/sec!

Sort

•  50linesofcode•  Map‐>key+textline

•  Reduce‐>IdenKty•  M=15000,R=4000

– ParKKononinitbytesofintermediatekey

•  Sortsin891sec!

WhataboutBackupTasks?

Andwait…it’suseful!

NB:August2004

OpenSourceImplementaKon

•  Hadoop•  hhp://hadoop.apache.org/core/

– ReliesonHDFS– AllinterfaceslookalmostexactlylikeMapReducepaper

•  Thereisevenatalkaboutittoday!– 4:15B17CSColloquium:MikeCafarella(Uwash)

AcBveDisksforLarge‐ScaleDataProcessing

TheConcept

•  Useaggregateprocessingpower– Networkeddisksallowforhigherthroughput

•  WhynotmovepartoftheapplicaKonontothediskdevice?– Reducedatatraffic

–  Increaseparallelismfurther

ShrinkingSupportHardware…

ExampleApplicaKons

•  MediaDatabase– Findsimilarmediadataby“fingerprint”

•  RealTimeApplicaKons– CollectmulKplesensordataquickly

•  DataMining– POSAnalysisrequiredadhocdatabasequeries

Approach

•  Leveragetheparallelismavailableinsystemswithmanydisks

•  Operatewithasmallamountofstate,processingdataasitstreamsoffthedisk

•  ExecuterelaKvelyfewinstrucKonsperbyteofdata

Results‐NearestNeighborSearch

•  Problem:DeterminekitemsclosesttoaparKculariteminadatabase– Performcomparisonsonthedrive– Returnsthedisksclosestmatches– Serverdoesfinalmerge

MediaMiningExample

•  Performlowlevelimagetasksonthedisk!

•  EdgeDetecKonperformedondisk– Senttoserverasedgeimage

– Serverdoeshigherlevelprocessing

WhynotjustuseabunchofPC’s?

•  Theperformanceinceaseissimilar•  Infact,thepaperessenKallyusedthissetuptoactuallybenchmarktheirresults!

•  Supposedlythiscouldbecheaper•  Thepaperdoesn’treallygiveagoodargumentforthis…– PossiblyreducedbandwidthondiskIOchannel– Butwhocares?

SomeQuesKons

•  Whatcouldadiskpossiblydobeherthanthehostprocessor?

•  WhataddedcostisassociatedwiththismediocreprocessorontheHDD?

•  Arenewdependenciesareintroducedonhardwareandsosware?

•  Perhapsother(beher)placestodothistypeoflocalparallelprocessing?

•  Maybein2001thismademoresense?