Map Reduce for data-intensive computing

(Some of the content is adapted from the original authors’ talk at OSDI 04)

Motivation

• Google needed to process web-scale data– Data much larger than what fits on one machine– Need parallel processing to get results in a

reasonable time– Use cheap commodity machines to do the job

Requirements

• Solution must– Scale to 1000s of compute nodes– Must automatically handle faults– Provide monitoring of jobs– Be easy for programmers to use

MapReduce Programming model

• Input and Output are sets of key/value pairs• Programmer provides two functions– map(K1, V1) -> list(K2, V2)• Produces list of intermediate key/value pairs for each

input key/value pair– reduce(K2, list(V2)) -> list(K3, V3)• Produces a list of result values for all intermediate

values that are associated with the same intermediate key

MapReduce Pipeline

Map Shuffle Reduce

Read from DFS

Write to DFS

MapReduce in Action

Map (k1, v1) list(k2, v2)• Processes one input key/value pair • Produces a set of intermediate key/value pairs

Reduce (k2, list(v2)) list(k3, v3) • Combines intermediate values for one particular key • Produces a set of merged output values (usually one)

MapReduce Architecture

MapReduce MapReduce MapReduce MapReduce

Distributed File System

Network

MapReduceJob Tracker

Software Components

• Job Tracker (Master)– Maintains Cluster membership of workers– Accepts MR jobs from clients and dispatches tasks to

workers– Monitors workers’ progress– Restarts tasks in the event of failure

• Task Tracker (Worker)– Provides an environment to run a task– Maintains and serves intermediate files between Map

and Reduce phases

MapReduce Parallelism

Hash Partitioning

Example 1: Count Word Occurrences

• Input: Set of (Document name, Document Contents)• Output: Set of (Word, Count(Word))• map(k1, v1):

for each word w in v1emit(w, 1)

• reduce(k2, v2_list):int result = 0;for each v in v2_list

result += v;emit(k2, result)

Map

Example 1: Count Word Occurrences

Map

Reduce

Reduce

this is a line

this is another line

another line

yet another line

this, 1is, 1a, 1line, 1this, 1is, 1another, 1line, 1

another, 1line, 1

yet, 1another, 1line, 1

a, 1another, 1is, 1is, 1

line, 1line, 1this, 1this, 1

another, 1another, 1

line, 1line, 1yet, 1

a, 1another, 3is, 2

line, 4this, 2yet, 1

Example 2: Joins

• Input: Rows of Relation R, Rows of Relation S• Output: R join S on R.x = S.y• map(k1, v1)

if (input == R)emit(v1.x, [“R”, v1])

elseemit(v1.y, [“S”, v2])

• reduce(k2, v2_list)for r in v2_list where r[1] == “R”

for s in v2_list where s[1] == “S”emit(1, result(r[2], s[2]))

Other examples

• Distributed grep• Inverted index construction• Machine learning• Distributed sort• …

Fault Tolerant Evaluation

• Task Fault Tolerance achieved through re-execution

• All consumers consume data only after completely generated by the producer– This is an important property to isolate faults to

one task• Task completion committed through Master• Cannot handle master failure

Task granularity and pipelining

• Fine granularity tasks– Many more map tasks than machines• Minimizes time for fault recovery• Pipelines shuffling with map execution• Better load balancing

Optimization: Combiners

• Sometimes partial aggregation is possible on the Map side

• May cut down amount of data to be transferred to the reducer (sometimes significantly)

• combine(K2, list(V2)) -> K2, list(V2)• For Word Occurrence Count example,

Combine == Reduce

Map

Example 1: Count Word Occurrences (With Combiners)

Map

Reduce

Reduce

this is a line

this is another line

another line

yet another line

this, 1is, 1a, 1line, 1this, 1is, 1another, 1line, 1

another, 1line, 1

yet, 1another, 1line, 1

a, 1another, 1is, 2

line, 2this, 2

another, 2

line, 2yet, 1

a, 1another, 3is, 2

line, 4this, 2yet, 1

Optimization: Redundant Execution

• Slow workers lengthen completion time• Slowness happens because of– Other jobs consuming resources– Bad disks/network etc

• Solution: Near the end of the job spawn extra copies of long running tasks– Whichever copy finishes first, wins.– Kill the rest

• In Hadoop this is called “speculative execution”

Optimization: Locality Optimization

• Task scheduling policy– Ask DFS for locations of replicas of input file blocks– Map tasks scheduled so that input blocks are

machine local or rack local• Effect: Tasks read data at local disk speeds• Without this rack switches limit data rate

Distributed Filesystem

• Used as the “store” of data• MapReduce reads input from DFS and writes

output to DFS• Provides a “shared disk” view to applications

using local storage on shared-nothing hardware

• Provides redundancy by replication to protect from node/disk failures

DFS Architecture

Taken from Ghemawat’s SOSP’03 paper (The Google Filesystem)

• Single Master (with backups) that track DFS file name to chunk mapping• Several Chunk servers that store chunks on local disks• Chunk Size ~ 64MB or larger• Chunks are replicated• Master only used for chunk lookups – Does not participate in transfer of data

Chunk Replication

• Several Replicas of each Chunk– Usually 3

• Replicas usually spread across racks and data centers to maximize availability

• Master tracks location of each replica of a chunk• When chunk failure is detected, master automatically

rebuilds new replica to maintain replication level• Automatically picks chunk servers for new replicas

based on utilization

Offline Reading

• Original MapReduce Paper– “Simplified Data Processing on Large Clusters” by

Jeffrey Dean and Sanjay Ghemawat in OSDI’04• Original DFS Paper– “The Google Filesystem” by Sanjay Ghemawat,

Howard Gobioff, and Shun-Tak Leung in SOSP ‘03• CACM Papers on MapReduce and Parallel DBMSs– MapReduce and Parallel DBMSs: Friends or Foes?– MapReduce: A Flexible Data Processing Tool