An Introduction to MapReduce 2 and YARN

An Introduction to MapReduce 2 and

YARN

Tom WhiteApril 25, 2012

Seattle Hadoop / Scalability / NoSQL Meetup

Wednesday, April 25, 2012

First, whatʼs MapReduce 1?


Whatʼs wrong with MR1?


Motivation

•Scaling >4000 nodes

•HA of Job Tracker

•Poor resource utilization


Yet Another Resource Negotiator


Node Manageris a generalized Task Tracker• Task Tracker

• fixed number of map or reduce slots

• Node Manager

• containers with variable resource limits


MR is user spaceYARN is kernel


Bonus Apps

•Distributed shell

•MPI (MAPREDUCE-2911)

•Master-worker (MAPREDUCE-3315)

•Apache Giraph, Hama


Old API ≠ MR1New API ≠ MR2


Old APIo.a.h.mapred

New APIo.a.h.mapreduce

MR1 ✓ ✓

MR2 ✓ ✓


Try out MR2

•Apache Hadoop 0.23.1

•CDH4 Beta 2


<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>1.0.2</version></dependency>

<dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>0.23.1</version></dependency>

MR1

MR2


TODO

• Still alpha status

• Performance tuning

• Usability bug fixes

• RM recovery

• Security in MR2 not complete


An Introduction to MapReduce 2 and YARN

Technology

Transcript of An Introduction to MapReduce 2 and YARN