Hadoop 101 v1

HADOOP 101A really quick overview of what it is

(and why you should care...)

Monday, 30 September, 13

HADOOP IS A

TOOL TO SOLVE

BIG DATA PROBLEMS


HADOOP IS A(ONE KIND OF)TOOL TO SOLVE

(SOME KINDS OF)BIG DATA PROBLEMS


OK....

SO WHAT IS A

BIG DATA PROBLEM?


“A PROBLEM IS A BIG DATA PROBLEM

WHEN THE SIZE OF THE DATA IS PART OF THE PROBLEM”


A few Terabytes of Data...Monday, 30 September, 13

Text processing--a few hours?


But what if you have more data?


Network Storage--Petabytes!


What if you need compute power for complex algorithms?


8 core? 16 Cores? 64 cores? 512 GB RAM?


More processing in one box--price grows exponentially

0

75,000

150,000

225,000

300,000

16 32 48 64 80 96 112 128

Single Machine


Compute cost for many commodity machines scales linearly

0

75,000

150,000

225,000

300,000

16 32 48 64 80 96 112 128

Single Machine Many Commodity Maciines


A network of commodity computers


Run jobs on part of the data on each one. Compile the results.


Let’s add a computer to manage the process of job delegation, merging the results...


We also need something to keep track of what files are where, so we know what data needs to be computed...


When you have a lot of computers, and even more hard drives, one thing I can guarantee...


Computers will eventually fail.


Hard drives will eventually fail.


Even whole racks will fail.


If a computer fails and you only have one copy of your data...


You will be very, very unhappy.


So lets store multiple copies of the data. Hard drives are CHEAP!


If one hard drive fails... we are still OK


If one computer fails... we are still OK


Even if a whole rack fails... we are still OK


Once we find a failure let’s have the system recopy the copies.


Guess, what? We’ve just invented Hadoop!


So let’s talk about the pieces of Hadoop.


Data nodes store and manage the data on a single “slave” computer

Data Node


Task trackers manage the compute

Data Node

Task Tracker


Job tracker manages task trackers, ships code to compute nodes

Data Node

Task TrackerJob Tracker


Name node manages distribution and replication on the data nodes

Data Node


Name Node


Map Reduce



HDFS (Hadoop Distributed File System)

Data Node

Name Node


VISUAL EXAMPLEMonday, 30 September, 13

MAPMonday, 30 September, 13

SHUFFLEMonday, 30 September, 13

REDUCEMonday, 30 September, 13

PUTTING IT ALL TOGETHERMonday, 30 September, 13

SO WHAT?


THAT’S WHAT!Monday, 30 September, 13

OPPORTUNITY!Monday, 30 September, 13

HADOOP ECOSYSTEMMonday, 30 September, 13

Hadoop 101 v1

Technology

Transcript of Hadoop 101 v1