Hadoop for humans
-
Upload
kien-pham -
Category
Technology
-
view
297 -
download
1
description
Transcript of Hadoop for humans
![Page 1: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/1.jpg)
Hadoop for humans
Kien PhamSoftware Engineer - R&D Anaheim, CA
10/04/2013
Friday, October 4, 13
![Page 2: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/2.jpg)
Hadoop?
Friday, October 4, 13
![Page 3: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/3.jpg)
isa framework
HDFSMap /Reduce
http://www.flickr.com/photos/d90nikon/6195610430/sizes/o/in/photostream/
Friday, October 4, 13
![Page 4: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/4.jpg)
Map / ReduceFriday, October 4, 13
![Page 5: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/5.jpg)
MapperI like SendGrid and email, you like SendGrid and email too
1 1 1 1 1
Friday, October 4, 13
![Page 6: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/6.jpg)
MapperI like SendGrid and email, you like SendGrid and email too
1 1 1 1 1
I like SendGrid and email, you like SendGrid and email too1 1 1 1 1
I like SendGrid and email, you like SendGrid and email too1 1 1 1 1
worker 1
worker 2
worker 3
Friday, October 4, 13
![Page 7: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/7.jpg)
Reducer1like
SendGridemailSendGridemail
1111
1likeSendGridemail
22
Friday, October 4, 13
![Page 8: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/8.jpg)
1likeSendGridemail
22
key value
Friday, October 4, 13
![Page 9: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/9.jpg)
key value{"d": "2013-09-01", "t": "j"}
{"d": "2013-09-02", "t": "j"}
{"d": "2013-09-01", "t": "x"}
{"d": "2013-09-02", "t": "x"}
764872
269661
190889
71693
Friday, October 4, 13
![Page 10: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/10.jpg)
HDFS
Friday, October 4, 13
![Page 11: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/11.jpg)
HDFS
Friday, October 4, 13
![Page 12: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/12.jpg)
HDFS @ SG138 TB
Friday, October 4, 13
![Page 13: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/13.jpg)
1 TB = 1,024 GB138TB = 141,312 GB
300GB / day
141,312 GB / 300 GB = 471 daysFriday, October 4, 13
![Page 14: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/14.jpg)
S3Friday, October 4, 13
![Page 15: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/15.jpg)
2015
50% of the world’s data
Hadoop will process
http://www.flickr.com/photos/tisdale53/4737492082/
Friday, October 4, 13
![Page 16: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/16.jpg)
custom jobs?
Friday, October 4, 13
![Page 17: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/17.jpg)
mrgumble
Friday, October 4, 13
![Page 18: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/18.jpg)
abstract Hadoop process
Friday, October 4, 13
![Page 19: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/19.jpg)
startstop
statusresult
Friday, October 4, 13
![Page 20: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/20.jpg)
mrgumble start -j my_cool_job
Friday, October 4, 13
![Page 21: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/21.jpg)
mrgumble stop -j my_cool_job
Friday, October 4, 13
![Page 22: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/22.jpg)
mrgumble status --job_id 1234
Friday, October 4, 13
![Page 23: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/23.jpg)
mrgumble result -j job_name
Friday, October 4, 13
![Page 24: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/24.jpg)
excited?
Friday, October 4, 13
![Page 25: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/25.jpg)
template.pyhadoop-jobs repo jobs/
Friday, October 4, 13
![Page 26: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/26.jpg)
import mrgumbleimport sgstats-hadoop
Friday, October 4, 13
![Page 27: Hadoop for humans](https://reader033.fdocuments.us/reader033/viewer/2022052522/548baf91b4795991048b4597/html5/thumbnails/27.jpg)
Live Demo
Friday, October 4, 13