1 Motivation Slide 1) Motivation Slide 2) Motivation Basics • What is ...
The Motivation For Hadoop Hadoop: Basic Concepts What Is...
Transcript of The Motivation For Hadoop Hadoop: Basic Concepts What Is...
![Page 1: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/1.jpg)
1compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
Section 9 : Case Study #Objectives of this Session The Motivation For Hadoop
What problems exist with traditional large-scale computing systemsWhat requirements an alternative approach should haveHow Hadoop addresses those requirements
Hadoop: Basic ConceptsWhat Is Hadoop?The Hadoop Distributed File System (HDFS)How Google MapReduce Algorithm worksAnatomy of a Hadoop Cluster
Who uses Hadoop ?
db.suven.net# Not a part of 1Z0-061 or 1Z0-144 Certification test , but very important technology in BIG DATA Analysis
![Page 2: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/2.jpg)
• Hadoop Solutions
– The most common problems Hadoop can solve
– The types of analytics often performed with Hadoop
– Where the data comes from ?
– The benefits of analyzing data with Hadoop
– How some real-world companies use Hadoop
• Hadoop Ecosystem
• Cloudera Software (All Open-Source)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 2
Objectives of this Session … contd…
![Page 3: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/3.jpg)
The Motivation For Hadoop
3compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 4: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/4.jpg)
*MPI: Message Passing InterfacePVM: Parallel Virtual Machine
4compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 5: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/5.jpg)
Major Problem
5compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 6: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/6.jpg)
1 GB = 1000 MB , 1 TB = 1000 GB , 1 PT = 1000 TB , 1 Exabyte = 1000 PTPT => petabyte , TB => teraByte
6compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 7: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/7.jpg)
7compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 8: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/8.jpg)
The Motivation For Hadoop
8compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 9: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/9.jpg)
1.
2.
9compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 10: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/10.jpg)
3.
4.
5.
10
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
![Page 11: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/11.jpg)
Hadoop History
11compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 12: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/12.jpg)
Core Hadoop Concepts
12compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 13: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/13.jpg)
Hadoop Components
13compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 14: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/14.jpg)
HDFS
14compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 15: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/15.jpg)
HDFSConcepts
15compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 16: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/16.jpg)
HDFS : How Files Are Stored ?
16compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 17: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/17.jpg)
How Files Are Stored: Example
17compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 18: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/18.jpg)
IMP :
How MapReduce Work ?
18compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 19: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/19.jpg)
MapReduce: The Mapper
19compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
![Page 20: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/20.jpg)
Example :
20compiled by Rocky Jagtiani Tech Head for
SCTPL , 9892544177
![Page 21: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/21.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 21
![Page 22: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/22.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 22
![Page 23: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/23.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 23
![Page 24: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/24.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 24
![Page 25: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/25.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
25
Anatomy of a Hadoop Cluster :
![Page 26: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/26.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 989254417726
![Page 27: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/27.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 27
![Page 28: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/28.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 28
Who uses Hadoop ?
![Page 29: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/29.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
29
Hadoop Solutions
![Page 30: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/30.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 30
A
![Page 31: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/31.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
31
B What is Problem if the data is coming ?
![Page 32: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/32.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
32
C
![Page 33: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/33.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 33
The most common problems Hadoop can solve :
We understand how each problem is solved using Hadoop in brief
D
![Page 34: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/34.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 34
![Page 35: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/35.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
35
![Page 36: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/36.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 36
![Page 37: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/37.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 37
![Page 38: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/38.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 38
![Page 39: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/39.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 39
![Page 40: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/40.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
40
![Page 41: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/41.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
41
![Page 42: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/42.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
42
How some real-world companies use HadoopE
![Page 43: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/43.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
43
Hadoop Ecosystem
![Page 44: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/44.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 44
Cloudera Software (All Open-Source)
![Page 45: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/45.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177
45
*enterprise data
warehouse (EDW)
Conclusion :
![Page 46: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/46.jpg)
1) Input to mapper is
"Google is one of the richest companies "
"one who works with the Google is technical expert "
what will be the out put after reducing ?
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 46
Questions
![Page 47: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/47.jpg)
2) Input to mapper is
"Cat is eating milk"
"Cat is very sweet and she likes milk"
"milk is in bottle"
what will be the out put after reducing ?
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 47
![Page 48: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/48.jpg)
3) Input to mapper is
"Dollar is national currency for USA"
"Rupee is national currency for India"
"Dollar is ahead of Rupee in economy"
"India is developing country"
what will be the out put after Mapping ?
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 48
![Page 49: The Motivation For Hadoop Hadoop: Basic Concepts What Is ...db.suvenconsultants.com/section_9_ApacheHadoop.pdf · Objectives of this Session The Motivation For Hadoop What problems](https://reader033.fdocuments.us/reader033/viewer/2022042316/5f0471e77e708231d40e0042/html5/thumbnails/49.jpg)
compiled by Rocky Jagtiani Tech Head for SCTPL , 9892544177 49
what will be the out put after reducing ?
what will be the out put after shuffling?