Apache hadoop
-
Upload
sai-koppuravuri -
Category
Technology
-
view
491 -
download
2
description
Transcript of Apache hadoop
Apache Hadoop
- Large Scale Data Processing
Sharath Bandaru & Sai Dinesh Koppuravuri
Advanced Topics PresentationISYE 582 :Engineering Information Systems
Overview Understanding Big Data
Structured/Unstructured Data
Limitations Of Existing Data Analytics Structure
Apache Hadoop
Hadoop Architecture
HDFS
Map Reduce
Conclusions
References
Understanding Big Data
Big DataIs creating
Large And Growing Files
Measured in:Petabytes (10^12)Terabytes (10^15)
Which is largely unstructured
Structured/Unstructured Data
Why now ?D
ata
Gro
wth
STRUCTURED DATA – 20%
1980 2013
UNSTRUCTURED DATA – 80%
Source : Cloudera, 2013
Challenges posed by Big Data
Velocity
Volume
Variety
400 million tweets in a day on Twitter1 million transactions by Wal-Mart every hour
2.5 peta bytes created by Wal-Mart transactions in an hour
Videos, Photos, Text messages, Images, Audios, Documents, Emails, etc.,
Limitations Of Existing Data Analytics Architecture
BI Reports + Interactive Apps
RDBMS (aggregated data)
ETL Compute Grid
Storage Only Grid ( original raw data )
Collection
Instrumentation
Moving Data To Compute Doesn’t Scale
Can’t Explore Original High Fidelity Raw Data
Archiving=Premature Data Death
So What is Apache ?
• A set of tools that supports running of applications on big data.
• Core Hadoop has two main systems:
- HDFS : self-healing high-bandwidth clustered storage.
- Map Reduce : distributed fault-tolerant resource management and scheduling coupled with a scalable data programming abstraction.
History
Source : Cloudera, 2013
The Key Benefit: Agility/Flexibility
Schema-on-Write (RDBMS): Schema-on-Read (Hadoop):• Schema must be created before any data can be loaded.
• An explicit load operation has to take place which transforms data to DB internal structure.
• New columns must be added explicitly before new data for such columns can be loaded into the database
• Data is simply copied to the file store, no transformation is needed.
• A SerDe (Serializer/Deserlizer) is applied during read time to extract the required columns (late binding).
• New data can start flowing anytime and will appear retroactively once the SerDe is updated to parse it.
• Read is Fast• Standards/Governance
• Load is Fast• Flexibility/Agility
Pros
Use The Right Tool For The Right Job
Relational Databases: Hadoop:
Use when:• Interactive OLAP Analytics (< 1 sec)• Multistep ACID transactions• 100 % SQL compliance
Use when:• Structured or Not (Flexibility)• Scalability of Storage/Compute• Complex Data Processing
Traditional Approach
Big Data
Powerful ComputerProcessing limit
Enterprise Approach:
Hadoop Architecture
Task Tracker
Job Tracker
Name Node
Data Node
Master
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Slaves
Map Reduce
HDFS
Hadoop Architecture
Task Tracker
Job Tracker
Name Node
Data Node
Master
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Slaves
Application
Job Tracker
Task Tracker
Job Tracker
Name Node
Data Node
Master
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Slaves
Application
Job Tracker
Task Tracker
Job Tracker
Name Node
Data Node
Master
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Slaves
Application
HDFS: Hadoop Distributed File System
• A given file is broken into blocks (default=64MB), then blocks are replicated across cluster(default=3).
1
2
3
4
5
HDFS
3
4
5
1
2
5
1
3
4
2
4
5
1
2
3
Optimized for :• Throughput• Put/Get/Delete• Appends
Block Replication for :• Durability• Availability• Throughput
Block Replicas are distributed across servers and racks
Fault Tolerance for Data
Task Tracker
Job Tracker
Name Node
Data Node
Master
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Slaves
HDFS
Fault Tolerance for Processing
Task Tracker
Job Tracker
Name Node
Data Node
Master
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Slaves
Map Reduce
Fault Tolerance for Processing
Task Tracker
Job Tracker
Name Node
Data Node
Master
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Slaves
Tables are backed up
Map Reduce
Input Data
Map Map Map Map Map
Shuffle
Reduce Reduce
Results
Understanding the concept of Map Reduce
Mother
Sam
An Apple
• Believed “an apple a day keeps a doctor away”
The Story Of Sam
Understanding the concept of Map Reduce
• Sam thought of “drinking” the apple
He used a to cut the
and a to make
juice.
Understanding the concept of Map Reduce
Next day• Sam applied his invention to all the fruits he could find in the fruit basket
(map ‘( )’)
(reduce ‘( )’) Classical Notion of Map Reduce in Functional Programming
A list of values mapped into another list of values, which
gets reduced into a single value
Understanding the concept of Map Reduce
18 Years Later
• Sam got his first job in “Tropicana” for his expertise in making juices.
Now, it’s not just one basket
but a whole container of fruits
Also, they produce a list of
juice types separately
Fruits
NOT ENOUGH !! But, Sam had just ONE
and ONE
Large data and list of values for output
Wa i t !
Understanding the concept of Map Reduce
Brave Sam
Fruits
(<a, > , <o, > , <p, > , …)
Each input to a map is a list of <key, value> pairs
Each output of a map is a list of <key, value> pairs
(<a’, > , <o’, > , <p’, > , …)Grouped by keyEach input to a reduce is a <key, value-list> (possibly a list of these, depending on the grouping/hashing mechanism)e.g. <a’, ( …)>Reduced into a list of values
Implemented parallel version of his innovation
Understanding the concept of Map Reduce
• Sam realized,
– To create his favorite mix fruit juice he can use a combiner after the reducers
– If several <key, value-list> fall into the same group (based on the grouping/hashing algorithm) then use the blender (reducer) separately on each of them
– The knife (mapper) and blender (reducer) should not contain residue after use – Side Effect Free
Source: (Map Reduce, 2010).
Conclusions• The key benefits of Apache Hadoop:
1) Agility/ Flexibility (Quickest Time to Insight)
2) Complex Data Processing (Any Language, Any Problem)
3) Scalability of Storage/Compute (Freedom to Grow)
4) Economical Storage (Keep All Your Data Alive Forever)
• The key systems for Apache Hadoop are:
1) Hadoop Distributed File System : self-healing high-bandwidth clustered storage.
2) Map Reduce : distributed fault-tolerant resource management coupled with scalable data processing.
References
• Ekanayake, S. (2010, March). Map Reduce : The Story Of Sam. Retrieved April 13, 2013, from http://esaliya.blogspot.com/2010/03/mapreduce-explained-simply-as-story- of.html.
• Jeffrey Dean and Sanjay Ghemawat. (2004, December). Map Reduce : Simplified Data Processing on Large Clusters.
• The Apache Software Foundation. (2013, April). Hadoop. Retrieved April 19, 2013, from http://hadoop.apache.org/.
• Isabel Drost. (2010, February). Apache Hadoop : Large Scale Data Analysis made Easy. retrieved April 13, 2013, from http://www.youtube.com/watch?v=VFHqquABHB8.
• Dr. Amr Awadallah. (2011, November). Introducing Apache Hadoop : The Modern Data Operating System. Retrieved April 15, 2013, from http://www.youtube.com/watch?v=d2xeNpfzsYI