Understanding hdfs
-
Upload
thirunavukkarasu-ps -
Category
Documents
-
view
1.162 -
download
0
Transcript of Understanding hdfs
![Page 1: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/1.jpg)
HDFS(Hadoop Distributed File System)
Thiru
![Page 2: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/2.jpg)
Agenda
Typical Work flow
Writing file in to HDFS
Reading file from
HDFS
Rack Awareness
Planning for a
ClusterQ & A
![Page 3: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/3.jpg)
Client
Map Reduce{Job Tracker}
HDFS{Name Node}
{Secondary Name Node}
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Data Node
Task Tracker
Masters
Slaves
Hadoop Server Roles
![Page 4: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/4.jpg)
Hadoop Cluster Name
NodeJob
TrackerSecondary
NNHadoop Client
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
DN + TT
Hadoop Client
![Page 5: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/5.jpg)
Sample HDFS Workflow
Write data in to cluster (HDFS)
Analyze the data (Map Reduce)
Store the result in to cluster (HDFS)
Read the result from cluster (HDFS)
Sample scenario: How many times customer called to customer care enquiring about a recently launched product? Compare it against the AD campaign in the television. Correlate both and find the best time to run the AD
CRM Data entry SQOO
P
HDFS
Map Reduce
ResultResult
![Page 6: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/6.jpg)
Write data in to cluster (HDFS
Hadoop Client
File size 200 MB
I want to write file
Name Node
Ok! Block size is 64 MB. Split the file in to 3 and write in to node 1,4,5
Data Node 1
Data Node 2
Data Node 3
Data Node 4
Data Node 5
Data Node 6
Client Consults
Name node
Client write data to one data node
Data node replicates
as per replication factor and intimates
Name node
Cycle repeats for every block
![Page 7: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/7.jpg)
Rack Awareness
Never loose data when a rack is down
Keep bulky flows within Rack when possible
Assumption in rack has higher bandwidth, and low latency
DN 1
DN 2
DN 3
DN 4
DN 5
DN 6
DN 7
DN 8
DN 9
DN 10
DN 11
DN 12
Name Node
Rack Aware: Rack 1: Data node 1Data node 2Rack 2:Data node 5
A
A
A
CB B
B
C
C
![Page 8: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/8.jpg)
Multi Block Replication
Hadoop Client
File size 200 MB
File.txt
Name Node
A CB
DN 1
DN 2
DN 3
DN 4
DN 5
DN 6
DN 7
DN 8
DN 9
DN 10
DN 11
DN 12
A
Replicate in 3,8
A
A
![Page 9: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/9.jpg)
Name Node
Data node sends hearth beatsEvery 10th heart beat is Block reportName node builds meta data from block report
If name node is down, HDFS is downMissing heartbeats signify lost nodes
Name node consults metadata and finds affected data
Name node consults rack awareness script
Name node tells data node to replicate
![Page 10: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/10.jpg)
Name Node & Secondary Name node Not a hot standby for the name node* (Zoo keeper)
Connects to name node every one hour* (Configurable)
Housekeeping, backup of Name node meta data
Saved meta data can be used to rebuild name node
Primary Name node
Secondary Name Node
File System Metadata:
File.txt = A0 {1,5,7}A1 {1,7,9}
A2{5,10,15}
It’s been 1 hr, give
your data
![Page 11: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/11.jpg)
Understanding Secondary name node house keeping
fsimageedits
edits-new
fsimageedits
Fsimage.ckpt
Fsimage.ckpt
edits Fsimage.ckpt
Primary Name Node Secondary Name Node
![Page 12: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/12.jpg)
Reading data from HDFS Cluster
Hadoop Client
I want to read file file.txt
Name Node
Ok! File.txt =
blck a {1,5,6}Blck b
{8,1,2}Blck c
{5,8,9}
Data Node 1
Data Node 2
Data Node 3
Data Node 4
Data Node 5
Data Node 6
Client Consults
Name node
Client receives
DN list for each block
Client picks first node
of list
Client reads data sequentiall
y
Data Node 7
Data Node 8
Data Node 9C
B
A
A A
BB
C
C
![Page 13: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/13.jpg)
Choosing right hardware
Master node
Single Point of Failure
Dual power supply for
redundancy
No Commodity hardware
Regular Data backup
RAM thumb rule – 1 GB/
Million blocks of
data
# Task per node•1 core can run 1.5 Mapper or Reducer
![Page 14: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/14.jpg)
Practice at Yahoo!
![Page 15: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/15.jpg)
HDFS clusters at Yahoo! include about 3500 nodes
A typical cluster node has:
· 2 quad core Xeon processors @ 2.5ghz · Red Hat Enterprise Linux Server Release 5.1 · Sun Java JDK 1.6.0_13-b03 · 4 directly attached SATA drives (one terabyte each) · 16G RAM · 1-gigabit Ethernet
Practice at YAHoo!
![Page 16: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/16.jpg)
70 percent of the disk space is allocated to HDFS. The remainder is reserved for the operating system (Red Hat Linux), logs, and space to spill the output of map tasks. (MapReduce intermediate data are not stored in HDFS.)
For each cluster, the NameNode and the BackupNode hosts are specially provisioned with up to 64GB RAM; application tasks are never assigned to those hosts.
In total, a cluster of 3500 nodes has 9.8 PB of storage available as blocks that are replicated three times yielding a net 3.3 PB of storage for user applications. As a convenient approximation, one thousand nodes represent one PB of application storage.
Practice at YAHoo!
![Page 17: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/17.jpg)
Durability of Datauncorrelated node failures
Replication of data three times is a robust guard against loss of data due to uncorrelated node failures.correlated node failures, the failure of a rack or core switch.
HDFS can tolerate losing a rack switch (each block has a replica on some other rack).loss of electrical power to the cluster
a large cluster will lose a handful of blocks during a power-on restart.
Practice at YAHoo!
![Page 18: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/18.jpg)
Benchmarks
Practice at YAHoo!
![Page 19: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/19.jpg)
Benchmarks
NameNode Throughput benchmark
Practice at YAHoo!
![Page 20: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/20.jpg)
Automated failover
plan: Zookeeper, Yahoo’s distributed consensus technology to build an automated failover solution Scalability of the NameNode
Solution: Our near-term solution to scalability is to allow multiple namespaces (and NameNodes) to share the physical storage within a cluster.
Drawbacks: The main drawback of multiple independent namespaces is the cost of managing them.
Future work
![Page 21: Understanding hdfs](https://reader036.fdocuments.us/reader036/viewer/2022081508/5576ca7ed8b42ae3108b4fe7/html5/thumbnails/21.jpg)
Thank you