Big Data/Hadoop Infrastructure Considerations
-
Upload
richard-mcdougall -
Category
Documents
-
view
936 -
download
3
description
Transcript of Big Data/Hadoop Infrastructure Considerations
© 2009 VMware Inc. All rights reserved
Big Data in a Software-Defined Datacenter
Richard McDougall
Chief Architect, Storage and Application Services
VMware, Inc
@richardmcdougll
2
Trend: New Data Growing at 60% Y/Y
Source: The Information Explosion, 2009
medical(imaging,(sensors(
cad/cam,(appliances,(machine(data,(digital(movies(
digital(photos(
digital(tv(
audio(
camera(phones,(rfid(
satellite(images,(logs,(scanners,(twi7er(
Exabytes of information stored 20 Zetta by 2015 1 Yotta by 2030 Yes, you are part of the yotta generation…
3
Trend: Big Data – Driven by Real-World Benefit
4
Hadoop batch analysis
Big Data Family of Frameworks
File System/Data Store
Host Host Host Host Host Host
HBase real-time queries
NoSQL Cassandra, Mongo, etc Big SQL
Impala, Pivotal HawQ
Compute layer
Distributed Resource Management
Host
Other Spark, Shark, Solr,
Platfora, Etc,…
5
Big (Data) problems: making management easier
6
Log Processing / Click Stream Analytics
Machine Learning / sophisticated data mining
Web crawling / text processing
Extract Transform Load (ETL) replacement
Image / XML message processing
Broad Application of Hadoop technology
General archiving / compliance
Financial Services
Mobile / Telecom
Internet Retailer
Scientific Research
Pharmaceutical / Drug Discovery
Social Media
Vertical Use Cases Horizontal Use Cases
Hadoop’s ability to handle large unstructured data affordably and efficiently makes it a valuable tool kit for enterprises across a number of applications and fields.
7
How does Hadoop enable parallel processing?
Source: http://architects.dzone.com/articles/how-hadoop-mapreduce-works
! A framework for distributed processing of large data sets across clusters of computers using a simple programming model.
8
Hadoop System Architecture
! MapReduce: Programming framework for highly parallel data processing
! Hadoop Distributed File System (HDFS): Distributed data storage
! Other Distributed Storage Options: • Alternatives to HDFS • MAPR (SW), Isilon (HW)
9
Host%1 Host%2 Host%3
%Input%File
Input%File
Job Tracker Schedules Tasks Where the Data Resides
Job Tracker
Job
DataNode
Task%%Tracker
Split%1%–%64MB
Task%<%1 Split%2%–%64MB Split%3%–%64MB
Task%%Tracker
Task%%Tracker
DataNode DataNode
Block%1%–%64MB Block%2%–%64MB Block%3%–%64MB
Task%<%2 Task%<%3
10
Hadoop Data Locality and Replication
11
Hadoop Topology Awareness
12
Rules of Thumb: Sizing for Hadoop
! Disk: • Provide about 50Mbytes/sec of disk bandwidth per core • If using SATA, that’s about one disk per core
! Network • Provide about 200mbits of aggregate network bandwidth per core
! Memory • Use a memory:core ratio of about 4Gbytes:core
! Example: 100 node cluster • 100 x 16 cores = 1600 cores
• 1600 x 50Mbytes/sec = 80Gbytes/sec(!) • 1600 x 200mbits = 320gbits of network traffic
13
Big Data Frameworks and Characteristics
Framework Scale of data
Scale of Cluster
Computable Data?
Local Disks?
File System: Gluster, Isilon, HDFS, etc,…
10s PB 10s to 100s Some Yes, for cost
Map-reduce: Hadoop
100s PB 10s to 1,000s Yes Yes, for cost, bandwidth and availability
Big-SQL: HawQ,, Aster Data, Impala, …
PB’s 10s to 100s Some Yes, for cost and bandwidth
No-SQL: Cassandra, hBase, …
Trilions Of rows
10s to 100s Some Yes, for cost and availability
In-Memory: Redis, Gemfire, Membase, …
Billions of rows
10s-100s Yes Primarily Memory
14
Big Data Virtual Infrastructure
15
Virtualization enables a Common Infrastructure for Big Data
Single purpose clusters for various business applications lead to cluster sprawl.
Virtualization Platform
! Simplify • Single Hardware Infrastructure
• Unified operations
! Optimize • Shared Resources = higher utilization
• Elastic resources = faster on-demand access
MPP DB Hadoop HBase
Virtualization Platform
Scaleout SQL
Hadoop
HBase
Cluster Sprawling
Cluster Consolidation
16
Native versus Virtual Platforms, 24 hosts, 12 disks/host
0
50
100
150
200
250
300
350
400
450
TeraGen TeraSort TeraValidate
Elap
sed
tim
e, s
econ
ds
(low
er is
bet
ter)
Native 1 VM 2 VMs 4 VMs
Native versus Virtual Platforms, 24 hosts, 12 disks/host
17
Why Virtualize Hadoop?
! Shrink and expand cluster on demand
! Resource Guarantee ! Independent scaling of
Compute and data
Mixed Workloads
! No more single point of failure
! One click to setup
! High availability for MR Jobs
Highly Available
! Rapid deployment ! Unified operations
across enterprise
! Easy Clone of Cluster
Simple to Operate
18
Ad hoc data mining
In-house Hadoop as a Service “Enterprise EMR” – (Hadoop + Hadoop)
Compute layer
Data layer
HDFS
Host Host Host Host Host Host
Production recommendation engine
Production ETL of log files
Virtualization platform
HDFS
19
Short-lived Hadoop compute cluster
Integrated Hadoop and Webapps – (Hadoop + Other Workloads)
HDFS
Host Host Host Host Host Host
Web servers for ecommerce site
Compute layer
Data layer
Hadoop compute cluster
Virtualization platform
20
Hadoop batch analysis
Integrated Big Data Production – (Hadoop + other big data)
HDFS
Host Host Host Host Host Host
HBase real-time queries NoSQL –
Cassandra, Mongo, etc
Big SQL – Impala
Compute layer
Data layer
Virtualization
Host
Other Spark, Shark, Solr,
Platfora, Etc,…
21
Serengeti: Deploy a Hadoop Cluster in under 30 Minutes
Deploy vHelperOVF to vSphere
Select configuration template
Automate deployment
Select Compute, memory, storage and network
Done
Step 1: Deploy Serengeti virtual appliance on vSphere.
Step 2: A few simple commands to stand up Hadoop Cluster.
22
Why Virtualize Hadoop?
! Shrink and expand cluster on demand
! Resource Guarantee ! Independent scaling of
Compute and data
Mixed Workloads
! No more single point of failure
! One click to setup
! High availability for MR Jobs
Highly Available
! Rapid deployment ! Unified operations
across enterprise
! Easy Clone of Cluster
Simple to Operate
23
vMotion, HA, FT enables High Availability for the Hadoop Stack
HDFS (Hadoop Distributed File System)
HBase (Key-Value store)
MapReduce (Job Scheduling/Execution System)
Pig (Data Flow) Hive (SQL)
BI Reporting ETL Tools
Man
agem
ent S
erve
r
Zook
eepr
(Coo
rdin
atio
n) HCatalog
RDBMS
Namenode
Jobtracker
Hive MetaDB
Hcatalog MDB
Server
24
Why Virtualize Hadoop?
! Shrink and expand cluster on demand
! Resource Guarantee ! Independent scaling of
Compute and data
Mixed Workloads
! No more single point of failure
! One click to setup
! High availability for MR Jobs
Highly Available
! Rapid deployment ! Unified operations
across enterprise
! Easy Clone of Cluster
Simple to Operate
25
Containers with Isolation are a Tried and Tested Approach
Host Host Host Host Host Host
vSphere, DRS
Host
Hungry Workload 1 Reckless Workload 2
Nosy Workload 3
26
Mixing Workloads: Three big types of Isolation are Required
! Resource Isolation • Control the greedy noisy neighbor • Reserve resources to meet needs
! Version Isolation • Allow concurrent OS, App, Distro versions
! Security Isolation • Provide privacy between users/groups
• Runtime and data privacy required
Host Host Host Host Host Host
vSphere, DRS
Host
27
Elasticity Enables Sharing of Resources
28 O
ther
VM
Oth
er V
M
Oth
er V
M
“Time Share”
Oth
er V
M
Oth
er V
M
Oth
er V
M
Oth
er V
M
Oth
er V
M
Oth
er V
M
Oth
er V
M
Oth
er V
M
Host Host Host
Oth
er V
M
Oth
er V
M
Oth
er V
M
Oth
er V
M
Had
oop
Had
oop
Had
oop
Had
oop
Had
oop
Had
oop
While existing apps run during the day to support business operations, Hadoop batch jobs kicks off at night to conduct deep analysis of data.
VMware vSphere
vHelper
HDFS HDFS HDFS
29
Big Data Impact to Storage
30
Traditional Storage – VMDK’s on SAN/NAS
VMDK (block)
VMDK (block)
VMDK (block)
App VM DB VM
Boot Data Boot
VMFS (SAN or NFS)
EMC NTAP
Archive
VMCB VM Level Archive
• VMs have just a few self-contained VMDKs • Data is not shared between VMs • VMs have limited individual bandwidth needs
31
Rapid Growth of Big Data Capacity Necessitates New storage
Source: The Information Explosion, 2009
medical(imaging,(sensors(
cad/cam,(appliances,(machine(data,(digital(movies(
digital(photos(
digital(tv(
audio(
camera(phones,(rfid(
satellite(images,(logs,(scanners,(twi7er(
Exabytes of information stored 20 Zetta by 2015 1 Yotta by 2030 Yes, you are part of the yotta generation…
32
Use Local Disk where it’s Needed
SAN Storage
$2 - $10/Gigabyte
$1M gets: 0.5Petabytes
200,000 Disk IOPS 8Gbyte/sec
NAS Filers
$1 - $5/Gigabyte
$1M gets: 1 Petabyte
200,000 Disk IOPS 10Gbyte/sec
Local Storage
$0.05/Gigabyte
$1M gets: 10 Petabytes
400,000 Disk IOPS 250 Gbytes/sec
33
Storage Economics
$-
$0.50
$1.00
$1.50
$2.00
$2.50
$3.00
$3.50
$4.00
$4.50
$5.00
$5.50
0.5 1 2 4 8 16 32 64 128
Cost per GB
Petabytes Deployed
Traditional SAN/NAS VMware vSAN
Distributed Object Storage
HDFS MAPR CEPH Scale-out NAS
Isilon, NTAP
34
Scalable Storage Architecture: Hadoop Demands
! Each node needs approximately ~1 disk of bandwidth per core • 16 core node needs ~1GByte/sec of bandwidth
! A 1,000 node Hadoop cluster needs 1Terabyte/sec of bandwidth • Local disks: $800 of local disks per node
• Traditional SAN: Would require an est. $50k of SAN storage per node
Data Node
Com
pute
Com
pute
Com
pute
Data Node C
ompu
te
Com
pute
Com
pute
Data Node
Com
pute
Com
pute
Com
pute
Data Node
Com
pute
Com
pute
Com
pute
35
Storage-Servers: Configurations and Capabilities
16-24 core server 12-24 SATA 2-4TB Disks 10 GbE adapter Throughput: 2.5Gbytes/sec Capacity: 96TB
16-24 core server 80 SATA 2-4TB Disks 10 GbE adapter Throughput: 2.5Gbytes/sec Capacity: 320TB
Typical Storage Server
High Capacity Server
36
Scalable Storage Bandwidth is Important for Big Data
0
20
40
60
80
100
120
1 10 20 30 40 50
Scalable Storage Single SAN/NAS
# Hosts
GBytes/sec
37
Hadoop has Significant Ephemeral Data
%75%%of%Disk%Bandwidth%
Job%
Map%Task%
Map%Task%
Map%Task%
Map%Task%
Reduce%
Reduce%
HDFS%
DFS%Input%Data%
DFS%Output%Data%%
12%%of%Bandwidth%
%12%%of%Bandwidth%
Spills%&%Logs%spill*.out*
Spills%
Map%Output%file.out*
Shuffle%Map_*.out*
Sort%
Combine%Intermediate.out*
38
Impact of Temporary Data
! Leverage local-disks when shared-storage is used • As much as 75% of the bandwidth will be transient • No-need for reliable, replicated storage
• Just use unprotected local-disk
HDFS%
Shared Data (Shared, Network Storage)
Temporary Data (Local Disks)
39
Extend Virtual Storage Architecture to Include Local Disk
! Shared Storage: SAN or NAS • Easy to provision • Automated cluster rebalancing
! Hybrid Storage • SAN for boot images, VMs, other
workloads
• Local disk for Hadoop & HDFS • Scalable Bandwidth, Lower Cost/GB
Host
Had
oop
Oth
er V
M
Oth
er V
M
Host
Had
oop
Had
oop
Oth
er V
M
Host
Had
oop
Had
oop
Oth
er V
M
Host
Had
oop
Oth
er V
M
Oth
er V
M
Host
Had
oop
Had
oop
Oth
er V
M
Host
Had
oop
Had
oop
Oth
er V
M
40
Scale-out Storage for Big Data: Local Disk or Scale-out-NAS
Host%
Host%
Host%
Host%
Host%
Host%
Host%
Top%of%Rack%Switch%
Servers with Local Disks Big-Data using Scale-out NAS
Host%
Host%
Host%
Host%
Host%
Host%
Top%of%Rack%Switch%
Scale<out%NAS%
Host%
Host%
Host%
Host%
Host%
Host%
Top%of%Rack%Switch%
Scale<out%NAS%
Temp Data
Shared Data
41
Big-Data using Local Disks
Host%
Host%
Host%
Host%
Host%
Host%
Host%
Top%of%Rack%Switch%
Servers with Local Disks
16-24 core server 12-24 SATA 2-4TB Disks 10 GbE adapter iSCSI/NFS for Shared Storage for vMotion etc,…
High Performance 10GBE Switch per Rack
42
Big Data with Scale-out-NAS
Big-Data using Scale-out NAS
Host%
Host%
Host%
Host%
Host%
Host%
Top%of%Rack%Switch%
Scale<out%NAS%
Host%
Host%
Host%
Host%
Host%
Host%
Top%of%Rack%Switch%
Scale<out%NAS%
Temp Data
Shared Data
Isilon Scale-out
NAS
Local Disk or SSD In each Host
For Transient Data
43
Big Data Impact to Networking
44
Hadoop has Specific Network Demands that must be Considered
Source: http://architects.dzone.com/articles/how-hadoop-mapreduce-works
! A framework for distributed processing of large data sets across clusters of computers using a simple programming model.
45
Host%
Host%
Host%
Top%of%Rack%Switch%
Host%
L2%Switch%
Top%of%Rack%Switch%
L2%Switch%
Host%
Host%
Host%
Host%
Top%of%Rack%Switch%
Host%
Host%
Host%
Host%
Top%of%Rack%Switch%
Host%
Host%
Host%
Host%
L2%Switch% L2%Switch%
AggregaQng%Switch%
AggregaQng%Switch%
A Typical Network Architecture
46
Host%
Host%
Host%
Top%of%Rack%Switch%
Host%
L2%Switch%
Top%of%Rack%Switch%
L2%Switch%
Host%
Host%
Host%
Host%
Top%of%Rack%Switch%
Host%
Host%
Host%
Host%
Top%of%Rack%Switch%
Host%
Host%
Host%
Host%
L2%Switch% L2%Switch%
AggregaQng%Switch%
AggregaQng%Switch%
A Typical Network Architecture
! Hadoop Requirements • Hadoop can generate up to 200mbits per core of network traffic • Bursts of bandwidth during remote data read and shuffle phases
• Data locality can help minimize net reads, but shuffle and reduce are global
! Inside Rack Network • Today’s core-counts require 10GBe to hosts inside Rack
• High throughput switches required to meet aggregate bandwidth
• Switch Network Buffers sizing is important due to coordinated bursts
! Rack-to-rack Network • Two level switch aggregation can be a bottleneck
• Easy to saturate, especially during shuffle phases
! Link-Speed Sizing • Hadoop originally designed around 4 cores, 1Gbit uplink
• Now, 16-24 cores per box means 10Gbe is required for uplink
47
Host%
Host%
Host%
Top%of%Rack%Switch%
Host%
L2%Switch%
Top%of%Rack%Switch%
L2%Switch%
Host%
Host%
Host%
Host%
Top%of%Rack%Switch%
Host%
Host%
Host%
Host%
Top%of%Rack%Switch%
Host%
Host%
Host%
Host%
L2%Switch% L2%Switch%
AggregaQng%Switch%
AggregaQng%Switch%
Two Level Aggregated Network Characteristics
Top of Rack: Overprovisioning Possible 10Gbe High End Switch >500Gbits/sec
Between Racks: Bandwidth can be saturated
48
Future Network Designs, optimized for Big-Data
Traditional Tree-structured networks
Clos network between Aggregation and Intermediate (core) switches
LA – (physical) Locator address AA – (flat, virtual) Application address
Pros: full-bisection bandwidth, non-blocking switching fabric, path diversity! Formalized in 1953
Cons: High oversubscription of upper-level (Aggregation) and core switches. Hard to get full-bisection bandwidth Images courtesy Greenberg et al.:http://research.microsoft.com/pubs/80693/vl2-sigcomm09-final.pdf and http://www.stanford.edu/class/ee384y/Handouts/clos_networks.pdf
49
In Conclusion
! Software Defined Datacenter enables Unified Big Data Platform ! Plan to build a Software Defined “Big Data”, Datacenter • Enable Hadoop and other Big-Data ecosystem components
! Values • Consolidated Hardware, reduced SKU
• Higher Utilization: Can leverage a shared cluster for bursty workloads • Rapid Deployment: Use APIs to deploy software-defined clusters rapidly
• Flexible Cluster Use: Share the same infrastructure for different workload types
! References www.projectserengeti.org vmware.com/hadoop VMware Hadoop Performance Whitepaper VMware Hadoop HA/FT Paper
© 2009 VMware Inc. All rights reserved
Big Data in a Software-Defined Datacenter
Richard McDougall
Chief Architect, Storage and Application Services
VMware, Inc
@richardmcdougll