Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with...
Transcript of Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with...
![Page 1: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/1.jpg)
Improve Hadoop Economics, Performance,
and Security with Compression and
Encryption
Ravi Lambi
Director of Software Engineering
Data Compression and Security Business Unit
Exar Corporation
Santa Clara, CA USA
November 2014
1
![Page 2: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/2.jpg)
The Storage IO BottleneckPerformance Gap
1
10
100
1000
10000
100000
1000000
10000000
100000000
Processor
Traditional Disk
Santa Clara, CA USA
November 2014 2
![Page 3: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/3.jpg)
The Storage IO BottleneckCurrent Server Solution
More Disks and more rack space
This will increase management cost, and also require more expensive
storage controller. Additionally, there is a limit to scale the width – each
server has a hard physical limit.
Santa Clara, CA USA
November 2014 3
![Page 4: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/4.jpg)
The Storage IO Bottleneck
It Is Difficult to Balance Performance, Capacity
Scaling and Cost Associated with the Storage IO
Summarized Challenge
Performance
Cost Capacity
Santa Clara, CA USA
November 2014 4
![Page 5: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/5.jpg)
Ingest
Map
Compress/Distribute
Decompress/Compute
Reduce
Output
Compression
CodecDiskNetwork
Santa Clara, CA USA
November 2014 5
Compression TechnologyWhere In Hadoop To Apply Compression?
![Page 6: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/6.jpg)
Exar’s Hadoop Acceleration - AltraHD
Ingest
Map
Compress/Distribute
Decompress/Compute
Reduce
Output
Compression
Codec
File System
Filter Driver
Driver
DiskNetwork
File System Filter
DriverCompression
Codec
5 GB/sec HW
Compression &
Encryption
Accelerator
Santa Clara, CA USA
November 2014 6
Offload Compression and Accelerate
![Page 7: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/7.jpg)
AltraHD Overview
Storage Volume
Native File System
File System Filter
Driver
Driver
Applications
Native Linux Kernel
• File System Filter Driver
– Kernel plug-in at the file system layer
– Compresses/decompresses ALL files
independent of application
• Transparent to the Application
– No modification to Applications or
Workflow
• Seamlessly Layers over File System
– Supports EXT3, EXT4, or XFS
• Fast, Easy Deployment
– No APIs – Software installs in minutes
• Hardware Acceleration Offloads Host
CPU
Exar Compression
& Encryption
Acceleration Card
Core Technology
Santa Clara, CA USA
November 2014 7
![Page 8: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/8.jpg)
MapReduce 1 Terasort Benchmark
93% 35% 27% 51% 21% 22%
0
200
400
600
800
1000
1200
1400
EXT3 8 Disk XFS 8 Disk EXT4 8 Disk EXT3 12 Disk XFS 12 Disk EXT4 12 Disk
SE
CO
ND
S
Native AltraHD % Improvement
Value Proposition – Performance
Santa Clara, CA USA
November 2014 8
![Page 9: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/9.jpg)
Value Proposition - PerformanceMarReduce2 Job Execution Time
27% 34% 31% 18% 18% 24%
0
200
400
600
800
1000
1200
1400
1600
1800
2000
2200
EXT3 8 Disk XFS 8 Disk EXT4 8 Disk EXT3 12 Disk XFS 12 Disk EXT4 12 Disk
SE
CO
ND
S
Native AltraHD % Improvement
Santa Clara, CA USA
November 2014 9
![Page 10: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/10.jpg)
Value Proposition – Storage
Santa Clara, CA USA
November 2014 10
Increased Storage Capacity
672
192
0 100 200 300 400 500 600 700
TERABYTES
MR2 Effective Storage Capacity
Native AltraHD
1344
192
0 300 600 900 1200 1500
TERABYTES
MR1 Effective Storage Capacity
Native AltraHD
![Page 11: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/11.jpg)
Value Proposition – Storage
Santa Clara, CA USA
November 2014 11
Increased Storage Capacity
Native - Storage
AltraHD – Effective Storage
![Page 12: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/12.jpg)
Value Proposition – Security
Exar’s Compression Acceleration Card Supports
Compression, Encryption, and Hashing in a Single
Pass
Aligned with Hadoop Security Roadmap
Santa Clara, CA USA
November 2014 12
Compression
Encryption
Hashing
![Page 13: Improve Hadoop Economics, Performance, and … Hadoop Economics, Performance, and Security with Compression and Encryption Ravi Lambi Director of Software Engineering Data Compression](https://reader031.fdocuments.us/reader031/viewer/2022020412/5ae129f17f8b9a1c248e091d/html5/thumbnails/13.jpg)
Value Proposition – Indirect Values Other Savings
Reduce Indirect Costs:
• Power
• Rack Space
• Cooling
• Disk Life
• etc.
Santa Clara, CA USA
November 2014 13