Tachyon: Reliable File Sharing at Memory- Speed Across Cluster...
Transcript of Tachyon: Reliable File Sharing at Memory- Speed Across Cluster...
![Page 1: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/1.jpg)
Tachyon: Reliable File Sharing at Memory-
Speed Across Cluster Frameworks
Haoyuan Li, Ali Ghodsi, Matei Zaharia,
Scott Shenker, Ion Stoica
UC Berkeley
![Page 2: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/2.jpg)
Outline
Outline | Motivation| Design | Results| Status| Future
• Motivation
• System Design
• Preliminary Results
• Alpha Status
• Future Direction
![Page 3: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/3.jpg)
Outline| Motivation | Design | Results| Status| Future
Memory is King
![Page 4: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/4.jpg)
Memory Trend
Outline| Motivation | Design | Results| Status| Future
• RAM throughput increasing exponentially
![Page 5: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/5.jpg)
Disk Trend
Outline| Motivation | Design | Results| Status| Future
• Disk throughput increasing slowly
![Page 6: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/6.jpg)
Consequence
Outline| Motivation | Design | Results| Status| Future
• Memory locality key to achieving
– Interactive queries
– Fast query response times
![Page 7: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/7.jpg)
Current Big Data Eco-system
Outline| Motivation | Design | Results| Status| Future
• Many frameworks already leverage memory
– e.g. Spark, Shark, and other projects
• File sharing between jobs replicated to disk
– Replication enables fault-tolerance
• Problem
– Disk replication becomes main bottleneck as more frameworks move to memory
![Page 8: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/8.jpg)
Tachyon Project
Outline| Motivation | Design | Results| Status| Future
• Reliable file sharing at memory-speed across cluster frameworks/jobs
• Challenge
– How to achieve reliability without replication?
![Page 9: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/9.jpg)
Idea
Outline| Motivation | Design | Results| Status| Future
• File system keeps track of lineage
– Metadata about the jobs that created the files
• Keep one in-memory copy of files
– Upon failure use lineage to recompute files
– High-throughput and reliability at the same time
• File system with integrated workflow manager
– Filesystem can launch jobs to recompute data
![Page 10: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/10.jpg)
Spark & Shark with Tachyon
Outline| Motivation | Design | Results| Status| Future
• Shark and Spark use Tachyon as a RDD store
• Benefits of using Tachyon
– Memory-throughput file sharing between jobs
– Fault-isolation between jobs (JVM restart/crash)
![Page 11: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/11.jpg)
Tachyon
Outline| Motivation | Design | Results| Status| Future
![Page 12: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/12.jpg)
System Architecture
Outline| Motivation | Design | Results| Status| Future
![Page 13: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/13.jpg)
Lineage
Outline| Motivation | Design | Results| Status| Future
![Page 14: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/14.jpg)
Generic Lineage API
Outline| Motivation | Design | Results| Status| Future
• Binary program
• Configuration
• Input Files List
• Output Files List
• Dependency Type
![Page 15: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/15.jpg)
Fault Recovery Time
Outline| Motivation | Design | Results| Status| Future
Recomputation Cost?
![Page 16: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/16.jpg)
An Example
Outline| Motivation | Design | Results| Status| Future
![Page 17: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/17.jpg)
Asynchronous Checkpointing
Outline| Motivation | Design | Results| Status| Future
1. Better than using existing solutions even
under failure.
2. Bounded recovery time.
3. Failure is relatively rare.
![Page 18: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/18.jpg)
Implementation
Outline| Motivation | Design | Results| Status| Future
JAVA + Maven + Thrift
![Page 19: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/19.jpg)
Spark Sequential Read
Outline| Motivation | Design | Results | Status| Future
![Page 20: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/20.jpg)
Spark Sequential Write
Outline| Motivation | Design | Results | Status| Future
![Page 21: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/21.jpg)
Simulated Workflow using Spark
Outline| Motivation | Design | Results | Status| Future
![Page 22: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/22.jpg)
Conviva Spark Query (I/O intensive)
Outline| Motivation | Design | Results | Status| Future
More than
75x speedup
Tachyon
outperforms
Spark cache
![Page 23: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/23.jpg)
Conviva Spark Query (less I/O intensive)
Outline| Motivation | Design | Results | Status| Future
12x speedup
GC kicks
in earlier
for Spark
cache
![Page 24: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/24.jpg)
Alpha Status
Outline| Motivation | Design | Results | Status | Future
• Releases
– Developer Preview: V0.2.0 (4/10/2013), V0.2.1(4/25/2013)
• Step one in our approach:
– First read of files cached in-memory
– Writes go synchronously to HDFS (No lineage
information in Developer Preview release)
– Spark and MapReduce can run without any code
change (ser/de becomes the new bottleneck)
– data=SparkContext.textfile(HdfsPath)
![Page 25: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/25.jpg)
Alpha Status
Outline| Motivation | Design | Results | Status | Future
• Releases
– Developer Preview: V0.2.0 (4/10/2013), V0.2.1(4/25/2013)
• Step one in our approach:
– First read of files cached in-memory
– Writes go synchronously to HDFS (No lineage
information in Developer Preview release)
– Spark and MapReduce can run without any code
change (ser/de becomes the new bottleneck)
– data=SparkContext.textfile(TachyonPath)
![Page 26: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/26.jpg)
Current Features
Outline| Motivation | Design | Results | Status | Future
• Java-like file API
• Compatible with Hadoop
• Native support for raw tables
• Pluggable underlayer file system
• Command line interaction
• Web user interface
• WhiteList, PinList
![Page 27: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/27.jpg)
Shark Close Integration
Outline| Motivation | Design | Results | Status | Future
• Next Shark release can store tables in Tachyon
(fast columnar Ser/De).
Spark Cache Tachyon
Table Full Scan 1.4 sec 1.5 sec
GroupBy (10 GB Shark Memory)
50 – 90 sec 45 – 50 sec
GroupBy (15 GB Shark Memory)
44 – 48 sec 37 – 45 sec
![Page 28: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/28.jpg)
Web User Interface (Demo)
Outline| Motivation | Design | Results | Status | Future
http://ec2-50-17-67-227.compute-1.amazonaws.com:19999/home
![Page 29: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/29.jpg)
Web User Interface (Demo)
Outline| Motivation | Design | Results | Status | Future
![Page 30: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/30.jpg)
Jenkins
Outline| Motivation | Design | Results | Status | Future
https://amplab.cs.berkeley.edu/jenkins/view/Tachyon/
![Page 31: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/31.jpg)
Short Term Roadmap
Outline| Motivation | Design | Results | Status | Future
• Improve the current implementation. (e.g: more
FS method implementation; enhance unit tests;
enrich code document and wiki.)
• Enhance HDFS FS interface Implementation.
• Master fault tolerant.
• Efficient Ser/De support.
• Different components performance
benchmarks.
![Page 32: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/32.jpg)
More Information
Outline| Motivation | Design | Results | Status | Future
• http://tachyon-project.org/
• https://github.com/amplab/tachyon
• https://github.com/amplab/tachyon/wiki
• Run Tachyon Locally:
https://github.com/amplab/tachyon/wiki/Running-Tachyon-Locally
![Page 33: Tachyon: Reliable File Sharing at Memory- Speed Across Cluster Frameworksfiles.meetup.com/3138542/Tachyon_2013-05-09_Spark_Meetup.pdf · 2013-05-12 · Tachyon: Reliable File Sharing](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea78b914de7717f871f6c71/html5/thumbnails/33.jpg)
UC Berkeley
Thank
you!