Tachyon: Reliable File Sharing at Memory- Speed Across...
Transcript of Tachyon: Reliable File Sharing at Memory- Speed Across...
![Page 1: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/1.jpg)
Tachyon: Reliable File Sharing at Memory-
Speed Across Cluster Frameworks
Haoyuan Li
UC Berkeley
![Page 2: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/2.jpg)
Outline
Outline | Motivation| Design | Results| Status| Future
• Motivation
• System Design
• Evaluation Results
• Release Status
• Future Directions
![Page 3: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/3.jpg)
Outline| Motivation | Design | Results| Status| Future
Memory is King
![Page 4: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/4.jpg)
Memory Trend
Outline| Motivation | Design | Results| Status| Future
• RAM throughput increasing exponentially
![Page 5: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/5.jpg)
Disk Trend
Outline| Motivation | Design | Results| Status| Future
• Disk throughput increasing slowly
![Page 6: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/6.jpg)
Consequence
Outline| Motivation | Design | Results| Status| Future
• Memory locality key to achieve
– Interactive queries
– Fast query response
![Page 7: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/7.jpg)
Current Big Data Eco-system
Outline| Motivation | Design | Results| Status| Future
• Many frameworks already leverage memory
– e.g. Spark, Shark, and other projects
• File sharing among jobs replicated to disk
– Replication enables fault-tolerance
• Problems
– Disk scan is slow for read.
– Synchronous disk replication for write is even slower.
![Page 8: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/8.jpg)
Tachyon Project
Outline| Motivation | Design | Results| Status| Future
• Reliable file sharing at memory-speed across cluster frameworks/jobs
• Challenge
– How to achieve reliable file sharing without replication?
![Page 9: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/9.jpg)
Idea
Outline| Motivation | Design | Results| Status| Future
Re-computation (Lineage) based storage using memory aggressively.
1. One copy of data in memory (Fast)
2. Upon failure, re-compute data using lineage (Fault tolerant)
![Page 10: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/10.jpg)
Stack
Outline| Motivation | Design | Results| Status| Future
![Page 11: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/11.jpg)
System Architecture
Outline| Motivation | Design | Results| Status| Future
![Page 12: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/12.jpg)
Lineage
Outline| Motivation | Design | Results| Status| Future
![Page 13: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/13.jpg)
Lineage Information
Outline| Motivation | Design | Results| Status| Future
• Binary program
• Configuration
• Input Files List
• Output Files List
• Dependency Type
![Page 14: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/14.jpg)
Fault Recovery Time
Outline| Motivation | Design | Results| Status| Future
Re-computation Cost?
![Page 15: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/15.jpg)
Example
Outline| Motivation | Design | Results| Status| Future
![Page 16: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/16.jpg)
Asynchronous Checkpoint
Outline| Motivation | Design | Results| Status| Future
1. Better than using existing solutions even
under failure.
2. Bounded recovery time (Naïve and Snapshot
asynchronous checkpointing).
![Page 17: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/17.jpg)
Master Fault Tolerance
Outline| Motivation | Design | Results| Status| Future
• Multiple masters
– Use ZooKeeper to elect a leader
• After crash workers contact new leader
– Update the state of leader with contents of caches
![Page 18: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/18.jpg)
Implementation Details
Outline| Motivation | Design | Results| Status| Future
• 15,000+ lines of JAVA
• Thrift for data transport
• Underlayer file system supports HDFS, S3,
localFS, GlusterFS
• Maven, Jenkins
![Page 19: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/19.jpg)
Sequential Read using Spark
Outline| Motivation | Design | Results | Status| Future
Flat Datacenter
Storage
Theoretical Maximum
Disk Throughput
![Page 20: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/20.jpg)
Sequential Write using Spark
Outline| Motivation | Design | Results | Status| Future
Flat Datacenter
Storage
Theoretical Maximum
Disk Throughput
![Page 21: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/21.jpg)
Realistic Workflow using Spark
Outline| Motivation | Design | Results | Status| Future
![Page 22: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/22.jpg)
Realistic Workflow Under Failure
Outline| Motivation | Design | Results | Status| Future
![Page 23: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/23.jpg)
Conviva Spark Query (I/O intensive)
Outline| Motivation | Design | Results | Status| Future
More than
75x speedup
Tachyon
outperforms
Spark cache
because of
JAVA GC
![Page 24: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/24.jpg)
Conviva Spark Query (less I/O intensive)
Outline| Motivation | Design | Results | Status| Future
12x speedup
GC kicks
in earlier
for Spark
cache
![Page 25: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/25.jpg)
Alpha Status
Outline| Motivation | Design | Results | Status | Future
• Releases
– Developer Preview: V0.2.1 (4/25/2013)
– Contributions from:
![Page 26: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/26.jpg)
Alpha Status
Outline| Motivation | Design | Results | Status | Future
• First read of files cached in-memory
• Writes go synchronously to HDFS (No lineage
information in Developer Preview release)
• MapReduce and Spark can run without any code
change (ser/de becomes the new bottleneck)
![Page 27: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/27.jpg)
Current Features
Outline| Motivation | Design | Results | Status | Future
• Java-like file API
• Compatible with Hadoop
• Master fault tolerance
• Native support for raw tables
• WhiteList, PinList
• Command line interaction
• Web user interface
![Page 28: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/28.jpg)
Spark without Tachyon
Outline| Motivation | Design | Results | Status | Future
val file = sc.textFile(“hdfs://ip:port/path”)
![Page 29: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/29.jpg)
Spark with Tachyon
Outline| Motivation | Design | Results | Status | Future
val file = sc.textFile(“tachyon:// ip:port/path”)
![Page 30: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/30.jpg)
Shark without Tachyon
Outline| Motivation | Design | Results | Status | Future
CREATE TABLE orders_cached AS SELECT * FROM orders;
![Page 31: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/31.jpg)
Shark with Tachyon
Outline| Motivation | Design | Results | Status | Future
CREATE TABLE orders_tachyon AS SELECT * FROM orders;
![Page 32: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/32.jpg)
Experiments on Shark
Outline| Motivation | Design | Results | Status | Future
• Shark (from 0.7) can store tables in Tachyon
with fast columnar Ser/De
20 GB data / 5 machines Spark Cache Tachyon
Table Full Scan 1.4 sec 1.5 sec
GroupBys (10 GB Shark Memory) 50 – 90 sec 45 – 50 sec
GroupBys (15 GB Shark Memory) 44 – 48 sec 37 – 45 sec
![Page 33: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/33.jpg)
Experiments on Shark
Outline| Motivation | Design | Results | Status | Future
• Shark (from 0.7) can store tables in Tachyon
with fast columnar Ser/De
20 GB data / 5 machines Spark Cache Tachyon
Table Full Scan 1.4 sec 1.5 sec
GroupBys (10 GB Shark Memory) 50 – 90 sec 45 – 50 sec
GroupBys (15 GB Shark Memory) 44 – 48 sec 37 – 45 sec
4 * 100 GB TPC-H data / 17 machines Spark Cache Tachyon
TPC-H Q1 65.68 sec 24.75 sec
TPC-H Q2 438.49 sec 139.25 sec
TPC-H Q3 467.79 sec 55.99 sec
TPC-H Q4 457.50 sec 111.65 sec
![Page 34: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/34.jpg)
Future
Outline| Motivation | Design | Results | Status | Future
• Efficient Ser/De support
• Fair sharing for memory
• Full support for lineage
• Next release is coming soon
![Page 35: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/35.jpg)
Acknowledgment
Outline| Motivation | Design | Results | Status | Future
Research Team: Haoyuan Li, Ali Ghodsi, Matei
Zaharia, Eric Baldeschwieler , Scott Shenker, Ion
Stoica
Code Contributors: Haoyuan Li, Calvin Jia, Bill
Zhao, Mark Hamstra, Rong Gu, Hobin Yoon,
Vamsi Chitters, Reynold Xin, Srinivas Parayya,
Dilip Joseph
![Page 36: Tachyon: Reliable File Sharing at Memory- Speed Across ...ampcamp.berkeley.edu/wp-content/uploads/2013/07/Tachyon_2013-0… · Tachyon: Reliable File Sharing at Memory-Speed Across](https://reader034.fdocuments.us/reader034/viewer/2022050917/5a9e53527f8b9a2e688d90d4/html5/thumbnails/36.jpg)
Questions?
http://tachyon-project.org
https://github.com/amplab/tachyon