Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow...
Transcript of Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow...
![Page 1: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/1.jpg)
AMBRY: LinkedIn’s Immutable Blob Storage System
Arjun ShenoyData SRE
![Page 2: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/2.jpg)
Motivation
Filers
Metadata DBMedia-Frontend
Clients
NFS Mounted
Media Server
● Not Cost effective
● High MTTR
● Scalability issue
● Operational Overhead
Issues Faced
![Page 3: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/3.jpg)
Reliability Scalability Low MTTR
Motivation
Low Cost
$Active - Active
![Page 4: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/4.jpg)
Evaluation
![Page 5: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/5.jpg)
#blobstore
Inspiration
![Page 6: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/6.jpg)
AMBRY
![Page 7: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/7.jpg)
1015
Handle Based Horizontally Scalable Petabyte Scalable
Metadata Support Large Blob SupportActive-Active
![Page 8: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/8.jpg)
ARCHITECTURE
![Page 9: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/9.jpg)
CDN
Clients
Ambry-Frontend
Ambry-Server
ClustermapManager
Clients
CDN
Ambry-Frontend
Ambry-Server
ClustermapManager
Data Center 1 Data Center 2
Cross-Colo Get
Replication
![Page 10: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/10.jpg)
COMPONENTS
![Page 11: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/11.jpg)
Ambry-Frontend
Coordination Security Non Blocking
![Page 12: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/12.jpg)
Ambry-Front End
Non Blocking
NIO Layer
Scaling Layer
Remote Service Layer
Ambry-frontend (Non-Blocking Model)
![Page 13: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/13.jpg)
Storage
Replication
JBOD
Ambry-Server
Storage
Responsible Built on
![Page 14: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/14.jpg)
Clustermap-Manager
![Page 15: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/15.jpg)
● HardwareLayout file
● Node State
● Disk Capacity and State
![Page 16: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/16.jpg)
● PartitionLayout file
● Mapping to nodes
● Partition State
![Page 17: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/17.jpg)
HOW DO THEY INTERACT?
![Page 18: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/18.jpg)
PutBlob WorkFlow
Client
Frontend
1. PutBlob
2. Choose Partition
3. Generate BlobId
DataNode DataNode DataNode
4. Send PUT Request in Parallel to all 3 replicas
5. Wait for 2 acks
6. Return BlobId
Sample Blob-Id: /AAEAAQAAAAAAAADFAAAAJDMyYWZiOTJmLTBkNDYtNDQyNS1iYzU0LWEwMWQ1Yzg3OTJkZQ.gif
![Page 19: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/19.jpg)
PutBlob Statistics
● Average QPS: 370
● Latencies(95th Percentile):● SmallBlob (<100KB ): <10ms● MediumBlob(100KB - 4MB): <10ms (Goes up to 20ms
extreme cases)
● LargeBlob (>4MB): <50ms (Goes up to 80ms extreme cases)
![Page 20: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/20.jpg)
GetBlob Workflow
Client
Frontend
1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif)
2. Determination of Partition Based on the Blob Id
DataNode DataNode DataNode
3. Send Requests to Replicas in Parallel
4. Wait for at least 1 successful response
5. Return Blob
![Page 21: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/21.jpg)
GetBlob Workflow
Client
Frontend
DataNode DataNodeDataNode DataNode
DC1 DC2
GET404 Cross-colo Get
Response
![Page 22: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/22.jpg)
GetBlob Statistics
● Average QPS: 24K
● Latencies(95th Percentile):● SmallBlob: <15ms● MediumBlob: <75ms (Goes up to 700ms extreme cases)● LargeBlob: <200ms (Goes up to 1.5s extreme cases)
![Page 23: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/23.jpg)
Replication Workflow
DataNode 1
Replication Thread
Store
DataNode 7
Replication Thread
Store
1. GetBlobSince(Offset)
2. GetBlobSince(Offset)3. BlobSinceContext
4. BlobSinceContext
5. Find BlobInfo 6. Context
![Page 24: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/24.jpg)
DataNode 1
Replication Thread
Store
DataNode 7
Replication Thread
Store
7. FetchBlobAndMeta
8. FetchBlobAndMeta
9. Stream
10. Stream
Replication Workflow
● Inter DC Replication Time: 20-100ms● Intra DC Replication Time: <10ms
![Page 25: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/25.jpg)
SOME STATISTICS
![Page 26: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/26.jpg)
Use-cases: Images, PDFs, Static files, Videos so on...
Total Capacity: 900TB Serving: 300TB
Growth Rate: 1160GB per day
![Page 27: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/27.jpg)
Git Link: github.com/linkedin/ambry/wiki
Blog Post: engineering.linkedin.com/blog/2016/05/introducing-and-open-sourcing-ambry---linkedins-new-distributed-
We’re Open Source!
![Page 28: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/28.jpg)
Thank You!!
![Page 29: Data SRE Arjun Shenoy AMBRY: LinkedIn’s Immutable Blob … · 2019. 12. 18. · GetBlob Workflow Client Frontend 1. GET(/AAAEQ...zU0wMWQ1Yzg3OTJkZQ.gif) 2. Determination of Partition](https://reader036.fdocuments.us/reader036/viewer/2022071117/600372904c04292abe03c7ab/html5/thumbnails/29.jpg)
The information in this presentation was compiled from sources believed to be reliable for informational purposes only. It does not constitute legal or professional advice. The views and opinions expressed in this presentation are those of the authors and do not necessarily represent official policy or position of Linkedin
All product and company names are trademarks™ or registered® trademarks of their respective holders.
DISCLAIMER