PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with...
-
Upload
virgil-powers -
Category
Documents
-
view
226 -
download
0
Transcript of PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with...
![Page 1: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/1.jpg)
PrincetonUniversity
From application requests to Virtual IOPs:Provisioned key-value storage with Libra
David Shue* and Michael J. Freedman
(*now at Google)
![Page 2: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/2.jpg)
Shared Cloud
Tenant A Tenant BVM VM VM VM VM VM
Tenant CVM VM VM
![Page 3: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/3.jpg)
Shared Cloud Storage
Tenant A Tenant BVM VM VM VM VM VM
Key-Value Storage
Block Storage
SQL Database
Tenant CVM VM VM
![Page 4: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/4.jpg)
Unpredictable Shared Cloud Storage
Tenant A Tenant BVM VM VM VM VM VM
Key-Value Storage
Block Storage
SQL Database
Tenant CVM VM VM
Disk IO-bound TenantsSSD-backed storage
Key-Value Storage
![Page 5: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/5.jpg)
Provisioned Shared Key-Value Storage
Tenant AVM VM VM
Tenant BVM VM VM
Tenant CVM VM VM
Application Requests
Shared Key-Value Storage
Low-level IO
SSDSSD SSD SSD SSD
GET/sPUT/s
IOPsBW
ReservationA ReservationB ReservationC
(1KB normalized)
![Page 6: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/6.jpg)
6
Libra Contributions
•Libra IO Scheduler- Provisions low-level IO allocations for app-
request reservations w/ high utilization.
- Supports arbitrary object distributions and workloads.
•2 key mechanisms- Track per-tenant app-request resource profiles.
- Model IO resources with Virtual IOPs.
![Page 7: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/7.jpg)
7
Related Work
Storage Type
App-reques
ts
WorkConservin
g
Media
Maestro Block N N HDD
mClock Block N Y HDD
FlashFQ Block N Y SSD
DynamoDB
Key-Value
Y N SSD
![Page 8: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/8.jpg)
Provisioned Distributed Key-Value Storage
Tenant A Tenant BVM VM VM VM VM VM
ReservationA ReservationB Tenant B
VM VM VM
Storage Node N
Storage Node 1 ...Shared Key-Value Storage
Global Reservation Problem
Local Reservation Problem
[Pisces OSD1 ’12]
![Page 9: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/9.jpg)
9
Storage Node N
Provisioned Distributed Key-Value Storage
ReservationA ReservationB
Storage Node 1
data partitionsdem
an
d
...
Tenant AVM VM VM
Tenant BVM VM VM
![Page 10: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/10.jpg)
10
Storage Node N
Storage Node 1
Provisioned Distributed Key-Value Storage
ReservationA ReservationB
...
ReservationA ReservationB
![Page 11: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/11.jpg)
11
Storage Node N
Storage Node 1
Provisioned Distributed Key-Value Storage
...
ResAn ResBnResA1 ResB1 ReservationA ReservationB
ReservationA ReservationB
![Page 12: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/12.jpg)
12
Key-Value Protocol
Persistence Engine
IO Scheduler
Physical Disk
Retrieve K
Read l337
IO operation
Provisioned Local Key-Value Storage
GET K
GET 1001100
Libra IO Scheduler
![Page 13: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/13.jpg)
13
blah
blah
Libra Design
LibraProvisionin
gPolicy
Libra IO Scheduler
GET
PU
T
How much IO to consume?
Reservation
Distribution Policy
How much IO to provision?
Persistence Engine
Physical Disk
DRR
![Page 14: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/14.jpg)
14
Provisioning App-request Reservations is Hard
IO Amplification
IO Interference
Non-linear IO Performance
1 KB PUT ≥ 1 KB Write
Variable IO throughput
Underestimate provisionable IO
Non-linear cost per KB
Model IO with Virtual IOPs
Track tenant app-request resource profiles
![Page 15: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/15.jpg)
15
Workload-dependent IO AmplificationPU
T PUT K,V3
V3V2
FLU
SH
V3 indexA M
LevelDB (LSM-Tree)
CO
MPA
CT
V1 indexH V
V0 indexF O
V3 index
A O
![Page 16: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/16.jpg)
16
Workload-dependent IO AmplificationPU
TFL
US
HC
OM
PA
CT
PUT K,V3
V3V2
V3 indexA M
V1 indexH V
V0 indexF O
V3 index
A O
LevelDB (LSM-Tree)
![Page 17: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/17.jpg)
17
Workload-dependent IO AmplificationPU
TFL
US
HC
OM
PA
CT
PUT K,V3
V3V2
V3 indexA M
V1 indexH V
V0 indexF O
V3 index
A O
LevelDB (LSM-Tree)
![Page 18: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/18.jpg)
18
Workload-dependent IO Amplification
GET K
indexA M
indexH V
indexF O
K indexG Z
![Page 19: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/19.jpg)
19
Libra Tracks App-request IO Consumption to Determine IO
Allocations
5 GET
25 PUT
FLUSH
COMPACT
Per-PUT
Per-GET
Compute app-request IO
profiles
GET
x
x PU
T IO=
Tenant A 5 IO units
PUT
FLUSH
Track IOconsumption
Provision IOallocations
blah
blah
LibraProvisionin
gPolicy
IOTenant A 500 IO/s
5
100
50
1
+6
10.5
80
70 500
![Page 20: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/20.jpg)
20
Unpredictable IO Interference
Die-level parallelism, low latency IOPs
4 read/4 write tenants
Shared-controller and bus contention
Erase-before-write overhead
FTL and read-modify-write garbage colleciton
![Page 21: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/21.jpg)
21
Unpredictable IO Interference
21
![Page 22: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/22.jpg)
22
Libra Underestimates IO Capacity to Ensure Provisionable Throughput
Provisionable IO throughput = floor(workloads)
(18 Kop/s)Pro
vis
ionable
IO
lim
it
![Page 23: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/23.jpg)
23
Libra Underestimates IO Capacity to Ensure Provisionable Throughput
Provisionable IO throughput = floor(workloads)
(18 Kop/s)Pro
vis
ionable
IO
lim
it
![Page 24: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/24.jpg)
24
Libra Underestimates IO Capacity to Ensure Provisionable Throughput
Provisionable IO throughput = floor(workloads)
(18 Kop/s)Pro
vis
ionable
IO
lim
it
![Page 25: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/25.jpg)
25
Non-linear IO Performance
Max BW Max IOP/s
![Page 26: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/26.jpg)
26
Non-linear IO Performance
Max BW Max IOP/s
![Page 27: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/27.jpg)
27
Libra Uses Virtual IOPs to Model IO Resources
Unifies IO cost into a single metric
Captures non-linear IO performance
Provides IO insulation
2 equal-allocation tenantsIO Insulation = 1/2 Max
Read/Write
![Page 28: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/28.jpg)
28
Libra Uses Virtual IOPs to Model IO Resources
2 equal-allocation tenantsIO Insulation = 1/2 Max
Read/Write
![Page 29: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/29.jpg)
29
blah
blah
Libra Design
LibraProvisionin
gPolicy
Libra IO Scheduler
Persistence Engine
Physical Disk
Charge tenant IOPs based on
VOP cost
Track app-request VOP consumption
Provision VOPs within
provisionable limit
Update tenant VOP
allocations
![Page 30: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/30.jpg)
30
Evaluation
• Does Libra's IO resource model achieve accurate resource allocations?
• Does Libra's IO threshold make an acceptable tradeoff of performance for predictability in a real storage stack?
• Can Libra ensure per-tenant app-request reservations while achieving high utilization?
![Page 31: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/31.jpg)
31
Interference-free Ideal
Libra Achieves Accurate IO Allocations
Throughput Ratio = Actual / Expected (IO Insulation)
Read 1 KB
even
![Page 32: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/32.jpg)
32
Interference-free Ideal
Libra Achieves Accurate IO Allocations
Write 1-256 KB
Throughput Ratio = Actual / Expected (IO Insulation)
![Page 33: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/33.jpg)
33
Libra Achieves Accurate IO Allocations
![Page 34: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/34.jpg)
34
Libra Achieves Accurate IO Allocations
![Page 35: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/35.jpg)
35
Libra Achieves Accurate IO AllocationsMin-Max Ratio =
Min Throughput Ratio / Max Throughput Ratio
![Page 36: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/36.jpg)
36
Libra Trades-off Nominal IO Throughput For Predictability
![Page 37: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/37.jpg)
37
Libra Trades-off Nominal IO Throughput For Predictability
< 10th percentile covered by SLA and higher-level policies
PercentileWorkloa
d10th 50th 80th All
99:1 1.6% 30.5% 40.5% 45.8%
25:75 1.4% 14.9% 25.0% 34.7%
1:99 0.7% 12.2% 19.5% 28.1%
Unprovisionable Throughput As a Percentage of Total Throughput
![Page 38: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/38.jpg)
38
Libra Achieves App-request Reservations
Read Heavy Mixed Write Heavy
1.5x0.5x
Work-conserving
consumption of unprovisioned
resources
Fully provisioned allocations
![Page 39: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/39.jpg)
39
Libra Achieves App-request Reservations
Read Heavy Mixed Write Heavy
![Page 40: PrincetonUniversity From application requests to Virtual IOPs: Provisioned key-value storage with Libra David Shue * and Michael J. Freedman (*now at Google)](https://reader031.fdocuments.us/reader031/viewer/2022020320/5697c0191a28abf838cce7f4/html5/thumbnails/40.jpg)
40
•Libra IO Scheduler- Provisions IO allocations for app-request
reservations w/ high utilization.
- Supports arbitrary object distributions and workloads.
•2 key mechanisms
- Track per-tenant app-request resource profiles.
- Model IO resources with Virtual IOPs.
•Evaluation
- Achieves accurate low-level IO allocations.
- Provisions the majority of IO resources over a wide range of workloads
- Satisfies app-request reservations w/ high utilization.
Conclusion