Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD...
Transcript of Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD...
![Page 1: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/1.jpg)
Key Value SSD Explained – Concept, Device, System, and Standard
09/14/2017Yang Seok Ki
Director of Memory Solutions LabSamsung Semiconductor Inc.
![Page 2: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/2.jpg)
Disclaimer
2
This presentation and/or accompanying oral statements by Samsung representatives collectively, the “Presentation”) is intended to provide information concerning the SSD and memory industry and Samsung Electronics Co., Ltd. and certain affiliates (collectively, “Samsung”). While Samsung strives to provide information that is accurate and up-to-date, this Presentation may nonetheless contain inaccuracies or omissions. As a consequence, Samsung does not in any way guarantee the accuracy or completeness of the information provided in this Presentation.
This Presentation may include forward-looking statements, including, but not limited to, statements about any matter that is not a historical fact; statements regarding Samsung’s intentions, beliefs or current expectations concerning, among other things, market prospects, technological developments, growth, strategies, and the industry in which Samsung operates; and statements regarding products or features that are still in development. By their nature, forward-looking statements involve risks and uncertainties, because they relate to events and depend on circumstances that may or may not occur in the future. Samsung cautions you that forward looking statements are not guarantees of future performance and that the actual developments of Samsung, the market, or industry in which Samsung operates may differ materially from those made or suggested by the forward-looking statements in this Presentation. In addition, even if such forward-looking statements are shown to be accurate, those developments may not be indicative of developments in future periods..
![Page 3: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/3.jpg)
Agenda• Cloud: A New Era• Scalability: A New Challenge• Key Value SSD: A New Technology
– Samsung Key Value SSD
• Ecosystem• Use Case and Performance Studies• Q&A
![Page 4: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/4.jpg)
Agenda• Background• Concept• Key Value SSD• Ecosystem• Use Case and Performance Studies• Standards• Q&A
![Page 5: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/5.jpg)
Cloud: A New Era & Challenges
![Page 6: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/6.jpg)
What happens in an internet minute?
6
1.5X
1.3X
1.3X
1.3X
![Page 7: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/7.jpg)
BC/AD in IT
6
Source: Human Computer Interaction % Knowledge Discovery
![Page 8: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/8.jpg)
Challenges in Cloud Era
Scalable TCO
Capacity
Throughput
Latency
Power
![Page 9: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/9.jpg)
Block: Parking Lot/Structure• A driver (host) is responsible for parking (data management)
9
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
![Page 10: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/10.jpg)
Object: Valet Parking• A parking facility (storage) is responsible for parking (data
management)
10
“music”
“photo”
“zip file”
“document”
![Page 11: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/11.jpg)
Key Value SSD: New Scalable Technology
![Page 12: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/12.jpg)
Everything is object!
12
OSD Object Storage
ID Attributes User Data
KV Storage
Key Value
![Page 13: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/13.jpg)
Key Value Stores are Common in Systems at Scale
![Page 14: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/14.jpg)
Key Value in Systems at Scale: Twitter Timeline Service
14
Write API
Fanout
twemproxy
Followers
Key Value Store
![Page 15: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/15.jpg)
Key Idea
Block Device Driver
Block Device
TX/s WAF, RAF, Latency
15
Host S/W
Traditional KV Store KV Stacks
Key Value Store is everywhere!
Thin KV Library
KV Device Driver
KV Device
KV
![Page 16: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/16.jpg)
Samsung KV-PM983 Prototype
16
NGSFF KV SSD
Form factor: NGSFF/U.2Capacity: 1-16TBInterface: NVMe PCIe Gen.3
![Page 17: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/17.jpg)
Physical Location / Offset
Index
User/Device Hash Key
NAND Page (32KB)
Lookup / Check hash collision
< NAND >
Read/Write User Data
Key Size Range ? Value Size Range ?
Key Value I/F Command
Key Value SSD device driver
Value Size
Meta data Key Value
Key Size
Get (key) / Put (Key, Value)
KV SSD Design Overview• Key/Value Range
– Key : 4~255B– Value : 64B~2GB (32B granularity)– The large value is stored into multiple NAND pages
Storage Server Key Value SSD
![Page 18: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/18.jpg)
Key Value SSD is a Scalable Solution with Better TCO
18
• Performance• Capability
KV SSD
• Capacity• Performance
Scale-Up• CPU• Server
Scale-In
• TCO• Power
Scale-Down• Capacity• Performance
Scale-Out
TCO($)
![Page 19: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/19.jpg)
KV SSD Ecosystem
![Page 20: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/20.jpg)
Ecosystem in Block and KV Device Era
BlockSSD
KVSSD
![Page 21: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/21.jpg)
KV SSD Ecosystem
Key Value SSD
Standard
Product
ApplicationsSDK
Partners
21
![Page 22: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/22.jpg)
KV SSD Ecosystem
Key Value SSD
Standard
Product
ApplicationsSDK
Partners
22
![Page 23: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/23.jpg)
Applications for KV SSD
23
Distributed DB Object Storage SystemNoSQL DB
KV Device
KV Stacks
OSD
API
KV Adapter
KV Device
KV Stacks
API
KV Adapter
KV Device
KV Stacks
API
Storage EngineStorage Engine
Swift API
![Page 24: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/24.jpg)
KV SSD Ecosystem
Key Value SSD
Standard
Product
ApplicationsSDK
Partners
24
![Page 25: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/25.jpg)
Key Value SW Stacks• SSD with native key value interface through hardware software co-design
Datacenter S/W Infra
Storage Plugin Interface
Block Interface
POSIX API
Key Value API
S/W Key Value Store
Key Value Glue Logic
File System
Block Device Driver
Index Log
JournalBlock Map
Command Protocol
Block DeviceMap Log
Datacenter S/W Infra
Storage Plugin Interface
KV Interface
Key Value APIThin KV Library
Key Value Glue Logic
KV Device Driver
Command Protocol
KV DeviceIndex Log
TX/s WAF, RAF, Latency
Index Log
Block Map
Index Log
25
![Page 26: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/26.jpg)
Key Value Software Development Stacks
Key Value Library & Tools
Cache AIO Multi-Queue Multi-Device Memory Manager
KV Abstract Device Interface (ADI)
Linux Kernel Device Driver Linux User-space Device Driver Windows Device Driver
store/retrieve/delete/exist KV Pair namespace
Tools
![Page 27: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/27.jpg)
KV API/Lib
KV ADI
kvbench: Key Value Benchmark Suite
Performance Profiler
Workload Generator
…RocksDB ForestDBWiredTiger
Filesystem
Block Device Driver
Block Device KV Device
KV Device Driver
![Page 28: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/28.jpg)
KV devices
Application
KV Virtualization
28
KV Virtualization
Key Space Distribution Load BalancingCapacity Management
![Page 29: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/29.jpg)
Key Value Software Development Stacks
Key Value Library & Tools
Cache AIO Multi-Queue Multi-Device Memory Manager
KV Abstract Device Interface (ADI)
Linux Kernel Device Driver Linux User-space Device Driver Windows Device Driver
store/retrieve/delete/exist KV Pair namespace
Tools
![Page 30: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/30.jpg)
Key Value SSD Use Case Studies
![Page 31: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/31.jpg)
Device
Key ValueStore
Benchmark
Use Case StudyScale-UpSingle Scale-Out
KV Stacksvs
NVMeoF
KVBench
![Page 32: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/32.jpg)
Device
Key ValueStore
Benchmark
Use Case StudySingle
or
NVMeoF
KV Stacks
KVBench
Scale-Up Scale-OutSingle
![Page 33: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/33.jpg)
Single Component Performance: RocksDB vs. KV Stacks
• RocksDB– Originated by Facebook and Actively used in their infrastructure– Most popular embedded NoSQL database – Persistent Key-Value Store– Optimized for fast storage (e.g., SSD)– Uses Log Structured Merge Tree architecture
• KV Stacks on KV SSD– Benchmark tool directly operates on KV SSD through KV Stacks
33
![Page 34: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/34.jpg)
RocksDB vs. KV Stacks Performance Measurement• Better Performance
– Lean software stacks– Overhead moved to device
• IO Efficiency– Reduction of host traffic to
devices
KV SSD
KV-PM983
Block SSD
PM983
VS.
Block Driver
RocksDB
Filesystem
Client: kvbench
Key Value ADI
Key Value API
KV Driver
KV Stacks
Hardware Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz96 GB RAMPM983(Block) & KV-PM983 SSD
Software Ubuntu 16.04RocksDB v5.0.2 on XFS50M records, 16B Key, 4KB value
![Page 35: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/35.jpg)
0
1
2
3
4
5
6
7
8
9
RocksDB(PM983) KV Stacks(KV-PM983)0
2
4
6
8
10
12
14
RocksDB(PM983) KV Stacks(KV-PM983)
Performance: Random PUT• 8x more QPS (Query Per Second) with KV Stacks than RocksDB on
block SSD• 90+% less traffic goes from host to device with KV SSD than
RocksDB on block device
8x
* Workload: 100% random put, 16 byte keys of random uniform distribution, 4KB-fixed values on single PM983 and KV-PM983 in a clean state
12.7
1.0
Rela
tive
QPS
8.0
Dev
ice
IO/U
ser I
O
![Page 36: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/36.jpg)
Device
Key ValueStore
Benchmark
Use Case Study
NVMeoF
vsKV Stacks
KVBench
Scale-OutScale-UpScale-UpSingle
![Page 37: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/37.jpg)
Scale-Up Storage: RocksDB• Linear Scaling
– More devices, more throughput and capacity
• IO Efficiency– Reduction of host traffics
to devices• Less CPU utilization
– Small number of cores or less CPU utilization for performanceXeon E5 Skylake (24 Cores) Xeon E5 Skylake (24 Cores)
DRAM 768 (GB)SSD(1.92 TB)
SSD(1.92 TB)
SSD(1.92 TB)2.5” SSD(1.92 TB)2.5” SSD(1.92 TB)
NGSFF(1 TB)
SSD(1.92 TB)
SSD(1.92 TB)
SSD(1.92 TB)2.5” SSD(1.92 TB)2.5” SSD(1.92 TB)NGSFF KV
(1 TB)18 EA 18 EA
vs
RocksDB
XFS
Page Cache
RAID0
Block Driver
RocksDBRocksDBRocksDB
Client: kvbench
![Page 38: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/38.jpg)
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
1 6 12 18
RocksDB (PM983) - R KV Stacks (KV-PM983)
0
1
2
3
4
5
6
7
8
RocksDB (PM983) KV Stacks (KV-PM983)
Scale-up Performance: Random Key PUT• 15x IO performance over S/W key value store on block devices
# of SSDs
Rela
tive
QPS
Relative performance to the maximum aggregate RocksDB random Put QPS for 1 SSD with a default configuration for 1 PM983 SSD in a clean state.System: Ubuntu 16.04.2 LTS, , Ext4, RAID0 for block SSDs, Actual CPU utilization could be 70-90% at CPU saturation point.Workload: 100% puts, 16 byte keys of random uniform distribution for RocksDB v. 5.0.2, 4KB-fixed values, 24 RocksDB instances with 4 client threads, 50GB/Instance or 1.2TB Data is used
6.8-7.0
1.0
15x
Dev
ice
IO/U
ser I
O
![Page 39: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/39.jpg)
0
0.5
1
1.5
2
2.5
RocksDB (PM983) KV Stacks (KV-PM983)
Scale-up Performance: Sequential Key PUT• 3.4x IO performance over S/W key value store on block devices
39
Rela
tive
QPS
1.0
2.0
Dev
ice
IO/U
ser I
O
Relative performance to the maximum aggregate RocksDB random Put QPS for 1 SSD with a default configuration for 1 PM983 SSD in a clean state.System: Ubuntu 16.04.2 LTS, , Ext4, RAID0 for block SSDs, Actual CPU utilization could be 90% at CPU saturation point.Workload: 100% puts, 16 byte keys of random uniform distribution for RocksDB v. 5.0.2, 4KB-fixed values, 36 RocksDB instances with 1 client thread, 34GB/Instance or 1.2TB Data is used
0.00
5.00
10.00
15.00
20.00
25.00
30.00
35.00
1 6 12 18
RocksDB (PM983) - S KV Stacks (KV-PM983)
3.4x
# of SSDs
![Page 40: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/40.jpg)
Device
Key ValueStore
Benchmark
Use Case StudyScale-Out
NVMeoF
vsKV Stacks
KVBench
Scale-Up Scale-OutSingle
![Page 41: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/41.jpg)
DBBench/KVBenchDBBench/KVBench
Scale-Out: RocksDB & KV Stacks Configuration
RDMA Fabric
RocksDBvs
KV Stacks
NVMeoFover
RDMA
…
Mission PeakKV-PM983 SSDs
…Software CentOS 7.3 w/ KV Target
NICs 2x 100GbE + 2x 50GbECPU Xeon 8160 CPU @2.10GHz
1-node 2-socket, 48-core 96-thread# SSDs 36x 1TB
Software CentOS 7.3 w/ KV SWNIC 2x 100GbE
CPUXeon E5-2699V4 CPU @2.20GHz1-node 2-socket44-core 88-thread
DRAM 256 GB
Client: kvbench Client: kvbench Client: kvbench Client: kvbench
![Page 42: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/42.jpg)
Local vs NVMeoF PUT Latency
0
200
400
600
800
1000
1200
1 2 4 8 16 32 64 128
Average Latency
Local Avg
RDMA Avg
Queue Depth
Mic
rose
cond
s @Qdepth: 1-8Overhead: 4-7us
RDMASwitch
kvbench
![Page 43: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/43.jpg)
0
2000
4000
6000
8000
10000
1 - Client 2 - Client 3 - Client 4 - Client
# of KV Clients
Scaling w/ 2 KV Servers
RocksDB KV Stacks
Performance and Capacity Scale-Out: PUT Throughput
13.8X
0
2000
4000
6000
8000
10000
1 - Client 2 - Client 3 - Client 4 - Client
# of KV Clients
Scaling w/ 2 KV Servers
RocksDB KV Stacks
5.87X
Client RocksDB: CentOS 7.3, Ext4, RAID0 for block SSDs, Workload: 100% puts, 16 byte keys of random uniform distribution for RocksDB, 4KB-fixed values, 24 RocksDB instances with 8 client threads, 50GB/Instance or 1.2TB Data is used, Client KV Stacks: CentOS 7.3, KV Load Generator, 100% 4K PUTs, 16 byte keys, KV Server: Mission Peak w/ NVMeoF KV Target
KQPS
KQPS
9.4M 9.4M
1.6M680K
![Page 44: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/44.jpg)
0102030405060708090
100
CPU Utilization for Clients
0102030405060708090
100
1 38 75 112
149
186
223
260
297
334
371
408
445
482
519
556
0102030405060708090
100
1 41 81 121
161
201
241
281
321
361
401
441
481
521
561
Avg Utilization 10% Higher
Time Time Time
2.1 M QPS
![Page 45: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/45.jpg)
Key Value SSD Standards
Bill MartinPrincipal Engineer
Memory Solutions Lab
![Page 46: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/46.jpg)
Key Value SSD layers
e.g. KV SSD
KV Device
KV Protocol Provider Interface
KV Wire Protocol
SNIA KV Library
Application(s)
KV Protocol Client Interface
e.g. SNIA KV API
e.g. NVMe KV commands
e.g. C Library, Java, etc.Written by vendors, open source, etc.
KV API
![Page 47: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/47.jpg)
Key Value SSD Standard Activities• NVMe
– Work on a technical proposal is being discussed by the NVMe working group
– The group is defining the scope of the work– This will be a new device type
• SNIA– A proposal for a Key Value API has been submitted to the SNIA Object
Drive Technical Working Group– Discussion on the minimum necessary commands to meet basic Key
Value needs is progressing
![Page 48: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/48.jpg)
Key Value, not Object Drive• Both standards efforts are focused on Key Value SSD not
Object Drive– Key Value is a means to submit a Key and put or get a Value– Object Drive would include more extensive commands to query the
Key Value database
48
![Page 49: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/49.jpg)
NVMe Extension for Key Value SSD• Defines a new device type for a Key Value device• A controller performs either KV or traditional block storage
commands
PUT GET DELETE EXISTS
Admin command
Identify commands for KV
Other non-block specific commands
New Key Value Commands
Existing Command Extension
![Page 50: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/50.jpg)
SNIA Key Value API• The Key Value API (Application Programming Interface) has been presented to SNIA for consideration in the
Object Drive Technical Working Group• Defines a Tuple
– Key– Value
• Defines KV specific constants– Max Key Lebngth– Alignment Unit
• Key type supported– 4 byte fixed– 8 byte fixed– Variable length character string– Variable length binary string
• The API defines the calls that an application may make to the Key Value device interface– These calls are independent of any specific implementation– These calls support the basic commands proposed for the NVMe standard
• Open/Close• Store/Retreive• Exist• Delete• Containers/groups
![Page 51: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/51.jpg)
Call for Participation• NVMe work is proceeding in the NVMe working group
– www.nvmexpress.org• Contributors and Promoters have access to working proposals
• SNIA work is proceeding in SNIA Object Drive Technical Working group– www.snia.org
• Members may join the Object Drive TWG and have access to working proposals
51
![Page 52: Key Value SSD Explained – Concept, Device, System, and … · · 2017-09-08Key Value SSD Explained – Concept, Device, System, and Standard. ... Physical Location / Offset. Index.](https://reader031.fdocuments.us/reader031/viewer/2022012923/5afd065b7f8b9a8b4d8cfeab/html5/thumbnails/52.jpg)
Key Value SSD is a Scalable Solution with Better TCO
52
TCO reduction
CPU or server reduction
Dense performance and capacity scaling
Linear performance and capacity scaling
Lean host software stacksKV SSD
Scale-Up
Scale-In
Scale-Down
Scale-Out