Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage...
Transcript of Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage...
![Page 1: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/1.jpg)
Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck
Byung S. Kim, Jaeho Kim, Sam H. Noh
UNIST(Ulsan National Institute of Science & Technology)
![Page 2: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/2.jpg)
Motivation & Observation
Our Idea• Provide full network performance
• Eliminate inconsistent performance
Evaluation• Full network bandwidth
• Consistent performance
Summary & Future work
Outline
![Page 3: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/3.jpg)
Motivation & Observation
Our Idea• Provide full network performance
• Eliminate inconsistent performance
Evaluation• Full network bandwidth
• Consistent performance
Summary & Future work
Outline
![Page 4: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/4.jpg)
SSD Garbage Collection: Degraded Performance
Lies, Damn Lies And SSD Benchmark Test Result, “http://www.seagate.com/kr/ko/tech-insights/lies-damn-lies-and-ssd-benchmark-master-ti/.”
![Page 5: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/5.jpg)
SSD Garbage Collection: Inconsistent Performance
Intel Solid-State Drive DC S3700 Series – Quality of Service, “http://www.intel.com/content/dam/www/public/us/en/documents/technology-briefs/ssd-dc-s3700-quality-service-tech-brief.pdf.”
Time
IOP
S
Sudden drop in IOPS
![Page 6: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/6.jpg)
Rise of All-Flash Array
GoogleDatacenter
Improving performancefor application request
![Page 7: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/7.jpg)
Architecture of All-Flash Array
![Page 8: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/8.jpg)
Architecture of All-Flash Array
![Page 9: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/9.jpg)
Interface Bandwidth Growth Trend
![Page 10: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/10.jpg)
Interface Bandwidth Growth Trend
Network becomes bottleneck
![Page 11: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/11.jpg)
Commercial SSD Trend
SSD product Read performance Write performance
Product A 6.8GB/s 4.8GB/s
Product B 3.5GB/s 2GB/s
Product C 2.7GB/s 1.5GB/s
10Gbps Ethernet: 1.25GB/sFibre channel: 1GB/s
![Page 12: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/12.jpg)
• Up to 34 SSDs per network port
Commercial All-Flash Array Trend
All-Flash Array products # of SSDs # of network ports
Product A 10 Up to 4
Product B 150 Up to 48
Product C 68 Up to 2
Product D 96 Up to 4
![Page 13: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/13.jpg)
• Up to 34 SSDs per network port
Commercial All-Flash Array Trend
Does these many SSDs really help?
All-Flash Array products # of SSDs # of network ports
Product A 10 Up to 4
Product B 150 Up to 48
Product C 68 Up to 2
Product D 96 Up to 4
![Page 14: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/14.jpg)
Performance of RAID 0 with 4 SSDs
RAID 0 config. by 4 NVMe SSDs (spec. read: 2400MB/s, write: 1200MB/s)(Measured read: 2000MB/s, write 1000MB/s)Ideal performance
Sequential write with128KB I/O size
![Page 15: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/15.jpg)
RAID 0: Inconsistent Performance
RAID 0 config. by 4 NVMe SSDs (spec. read: 2400MB/s, write: 1200MB/s)(Measured read: 2000MB/s, write 1000MB/s)
Inconsistent performance
Ideal performance
Sequential write with128KB I/O size
![Page 16: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/16.jpg)
Performance of RAID 0 with 4 SSDs
RAID 0 config. by 4 NVMe SSDs (spec. read: 2400MB/s, write: 1200MB/s)(Measured read: 2000MB/s, write 1000MB/s)
10GbE (1.25GB/s)
Ideal performance
Does not saturatenetwork bandwidth
Sequential write with128KB I/O size
![Page 17: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/17.jpg)
Performance is limited by network bandwidth
Limited by Network Bandwidth
Another motivation: provide full network performance
under network connection
![Page 18: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/18.jpg)
Performance is limited by network bandwidth
Limited by Network Bandwidth
Another motivation: provide full network performance
under network connection
Sustained full network performance
![Page 19: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/19.jpg)
All-Flash Array suffers from inconsistent, limited performance
We want consistent, full network performance!
Problem and Our Goal
![Page 20: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/20.jpg)
Motivation & Observation
Our Idea• Provide full network performance
• Eliminate inconsistent performance
Evaluation• Full network bandwidth
• Consistent performance
Summary & Future work
Outline
![Page 21: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/21.jpg)
Serial configuration
Our Solution
Front-end SSD Back-end SSDs
![Page 22: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/22.jpg)
Provide full network performance• Absorb all writes with Front-end SSD
• Log-structured manner
Provide consistent performance• Eliminate GC within Front-end SSD
• Propose xGC
Serial Configuration Design
![Page 23: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/23.jpg)
Absorb all writes with Front-end SSD• Network bandwidth < Front-end SSD bandwidth
Log-structured manner• Sequential, append only writes
Full Network Performance
![Page 24: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/24.jpg)
Front-end SSD will eventually fill up• Garbage Collection?
Managing Front-end SSDs• Selecting next Front-end
• Making space available
Issues in Providing High Performance
![Page 25: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/25.jpg)
Handling Garbage Collection
SSD 1
Front-End(Write ptr)
SSD 2 SSD 3
Eventually, the Front-end SSD becomes fulland garbage collection is needed
External GC (xGC)• Garbage collection never occurs at Front-end SSD
![Page 26: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/26.jpg)
Handling Garbage Collection
SSD 1 SSD 2 SSD 3
When the Front-end SSD fills up• New Front-end is selected
• Old Front-end SSD becomes a Back-end SSD
External GC (xGC) is performed between Back-end SSDs
xGC
Front-End(Write ptr)
![Page 27: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/27.jpg)
Handling Garbage Collection
SSD 1
Front-End(Write ptr)
SSD 2 SSD 3
When all valid data is moved• Old front-end is cleaned by issuing TRIM command
Trim
![Page 28: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/28.jpg)
Front-end performance is not affected by GC
Front-end always provides consistent performance
Effect of xGC
![Page 29: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/29.jpg)
Motivation & Observation
Our Idea• Provide full network performance
• Eliminate inconsistent performance
Evaluation• Full network bandwidth
• Consistent performance
Summary & Future work
Outline
![Page 30: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/30.jpg)
Evaluation Settings
Description
Storage Server Host Server
CPU Intel Xeon E5-2609 Intel i5-6600k
RAM 64GB DRAM 16GB DRAM
Ethernet 10Gbps
OS Linux kernel 4.4.43 Linux kernel 4.3.3
SSDIntel 750 400GB NVMe SSD × 4(spec. read: 2400MB/s, write: 1200MB/s)(Measured read: 2000MB/s, write 1000MB/s)
![Page 31: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/31.jpg)
Observe effect of network connection
Observe effect of serial configuration
Evaluation Settings
Storage serverHost
10GbEFTP
Application
SSD
SSD
SSD
SSD
Managing technique
I/O request
Storage server
SSD
SSD
SSD
SSD
Managing technique
GC IncurringApplication I/O request
![Page 32: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/32.jpg)
Transfer 10 files (respectively, 10GB) with 10 threadsto storage server via FTP protocol
Measurement point is the storage server
Effect of Network Bandwidth
Storage serverHost
10GbEFTPApplication
SSD
SSD
SSD
SSD
Managing technique
I/O request
![Page 33: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/33.jpg)
Effect of Network Bandwidth
4 NVMe SSDs (spec. read: 2400MB/s, write: 1200MB/s)(Measured read: 2000MB/s, write 1000MB/s)
Ideal performance of RAID 0
Typical measured performance
![Page 34: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/34.jpg)
Effect of Network Bandwidth
4 NVMe SSDs (spec. read: 2400MB/s, write: 1200MB/s)(Measured read: 2000MB/s, write 1000MB/s)
10Gb Ethernet(1.25GB/s)
![Page 35: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/35.jpg)
Effect of Network Bandwidth
4 NVMe SSDs (spec. read: 2400MB/s, write: 1200MB/s)(Measured read: 2000MB/s, write 1000MB/s)
Avg. 924MB/s (4 SSDs)
Avg. 895MB/s (Front-end, 1 SSD)
10Gb Ethernet(1.25GB/s)
![Page 36: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/36.jpg)
Performance is determined by networkindependent of performance of storage
• Performance of our approach is similar to that of RAID 0 with 4 SSDs
Conclusion of First Evaluation
![Page 37: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/37.jpg)
Performance with network effect removed• Verify performance of serial configuration
Synthetic workload generated by FIO benchmark tool• Perform I/O for 30 minutes after aging
− 1200GB footprint
− 256KB random writes
• Measure performance of random write workload with 64KB I/O size
Observe Effect of Serial Configuration
Storage server
SSD
SSD
SSD
SSD
Managing technique
GC IncurringApplication I/O request
![Page 38: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/38.jpg)
Verifying Consistent Performance
4 NVMe SSDs (spec. read: 2400MB/s, write: 1200MB/s)(Measured read: 2000MB/s, write 1000MB/s)
Aging workload
10Gb Ethernet(1.25GB/s)
Typical measured performance
![Page 39: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/39.jpg)
In Contrast to RAID 0 Configuration
Ideal performance
10GbE (1.25GB/s)
![Page 40: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/40.jpg)
Aging workload
10Gb Ethernet(1.25GB/s)
Verifying Consistent Performance
4 NVMe SSDs (spec. read: 2400MB/s, write: 1200MB/s)(Measured read: 2000MB/s, write 1000MB/s)
• Observe consistent performance• Observe no internal GC effect• Observe no computing overhead for xGC
Typical measured performance
![Page 41: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/41.jpg)
Motivation & Observation
Our Idea• Provide full network performance
• Eliminate inconsistent performance
Evaluation• Full network bandwidth
• Consistent performance
Summary & Future work
Outline
![Page 42: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/42.jpg)
All-Flash Array is faster but limited by network bandwidth
All-Flash Array suffers from inconsistent performancedue to garbage collection
Proposed technique that satisfiesboth full network performance and consistent performance
Summary
![Page 43: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/43.jpg)
Fault-tolerance
Parallelization of serial configuration• One serial configuration per network port
Scalability• More than 4 SSDs
• Heterogeneous SSDs
Metadata management
Latency performance
Effect on read performance
Future Work
![Page 44: Managing Array of SSDs When the Storage Device is No ... · Managing Array of SSDs When the Storage Device is No Longer the Performance Bottleneck Byung S. Kim, Jaeho Kim, Sam H.](https://reader035.fdocuments.us/reader035/viewer/2022062506/5f0343287e708231d408579d/html5/thumbnails/44.jpg)
Thank you