Adding Advanced Storage Controller Functionality via Low ... › legacy › events › fast12 ›...
1
Executive Summary Executive Summary To meet new requirements, new functions are often added to existing storage systems Examples: file serving, database, in-line deduplication, in-line compression Traditional approaches • Code-level integration • Run new functions on external gateways Our approach: Run new functions on virtual machines • All the pros of the traditional approaches with none of the cons • Main concern is impact on I/O performance We demonstrate a set of mechanisms and techniques that achieve near-zero performance overhead Conclusions • It is feasible to use a virtual infrastructure to integrate new functions into a storage controller • Benefit from performance and hardware cost of deep integration • Benefit from shorter time to market, isolation, and simpler development model of the gateway approach • Future work : test with multiple VMs, ELI integration Network Micro-benchmarks Network Micro-benchmarks 21 μs 21 μs Polling 49 μs 24 μs No polling Guest Bare metal 6.6 μs 49 μs Added Latency 15.9 μs 50 μs Total Latency Optimized Initial Muli Ben-Yehuda, Michael Factor, Eran Rom, and Avishay Traeger IBM Research – Haifa Eran Borovik and Ben-Ami Yassour Adding Advanced Storage Controller Functionality via Low-Overhead Virtualization Adding Advanced Storage Controller Functionality via Low-Overhead Virtualization File Server Workload File Server Workload 0 50000 100000 150000 200000 250000 300000 350000 400000 IOPS 1 2 4 8 16 notifications guest-poll # Threads 0 50000 100000 150000 200000 250000 300000 350000 400000 IOPS 1 2 4 8 16 notifications host-poll guest-poll full-poll # Threads Workload: 4KB read cache miss We achieve bare-metal performance running a file server in a VM with 4 and 6 cores 6 cores 4 cores Network latency: ping flood 0% latency overhead Block latency: 4KB sync writes ~5% overhead for fastest SPC-1 cache hits (130 μs) Netperf req-resp & send Negligible throughput overhead Block throughput: 4KB sync I/Os Contact: Avishay Traeger [email protected] Contact: Avishay Traeger [email protected] Block Micro-benchmarks Block Micro-benchmarks Overview of Our Optimizations Overview of Our Optimizations Allocate a core on the guest for polling the network device, as well as block completions coming from the host Use the storage controller’s existing polling thread to poll for block requests coming from the guest Statically allocate CPU cores and memory Use HugePages for backing the guest’s memory to avoid page faults Boot guest kernel with idle=poll to avoid exits from halt/mwait Modify thread priorities and affinities Cache Miss Workflow Cache Miss Workflow 10GbE NIC Guest (KVM Process) Host (Controller Process) 1 Read IOPS: 350K (↑7.3x) Write IOPS: 284K (↑6.5x) Core 1: File Server File Server 3 Put block request in shared memory 4 Core 2: Guest Polling Thread Poll block completions 8 2 Poll NIC driver Core 3: Controller Storage Controller 6 Put completion in shared memory 7 Core 4: Host Polling Thread Poll file server block requests 5 (Poll other sources)
Transcript of Adding Advanced Storage Controller Functionality via Low ... › legacy › events › fast12 ›...
-
Executive SummaryExecutive Summary
To meet new requirements, new functions are often added to existing storage systemsExamples: file serving, database, in-line deduplication, in-line compression
Traditional approaches• Code-level integration• Run new functions on external
gateways
Our approach: Run new functions on virtual machines• All the pros of the traditional
approaches with none of the cons• Main concern is impact on I/O
performance
We demonstrate a set of mechanisms and techniques that achieve near-zero performance overhead
Conclusions
• It is feasible to use a virtual infrastructure to integrate new functions into a storage controller
• Benefit from performance and hardware cost of deep integration
• Benefit from shorter time to market, isolation, and simpler development model of the gateway approach
• Future work: test with multiple VMs, ELI integration
Network Micro-benchmarksNetwork Micro-benchmarks
21 μs21 μsPolling
49 μs24 μsNo polling
GuestBare metal
6.6 μs49 μsAdded Latency
15.9 μs50 μsTotal Latency
OptimizedInitial
Muli Ben-Yehuda, Michael Factor,Eran Rom, and Avishay TraegerIBM Research – Haifa
Eran Borovik and Ben-Ami Yassour
Adding Advanced Storage Controller Functionality via Low-Overhead VirtualizationAdding Advanced Storage Controller Functionality via Low-Overhead Virtualization
File Server WorkloadFile Server Workload
0
50000
100000
150000
200000
250000
300000
350000
400000
IOP
S
1 2 4 8 16 notifications
guest-poll
# Threads
0
50000
100000
150000
200000
250000
300000
350000
400000
IOPS
1 2 4 8 16notif ications
host-pollguest-poll
f ull-poll
# Threads
Workload: 4KB read cache miss
We achieve bare-metal performance running a file
server in a VM with4 and 6 cores
6 c
ore
s4
co
res
Network latency: ping flood
0% latency overhead
Block latency: 4KB sync writes
~5% overhead for fastestSPC-1 cache hits (130 μs)
Netperf req-resp & send
Negligible throughput overhead
Block throughput: 4KB sync I/Os
Contact: Avishay Traeger [email protected]: Avishay Traeger [email protected]
Block Micro-benchmarksBlock Micro-benchmarks
Overview of Our OptimizationsOverview of Our Optimizations
Allocate a core on the guest for polling the network device, as well as block completions coming from the host
Use the storage controller’s existing polling thread to poll for block requests coming from the guest
Statically allocate CPU cores and memory
Use HugePages for backing the guest’s memory to avoid page faults
Boot guest kernel with idle=poll to avoid exits from halt/mwait
Modify thread priorities and affinities
Cache Miss WorkflowCache Miss Workflow
10GbE NIC
Guest(KVM Process)
Host(Controller Process)
1
Read IOPS: 350K (↑7.3x)Write IOPS: 284K (↑6.5x)
Core 1:File Server
File Server3
Put block requestin shared memory
4
Core 2:Guest Polling Thread
Poll blockcompletions
8
2 Poll NIC driver
Core 3:Controller
Storage Controller 6
Put completionin shared memory
7
Core 4:Host Polling Thread
Poll file serverblock requests
5
(Poll other sources)
/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 300 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile () /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False
/SyntheticBoldness 1.000000 /Description > /Namespace [ (Adobe) (Common) (1.0) ] /OtherNamespaces [ > /FormElements false /GenerateStructure true /IncludeBookmarks false /IncludeHyperlinks false /IncludeInteractive false /IncludeLayers false /IncludeProfiles true /MultimediaHandling /UseObjectSettings /Namespace [ (Adobe) (CreativeSuite) (2.0) ] /PDFXOutputIntentProfileSelector /NA /PreserveEditing true /UntaggedCMYKHandling /LeaveUntagged /UntaggedRGBHandling /LeaveUntagged /UseDocumentBleed false >> ]>> setdistillerparams> setpagedevice
