Managing NetApp Cluster With NetApp OnCommand System Manager
PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories...
Transcript of PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories...
![Page 1: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/1.jpg)
Michio Honda(NECLaboratoriesEurope/NetApp*),LarsEggertandDouglasSantry (NetApp)
ACMHotNets 2016November9-10th,Atlanta,GA
PASTE:NetworkStacksMustIntegratewithNVMM
Abstractions
*workdone
![Page 2: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/2.jpg)
https://www.hpe.com/us/en/servers/persistent-memory.html
Motivation
• Non-VolatileMainMemories(NVMMs)• Persistent• Byte-addressable• Lowlatency• 10s-1000sofns
• Shiftfromblock- tobyte-granularitypersistency• OSabstractions• Direct accesstommap()-ed files
• Datastructures• Filesystemsanddatabases
Whatareimplicationsfornetworking?
![Page 3: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/3.jpg)
Case Study: Write-Ahead Logging
• Persistclient’srequestpriortoacknowledgment• Durablystoredataintoalogfiletomaskoverheadofupdatingprimarydatabase(e.g.B-tree)totheclient
• 1KBcommit
• 2030us• Networkingtakes40us
client
DRAM
Networkstack
SSD/DIsk
App
NIC(1)
(2)(5)
Storagestack
(4)
write()/fsync()ormemcpy()/msync()
(3)
read()
![Page 4: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/4.jpg)
write()/fsync()ormemcpy()/msync()
Case Study: Write-Ahead Logging
• 2000 42 us• Networkingtakes40us
• This2usisnotsmall
client
DRAM
Networkstack
Storagestack
NVMM
App
NIC(1)
(2)
(5)EmulatedusingareservedregionofDRAM
(3)
read()
(4)
• Persistclient’srequestpriortoacknowledgment• Durablystoredataintoalogfiletomaskoverheadofupdatingprimarydatabase(e.g.B-tree)totheclient
• 1KBcommit
![Page 5: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/5.jpg)
1 5 10 15 20 250
20406080
100120140
TK
rou
gK
pu
t [1
. t
ran
V/V]
1 5 10 15 20 250
50100150200250300350
/ate
ncy
[µV]
# of Concurrent ConnectLonV
1et. 2nOy1et. + read()/mVync() on 1V00 (emuO.)1et. + memcpy()/mVync() on 1V00 (emuO.)
• Parallelrequestsareserializedoneachcore
Case Study: Write-Ahead Logging
33%throughputdecrease,50%latencyincrease
![Page 6: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/6.jpg)
Data Copies Matter
• CacheMisses• Copytotmp buffer(e.g.,read())ischeap• Loggingalwayshappenstoadifferentdestination
appbufferkernelbuffer
read()
logfile(mmap()-ed)
memcpy()
Overall cachemisses LargestContributor
Networkingonly 0.0004 % net_rx_action()(84%)
Networking+NVMM(read() +memcpy()+msync())
4.4121% memcpy()(98%)
Networking+NVMM(read()+msync())
8.3451% sys_read()(99%)
Wemustavoiddatacopyforlogging!
![Page 7: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/7.jpg)
Packet Store (PASTE) Overview
• StaticpacketbuffersonanamedNVMMregion• DMAtoNVMM
• Zero-copyAPIs• Fastlogging
client
Networkstack
Storagestack
NVMM
App
NIC(1)
(2)
(3)
/mnt/nvmm/pktbufs
/mnt/pmem/appmd
(4)
metadataonly(e.g.,bufferindex)
![Page 8: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/8.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
![Page 9: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/9.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
UnreadReadorwritten
![Page 10: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/10.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
UnreadReadorwritten
![Page 11: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/11.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
UnreadReadorwritten
![Page 12: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/12.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
Flushed
DMAisperformedtoL3cache(DDIO)
UnreadReadorwritten
![Page 13: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/13.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
DMAisperformedtoL3cache(DDIO)
Flushed
UnreadReadorwritten
![Page 14: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/14.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
DMAisperformedtoL3cache(DDIO)
Flushed
UnreadReadorwritten
![Page 15: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/15.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
DMAisperformedtoL3cache(DDIO)
Flushed
UnreadReadorwritten
![Page 16: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/16.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
DMAisperformedtoL3cache(DDIO)
Flushed
UnreadReadorwritten
![Page 17: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/17.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
Idempotentrequest
DMAisperformedtoL3cache(DDIO)Unnecessarydataisnotflushed toDIMM
Flushed
UnreadReadorwritten
![Page 18: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/18.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
DMAisperformedtoL3cache(DDIO)Unnecessarydataisnotflushed toDIMM
Flushed
UnreadReadorwritten
![Page 19: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/19.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
DMAisperformedtoL3cache(DDIO)Unnecessarydataisnotflushed toDIMM
Flushed
UnreadReadorwritten
![Page 20: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/20.jpg)
Fast Logging with PASTE
/mnt/nvmm/myapp_metadata
buf_idx off len0 100 11351 100 9323 100 1024
/mnt/nvmm/pktbufsbuf_ofs: 123
/mnt/nvmm/pktbufs
packetbuffers(static)
metadataheader
metadataentries
NICring
Application
(2)Writemetadataentry(3)Flush(bufferandmetadata)
netmap APImmap()
(1)Readdata(zerocopy)
Kernel
User
TCP/IPinputand
output
DMAisperformedtoL3cache(DDIO)Unnecessarydataisnotflushed toDIMM
Flushed
UnreadReadorwritten
![Page 21: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/21.jpg)
10-88%throughputincrease,9-46%latencyreduction
Preliminary Results
• Implementation• Extendthenetmap framework
• Stackmap forTCP/IP
![Page 22: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/22.jpg)
Related Work
• Enhancednetworkstacks• MegaPipe (OSDI’12),Stackmap (ATC’16),Fastsocket(ASPLOS’16)• IXandArrakis (OSDI’14),mTCP (NSDI’13),Sandstorm(SIGCOMM’14),MICA (NSDI’14)
• NVMMfilesystems• BPFS(SOSP’09),NOVA(FAST’15)
• NVMMdatabases• NVWAL(ASPLOS’15),REWIND(VLDB’15),NV-Tree (FAST’15)
NoNVMMaware
Nonetworkingaware
![Page 23: PASTE - Keio Universitymicchie/talks/paste-hotnets.pdf · MichioHonda (NEC Laboratories Europe/NetApp*), Lars Eggert and Douglas Santry(NetApp) ACM HotNets2016 November 9-10th, Atlanta,](https://reader034.fdocuments.us/reader034/viewer/2022042317/5f067f267e708231d4184853/html5/thumbnails/23.jpg)
Conclusion
• Implications• Networkstacksarenowabottleneckfordurablystoringdata• Improvingnetworkandstoragestacksinisolationisnotenough• Weneednewstacksdesign
PASTE:FastloggingwithnamedpacketbuffersonNVMMandzero-copyAPI