Changpeng Liu, Cloud Software Engineer Piotr Pelpliński ... · vhosttarget (kernel or userspace)...
Transcript of Changpeng Liu, Cloud Software Engineer Piotr Pelpliński ... · vhosttarget (kernel or userspace)...
Changpeng Liu, Cloud Software Engineer
Piotr Pelpliński, Cloud Software Engineer
• Introduction to VirtIO and Vhost
• SPDK Vhost Architecture
• Use cases for vhost
• Benchmarks
• Next steps
QEMU VIRTIO
Vhost (KERNEL)
vhost (USERSPACE)
Virtio• Paravirtualized driver specification
• Common mechanisms and layouts for device discovery, I/O queues, etc.
• virtio device types include:
• virtio-net
• virtio-blk
• virtio-scsi
• virtio-gpu
• virtio-rng
• virtio-cryptoHypervisor (i.e. QEMU/KVM)
Guest VM(Linux*, Windows*, FreeBSD*, etc.)
virtio front-end drivers
virtio back-end drivers
device emulation
virtqueuevirtqueuevirtqueue
5
QEMU
Kernel
Guest VM
Guest kernel
Application
virtqueue
I/O Processing
AIO
QEMU VirtIO SCSI
6
1. Add IO to virtqueue
2. IO processed by QEMU
3. IO issued to kernel
4. Kernel pins memory
5. Device executes IO
6. Guest completion interrupt
QEMU VIRTIO
Vhost (KERNEL)
vhost target (kernel or userspace)
Vhost• Separate process for I/O processing
• vhost protocol for communicating guest VM parameters
• memory
• number of virtqueues
• virtqueue locations
Hypervisor (i.e. QEMU/KVM)
Guest VM(Linux*, Windows*, FreeBSD*, etc.)
virtio front-end drivers
device emulation
virtio back-end drivers
virtqueuevirtqueuevirtqueue
vhostvhost
QEMU
Kernel
Guest VM
Guest kernel
Application
virtqueue
vhost-kernel AIO
Kernel VHOST
9
1. Add IO to virtqueue
2. Write virtio doorbell
3. Wake vhost kernel
4. Kernel pins memory
5. Device executes IO
6. Guest completion interrupt
kvm
QEMU VIRTIO
Vhost (KERNEL)
vhost (USERSPACE)
11
Host Memory
QEMU
Guest VM
virtio-scsi
Shared Guest VMMemory
SPDK vhost
vhost DPDK vhost
virtio-scsi
virtqueuevirtqueuevirtqueue
eventfd
UNIX domain socket
SPDK VHOST Architecture
QEMU
Kernel
Guest VM
Guest kernel
Application
virtqueue
SPDK Vhost
vhost i/o
SPDK VHOST
12
1. Add IO to virtqueue
2. Poll virtqueue
3. Device executes IO
4. Guest completion interrupt
kvm
SCSI
Block Device Layer (bdev)
scsi/bdev Translation
NVMebdev driver
mallocbdev driver
Ceph RBDbdevdriver
Linux AIO bdev driver
vhostDPDK
rte_vhost
JS
ON
RP
C
QEMU
SPDK VHOST Layers
13
COMPARISON with existing solutions
Host Kernel
QEMU
Guest VM
Guest Kernel
VIRTIO_SCSI
VIRTIO_SCSI_PCI
NVME_MOD
Host Kernel
QEMU
VHOST_SCSI_PCI
NVME_MOD
QEMU
VHOST_USER_SCSI_PCI
LIO
VHOST
SPDK VHOST
PMD_NVME
SCSI
VHOST_USER
Guest VM
Guest Kernel
VIRTIO_SCSI
Guest VM
Guest Kernel
VIRTIO_SCSI
IOCTL SOCKET
14
QEMU VIRTIO SCSI Target VHOST Kernel Target VHOST Userspace Target
SPDK
SCSI
BDAL
SPDK Vhost
BlobstoreBlob Bdev
NVMe Driver
BDAL
NVMeBdev
VM
Intel® SSD for Datacenter
VM EPHEMERAL STORAGE• Increased efficiency yields
greater VM density
16
Intel® SSD for Datacenter
SPDK
SCSI
SPDK Vhost
NVMe-oFInitiator
BDAL
NVMe-oFBD
VM
NVMe-oFTarget
VM Remote Storage• Enable disaggregation and
migration of VMs using remote storage
17
Ceph Cluster
SPDK
Intel® SSD for Datacenter
Ceph RBD Driver
BDAL
CephBdev
SCSI
SPDK Vhost
VM
VM CEph Storage• Potential for innovation in
data services
• Cache
• Deduplication
18
0
2
4
6
8
10
12
14
16
QEMU VirtIO Vhost Kernel SPDK Vhost
Co
res
VM IO Processing
System configuration: 44x Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz (HT off); Cores per socket: 22; 8x Samsung 8GB DDR4 @2400 12x Intel SSD DC P3700 Series 1,5T @ FW 8DV101H0 DPDK: 17.02; Host Dist/Kernel: Fedora 25/Kernel 4.8.15-300; Guest Dist/Kernel: Ubuntu 16.04/Kernel 4.4.0-59-generic, mq enabled; Fio ver: fio-2.2.10; Fio workload: blocksize=4k, iodepth=512, iodepth_batch=128, iodepth_low=256, ioengine=libaio, size=10G, ramp_time=10, group_reporting, thread, numjobs=1, direct=1, rw=randread
Benchmarks
0
200000
400000
600000
800000
1000000
1200000
QEMU VirtIO Vhost Kernel SPDK Vhost
IO p
er
seco
nd
IO per second
20
• VFIO Support
• Support for vhost-blk protocol
• Live migration
• Performance tuning, including:
• multiqueue
• completion event coalescing
• Integration software
Next Steps
22
Unmodified guest VMs
Add data services and add value
Big improvement in VM I/O efficiency
Notices and DisclaimersIntel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com, or from the OEM or retailer.
No computer system can be absolutely secure.
Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit http://www.intel.com/performance.
Intel, the Intel logo, Xeon, and others are trademarks of Intel Corporation in the U.S. and/or other countries.
*Other names and brands may be claimed as the property of others.
© 2017 Intel Corporation.