Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand...
-
Upload
isabella-speller -
Category
Documents
-
view
215 -
download
1
Transcript of Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand...
![Page 1: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/1.jpg)
1
Differentiated I/O services in virtualized environments
Tyler Harter, Salini SK & Anand Krishnamurthy
![Page 2: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/2.jpg)
2
Overview• Provide differentiated I/O services for applications in
guest operating systems in virtual machines
• Applications in virtual machines tag I/O requests
• Hypervisor’s I/O scheduler uses these tags to provide quality of I/O service
![Page 3: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/3.jpg)
3
Motivation• Variegated applications with different I/O requirements
hosted in clouds
• Not optimal if I/O scheduling is agnostic of the semantics of the request
![Page 4: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/4.jpg)
4
Motivation
Hypervisor
VM 1 VM 2 VM 3
![Page 5: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/5.jpg)
5
Motivation
Hypervisor
VM 2
VM 3
![Page 6: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/6.jpg)
6
Motivation• We want to have high and low priority processes that
correctly get differentiated service within a VM and between VMs
Can my webserver/DHT log pusher’s IO be served differently
from my webserver/DHT’s IO?
![Page 7: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/7.jpg)
7
Existing work & Problems
• Vmware’s ESX server offers Storage I/O Control (SIOC)
• Provides I/O prioritization of virtual machines that access a shared storage pool
But it supports prioritization only at host granularity!
![Page 8: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/8.jpg)
8
Existing work & Problems
• Xen credit scheduler also works at domain level
• Linux’s CFQ I/O scheduler supports I/O prioritization
– Possible to use priorities at both guest and hypervisor’s I/O scheduler
![Page 9: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/9.jpg)
9
Original Architecture
QEMU VirtualSCSI Disk
Syscalls
I/O Scheduler(e.g., CFQ)
Syscalls
I/O Scheduler(e.g., CFQ)
GuestVMs
Host
HighLow HighLow
![Page 10: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/10.jpg)
10
Original Architecture
![Page 11: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/11.jpg)
11
Problem 1: low and high may get same service
![Page 12: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/12.jpg)
12
Problem 2: does not utilize host caches
![Page 13: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/13.jpg)
13
Existing work & Problems
• Xen credit scheduler also works at domain level
• Linux’s CFQ I/O scheduler supports I/O prioritization
– Possible to use priorities at both guest and hypervisor’s I/O scheduler
• Current state of the art doesn’t provide differentiated services at guest application level granularity
![Page 14: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/14.jpg)
14
Solution
Tag I/O and prioritize in the hypervisor
![Page 15: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/15.jpg)
15
Outline
• KVM/Qemu, a brief intro…• KVM/Qemu I/O stack• Multi-level I/O tagging• I/O scheduling algorithms• Evaluation• Summary
![Page 16: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/16.jpg)
16
KVM/Qemu, a brief intro..
Hardware
Linux Standard Kernel with KVM - Hypervisor
KVM module part of Linux kernel since
version 2.6
Linux has all the mechanisms a VMM
needs to operate several VMs.
Has 3 modes:- kernel, user, guest
kernel-mode: switch into guest-mode and handle exits due to I/O operations
user-mode: I/O when guest needs to access devices
guest-mode: execute guest code, which is the guest OS except I/O
Relies on avirtualization capable CPU with either Intel VT or AMD SVM extensions
![Page 17: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/17.jpg)
17
KVM/Qemu, a brief intro..
Hardware
Linux Standard Kernel with KVM - Hypervisor
KVM module part of Linux kernel since
version 2.6
Linux has all the mechanisms a VMM
needs to operate several VMs.
Has 3 modes:- kernel, user, guest
kernel-mode: switch into guest-mode and handle exits due to I/O operations
user-mode: I/O when guest needs to access devices
guest-mode: execute guest code, which is the guest OS except I/O
Relies on avirtualization capable CPU with either Intel VT or AMD SVM extensions
![Page 18: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/18.jpg)
18
KVM/Qemu, a brief intro..
Hardware
Linux Standard Kernel with KVM - Hypervisor
Each Virtual Machine is an user space process
![Page 19: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/19.jpg)
19
KVM/Qemu, a brief intro..
Hardware
Linux Standard Kernel with KVM - Hypervisor
libvirt
Other user
space ps
![Page 20: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/20.jpg)
KVM/Qemu I/O stack
Application in guest OS
Application in guest OS
System calls layer
read, write, stat ,…
VFS
FileSystem BufferCache
Block
SCSI ATA
Issues an I/O-related system call (eg: read(), write(), stat()) within a user-space context of the virtual machine.
This system call will lead to submitting an I/O request from within the kernel-space of theVM
The I/O request will reach a device driver - either an ATA-compliant (IDE) or SCSI
![Page 21: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/21.jpg)
KVM/Qemu I/O stack
Application in guest OS
Application in guest OS
System calls layer
read, write, stat ,…
VFS
FileSystem BufferCache
Block
SCSI ATA
The device driver will issue privileged instructions to read/write to the memory regions exported over PCI bythe corresponding device
![Page 22: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/22.jpg)
KVM/Qemu I/O stack
Hardware
Linux Standard Kernel with KVM - Hypervisor
These instructions will trigger VM-exits, that will be handled by the coreKVM module within the Host's kernel-space context
Qemu emulatorThe privileged I/O relatedinstructions are passed by the hypervisor to the QEMU machine emulator
A VM-exit will take place for each of the privilegedinstructions resulting from the original I/O request in the VM
![Page 23: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/23.jpg)
KVM/Qemu I/O stack
Hardware
Linux Standard Kernel with KVM - Hypervisor
Qemu emulator
These instructions will then beemulated by device-controller emulation modules within QEMU (either as ATA or as SCSI commands)
QEMU will generate block-access I/O requests, in a special blockdeviceemulation module
Thus the original I/O request will generate I/O requests to the kernel-space of the Host
Upon completion of the system calls, qemu will "inject" an interrupt into the VM that originally issued the I/O request
![Page 24: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/24.jpg)
Multi-level I/O tagging modifications
![Page 25: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/25.jpg)
Modification 1: pass priorities via syscalls
![Page 26: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/26.jpg)
Modification 2: NOOP+ at guest I/O scheduler
![Page 27: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/27.jpg)
Modification 3: extend SCSI protocol with prio
![Page 28: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/28.jpg)
Modification 2: NOOP+ at guest I/O scheduler
![Page 29: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/29.jpg)
Modification 4: share-based prio sched in host
![Page 30: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/30.jpg)
Modification 5: use new calls in benchmarks
![Page 31: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/31.jpg)
31
Scheduler algorithm-Stride - ID of application = Shares assigned to V – Virtual IO counter for = Global_shares/
Dispatch request(){
Select the ID which has lowest Virtual IO counterIncrease by if ( reaches threshold)
Reinitialize all to 0 Dispatch request in the queue
}
![Page 32: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/32.jpg)
32
Scheduler algorithm cntd
• Problem: Sleeping process can monopolize the resource once it wakes up after a long time
• Solution: – If a sleeping process wakes up, then set
= max( min(all which are non zero), )
![Page 33: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/33.jpg)
33
Evaluation• Tested on HDD and SSD• Configuration:
Guest RAM size 1GBHost RAM size 8GBHard disk RPM 7200SSD 35000 IOPS Rd, 85000 IOPS
WrGuest OS Ubuntu Server 12.10 LK 3.2Host OS Kubuntu 12.04 LK 3.2Filesystem(Host/Guest) Ext4Virtual disk image format qcow2
![Page 34: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/34.jpg)
34
Results • Metrics:– Throughput– Latency
• Benchmarks:– Filebench– Sysbench– Voldemort(Distributed Key Value Store)
![Page 35: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/35.jpg)
35
Shares vs Throughput for different workloads : HDD
![Page 36: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/36.jpg)
36
Shares vs Latency for different workloads : HDD
• Priorities are better respected if most of the read request hits the disk
![Page 37: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/37.jpg)
37
Effective Throughput for various dispatch numbers : HDD
• Priorities are respected only when dispatch numbers of the disk is lower than the number of read requests generated by the system at a time
• Downside: Dispatch number of the disk is directly proportional to the effective throughput
![Page 38: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/38.jpg)
38
Shares vs Throughput for different workloads : SSD
![Page 39: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/39.jpg)
39
Shares vs Latency for different workloads : SSD
• Priorities in SSDs are respected only under heavy load, since SSDs are faster
![Page 40: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/40.jpg)
40
Comparison b/w different schedulers
• Only Noop+LKMS respects priority! (Has to be, since we did it)
![Page 41: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/41.jpg)
41
Results
Hard drive/SSD
Webserver Mailserver Random Reads
Sequential Reads
Voldemort DHT Reads
Hard disk
Flash
![Page 42: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/42.jpg)
42
Summary• It works!!! • Preferential services are possible only when dispatch
numbers of the disk is lower than the number of read requests generated by the system at a time
• But lower dispatch number reduces the effective throughput of the storage
• In SSD, preferential service is only possible under heavy load• Scheduling at the lowermost layer yields better
differentiated services
![Page 43: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/43.jpg)
43
Future work• Get it working for writes• Get evaluations on VMware ESX SIOC and compare with
our results
![Page 44: Differentiated I/O services in virtualized environments Tyler Harter, Salini SK & Anand Krishnamurthy 1.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649c845503460f9493ac51/html5/thumbnails/44.jpg)
44