2

VM Performance: I/O

Arne Wiebalck

LHCb Computing Workshop, CERN

May 22, 2015

Arne Wiebalck -- VM Performance: I/O

3

In this talk: I/O only

• Most issues were I/O related- Symptom: High IOwait

• You can optimize this - Understand service offering- Tune your VM

• CPU being looked at- Host-side, too early

Arne Wiebalck -- VM Performance: I/O

4

Hypervisor limits

Disk

hypervisor

Arne Wiebalck -- VM Performance: I/O

• Two-disk RAID-1- Effectively one disk

• Disk is a shared resource- IOPS / #VMs

• Your VM isco-hosted with7 other VMs

User expectation: ephemeral disk ≈ local disk

RAM

RAM

CPUs

pro rata

IOPS

4 cores 8 GB

25 IOPS

virtual machine

5Arne Wiebalck -- VM Performance: I/O

Ask not only what theVM can do for you …

6

Tip 1: Minimize disk usage

• Use tmpfs

• Reduce logging

• Configure AFS memory caches- Supported in Puppet

• Use volumes

• …

Arne Wiebalck -- VM Performance: I/O

7

Tip 2: Check IO Scheduling• The “lxplus problem”

• VMs use ‘deadline’ elevator- Set by ‘virtual-guest’ tuned profile, RH’s default for VMs- Not always ideal (interactive machines):

‘deadline’ prefers reads, can delay writes (default: 5 secs)

- Made to allow reads under heavy load (webserver)- lxplus: sssd makes DB updates during login

• IO scheduler on the VM changed to CFQ- Completely Fair Queuing

• Benchmark: login loop

Arne Wiebalck -- VM Performance: I/O

8

‘deadline’ vs. ‘CFQ’

IO benchmarkstarts

Switch from ‘deadline’to ‘CFQ’

IO benchmarks continues

Arne Wiebalck -- VM Performance: I/O

9

Tip 3: Use volumes

• Volumes are networked virtual disks- Show up as block devices - Attached to one VM at a time- Arbitrary size (within your quota)

• Provided by Ceph (and NetApp)

• QoS for IOPS and bandwidth- Allows to offer different types

Arne Wiebalck -- VM Performance: I/O

10

Volumes types

Arne Wiebalck -- VM Performance: I/O

Name Bandwidth IOPS Comment

standard 80MB/s 100 std disk performance

io1 120MB/s 500 quota on request

cp1 80MB/s 100 critical power

cp2 80MB/s 100critical powerWindows only

(in preparation)

11

Ceph volumes at work

ATLAS TDAQ monitoring application

Y- Axis: CPU % spent in IOwait

Blue: CVI VM (h/w RAID-10 with cache)

Yellow: OpenStack VM

IOPS QoS changedfrom 100 to 500 IOPS

EGI Message Broker monitoring

Y- Axis: Scaled CPU load(5 mins of load / #cores)

IOPS QoS changedfrom 100 to 500 IOPS

Arne Wiebalck -- VM Performance: I/O

12

Tip 4: Use SSD block level caching

• SSDs as disks in hypervisors would solve all IOPS and latency issues

• But still (too expensive and) too small

• Compromise: SSD block level caching- flashcache (from Facebook, used at CERN for AFS before)- dm-cache (in-kernel since 3.9, rec. by RedHat, in CentOS7)- bcache (in kernel since 3.10)

Arne Wiebalck -- VM Performance: I/O

13

bcache

Disk100

IOPS

hypervisor

RAM

RAM

CPUs

SSD20k

IOPS

• Change cache mode at runtime- Think SSD replacements

• Strong error-handling- Flush and bypass on error

• Easy setup- Transparent for VM- Need special kernel for HV

Arne Wiebalck -- VM Performance: I/O

14

bcache in action (2)

Switch cache modefrom ‘none’

to ‘writeback’

On a 4 VM hypervisor:

~25 IOPS/VM ~1000 IOPS/VM

Benchmarking a caching system is non-trivial:

- SSD performance can vary over time

- SSD performance can vary between runs

- Data distribution important (c.f. Zipf)

Benchmark ended,#threads decreased

Arne Wiebalck -- VM Performance: I/O

15

bcache and Latency

SSD block level caching sufficientfor IOPS and latency demands.

Blue: CVI VM (h/w RAID-10 w/ cache)

Yellow: OpenStack VM

Red: OpenStack on bcache HV

Use a VM on a bcache hypervisor

Arne Wiebalck -- VM Performance: I/O

Caveat: SSD failures are fatal!

Clients: lxplus, ATLAS build service, CMS Frontier, root, … (16 tenants)

16

“Tip” 5: KVM caching

• I/O from the VM goes directly to the disk- Required for live migration- Not optimal for performance

• I/O can be cached onthe hypervisor- Operationally difficult- No live migration- Done for batch

Arne Wiebalck -- VM Performance: I/O

Disk

hypervisor

RAM

RAM

CPUs

4 cores 8 GB

virtual machine

17

KVM caching in action

Arne Wiebalck -- VM Performance: I/O

ATLAS SAM VM(‘none’ to ‘write-back’)

Batch nodes with and without KVM caching

18

Take home messages

• The Cloud service offers various options to improve the I/O performance of your VMs

• You need to analyze and pick the right one for your use case- Reduce I/O- Check I/O scheduler- Use volumes- Use SSD hypervisors- (Use KVM caching)

• Get in touch with the Cloud team in case you need assistance!

Arne Wiebalck -- VM Performance: I/O