Amnesic Online Synopses for Moving Objects Michalis Potamias, Kostas Patroumpas, and Timos Sellis
Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017...
Transcript of Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017...
![Page 1: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/1.jpg)
Image ©2013 CC-BY-NC-SA by Tony GrayImage ©2013 CC-BY-NC-SA by Tony Gray
Turtles all theTurtles all theWay DownWay Down
![Page 2: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/2.jpg)
Turtles all theWay Down
Misadventures with Trim and Thin
Steven EllisSenior Solution Architect – Red Hat [email protected] / @StevensHat
![Page 3: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/3.jpg)
![Page 4: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/4.jpg)
Agenda● SSDs + Trim + Thin● Turtle● Turtles● Virtual Turtles● Migrating Turtles● Nesting Turtles● Cows
● Redundant Turtles● KVM Serial Console
![Page 5: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/5.jpg)
SSD + Trim / Unmap / Discard● Critical to effective use of SSDs● Lack of Trim can impact
– Lifetime of disk● Blocks aren’t released / re-allocated correctly
– Performance
● OS + File-system need to support Trim– Delete requests are passed to SSD
![Page 6: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/6.jpg)
Turtles?
SSD
Partition
Encryption
LVM
thinpool
KVM
Virtual Disk
Partition
LVM
Filesystem
SSD
Partition
Encryption
LVM
Filesystem
SSD
Partition
LVM
Filesystem
SSD
Partition
Filesystem
![Page 7: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/7.jpg)
File-system Discard
SSD
Partition
Filesystem
![Page 8: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/8.jpg)
File-system Discard
SSD
Partition
Filesystem
● XFS / EXT4 / BRTFS / JFS– fstrim
– periodic discard● /etc/fstab options discard
● SWAP understands discard● F2FS for Android / IoT / Embedded
● /etc/fstab options discard
● Confirm working– sudo lsblk -o MOUNTPOINT,DISC-MAX,FSTYPE
![Page 9: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/9.jpg)
LVM Passthru
SSD
Partition
LVM
Filesystem
● Edit /etc/lvm/lvm.conf– issue_discards = 1
● Previous file-system tips apply
![Page 10: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/10.jpg)
dmcrypt / Luks
SSD
Partition
Encryption
LVM
Filesystem
● Edit /etc/crypttab– luks-blah-blah-blah UUID=blah-
blah-blah none luks,discard
● Recreate boot disks– dracut –force
● Reference– http://blog.christophersmart.com/201
3/06/05/trim-on-lvm-on-luks-on-ssd
![Page 11: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/11.jpg)
Thin LVM
SSD
Partition
Encryption
LVM
thinpool
Virtual DiskCreating a thin pool on VG myVG[root@t440s ~]# lvcreate -T -L 80G myVG/lv_thin
Logical volume "lv_thin" created.
New VM Disk Image[root@t440s ~]# lvcreate -V40G -T myVG/lv_thin -n New_VM
Logical volume "New_VM" created.
![Page 12: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/12.jpg)
Trim + libvirt
SSD
Partition
Encryption
LVM
KVM
Check your machine type > 2.1<type arch='x86_64' machine='pc-i440fx-2.4'>hvm</type>
Add discard to your hard disks<driver name='qemu' type='raw' discard='unmap'/>
Add scsi controller model virtio-scsi<controller type='scsi' index='0' model='virtio-scsi'>
![Page 13: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/13.jpg)
Trim in a VM
SSD
Partition
Encryption
LVM
thinpool
KVM
Virtual Disk
Partition
LVM
Filesystem
Treat like a physical environment
Same rules apply for
● fstrim vs discard● lvm.conf● Luks
![Page 14: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/14.jpg)
Encryption +Trim in a VM
SSD
Partition
Encryption
LVM
thinpool
KVM
Luks Virtual Disk
Partition
LVM
Filesystem
Don’t..
Just Don’t..
Please please...
DO NOT DO IT.
![Page 15: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/15.jpg)
Are you Trim?
SSD
Partition
Encryption
LVM
thinpool
KVM
Virtual Disk
Partition
LVM
Filesystem
Boot and test your VM[root@testvm ~]# lsblk -o MOUNTPOINT,DISC-MAX,FSTYPE
MOUNTPOINT DISC-MAX FSTYPE
/boot 1G xfs
[SWAP] 1G swap
/ 1G xfs
Run fstrim in your VM[root@testvm ~]# fstrim -v /
/: 10.4 GiB (11197124608 bytes) trimmed
![Page 16: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/16.jpg)
Fat -> Thin
SSD
Partition
Encryption
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Physical Host
Virtual Disk Image
![Page 17: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/17.jpg)
Fat -> Thinkpartx + fstrim
SSD
Partition
Encryption
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Physical Host
Virtual Disk Image
Painfully via kpartx + fstrim# kpartx -a /dev/myVG/myVMDisk
# cd /mnt; mkdir volume
# mount /dev/mapper/myVG-myVMDisk_BASE1 /mnt/volume/
# fstrim -v volume
boot: 356.5 MiB (373850112 bytes) trimmed
# umount volume/
![Page 18: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/18.jpg)
Fat -> Thinkpartx + fstrim
SSD
Partition
Encryption
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Physical Host
Virtual Disk Image
Repeat for other filesystemsmount /dev/guestVG/root /mnt/volume/
# fstrim -v ./volume
./volume: 9.8 GiB (10481520640 bytes) trimmed
# umount volume
![Page 19: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/19.jpg)
Fat -> Thinkpartx + fstrim
SSD
Partition
Encryption
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Physical Host
Virtual Disk Image
Clean up VGs and kpartx /dev/mapper entries# vgchange -a n guestVG
0 logical volume(s) in volume group "guestVG" now active
# kpartx -d /dev/mapper/myVG-myVMDisk_BASE
![Page 20: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/20.jpg)
Fat -> Thinvirt-sparsify
SSD
Partition
Encryption
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Physical Host
Virtual Disk Image
Existing FAT LVM# dd if=myVMDisk.img of=/dev/myVG/myVMDisk
# virt-sparsify --in-place /dev/myVG/myVMDisk
Fat image ->Thin LVM# virt-sparsify myVMDisk.img /dev/myVG/myVMDisk
# lvs /dev/myVG/myVMDisk
![Page 21: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/21.jpg)
Thin -> Thin
SSD
Partition
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Host B
SSD
Partition
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Host A
● Painful● Sparse● With Compression
(Encryption)(Encryption)
![Page 22: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/22.jpg)
Thin -> Thin
SSD
Partition
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Host B
SSD
Partition
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Host A
● Painful– dd via ssh or nc
● Results in a Fat LV
– Use kpartx/fstrim or virt-sparsify at destination
(Encryption)(Encryption)
![Page 23: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/23.jpg)
Thin -> Thin
SSD
Partition
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Host B
SSD
Partition
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Host A
Sparse[root@hostb]# nc -l 11900 | \
dd of=/dev/HostbVG/guestVG bs=16M conv=sparse
[root@hosta]# dd if=/dev/HostaVG/guestVG \
bs=16M | nc hostb 11900
[root@hostb]# virt-sparsify –in-place \
/dev/HostbVG/guestVG
(Encryption)(Encryption)
![Page 24: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/24.jpg)
Thin -> Thin
SSD
Partition
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Host B
SSD
Partition
LVM
thinpool
KVM
Thin Virtual Disk
Partition
LVM
Filesystem
Host B
Sparse + lzop[root@hostb]# nc -l 11900 | lzop -d -c - \
dd of=/dev/HostbVG/guestVG bs=16M conv=sparse
[root@hosta]# dd if=/dev/HostaVG/guestVG \
bs=16M | lzop --fast - | nc hostb 11900
[root@hostb]# virt-sparsify –in-place \
/dev/HostbVG/guestVG
(Encryption)(Encryption)
![Page 25: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/25.jpg)
Nesting Turtles?
CC by NC Christa Burnshttps://www.flickr.com/photos/christajoy42/5562852057/
![Page 26: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/26.jpg)
Nesting KVM
Virtual Disk
Partition
(Encryption)
LVM
Nested KVM
Virtual Disk
Partition
LVM
Filesystem
Virtual Host
SSD
Partition
LVM
thinpool
KVM
Virtual Disk
Physical Host
/etc/modprobe.d/kvm-intel.confoptions kvm-intel nested=1
options kvm-intel enable_shadow_vmcs=1
options kvm-intel enable_apicv=1
options kvm-intel ept=1
OR
/etc/modprobe.d/kvm-amd.confoptions kvm-amd nested=1
(Encryption)
(thinpool)
![Page 27: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/27.jpg)
But I like (q)Cows?
![Page 28: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/28.jpg)
Trim a qcow2 VM
SSD
Partition
LVM
KVM
qcow2 image
Partition
LVM
Filesystem
(Encryption)
(thinpool)
Same rules as we covered KVM and LVM● KVM
– Machine type / unmap / virtio-scsi
● Inside the VM– fstrim vs discard
– lvm.conf
● virt-sparsify– virt-sparsify disk.raw --convert qcow2 disk.qcow2
![Page 29: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/29.jpg)
Raid and Trim?
SSD
md0
mdadm raid1
Filesystem
LVM
Two key software raid technologies● mdadm● lvm raid
SSD SSD
partition
Filesystem
LVM raid1
SSD
partition
![Page 30: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/30.jpg)
mdadm raid1
SSD
md0
mdadm raid1
Filesystem
LVM
Linux software raid (mdadm)● Discard is passed-thru correctly
– Subject to your kernel version / distro
● Initialisation may write to all blocks– Potentially impacting performance
SSD
![Page 31: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/31.jpg)
LVM raid1
SSD
partition
Filesystem
LVM raid1
LVM based raid 1● Discard is passed-thru correctly
– Less initialisation overhead compared with mdadm
SSD
partition
![Page 32: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/32.jpg)
KVM Tips1) Trim / discard with libvirt + KVM
2) Using virt-sparsify
3) Nesting with KVM
4) Enable KVM serial Console
![Page 33: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/33.jpg)
KVM Tips1) Trim / discard with libvirt + KVM
2) Using virt-sparsify
3) Nesting with KVM
4) Enable KVM serial Console (including Grub)
![Page 34: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/34.jpg)
KVM serial console ● Need to modify /etc/default/grub
– GRUB_TERMINAL="serial console”
– GRUB_SERIAL_COMMAND="serial --speed=38400 --unit=0 --word=8 --parity=no –stop=1"
– GRUB_CMDLINE_LINUX● remove “rhgb quiet”
● Add “console=tty0 console=ttyS0,38400n8”
● Rebuild grub2 config– grub2-mkconfig -o /boot/grub2/grub.cfg
![Page 35: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/35.jpg)
KVM serial console + Ansible● Example playbook
– https://github.com/steven-ellis/ansible-playpen● grub_console.yaml
– Currently RHEL/Centos/Fedora centric● Pull requests welcomed
● Demo
![Page 36: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/36.jpg)
http://people.redhat.com/sellis
Questions?
![Page 37: Turtles all the Way Down - people.redhat.compeople.redhat.com/sellis/Presentations/LCA 2017 Turtles.pdf · 2017. 1. 16. · Turtles all the Way Down Misadventures with Trim and Thin](https://reader035.fdocuments.us/reader035/viewer/2022071413/610bad205f7f34218907ad6f/html5/thumbnails/37.jpg)
Open?