Virtualization Technique Virtualization Technique System Virtualization I/O Virtualization.
MySQL-and-virtualization
-
Upload
raghavendra-prabhu -
Category
Technology
-
view
871 -
download
0
description
Transcript of MySQL-and-virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
MySQL and VirtualizationPercona Live MySQL Conference and Expo 2013
Raghavendra [email protected]
Percona
23rd April, 2013
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Outline
Introduction
Virtualization
MySQL
Symbiosis
Instrumentation/Debugging
Instrumentation II
Application
Conclusion
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Overview
I WhyI Why notI Performance v/s EfficiencyI Overcommit!I High AvailabilityI Testing/Instrumentation
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Popular solutions
I Xen, KVM, VirtualBoxI Why KVM
I Massive reuse of kernel and userspaceI Compatibility in case of para-virt
I Most applies to Xen as wellI Important difference
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Native against Emulation
I Emulation: TCGI Valgrind
I KVMI Kernel module
I VT-x (Intel), AMD-VI Guest - Host context
I Exit - Enter latencyLazy approach
I Para-virtualization
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Development
I QEMUI Most active
I In-kernelI Native client
I More hypervisorsI KVM API - ioctl /dev/kvm
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Considerations
I PerformanceI Scaling - Horizontal or VerticalI Eggs and the BasketI Moore’s law
I CostI Saturation
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Memory
I KSMI How it works and is usedI How it can be used: MADV_MERGEABLE
I Memory Ballooning: VirtioI OvercommitI Memory statistics: QOM
I CachingI Affects I/OI Write{back,through}, none, directsync and unsafe
I Snapshot mode & commit: cache=unsafe
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Memory III KSM for MySQL
I Multiple buffer poolsI Shared storage - VirtFS
I With NUMAI Pinning + KSM = !
I ksmtuned and numadI Ballooning: In host’s shoes
I WhereI Migration & Upgrade - Hot/Cold
I With QGAI Feedback from MySQL: Para-virtualized process!
I CacheI InnoDB + cache=none = true O_DIRECT: for NFSI =writethrough: app fails to grok fsyncI =writeback: speed and benefits of none
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
CPU
I Processor pinningI Cgroups: Better
I Applies to othersI Dynamic
I HotplugI Overcommitting - smp
I Context switches
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
NetworkI Types
I VDEI TAP and BridgeI OpenvswitchI DumpI Multicast with UDP
I PerformanceI virtio-net
I Para-virtualized constructI Multi-queue
I vhost-netI Zero-copy transmit: DMAI No Qemu: kernel and KVMI Not just network
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Security
I MasqueradingI Isolation
I Namespaces
I Random NumbersI Why?I Hardware / Virtio RNG
I Resource exhaustion: DDoSI OOM: Cgroups
I Seccomp sandbox: Safety is only an illusion!I Guest to host vulnerabilities
I Encryption
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
I/O and StorageI The weakest linkI Virtio
I virtio-blk-pci data planeI Separate I/O thread
I PCI PassthroughI PCI devices in generalI IOMMU (AMD) & VT-d (Intel)I SR-IOV: multiple devices without having
I VFIO: in userspaceI No longer exclusiveI More secure: very importantI Interrupts closer to guest
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
I/O and Storage III VirtFS
I Filesystem passthroughI 9p protocol
I PossibilitiesI Fragmentation
I How: Hole punchingI Fallocate, pre-allocation Qcow
I Thin provisioningI Qcow lazy provisioningI Device mapper target
I Throttling: Errant guests!block_set_io_throttle
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
I/O and Storage III
I BackendsI RawI QcowI Block device attachment
I Performance close to bare metalI GlusterFSI Sheepdog
I Distributed StorageI iSCSII NBD
I Networked storageI Shared disk image
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
I/O and Storage IVI Scheduler
I DeadlineI CFQ - quantum and slice
I DiscardI Pass discard requests to host filesystemI Fully async
I AIOI pthread v/s native
I SwapI SwappinessI Less preferred to Ballooning and KSMI Poor with EPT
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Others
I Transparent Huge PagesI Issues?
I Possible fixes - MADV_NOHUGEPAGEI How can it help
I ivshmemI Fastest message passing
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Reliability
I What everyone wants: Uninterrupted Sleep - ‘D’ stateI What they usually get: Nagios alertsI Its like InsuranceI What can be doneI Backups
I Backup the original imageI Blockdev snapshotI guestfs-fsfreeze-{freeze,thaw} and FTWRL
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Reliability III High Availability
I System uptime v/s Application uptimeI Difference
I MigrationI Live migration: RequirementI Online block streaming/commit
I Drive MirroringI Live block copy
I Snapshots
I VM Snapshots: complete state
I Qcow2 snapshotsRaghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Upgrades
I Downtime
I Ballooning the buffer pool
I Migration
I Preserves connectionsI OpenvSwitch
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
TestingI Cluster
I PXC
I TopologiesI Army of nodesI Forest: Cluster of clusters
Data centers in microcosm
I Memory
I NUMA
I Page cache and Memory Pressure
I Dump guest memory: coreI Examine online
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Fault injection
I NMI and MCE
I Pulling the power cord!
I Filesystem
I Interesting cases
I Ext4I XFS
I PCI AER
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Others
I Jenkins and Vagrant
I Watchdog
I “Houston, we have a problem”
I With MySQL: Turtles all the way
I systemd
I ACPI tables
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
perf-kvm
I perf toolsuite
I Strace?
I Caveats
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
GDB monitor
I Remote Target and Single Stepping
I Kernel development
I bzImage and initrd
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Qemu Monitor (QMP/HMP)
I Ballooning
I Throttle
I PCI Hot-add
I Examining memory
I JSON API
I Endless possibilities with scriptingI Python scripts in repo
I Info commands
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
TracingI Ftrace
I LatencyI Virtio I/O
I Dtrace
I With systemtap
I LTTng
I Simple
I Stderr
I Dprintf
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Development
I QCOW
I Backing file hierarchy
I block stream
I Advantages
I Copy on read
I Lazy Refcounts
I Encryption and Compression
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Development III Caveats
I Raw v/s Qcow: Performance
I Rendering Qcow unusable
I Filesystem passthrough
I Share datadir
I Common code-tree / development directories
I Pass the entire root R/O!I Lightweight SnapshotI Useful to have newer FHS with /usr
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
WrappersI Scripting
I libvirt
I virsh / virt-manager
I Openstack
I Proxmox
I Candy
I Spice
I VNC
I oVirtI VDSM
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Thoughts?
I Questions
I Usage and Experience
Raghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
AppendixI Best practices: http://goo.gl/LY3tf
I Virtio Introduction: http://goo.gl/B9bCA
I KVM Kernel API: http://goo.gl/5A4Ma
I linux-kvm.org: http://goo.gl/yWPiU
I Internals blog: http://goo.gl/z07cy
I Qemu wiki: http://goo.gl/IGoKa
I RedHat Virtualization docs: http://goo.gl/rXRKc
I Docs from Qemu git tree: http://goo.gl/qD7nj
I KSM internals: http://goo.gl/ypyhkRaghavendra Prabhu Percona
MySQL Virtualization
Introduction Virtualization MySQL Symbiosis Instrumentation/Debugging Instrumentation II Application Conclusion
Me
I Raghavendra Prabhu
I http://wnohang.net/about
I Software Engineer at Percona and Product Lead of PXC
I https://launchpad.net/percona-xtradb-cluster
I Slides downloadable at http://goo.gl/jsult
Raghavendra Prabhu Percona
MySQL Virtualization