Performance Tuning, Management and Optimization in a Virtual Infrastructure
description
Transcript of Performance Tuning, Management and Optimization in a Virtual Infrastructure
![Page 1: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/1.jpg)
Performance Tuning, Management and Optimization in a
Virtual Infrastructure
Presented by David DavisDirector of Infrastructure
www.TrainSignal.com
![Page 2: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/2.jpg)
Who Is David Davis?
• Director of Infrastructure Train Signal, Inc --
the leader in Professional IT video training
• Over 15 years in enterprise infrastructure
management and years of real-world
virtualization experience
• Have obtained the following certifications:
CCIE#9369, VCP, MCSE and CISSP
![Page 3: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/3.jpg)
• Author of six video training courses and
hundreds of articles for well-known websites
such as: SearchVMware.com and
VirtualizationAdmin.com
• Best known for my Train Signal VMware
• ESX Server video training course
• Best of VMworld 2008 Awards Judge
• Company website: www.TrainSignal.com
• Personal website: www.VMwareVideos.com
Who Is David Davis?
![Page 4: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/4.jpg)
Abstract
• Virtualized Infrastructures can perform as well
or better than physical infrastructures, if
performance tuning and management are
done correctly.
• Today’s Applications are complex:
• Virtualized
• Distributed
• Intensive
• Tied to SLAs
![Page 5: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/5.jpg)
• We will cover…
• Performance Tuning and Management in the
virtual infrastructure
• Best practices for virtualization performance
• How to troubleshoot bottlenecks on existing
systems
• Tools that allow IT Pros to manage virtual
infrastructure properly
Abstract
![Page 6: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/6.jpg)
• What are common mistakes that can hurt
performance and how can you prevent them?
• What tools are available for managing the
performance of your virtual infrastructure?
• What are the best practices for configuring
your virtualization infrastructure to ensure
ideal performance?
Abstract
![Page 7: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/7.jpg)
Assumptions…
• Good understanding of server virtualization concepts
• May or may not already be using virtualization
• Have, or will have, performance concerns
(that’s everyone, right?)
What I Assume You Already Know
![Page 8: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/8.jpg)
By The End Of The Session, You’ll Know The Following:
• How to manage performances
• How to optimize performance
• How to troubleshoot performance issues
• How to design your VI so that you prevent performance issues
![Page 9: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/9.jpg)
Virtualization Basics: Virtualization Guest Encapsulation
![Page 10: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/10.jpg)
Virtualization Performance Basics
Hardware
Device Drivers
Storage Stack Network Stack
VMM VMMVMkernel
ResourceManagement
UserWorlds POSIX API
VM VM
Other Peripheral I/O
HostedManagementAgents and Interfaces
Service Console
![Page 11: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/11.jpg)
Managing Performance
![Page 12: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/12.jpg)
Managing Performance
![Page 13: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/13.jpg)
Managing Performance
![Page 14: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/14.jpg)
ESX 3.5 And Update 1
• Up to 32 logical processors per host (64 LP
experimental)
• Large memory support – 250 GB physical and
64Gb per VM
• Large page size – 2 MB VMKernel pages can be
allocated to guest OS
• 192 vCPUs per Host -- Update 1
![Page 15: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/15.jpg)
ESX “4.0”• CPU and RAM “hot add”
• Historical performance tracking and performance alerts
• Clustered VirtualCenter Servers
• ESX hosts profile management
• Cross-hosts virtual networking
• 8-way virtual SMP
• Virtual machines fault tolerance across multiple hosts
dubbed “vlockstep”
![Page 16: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/16.jpg)
Virtualization Overhead
• CPU -- special handling of instructions
• Memory -- additional management tasks
• Devices and resource management -- not
direct access to hardware
• Typically, difficult to notice -- in my
experience
![Page 17: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/17.jpg)
Memory Overhead
• Service console 272 MB (not in ESXi)
• VMKernel 100 MB+
• Per-VM memory overhead increases with:
• Number of VCPUs
• Size of guest memory
• 64-bit guest OS
![Page 18: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/18.jpg)
Virtual CPU Recommendations
• Single threaded app = uni proc VM
• Multi threaded app = SMP VM
• But only as many as required
• Unused VCPU in SMP VM = scheduling
overhead -- see KB 1077 and 1730
![Page 19: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/19.jpg)
• Make sure OS HAL matches number of CPUs
-- MP vs. UP HAL
• Use 64-bit guests, if possible -- more
registers, larger kernel tables
• Still, remember 64-bit OS vs. app
compatibilities
Virtual CPU Recommendations
![Page 20: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/20.jpg)
Performance Issues Due To Interrupts
• Any controller but usually USB
• Disable USB
• See KB 1290
![Page 21: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/21.jpg)
Overall Device Recommendations
• Disable / remove all unused devices
• USB, CDROM, Floppy
• Can consume CPU when idle
![Page 22: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/22.jpg)
Large Guest Pages Backed By Host
• New in ESX 3.5
• Significant performance improvement for
memory intensive apps
• Best to allocate large pages immediately
after VM boot
• Page sharing not supported for large pages
![Page 23: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/23.jpg)
Network Performance
• Check NICs for proper speed and duplex,
hardcoded
• NIC teaming distributes load and offers
passive failover
• Separate NICs avoid contention -- console,
VMKernel, and VM
• Tune VM to VM networking and rx/tx buffers
(KB 1428)
![Page 24: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/24.jpg)
• Use 32-bit vmxnet driver instead of vlance
• To use vmxnet, install tools
• E1000 is for 64-bit guests
• Enhanced vmxnet is offered for several
guests
Network Performance
![Page 25: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/25.jpg)
• Use a network adapter that supports the following:
• Checksum offload
• TCP segmentation offload (TSO)
• Jumbo frames (JF), available in enhanced vmxnet vNIC
• Capability to handle high memory DMA
• Capability to handle multiple scatter/gather elements per Tx frame
• 10 G
Network Performance
![Page 26: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/26.jpg)
• Vmxnet -- high speed net driver
• Memory balloon driver
• Improved graphics
• Timer sponge for correct accounting of time
• Timesync -- syncs time with host every
minute
Install VMware Tools
![Page 27: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/27.jpg)
Storage Performance
• Hardware configuration
affects storage
performance
• Consult SAN
Configuration Guides
• Ensure caching is enabled
• Consider tuning layout of
LUNs across RAID sets
• Spread I/O requests
across available paths
FC Switch
ESX ServerHBA1 HBA2 HBA3 HBA4
Storage array
SP2SP1
1 2 3 4
![Page 28: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/28.jpg)
Storage Performance• Fibre Channel SAN storage best practices
• Set LUN queue depth appropriately (KB 1267)
• Networked storage best practices (NFS, iSCSI)
• Ensure sufficient CPU for software-initiated iSCSI
and NFS
• Avoid link oversubscription
• Ensure consistent configuration across the full
network path
• Use multiple mount points with multiple VMs
![Page 29: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/29.jpg)
Benchmarking With VMark
• VMmark: A scalable benchmark for virtualized enterprise systems• Provides meaningful measurement of
virtualization performance
• Generates metric that scales with underlying system capacity
• Used to compare the performance of different hardware and virtualization platforms
• Employs realistic, diverse workloads running on multiple operating systems
![Page 30: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/30.jpg)
Storage Performance• VM Configuration• Choose placement of data disks and swap
files on LUNs appropriately• RAID type, spindles available, concurrent access
of LUNs etc.• Increase VM’s max outstanding disk requests
if needed (KB 1268)
• Esxtop enhancements
• Per device and path stats
• Per VM device stats
![Page 31: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/31.jpg)
DRS Performance• Ensure hosts in a cluster are VMotion compatible
• Minimize reservations if possible
• Use VM affinity and anti-affinity rules only when needed
• Migration threshold should be set less aggressively
when
• Hosts in the cluster are inhomogeneous
• VM resource utilization is highly variable in time
• More affinity and anti-affinity rules
• Use DRS to achieve max performance
![Page 32: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/32.jpg)
Troubleshooting Performance
• Know your applications
• Have a baseline
• Esxtop
• Decent tools in VI Client
• Find Bottleneck, CPU, Disk, RAM or Net
• Host or Guest?
• Some components may be out of your domain -- both SAN and Net are critical
![Page 33: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/33.jpg)
Designing VI To Prevent Performance Issues
• Capacity Planning is key
• Know your apps
• Understand the SAN
• Use DRS / Resource Pools
• Don’t skimp on hardware
![Page 34: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/34.jpg)
Prevent Common Performance Mistakes
• P2V
• Poor Improper Sizing
• Poor Hardware Selection
• Alerting not configured
• Not using DRS
![Page 35: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/35.jpg)
Performance Tools
• Esxtop
• vKernel Capacity Bottleneck Analyzer
• vKernel Modeler
• Solarwinds Orion VMware Edition and free VM
Monitor
• Veeam Monitor
• Nagios
• Vizioncore vCharter / vFoglight
• eG VM Monitor
![Page 36: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/36.jpg)
Performance Tool Demo
INSERT GRAPHICS – estimated to be about 5
slides
![Page 37: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/37.jpg)
David’s Five Performance Tips
1. Know how to use esxtop -- quick and simple
2. Know your applications and environment
3. Have a baseline
4. Don’t skimp on hardware
5. Use third-party performance tools -- historical
performance monitoring is required
![Page 38: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/38.jpg)
Conclusion
• Virtualization environments continue to grow
in complexity
• Managing performance doesn’t have to be
difficult
• Follow best practices, know your environment
and use third-party performance tools
• With that, performance can be improved and
troubleshooting can be simplified
![Page 39: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/39.jpg)
References
• Related papers for best practices and benchmarking:• ESX Server 3 performance tuning best practices • www.vmware.com/pdf/vi_performance_tuning.pdf• VMmark• www.vmware.com/pdf/vmmark_intro.pdf
• SAN Configuration Guide:• www.vmware.com/pdf/esx_san_cfg_technote.pdf• www.vmware.com/pdf/vi3_esx_san_cfg.pdf
![Page 40: Performance Tuning, Management and Optimization in a Virtual Infrastructure](https://reader035.fdocuments.us/reader035/viewer/2022062809/56815a1f550346895dc7638e/html5/thumbnails/40.jpg)
Questions?