Virtualizing Microsoft®
Exchange® Server 2007 on Dell™ Systems Running VMware® ESX A Dell Technical White Paper
End‐to‐End Solutions Engineering
By Ramesh Radhakrishnan and Tom King
Dell Product Group
May 2009
Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX
Page ii
THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.
© 2009 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.
Dell, the DELL logo, EqualLogic, and the DELL badge are trademarks of Dell Inc.; Microsoft, Outlook, and Windows are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries.
Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX
Page 1
Table of Contents Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX .......................................... i
Executive Summary ....................................................................................................................................................... 2
Introduction ................................................................................................................................................................... 2
Virtualized Exchange Server 2007 Building Block .......................................................................................................... 3
Solution Architecture ................................................................................................................................................. 3
Deployment Best Practices ........................................................................................................................................ 4
Network Setup and best practices ......................................................................................................................... 5
Storage Best Practices ........................................................................................................................................... 5
Performance Evaluation of Virtualized Mailbox Role .................................................................................................... 5
Test Methodology .................................................................................................................................................... 5
Scalability Testing .................................................................................................................................................... 6
Conclusion ..................................................................................................................................................................... 9
References ..................................................................................................................................................................... 9
Appendix A................................................................................................................................................................... 10
Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX
Page 2
Executive Summary This paper provides an overview of deploying Exchange 2007 Server in a virtualized environment using
VMware 3.5 Update 3 on Dell PowerEdge Servers and EqualLogic iSCSI storage array. In this paper a solution
building block is proposed for 4000 mailbox users to deploy in an ESX environment running on Dell hardware.
Various mailbox configurations of Exchange Server 2007 are tested to understand the sizing requirements for
Exchange Server 2007 in a virtualized environment and validate the building block.
Introduction There is a growing interest in deploying Exchange Server 2007 in virtualized environments with the recent
announcement regarding change in support policies for virtualized Exchange deployments1. In a virtualized
environment, Exchange users can consolidate multiple Exchange Server Mailbox roles on a single physical
server to take advantage of unused CPU cycles or available memory. Additional Exchange Server roles (Hub
Transport or Client Access Server) can be hosted on virtual machines (VM) and consolidated on the same
hardware to reduce costs. Benefits of running Exchange in a virtualized environment will depend on the
customer environment. Some of the advantages of virtualizing Exchange 2007 Server are:
• Cost savings thru server hardware consolidation
• Reducing performance bottlenecks by moving Mailbox VMs to systems with available CPU cycles
• Easier management of Exchange systems and infrastructure
In this paper a sample configuration is presented which is tested and validated to meet the performance and
stability requirements of an enterprise Exchange deployment. This sample configuration is scalable and can
be used as a building block for larger scale deployments. By deploying Exchange 2007 Server on ESX
environment hosted on Dell Servers and EqualLogic iSCSI SAN, organizations can reduce the management
complexity and improve utilization of hardware resources by consolidating on fewer hardware resources.
To provide sizing guidelines for Exchange Server 2007 the Mailbox role was deployed in a Virtualized
environment and tested using multiple Mailbox configurations. The trending data was used to understand
the CPU and memory requirements for Exchange Server 2007 running within a VM and how it differed
compared to a physical environment. For consolidation purposes, the performance of deploying Exchange
Server 2007 Mailbox roles in multiple VMs on a single server is also evaluated.
This white paper is intended for customers, IT architects, storage administrators and IT administrators who are evaluating, planning, or deploying Exchange 2007 Server on VMware ESX 3.5 platform and require guidance on available options and best practices. The goals of this white paper are to introduce customers to Dell’s building block for deploying Exchange 2007 Server on VMware ESX and simplify the decision making process for the customer when deploying Exchange 2007 in Dell hardware running ESX hypervisors.
1 Microsoft Support Policies and Recommendations for Exchange Servers in Hardware Virtualization Environments, http://technet.microsoft.com/en‐us/library/cc794548.aspx
Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX
Page 3
Virtualized Exchange Server 2007 Building Block This section presents an overview of the building block configuration for an Exchange Server 2007
deployment which uses VMware ESX to create a virtualized environment. The building block presented in
Figure 1 was tested using LoadGen tool to simulate a 4000 heavy user workload and it comfortably met the
performance requirements recommended by Microsoft for good quality of service. The Performance
Evaluation section describes how this building block configuration performed at the server, storage and
application level when LoadGen was used to simulate up to 4000 users.
Figure 1 – Architecture of Building Block for Exchange 2007 on ESX 3.5
Solution Architecture In this section the core components that make up the Dell|VMware Virtualized Exchange Building Block are
detailed. The architecture can be broken into four primary layers, each coexisting and working together to
provide an optimized platform for messaging deployment targeting small to mid‐size organizations or
businesses. The individual layers of the building block are –
• Layer 1: Dell PowerEdge Servers
• Layer 2: Dell|EqualLogic iSCSI SAN Storage
• Layer 3: VMware Infrastructure 3
Vmware ESX
W8OS
Active Directory VM
W8OS
CAS/Hub Transport VM
Virtualized Storage Environment
Clients
PC6248 PC6248
W8OS
Exchange Mailbox VM
EqualLogic PS6000XV
Dell PowerEdge R710
EqualLogic PS6000XV
Workload LAN
iSCSI Fabric
Networking Legend
Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX
Page 4
• Layer 4: Windows Server 2008 and Microsoft Exchange Server 2007
Layer 1: Selecting the right server for a virtualized Exchange environment requires careful consideration since
it could negatively impact the cost and performance of the overall solution. The proposed building block uses
the Dell PowerEdge R710 server with dual quad‐core processors (Intel Xeon 5500 series). The R710 server is
optimized for virtualization solutions with its enhanced memory and I/O features. Microsoft recommended
guideline should be followed when determining the CPU and memory resources being assigned to the
individual machines running the Exchange 2007 Roles. [1]
Layer 2: The storage solution used for the virtualization building block is based on the Dell EqualLogic 6000
Series iSCSI SAN. EqualLogic storage arrays provide a cost‐effective virtualized storage solution that offers
excellent performance for messaging environments. The number and size of volumes for Exchange 2007
mailbox databases and logs were based on best practices recommended by Dell and Microsoft and depends
on the number and size of user mailboxes. When planning storage architecture for Exchange 2007 the
Exchange 2007 Storage Requirements calculator is a valuable tool. [2]
Layer 3: VMware Virtualization Infrastructure 3 (VI3) was used as the virtualization foundation to create the
virtualized messaging environment. VI3 abstracts hardware resources (CPU, memory, storage and network)
to components that can serve multiple virtual machines providing greater hardware utilization. Using a
virtualized environment allows the hardware to run multiple virtual machines that host the individual pieces
of the Exchange 2007 solution. Multiple virtual machines can be used to run the Exchange 2007 Mailbox role,
each hosting a small number of users to allow more flexibility to the administrator in terms of migrating VMs
across physical servers. Some solution may use a larger VM configuration and host a larger number of users
per VM while still taking advantage of virtualization features like decoupling from hardware and increasing
hardware utilization levels.
Layer 4: The final layer consists of the guest OS running in the VMs and the application itself. The virtual
machines run Windows Server 2008 OS editions and Exchange Server 2007 roles are installed on the guest
VMs. Additional infrastructure server such as Active Directory server may be virtualized as well to increase
server consolidation. Exchange 2007 server best practices should be followed when implemented in a
virtualized environment as well to maximize performance and solution stability [3].
Additional consideration should be made for storage configuration as iSCSI connections can be established at
the guest OS level or it may be configured at the ESX host level. Similarly, network design should be planned
out ahead so that enough network devices are available for both workload LAN and storage fabric to meet
the performance requirements of the messaging solution.
Deployment Best Practices In this section, we discuss a few of the best practices and proceedures that should be followed when
implementing the network and storage infrastructure for the virtualized Exchange 2007 deployment.
Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX
Page 5
Network Setup and best practices Although not required or necessary, it is better to isolate the iSCSI traffic from the remaining virtual machine traffic. This allows more fine‐grained monitoring and control over the networks’ differing configuration, e.g. Jumbo Frames for iSCSI traffic versus “normal” MTU settings for the remaining traffic.
Further, each network should be divided with multiple physical paths for load balancing and failover capabilities. In our test bed, we used one PowerConnect 6248 for virtual machine network and management traffic connected to one pair of NICs on the ESX server, and another PowerConnect 6248 for iSCSI traffic connected to another pair of NICs from a dual‐port PCIe card on the ESX server. In highly mission‐critical installations, these networks can be further divided to ensure no single point of failure.
ESX provides highly flexible tools when configuring separate networks. See Appendix A, Figure 6 for details of how the network topology was setup using the ESX networking management tool.
Storage Best Practices Throughout each set of tests, each mailbox server was provided 80% of its allocated space for the mailbox database and 20% of its allocated space for the mailbox logs. Each mailbox server’s iSCSI target was not shared with other virtual machines. None of these particular iSCSI targets housed the virtual machines VMDK files.
The ESX array used RAID 10 across 14 disks with two hot spares. Prior testing using Jetstress showed that RAID 10 offers the best performance in similar environments.
Our testing covered three different storage access configurations:
‐ iSCSI initiator on guests with MPIO ‐ iSCSI initiator on ESX with VMFS file systems ‐ iSCSI initiator on ESX with Raw Device Maps (RDM)
If any external systems need access to the database and log files, using RDM is advantageous. Additionally, using RDM allows switching the file system to be accessed with a guest‐based iSCSI initiator utilizing a multi‐path I/O driver without reformatting. On the other hand, VMFS offers added high availability features such as integration with EqualLogic ASM/VMware editions to create VSS‐aware snapshots and backups.
Whereas using the ESX 3.5 iSCSI initiator allows only one session per target, using the guest‐based initiator allowed load balancing through MPIO. However, our tests showed there was no performance gain in doing so due to the increased overhead and added layers between the guests and the iSCSI targets.
Performance Evaluation of Virtualized Mailbox Role This study evaluates the scalability and performance characteristics of running Exchange Server 2007 in a VMware ESX environment. The performance of running Exchange Server roles (Mailbox, Hub Transport and CAS) in a virtual machine is compared against running them in a physical server to measure the virtualization overhead associated with a messaging workload.
Test Methodology
Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX
Page 6
The sizing of the VMs (CPU cores and memory) is based on Dell and Microsoft recommended guidelines for a
heavy user Mailbox profile and prior testing done at Dell for physical Exchange deployments. The test
configuration is monitored for bottlenecks by measuring Server, Storage and Exchange related counters
within the Virtual Machine (VM) and also at the host ESX level.
Microsoft Exchange Load Generator (LoadGen) tool (version 08.02.0045) is used to simulate user workloads
and evaluate performance of the Exchange Server 2007 server configurations. For sizing recommendations,
performance is evaluated at multiple levels—application, server, and storage. Performance is measured at
the application level by using Exchange related metrics measured by Exchange LoadGen. Microsoft Reliability
and Performance monitor is used to measure server and storage level performance metrics. Esxtop is used
to collect CPU and memory usage from the ESX server.
Table 1 lists events that were measured and thresholds that were used to determine if a configuration
performed as expected.
Table 1: Counters Used to Evaluate Performance
Performance Counter Expected Value (MS Recommended)
SERVER PERFORMANCE
Exchange VM: Processor(_Total)\% Processor Time Average less than 75%
ESX : Esxtop ‐ Physical Cpu(_Total)\% Processor Time Average less than 75%
APPLICATION PERFORMANCE LoadGen 95th Percentile Send Mail Latency Less than 1000 ms
MS ExchangeIS: RPC Averaged Latency Average less than 25 ms
MS ExchangeIS Mailbox: Messages Queued for Submission
Average less than 50
STORAGE PERFORMANCE Database Disks: Average Sec/Read Less than 20 ms
Database Disks: Average Sec/Write Less than 100 ms
Log Disks: Average Sec/Write Less than 10 ms
Scalability Testing
LoadGen was used to simulate 1000 to 4000 users and tested against the solution building block to study the scalability of the proposed solution configuration. Table 2 describes the VM configuration in terms of CPUs and memory that was allocated to the VM. These configurations are based on the resources that would be recommended for a physical deployment to satisfy similar number of users and workload.
Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX
Page 7
Table 2 ‐ Exchange Server 2007 Mailbox VM Configuration
1000 User VM 2000 User VM 3000 User VM 4000 User VMCPUs 2 vCPU 2 vCPU 4 vCPU 4 vCPU Memory 2GB(OS)+5GB 2GB(OS)+10GB 2GB(OS)+15GB 2GB(OS)+20GBHardDisk (OS) 20GB 20GB 20GB 20GB
Figure 2 illustrates the performance and scalability of the proposed architecture. The primary metrics used to compare performance were the host server’s processor utilization and the 95th‐percentile response time for the LoadGen SendMail task. The proposed architecture building block is observed to provide good quality of service as the 95th percentile’s average latency is less than 500 milliseconds and still has ample CPU headroom to handle spikes in the workload.
Figure 2 ‐ Scalability of Exchange 2007 Server in ESX Environment
Figure 3 illustrates two of the RPC specific counters that are commonly monitored to evaluate the health and performance of the messaging environment. RPC averaged latency represents the time in milliseconds that it takes Exchange (store.exe) to process the last 1024 packets received. It represents the shortest amount of time that clients have waited for a response from the server. The RPC averaged latency is well below the threshold of 25 ms for 4000 users indicating that the server is able to process requests in a reasonable amount of time.
The Messages Queued for Submission performance counter indicates the number of messages in the mailbox store that are ready to be sent and are waiting to be submitted to a transport server. The queued messages are observed to be well within the recommended threshold of 50 indicating a healthy configuration.
Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX
Page 8
Figure 3 ‐ Application Level Performance
Figure 3 illustrates the observed disk subsystem performance observed for the proposed solution building block using EqualLogic iSCSI Storage Arrays. Disk subsystem bottlenecks that usually manifest as high disk latencies can cause performance bottlenecks that lead to poorly performing Exchange deployments.
Figure 4 ‐ Storage Level Performance
The disks read and write latencies are within the 20 millisecond threshold that is recommended for optimal performance of the Exchange 2007 deployment. The mailbox database disk latencies are observed to increment as the number of users are increased from 1000 to 4000, but still remain well within the 20 ms threshold. The storage subsystem was designed to support 4000 heavy users and took in to account both the capacity requirements and the performance requirements for this workload. A RAID 10 configuration was used for both database and log disks to meet the performance requirements of the LoadGen heavy workload profile.
Finally, in Figure 5 we compare the performance of the virtualized Exchange 2007 solution to a physical deployment to quantify the impact of virtualization. SendMail latency, CPU loads and RPC latency is
Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX
Page 9
compared for virtual and physical deployments of 1000‐3000 users. SendMail latency is plotted on the primary axis and CPU Load and RPC latency is plotted on the secondary axis. The send mail latency is observed to increase from 64 ms for 1000 users to 94 ms for 3000 users for both physical and virtual Exchange deployments. There is practically no difference in user perceived latencies for either deployment. There is an increase in CPU load for the virtualized Exchange deployment and this is an artifact of both virtualization overhead and running additional Exchange roles on the server. The increase however is minimal, does not affect end user performance as observed from the SendMail latencies and leaves adequate CPU headroom to handle any spikes in the workload.
Figure 5 ‐ Comparing Performance of Virtualized Building Block to Physical Exchange Deployment
Conclusion This study provides an overview of best practices and using a building block approach to deploy Microsoft Exchange Server 2007 on Dell Servers running VMware ESX. Using the Exchange 2007 solution building block proposed in this paper, IT administrators can design, configure and deploy Exchange Server 2007 using Dell servers, EqualLogic storage and VMware ESX virtualization software.
References [1] Planning Processor Configurations, Microsoft TechNet http://technet.microsoft.com/en‐us/library/aa998874.aspx
[2] Exchange 2007 Mailbox Server Role Storage Requirements Calculator http://msexchangeteam.com/files/12/attachments/entry438481.aspx
Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX
Page 10
[3] Dell Exchange Server 2007 Solutions http://www.dell.com/exchange
Appendix A
Figure 6 ‐ Network topology for Virtualized Building Block
Virtualizing Microsoft® Exchange® Server 2007 on Dell™ Systems Running VMware® ESX
Page 11
Figure 7 ‐ Storage Configuration Setup and Details
Top Related