VIRTUALIZING MICROSOFT EXCHANGE 2010 ON DELL POWEREDGE™ SERVERS AND STORAGEA DELL TECHNICAL WHITE PAPER

End-to-End Solutions Engineering

Dell Product Group

September 2010

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

© 2010 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.

Dell, the DELL logo, and the DELL badge, PowerConnect, and PowerVault are trademarks of Dell Inc. Symantec and the SYMANTEC logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the US and other countries. Microsoft, Windows, Windows Server, and Active Directory are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell Inc. disclaims any proprietary interest in trademarks and trade names other than its own.

September 2010

Content

sA DELL TECHNICAL WHITE PAPER.......................................................................................11. Introduction...............................................................................................................42. Overview...................................................................................................................5

2.1 Hardware Components.....................................................................................53. Deployment Best Practices for Virtualized Exchange Server 2010............................84. Reference Architecture for 4000 Mailbox Virtualized deployments.........................104.1 Performance Study............................................................................................145. Summary.................................................................................................................176. Reference and Additional Information.....................................................................18

1. IntroductionEnterprises are seeking ways to streamline IT budgets without affecting operations and limiting features to end users. Microsoft Exchange® 2010 introduces new performance enhancements, which can significantly reduce storage requirements, and extends high-availability features. Exchange Server 2010 is officially supported by Microsoft on virtualized infrastructures and if designed correctly, can further reduce the total cost of ownership and maximize return on investment. In this white paper, we design and evaluate a 4000 mailbox Exchange 2010 deployment on Dell’s Virtualization Business Ready Configuration (BRC).

Virtualization Business Ready Configuration’s (BRC) from Dell are tools that help customers to minimize complexity during planning, ordering, deployment and installation of their virtual infrastructure. Dell offers multiple BRC’s that provides the customer a choice between multiple server and storage platforms, and for their virtualization infrastructure software. The BRC is optimized for running virtualized workloads and provide optimal redundancy, scalability and manageability. This document covers an approach to virtualize Exchange 2010 on the Dell Virtualization BRC for a Small Medium Business. The approach used in this document is hypervisor agnostic, and therefore the configuration can be used for hypervisors from Microsoft, VMware or any third party vendor that is validated by Microsoft to be used for Exchange Virtualization.

This white paper describes using the Business Ready Configuration for Virtualization to implement an Exchange Server 2010 messaging infrastructure on a VMware® vSphere™ environment. It documents the Exchange 2010 architecture, server and storage configurations, virtual machine sizing and best practices to enable an efficient virtualized Exchange 2010 messaging infrastructure. The best practices presented are the result of installing, sizing and evaluating Exchange Server 2010 components on a Virtualization BRC at the End-to-End Solutions Engineering Lab. Experimental results from the testing form the basis for sizing and performance guidance presented in this white paper to architect an optimal Exchange 2010 solution

2. OverviewDell offers multiple BRC’s that provide the customer a choice between multiple server, storage and virtualization platforms. Customers use the BRC’s from Dell for quick and efficient ordering, deployment and installation of their virtual infrastructure. The BRC used to host the virtualized Exchange 2010 building block presented in this document is available at http://content.dell.com/us/en/enterprise/d/virtualization/BusinessReadyDataCenter.aspx. This BRC combines the Dell PowerEdge™ M-Series blade servers and EqualLogic™ PS6000 storage area network (SAN) with Cisco® networking switches and VMware® vSphere™ 4 technology to help achieve an automated and efficient data center that can be used to host Exchange 2010 Server roles along with other virtualized workloads.

2.1 Hardware Components

P o w e r E d g eM1 0 0 0 e

9 1 01 2 1 1 1 23 4 1 3 1 45 6 1 5 1 67 8

0

1

00

1

0 0

1

00

1

0 0

1

00

1

0 0

1

00

1

0

0

1

00

1

0 0

1

00

1

0 0

1

00

1

0 0

1

00

1

0

VMware vSphere 4

Storage Fabric

Network

VM VM VM VM

Pow

erEdge M

1000e M

610 blade servers

EqualLogic P

S6000

Series S

AN

Figure 1: Typical vSphere 4 Architecture with PS series SAN

The PowerEdge M610 is a two-socket blade server designed with energy-optimized options supporting the new Intel® Xeon® 5600 series processors, DDR3 memory and advanced embedded management capability.

The Dell EqualLogic PS6000 is an iSCSI storage device from Dell designed for deployment in enterprise business environments. It consists of four 1Gb Ethernet network ports per controller for data, faster processors, 2 GB of cache per controller, support for RAID 6, increased drive capacity, and a new monitoring application, SAN HQ, at no additional cost. For more information on Dell EqualLogic storage, see www.dell.com/equallogic.

In the following table, we provide a summary specification of the pre-configured solution. In the rest of the document, we will discuss reference architectures and best practices for deploying and using these solutions. The solution ID can be used to order or view the complete virtualization solution to deploy the reference architecture.

Table 1: Solution SpecificationHardware

Solution ID 1131816.1

Blade Chassis (1) M1000eESX Blade Server (6) x M610Storage Device (4) PS6000XVManagement Blade Server (1) x M610

Chassis ConfigurationI/O module for A1, C1, A2, C2

Cisco M 3130G

I/O module for B1, B2 Cisco M 3130G with (2) 1000BAS-T SFP modulesStacking cables 6Management Redundant Chassis Management Controllers (CMC)KVM Integrated Avocent keyboard, video, and mouse

(iKVM) switchESX Blade Server Configuration

Blade Server Model M610

Processor 2 x Intel Xeon X5660, 2.8Ghz, 12M Cache,Turbo, HT, 1333MHz Memory

Memory 96 GB (12 x 8 GB, DDR3)

Add-in Controllers (2) Broadcom 5709 Dual Port GbE I/O Card for M-Series Blades

Local storage and controller

Diskless configuration

Hypervisor VMware ESXi 4.0

Storage ConfigurationStorage Device (4) PS6000XV

Drives 16 X 600GB, 15K SAS

Storage Capacity 9.6 Terabyte capacity per Device

Management Blade ServerBlade Server Model (1) M610

Processor (2) x Intel Xeon (Nehalem) E5520, 2.26Ghz, 8M Cache

Memory 8 GB

Network Controller (2) Broadcom 5709 Dual Port GbE I/O Card for M-Series Blades

Virtualization Management S/W

VMware vCenter 4.0, 3Yr Upgrade Subscription

Software and ServicesServices 3 Year ProSupport for IT and Mission Critical 4HR

7x24 Onsite Pack EqualLogic PS Array Installation M1000e Blade Chassis On-site installation service

3. Deployment Best Practices for Virtualized Exchange Server 2010Microsoft started supporting Exchange Server on Virtualized hardware with Exchange Server 2007. Exchange Server 2010 is supported as well, assuming certain conditions are fulfilled. In terms of hypervisors, Windows 2008 with Hyper-V Technology, Microsoft Hyper-V Server and any third party hypervisor (such as VMware and Citrix offerings) that are validated under Window’s SVVP (Server Virtualization Validation Program) are supported. The guest VM running the Exchange Server 2010 roles has to be deployed on Windows Server 2008 SP2 or R2 to be supported. All server roles are supported in a virtualized environment, except for Unified Messaging role, which has to be on a physical server.

Other important configuration or deployment scenarios that to keep in mind –

Dynamic VHD disks are not supported; neither are snapshots, differencing or delta mechanisms

Exchange 2010 HA features like Database Availability Groups (DAGs) cannot be combined with hypervisor based clustering solutions.

Overprovisioning of virtual processor is not supported for a ratio greater than 2:1 Minimum size of the fixed disk for the Exchange 2010 operating system virtual

disk has to be 15Gb in addition to page file size (equal to the memory allocated for the VM).

In a supported configuration, server based applications cannot be run on the physical root machine of a server that is hosting the Exchange 2010 roles inside virtual machines. It is okay to run management software like anti-virus, backup software etc on the root partition.

Operating System disk images for the virtual machines have to be stored on serparate storage array or disks from the Exchange database or log storage.

Sizing the Exchange Server Mailbox Roles/VMsExchange Server 2010 is not virtualization aware and therefore the majority of the design principles used for a physical deployment will translate to the virtual deployment design. Virtual machines are sized similar to physical servers, depending on the Exchange server role that is being hosted on the virtual machine. The number of cores and memory per virtual machine will depend on the message profiles and mailbox sizes.

The following guidelines are recommended by Microsoft when deploying Exchange 2010 on a Hyper-V environment to size virtual machines hosting Exchange roles–

The Hub and CAS server roles are combined in a single VM. This VM is assigned 4 virtual CPUs and 8GB of memory. The number of multi-role Hub/CAS deployed is the same as the number of Mailbox VMs deployed, i.e. the ratio of Hub/CAS VMs to Mailbox VMs is 1:1.

The Mailbox server role is assigned four virtual CPUs. The approximate maximum number of mailboxes recommended per Mailbox VM is 3000 to 4000 depending on the user profile and concurrency. The memory assigned to each Mailbox VM is 4GB (for OS) + memory needed for mailbox database cache. The requirements per user for mailbox database cache size ranges from 3MB-9MB (light profile to very heavy profile).

In the design presented in this paper, we use a multi-role Exchange VM which combines the HUB/CAS/Mailbox roles in a single machine. The goal of this design is to use a part of the resources available in the Virtualization infrastructure, and share the hardware with other virtualized applications and workloads.

The number of drives required for database and logs is calculated using the same principles as a physical deployment. The Microsoft storage calculator is used to determine the storage design in terms of the number of drives and RAID type for the solution. For internal or direct attach storage configurations SCSI pass-thru is recommended where the volumes are presented directly to the Mailbox VMs for database and log volumes. For iSCSI SAN, the initiator is configured on the host server disk is again presented to the guest VM as a pass-thru disk to database and log storage.

4. Reference Architecture for 4000 Mailbox Virtualized deployments

The Agile Consolidated model from Dell for Exchange Server 2010 deployments are targeted for the virtualized datacenters that have already invested heavily in virtualization1. These datacenters are already running their Tier-2 and Tier-3 applications on VMs and looking at moving their Tier-1 apps on to VMs as well. The storage infrastructure for this class of deployment is on Dell SAN products such as Dell EqualLogic™ PS arrays and Dell/EMC CX4 and AX4 arrays.

A sample reference architecture for the Agile Consolidated Dell model using internal storage is shown in Error: Reference source not found. This architecture is designed for a 4000 mailbox with an I/O profile of 150 messages per day and mailbox sizes of up to 2GB. Table 2 lists the input variables that were used for this configuration.

Table 2 - Input Requirements for 4000 User Configuration

Input DetailNumber of mailboxes 4,000I/O Profile 150 Messages/DaySize of Mailbox Up to 2GBAverage Message Size 75KBNumber of Copies 2-copy DAG (1 active, 1 passive)

The high-availability for mailboxes is provided using built-in Exchange resiliency features thru the Database Availabiliy Group (DAG). The 4000 mailboxes are evenly distributed between four Exchange VMs, hosted on two separate physical servers. Two Database Availability Groups (DAGs) are created across the 4 VMs so that the mailbox copy is hosted on a separate physical server. The hypervisors are not part of a cluster since virtualization features are not used in this design to provide high-availability. The hypervisors not hosting Exchange, can be part of a cluster and applications running on them can be load balanced thru the virtualization software.

Each physical server hosts a multi-role Mailbox/Hub/CAS server VM. Error: Referencesource not found details the host and VM configurations to support 4000 users. Full

1 http://www.dell.com/downloads/global/solutions/security/exchange_2010.pdf

provisioning of resources is configured so that any server can handle a scenario where all the passive copies become active in the event of failure happening on a server or individual databases. High Availbility in the local site is provided by Exchange Database Availability Groups (DAGs) which provide protection at the application and server layers as well as the storage and database layers.

M1000e enclosure

(4) Cisco Catalyst Switches

with stacking

VM Network

Management Network

(2) Cisco Catalyst Switches

with stacking

(16) 1Gb active and(16) 1Gb passive

redundant connectionwith load balancing

(4) E

qual

Logi

c PS

6000

XVM1000e enclosure with (6) M610blade servers for ESX, (1) M610 forvCenter and (9) half height slot forfuture expansion

ESXi

Bla

de-1

ESXi

Bla

de-4

Expa

nsio

nEx

pans

ion

Expa

nsio

n

Expa

nsio

n

vCen

ter B

lade

(4) 10Gb Uplinkswith EtherChannel. VM, management and VMotion network Separated through VLANs. (4) more 10Gb uplinks for future expansion

Expa

nsio

nEx

pans

ion

Expa

nsio

n

ESXi

Bla

de-6

ESXi

Bla

de-5

Expa

nsio

nEx

pans

ion

ESXi

Bla

de-3

ESXi

Bla

de-2

LUNs

Cisc

o Ca

taly

st Sw

itche

s

Exchange VMs

Virtual Machines

Figure 2 - Agile Consolidated Model on VMware

Figure 3 shows the consolidated hardware infrastructure on which the 4,000 user Exchange 2010 deployment is installed. Two PowerEdge M610 servers are used to host the multi-role Exchange Virtual Machine. Dell EqualLogic iSCSI SAN storage is used to host the Exchange Mailbox database copies, logs, and the VM OS images. The arrays

are configured using RAID-10 to maximize performance and provide resiliency against disk failures at the storage level. Multiple volumes are used for storing the Exchange database and log files in addition to other volumes to hold the VM images and storage for other VMs.

Figure 3 shows the layout of the Exchange 2010 server roles and distribution of virtual machines on the servers. The 4,000 users are distributed across four Exchange VMs, each host running two Mailbox VMs. In the event of a server failure, Exchange DAG will activate the passive copy on working server to restore service to the failed mailboxes.

Configuration details for the server, storage and virtual machines are shown in . Each PowerEdge M610 is configured with two quad-core CPUs and 96 GB of memory so it can handle additional load in case of server failure on the other host. Each Mailbox VM instance is configured with 4v CPUs and 48GB of memory and handles 1000 active mailbox users and 1000 passive users. The hosts are connected to the iSCSI SAN using the software iSCSI initiator in the host hypervisor.

1

0

Active Copy (first copy)

Passive Copy (second copy)

Exchange VM1

Exchange VM2

1

0

Exchange VM3

Exchange VM4

DB3 DB4

DB8 DB7

DB1 DB2

DB6 DB5

DB1 DB2

DB5 DB6

DB3 DB4

DB7 DB8

MB VM1 MB VM2 MB VM3 MB VM4

Pow

erE

dge

M61

0

Pow

erE

dge

M61

0

Equ

alLo

gic

PS

6000

XV

DAG2DAG1

Figure 3 - Exchange 2010 Reference Architecture for 4000 Users

Table 3 – Server and Storage Configurations

Solution Output DetailHost Server Configuration

2 PowerEdge M6102 x Quad-Core CPUs 6.4 GT/s 96 GB DDR4 Memory

Mailbox/HUB/CAS VM 2VM’s per Host4 vCPU, Memory per VM: 48GB 1000 Mailboxes Active and 1000 Mailboxes Passive per Virtual Machine

Storage Configuration 4 x EqualLogic PS6000XV iSCSI Enclosures16 x 3.5” 600 GB 15K RPM SAS disks per enclosure

RAID-10 across the entire arrayDatabase ConfigurationDatabases 4 database per Mailbox VM (data and logs combined per

database)Total 16 databases (8 active and 8 passive databases)

Mailboxes 500 Mailboxes per DBUp to 2GB capacity per Mailbox

4.1 Performance StudyA smaller configuration of the reference architecture presented was tested and evaluated in the Dell enterprise solutions lab to validate the architecture and gather experimental data. Figure 4 illustrates the configuration that was tested - VMware ESX Server v4.1 is installed on both Dell PowerEdge M610 servers. Each host is running a single Guest VM installed with Microsoft Windows Server 2008 R2 Enterprise Edition. Exchange 2010 roles are installed on the guest virtual machines and host 1000 active and 1000 passive mailboxes each. The mailbox VM is assigned four virtual CPUs and 24GB of memory. It was configured with 2 vNICs, one connected to the public server network and the other was a private network configured for the DAG (Database Availability Group). The Exchange 2010 database and logs are hosted on volumes created on the externally connected EqualLogic PS6000XV array.

1

0

Active Copy (first copy)

Passive Copy (second copy)

Exchange VM1

1

0

Exchange VM2

Pow

erEd

ge

M61

0

Pow

erEd

ge

M61

0

EqualLogic PS6000XV

VM1

DB2DB1 VM2

DB2 DB1

DAG

Figure 4 - Performance Study Configuration and Setup

Figure 5 shows the average SendMail and 95th Percentile Send Mail latencies measured when simulating a 1000 user mailbox deployment on each virtual machine. The 95 th

percentile latencies are well within the 1-second threshold recommended by Microsoft for adequate end user performance. The SendMail latencies are still within the threshold

DAG

DAG

when a Database Availability Group (DAG) is created across the two Exchange VMs and they host the passive copy of the databases.

70 69 71 77

223 218

245 249

0

50

100

150

200

250

300

VM1 1K Mailboxes

VM2 1K Mailboxes

VM1 2K Mailboxes

(1K Active + 1K Passive)

VM2 2K Mailboxes

(1K Active + 1K Passive)

Late

ncy

(ms)

Send Mail Latency 95th Percentile SendMail Latency

Figure 5 - Microsoft LoadGen Send Mail Latencies

Figure 6 illustrates the CPU and Memory usage for the ESX host and Virtual Machines when running 2000 mailbox LoadGen simulation on the standalone configuration and DAG configuration. As expected there is a minor increase in both CPU and memory usage for the DAG configuration as the Exchange server is hosting the passive copy in addition to the active mailboxes.

3 2 3 4

119

1315

25537 25376 25950 25922

18948 1877721029 20658

0

5000

10000

15000

20000

25000

30000

02468

10121416

VM1 1K Mailboxes

VM2 1K Mailboxes

VM1 2K Mailboxes

(1K Active + 1K Passive)

VM2 2K Mailboxes

(1K Active + 1K Passive)

Mem

ory C

onsu

mpti

on (

Mby

tes)

CPU

Util

izatio

n (%

)

Physical Server CPU Utilization VM CPU Utilization Memory Usage - Physical Server Memory Usage - VM

Figure 6 - CPU and Memory Usage for ESX Server and Exchange VM

The database and log performance metrics are shown in the following figures. In the stand alone configuration there are no log reads being performed. When a DAG is

DAG

DAG

established, we see log reads on both Exchange VMs as an artifact of the DAG functionality. As expected the DB and log I/O’s are significantly higher in a DAG configuration due to the maintaining a database copy. The DB read latencies are observed to increase in the DAG configuration as well, but are within the recommended thresholds by Microsoft for acceptable performance. The latencies presented in the DAG configuration is a sum of the attached and recovery latencies – the attached read latency is still within 10ms which is half of the recommended threshold by Microsoft for optimal end user Exchange performance.

114

42

14

47 47

87 86

0 0 2.9 2.7

67 6973

68

0

10

20

30

40

50

60

70

80

90

100

VM1 1K Mailboxes

VM2 1K Mailboxes

VM1 2K Mailboxes

(1K Active + 1K Passive)

VM2 2K Mailboxes

(1K Active + 1K Passive)

DB Reads/Sec DB Writes/SecLog Reads/Sec Log Writes/Sec

Figure 7 - Database Throughput Performance

8.5 9.0

13.3

21.1

2.11.2

2.0 1.50 0

2.6 2.41.01 0.71 0.78 0.82

0.0

5.0

10.0

15.0

20.0

25.0

VM1 1K Mailboxes

VM2 1K Mailboxes

VM1 2K Mailboxes

(1K Active + 1K Passive)

VM2 2K Mailboxes

(1K Active + 1K Passive)

Late

ncy

(ms)

DB Read Latency DB Write LatencyLog Read Latency Log Write Latency

Figure 8 - Database and Log Latency

Figure 9 shows how the ESX host and Exchange VM are affected when a DB failiure occurs on one of the Mailbox VMs. We simulate a failure on VM2 and this makes the copy on VM1 active, and it is therefore running 2000 active mailboxes. This was done to validate that the VM resources were adequate to handle a failure scenario at the database or host level. The chart plots the impact on resource usage over normal operation when the VM is running 1K active and 1K passive mailboxes.

2.3% 1.8% 0.0%5.4%

26.1%

63.4%

23.4%

91.7% 92.1%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

ESX Host CPU Utilization

VM CPU Utilization

Memory Usage ESX Host

Memory Usage Virtual

Machine

MS ExchIS RPC Avg Latency

MS ExchIS RPC Operations/sec

MS ExchIS Mailbox

Messages Queued

MS ExchIS Mailbox

Messages Delivered/Sec

MS ExchIS Mailbox

Messages Submitted/Sec

Perf

orm

ance

Impa

ct

Relative Performance of VM with 2K active Mailboxes vs. VM with 1K Active + 1K Passive Mailboxes

Figure 9 - Performance Impact when simulating a database faliure

At the Exchange application level we observe an increase in the RPC latency, operations/sec and messages being queued. This is expected behavior since all mailbox operations that was being handled by the two VMs are now being handled by a single VM. One VM is now hosting 2K active mailboxes, as compared to 1K active and 1K passive mailboxes. Hosting passive mailboxes does not incur RPC or Exchange related load on the VM, but has in impact on the storage subsystem. Since the VM was sized adequately to handle this addition load of 1K active mailboxes, there is only a minor increase in the resource utilization and end user performance is not impacted as verified by the SendMail latencies in the two scenarios. The DB read and write latencies were verified to be less than 10 ms which is well within the recommended threshold by Microsoft, proving that we have enough resources allocated to the VM to handle a database failure.

5. SummaryThis paper presented a reference architecture for deploying Exchange Server 2010 on virtualized Dell infrastructure. A four thousand user architecture was presented using the agile consolidated model from Dell. The high-availability for mailboxes was provided using built-in Exchange resiliency features thru the Database Availabiliy Group (DAG). The performance of the proposed architecture was evaluated using a scaled-down version and test results showed that the configuration was able to handle a 2000 mailbox deployment efficiently. The test setup can be scaled up to handle 4000 users without adding more blades since the CPU and memory usage was very minimal. Adding additional EqualLogic arrays result in a linear performance increase and will

handle the I/O requirements for 4000 mailboxes. This setup can be shared with other virtualized applications, and scaled to accommodate future growth for the organization.

6. Reference and Additional Information

Dell Exchange 2010 Sizing Guidehttp://www.dell.com/downloads/global/solutions/Introduction_to_Exchange_2010_Sizing.pdf

Microsoft’s Exchange 2010 Mailbox Server Role Requirements Calculatorhttp://msexchangeteam.com/archive/2009/11/09/453117.aspx

Dell Architecture Models for Exchange 2010http://www.dell.com/downloads/global/solutions/security/exchange_2010.pdf

VMware vSphere™ Reference Architecture for Small Medium Business http://i.dell.com/sites/content/business/solutions/engineering-docs/en/Documents/VMware-vSphere-Reference-Architecture-SMB.pdf

Documentation for VMware vSphere 4 for Dell PowerEdge Servers http://support.dell.com/support/edocs/software/eslvmwre/VS/VS.htm

VMware vSphere 4 iSCSI SAN Configuration Guide http://www.vmware.com/pdf/vsphere4/r40/vsp_40_iscsi_san_cfg.pdf

VMware Network Concepts http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

Configuring VMware vSphere Software iSCSI with Dell EqualLogic PS Series Storage http://www.equallogic.com/resourcecenter/assetview.aspx?id=8453