Using Clustering With Exchange 2003

8/8/2019 Using Clustering With Exchange 2003

1/40


2/40

Topics Discussed in This White Paper Updated: September 1, 2003

This paper provides detailed information about the hardware, software, and configuration of the Exchange

server cluster, and explains why OTG chose the configuration. It also includes best practices for using

server clusters, as well as information about several operational procedures that OTG uses to help make

messaging highly reliable and available. The following topics are discussed.

Reasons for Using Server Clusters

OTG Objectives

Server Cluster Overview

How OTG Configures the Exchange Server Cluster

How OTG Backs Up the Server Cluster

How OTG Monitors the Server Cluster

Best Practices for Server Clusters

Reasons for Using Server ClustersUpdated: September 1, 2003

This section discusses specific clustering features that help the Operations and Technology Group (OTG)

achieve their objectives.

A server cluster is a collection of servers known as nodes that together provide a single, highly available

system for hosting applications. Multiple nodes in a cluster remain in constant communication. A clusters

resources and applications are organized into functional units called resource groups. Each resource group

is assigned to a node and can only be owned by a single node at any point in time.

There are three primary benefits to using server clusters. Server clusters are highly available, scalable,

and easy to manage.

Highly available: Server clusters provide a highly available system for deploying applications.

You can use server clusters to protect against failures of hardware, operating systems, device

drivers, or applications. If one of the nodes in a cluster is unavailable as a result of failure or

maintenance, another node immediately begins providing service (a process known as failover).

Server clusters also allow you to upgrade the operating system and application software with

minimal downtime.

Scalable: Server clusters provide scalability for important resources and applications. When the

overall load of a cluster exceeds its capabilities, you can incrementally add additional nodes tothe cluster. In Windows Server 2003, server clusters can include up to eight nodes.

Easy to manage: In a server cluster, you can quickly inspect the status of all cluster resources

and move workloads around onto different nodes. Because you can move processing to alternate

nodes, you can perform rolling upgrades on the servers. In a rolling upgrade, a server cluster

continues to provide service while software is being upgraded on each node until all nodes have

been upgraded.


3/40

Microsoft has used these cluster features to help achieve its technical and business objectives. These

objectives are described in the following section.

OTG ObjectivesUpdated: September 1, 2003

The Operations and Technology Group (OTG) at Microsoft has used clustering to achieve the followingtechnical and business objectives for messaging within the company:

High availability. The server cluster discussed in this document provides a messaging solution

with a high level of availability to users. OTG strives to make the Exchange servers available

99.99% of the time.

Scalability . In a server cluster, administrators can add additional servers as necessary in a way

that is virtually transparent to the users. Because they have implemented clustering to manage

the Exchange servers (and for other reasons), OTG has been able to increase the scale limitation

of the mail store from 3000 mailboxes (100 MB per mailbox) to 5000 mailboxes (200 MB per

mailbox).

Manageability . By using a server cluster, administrators can manage and maintain the

Exchange servers through a process that is virtually transparent to users.

Consolidation. Server clusters can help administrators consolidate their resources. Server

clusters are helping OTG consolidate their worldwide messaging resources from 75 locations to

seven and from 115 servers to 30.

These objectives are similar to those of most businesses who use Exchange 2003. Therefore, the server

cluster information presented in this document should be valuable to businesses with similar goals.

Server Cluster OverviewUpdated: September 1, 2003

This section gives a general overview of server clusters and the Exchange server cluster at Microsoft that

supports messaging within the company.

The following terminology is used in this section and throughout the document.

Active node. The cluster server that currently owns cluster group resources and responds to

network requests made to those services.

Alternative passive node. The cluster server that does not currently own cluster group resources

but is available if the active fails over and the primary passive node is unavailable.

Primary passive node. The cluster server that does not currently own cluster group resources but

is available if the active node fails over.

Virtual server. A collection of services that appear to clients as a physical Windows-based server

but are not associated with a specific server. All virtual servers must include a Network Name

resource and an IP Address resource.

Server clusters can take two forms: active/passive clusters and active/active clusters. In active/passive

clustering, the cluster includes active nodes and passive nodes. The passive nodes are only used if an


4/40

active node fails. In active/active clusters, all nodes are active. In the event of a failover, the remaining

active node takes on the additional processing operations, which causes a reduction in the overall

performance of the cluster. Active/passive cluster configurations are generally recommended over

active/active configurations because they often increase performance, availability, and scalability.

The Exchange server cluster is based on an active/passive model, with a few alterations that maximize the

performance of the hardware. The Operations and Technology Group (OTG) has configured a multinodecluster that consists of four active nodes, one primary passive node, and two alternate passive nodes. The

four active nodes provide most of the processing operations for the messaging environment. Like most

active/passive configurations, the primary passive node is used only when one of the active nodes fails or

to support the process of rolling upgrades. The two alternate passive nodes are only used for messaging if

failover occurs and the primary passive node is not available. To take full advantage of the available

hardware, the alternate passive nodes are also used to facilitate the backup and upgrade processes. This

unique configuration supports a total of 16,000 mailboxes.

The following topics are described in this section:

Diagram 1. Exchange Server Cluster

Reasons for Using Alternative Passive Nodes

Diagram 2. Failover Process of the Server Cluster

How OTG Configures the Exchange Server Cluster


5/40

Diagram 1 Exchange Server Cluster

Diagram 1 shows the four active nodes, the primary passive node, and the two alternate passive nodes

that make up the Exchange server cluster. All of the nodes are connected to two storage area networks

(SANs). The two SANs provide communication between the nodes and the shared storage components,

which are two redundant Enterprise Virtual Arrays. For more information about how storage is configured

in this cluster, see Storage Components later in this document.


6/40


7/40

settings that are not specifically related to server clusters. However, each server cluster setting is

discussed in depth.

For information about hardware configuration, see Hardware in the Server Cluster.

For information about the software configuration, see Software in the Server Cluster.

For information about the server cluster configuration, see Server Cluster Settings.

Hardware in the Server Cluster

This section describes the hardware configuration for each of the following components of the server

cluster:

Active and Primary Passive Nodes

Alternate Passive Nodes

Network Adapters

Storage Components

Active and Primary Passive Nodes

The four active nodes and the primary passive node in the server cluster all use the same hardware

configuration, which is based on the HP ProLiant DL580-G2 processor with the following specification. Each

active node supports 4000 mailboxes. Because there are 4 active nodes, this server cluster supports

approximately 16,000 mailboxes.

Processors HP DL580-G2 Quad - 1.90 GHz, Hyperthreading-enabled

Memory 4 GB of Error Correction Coding (ECC) RAM

Controller Array controller - Integrated 5i

Hard disk drives 2 x 36 GB RAID-1:

C$ Windows Server 2003, Enterprise Edition

D$ Exchange Application

Explanation

OTG chose this hardware to provide optimized performance during peak usage periods, which are typically

experienced on Monday mornings. OTG enabled hyperthreading on these nodes, which has increased

performance of the node by 20-25%. (Hyperthreading is a technology that Intel developed that enables

multithreaded software applications to execute threads in parallel on a single multicore processor instead

of processing threads in a linear fashion). In addition, by increasing the performance, each node can

manage more mailboxes.


8/40

The hardware for the primary passive node was chosen for the same reasons as the active node

hardware. In the event of a failover, the passive node needs to perform exactly like an active node in

order to maintain dependable messaging.

Note

Users who are implementing Exchange 2003 or Windows Server 2003 should refer to the WindowsServer Catalog (http://go.microsoft.com/fwlink/?LinkId=4303) before buying hardware to ensurecompatibility. The Windows Server Catalog is the online source for customers and partners to findhardware and software that is compatible with the Windows Server 2003 family.

Alternate Passive Nodes

The two alternate nodes in the server cluster have a hardware configuration that is based on the HP

ProLiant DL380-G2 dual processor with the following specification.

Processors DL380-G2 dual - 2.4 GHz

Memory 2 GB RAM

Controller Array controller - Integrated 5i

Hard disk drives 2 x 36 GB RAID1:

C$ Windows Server 2003, Enterprise Edition

D$ Exchange Application

Explanation

As discussed earlier in this paper, OTG chose hardware for the alternate nodes to increase resource

efficiency. With this hardware, the alternate passive nodes can be used to facilitate the backup processupdates to the system.

For more information about why OTG uses alternative passive nodes, see Reasons for Using Alternative

Passive Nodes.

Note

Users who are implementing Exchange 2003 or Windows Server 2003 should refer to the WindowsServer Catalog (http://go.microsoft.com/fwlink/?LinkId=4303) before buying hardware. The WindowsServer Catalog is the online source for customers and partners to find hardware and software that iscompatible with the Windows Server 2003 family.

Network Adapters

Each node in the server cluster has two network adapters installed. OTG has used NC7770 Gigabit Server

Adapters that are connected to 100-MB switches.

The following settings appear on the General tab that is available when you configure or view the

properties of the adapters in Cluster Administrator. You can use Cluster Administrator to configure a

cluster and its nodes, resource groups, and resources.
http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303http://go.microsoft.com/fwlink?linkid=4303


9/40

Adapter 1

This network adapter enables node-to-node communication. It is connected to the private network that is

used exclusively in the cluster. Adapter 2 carries all other communication.

Configuration setting Explanation

Name:

Internal

OTG has changed the default name for this network connection to clearlyidentify that it is used for internal communication within the network.

Enabled for clusteruse is checked.

This adapter is available for the cluster to use.

Adapter 2

This network adapter enables both node-to-node (internal) and client-to-cluster (public) communication.

Because this network adapter carries both kinds of communication, it can serve as a backup for Adapter 1.


Name:

Internal + Public

OTG has changed the default name for this network connection to clearlyidentify that it is used for all communication within the network.

Enabled for cluster useis checked.

This adapter is available for the cluster to use.

Note

It is important to configure the private network to be the highest priority for internal clustercommunication in the Cluster Administrator. It is also important to make sure the public network isfirst in the TCP/IP binding order.

Explanation

OTG uses two networks to ensure network availability for internal cluster communications. Both adapters

are configured with static IP addresses. The use of dynamic IP addresses is not recommended in server

clusters.

Note

The nodes of a cluster must be on the same subnet, but you can use virtual LAN (VLAN) switches onthe interconnects between two nodes. (An interconnect is a private network that connects nodes in acluster). If you use a VLAN, the point-to-point, round-trip latency must be less than 1/2 second andthe link between two nodes must appear as a single point-to-point connection from the perspective of the Windows operating system running on the nodes. To avoid single points of failure, use independentVLAN hardware for the different paths between the nodes.

Storage Components

This section will discuss the storage design for the Exchange server cluster and the reasons for this

configuration.

The following terminology is used in this section and throughout the document.


10/40

Disk group. A group of physical disks. Virtual disks are created across the disk groups.

Storage group. An Exchange 2003 component that supports five databases (mail stores) using

either a single or multiple data device with a common log device.

Virtual server. A collection of services that appear to clients as a physical Windows-based server,

but are not associated with a specific server. All virtual servers must include a Network Name

resource and an IP Address resource.

Storage area network (SAN). A high-speed network that provides a direct connection between

servers and storage, including shared storage, clusters, and disaster-recovery devices.

All of the nodes in the cluster are connected to two SANs, which provide communication between the

nodes and the two shared storage components.

The shared storage components are two redundant Enterprise Virtual Arrays (HP StorageWorks EVA

5000s). Diagram 1 illustrates the relationship of the nodes and the storage components. Diagram 3

describes the storage configuration for each Exchange virtual server. OTG configured each EVA to support

four Exchange virtual servers and a total of 16,000 200-MB mailboxes.

The following table summarizes the storage configuration for each EVA. Each EVA has five disk groups that

are characterized by different numbers of hard disks, data types, and Vraid types.

EVA disk group Number of 72-GB disks Data type Vraid type

DG1 48 Data/SMTP 1

DG2 48 Data/SMTP 1

DG3 14 Log/Quorum 1

DG4 28 Backup 5

DG5 28 Backup 5

Explanation

OTG designed this cluster for optimum performance to support production requirements for peak loads.

They also strived to ensure maximum availability by eliminating as many single points of failure within the

environment. OTG achieved optimum performance by scaling the servers to support a realistic number of

mailboxes and optimizing the SAN design for peak disk transfers (4K random read/write activity). OTG

analyzed I/O requirements for each 200 MB mailbox and determined that each mailbox would generatebetween 1.0 and 1.2 disk transfers for peak loads (this is specific to the user environment at Microsoft).

With a plan to support 4000 mailboxes for each Exchange virtual server, OTG determined that each host

would need to sustain 4800 disk transfers per second and also have the ability to support any unexpected

increase in demand.


11/40

OTG needed to optimize the SAN design in order to anticipate and account for peak disk transfers while

maintaining optimum read and write latencies. They also needed to ensure that Exchange would not

encounter any bottlenecks.

After evaluating the performance of the HP StorageWorks EVA 5000s, OTG found that the EVA supported

12,000 transfers in an optimized configuration with acceptable latencies, even under an aggressive 4 KB

random I/O load. As a result, OTG implemented two EVA 5000s per cluster. These EVAs support four

Exchange virtual servers and a total of 16,000 200-MB mailboxes.

OTG uses HP StorageWorks Secure Path, which is a multipath application, to manage and maintain

continuous data access to the storage system. Specifically, they use Secure Path to assist with optimizing

disk I/O activity by ensuring disk distribution across fiber channel adapters (FCAs) and SAN controllers on

each node. (Fiber channel adapters are expansion cards used to connect the nodes to SANs). Disk

distribution has proven to be beneficial in maintaining an optimized environment. It has greatly reduced

peak read/write disk latency and has improved online backup to disk.

OTG also uses Secure Path to assist with eliminating single points of failure between nodes and the

storage. Secure Path helps maintain I/O even if a node fails. In the event of a node failure, Secure Pathdetects the failure and moves disks from the failed path to an available path. This process is called

failover. Failover keeps the resources available while requiring no downtime. Once the failed nodes are

replaced, the disks can be failed back using HPs Secure Path Manager to restore optimized I/O.


12/40

Diagram 3 Storage Configuration for Each Exchange Virtual Server

Diagram 3 depicts the storage configuration for one of the Exchange virtual servers. OTG configured each

EVA to support four Exchange virtual servers and a total of 16,000 200-MB mailboxes. Each Exchange

virtual server hosts four storage groups with five databases each. Each database is configured with a 200-

MB limit per mailbox. This design specifies a maximum of 200 mailboxes per database, so each Exchange

virtual server contains 4,000 mailboxes.

For additional information about the storage configuration for each virtual server, see Appendix A. Formore information about Enterprise Virtual Arrays, see Microsoft Exchange 2003 Storage Design for the

EVA5000 and other related white papers in the Storage Array Systems White Papers section of the

Hewlett-Packard Web site (http://go.microsoft.com/fwlink/?LinkId=19716).

Software in the Server Cluster

The nodes in the server cluster have the following software installed.
http://go.microsoft.com/fwlink/?linkid=19716http://go.microsoft.com/fwlink/?linkid=19716


13/40

Node Type Software

Active node and primary passive node Windows Server 2003, Enterprise Edition

Exchange 2003

Microsoft Operations Manager (MOM)

Agent

HP StorageWorks Secure Path

Alternative passive node Windows Server, 2003 Enterprise Edition

Exchange 2003

Microsoft Operations Manager (MOM)

Agent

HP StorageWorks Secure Path

Veritas Backup Exec

OTG has made the following operating system modifications on each node:

/3-GB switch set in the Boot.ini file.

/USERVA=3030 parameter set in the Boot.ini file.

In the registry, SystemPages is set to 0.

Cluster event log replication is set to 0 in Event Viewer. Because OTG uses Microsoft Operations

Manager to manage their clusters, this setting is set to 0 to eliminate event replication across all

nodes in the cluster.

Mount points are used to support the log, SMTP, and backup drives in order to reduce the

number of drive letters used.

Server Cluster Settings

This section describes how OTG has configured the settings for the Exchange server cluster and the

reasons why they chose the settings. It also explains why some settings are specific to OTG and are not

recommended for typical customer deployments. This section describes the settings that are configured

for the server cluster as a whole, the settings that are specific to each resource group in the cluster, and

each of the resource configurations.

The server cluster contains four Exchange virtual servers, four backup virtual servers, and one cluster

virtual server. A virtual server is a collection of services that appear to clients as a physical Windows-

based server, but are not associated with a specific server. All virtual servers must include a Network

Name resource and an IP Address resource.


14/40

Each of the virtual servers contains the following specified resources. For server clusters, a resource is a

physical or logical entity that is capable of being managed by a cluster, brought online and taken offline,

and moved between nodes. A resource can be owned only by a single node at any point in time.

Virtual Server Resources

Exchange virtualservers

Nine Physical Disk resources

One Network Name resource

One IP Address resource

The following Exchange 2003 resources:

Exchange Message Transfer Agent instance (this resource is only

in the first Exchange instance in the cluster)

Exchange Routing Service instance

SMTP virtual server instance Exchange HTTP virtual server instance

Exchange IMAP4 virtual server instance

Exchange POP3 virtual server instance

Search Service instance

Exchange System Attendant

Exchange Information Store

Backup virtualservers

Four Physical Disk resources Four File Share resources



Cluster virtualserver

One Physical Disk resource



One Distribution Transaction Coordinator resource

One Majority Node Set resource

The following sections describe how OTG has configured the settings in Cluster Administrator. You can use

Cluster Administrator to configure a cluster and its nodes, resource groups, and resources.

Cluster Properties


15/40

Exchange Virtual Server Properties

Backup Virtual Server Properties

Cluster Virtual Server Properties

Resource Settings

Cluster Properties

This section describes how OTG has configured the server cluster as a whole.

The following settings appear on the Quorum tab that is available when you configure or view the cluster

properties.

Configurationsetting Explanation

Quorumresource:

Majority NodeSet

In a majority node set server cluster, each node maintains its own copy of the clusterconfiguration data. The Majority Node Set resource, acting as the Quorum Resource,ensures that the cluster configuration data is kept consistent across the differentnodes. The majority node set cluster is only meant for use in targeted scenarios like ageographically dispersed cluster, which is a cluster that spans multiple sites.

OTG has used a majority node set server cluster primarily for validation and testingpurposes. Majority node set clusters also reduce the likelihood of a single point of failure in certain hardware devices (such a host bus adapters or fibre switches), whichcan cause the entire cluster to fail.

Important

Do not configure your cluster as a majority node set cluster unless you have aspecific need to do so and it is part of a cluster solution offered by your Original

Equipment Manufacturer (OEM), Independent Software Vendor (ISV), orIndependent Hardware Vendor (IHV). The single quorum device server clusterwill still cover the vast majority of your cluster deployment needs.

Reset quorumlog at:

4096 KB

The maximum size of the quorum log is 4096 KB. This is enough space to hold thecluster configuration information, such as which servers are part of the cluster, whatresources are installed in the cluster, and what state those resources are in (forexample, online or offline).

The following settings appear on the Network Priority tab that is available when you configure or view

the clusters properties.


Networks used for internalcluster communications:

Internal

Internal + Public

The Internal network has the highest priority. The Internal + Publicnetwork will only be used if the Internal network becomesunavailable.


16/40

The following settings appear on the Security tab that is available when you configure or view the

clusters properties.


Group or user names:

Administrators have

full Control

Network Service has

full Control

System has full

Control

These groups have permission to alter the configuration of thiscluster.

Exchange Virtual Server Properties

This section describes how OTG has configured the Exchange virtual servers in the server cluster.

OTG has configured four Exchange virtual servers, which each host four storage groups. These storage

groups contain a total of 20 databases. Each database is configured with a 200-MB limit per mailbox. This

design specifies a maximum of 200 mailboxes per database, so each Exchange virtual server can contain a

maximum of 4,000 mailboxes.

The following settings appear on the General tab that is available when you configure or view properties

for the Exchange virtual servers.


Preferred owners:

Ac

tive node 1

Pri

mary

passive

node

Alt

ernate

passive

node 1

Alt

ernate

passive

The Exchange virtual server will fail over to the indicated nodes in the specifiedorder. If the active node fails, the Exchange virtual server will fail over to theprimary passive node. If the primary passive node is unavailable, the Exchangevirtual server will fail over to the alternate passive nodes. If none of the nodesare available, this resource group will fail over to a randomly selected node (thisalso depends on the possible owner settings for its resources).


17/40

node 2

The following settings appear on the Failover tab that is available when you configure or view properties



Threshold setto 10

Period set to 6

The Exchange virtual server is allowed to fail 10 times within 6 hours before it is leftin a failed state. The 11th time a resource in the resource group fails, the Clusterservice fails all other resources in the group and leaves the entire group offlineinstead of failing over the group.

The following setting appears on the Failback tab that is available when you configure or view properties



PreventFailback ischecked.

The Exchange virtual server will not fail back when a failed node returns tooperation. Instead, the Exchange virtual server will continue to run on the alternatenode after the failed node comes back online. This requires administratorintervention to move the resource group back to the original, preferred node. OTGprefers this method because the administrator has complete control over when thefailback occurs and can coordinate this to give users the least amount of downtime.

Backup Virtual Server Properties

This section describes how OTG has configured the four backup virtual servers in the server cluster. For

more information about how the backup process works, see How OTG Backs Up the Server Cluster later in

this document.

The following setting appears on the General tab that is available when you configure or view the

properties for the backup virtual servers.


There are nopreferred ownersfor this resource

group.

OTG has created various scheduled tasks that control the movement of thebackup virtual servers between nodes. The scheduled tasks force the backupvirtual servers to move between the passive and active nodes, depending on

the stage of the backup process that is being completed. The movement of thebackup virtual servers between nodes is also determined by the possibleowner settings of their resources.

The following settings appear on the Failover tab that is available when you configure or view the



18/40


Threshold setto 10

Period set to 6

This resource group is allowed to fail 10 times within 6 hours before it is left in afailed state. The 11th time a resource in the group fails, the Cluster service fails all

other resources in the group and leaves the entire group offline instead of failingover the group.

The following setting appears on the Failback tab that is available when you configure or view the




This resource group will not fail back when a failed node returns to operation.Instead, it will continue to run on the alternate node after the failed node comesback online. This requires administrator intervention to move the group back to the

original, preferred node. OTG prefers this method because the administrator hascomplete control over when the failback occurs and can coordinate this to make surethat users experience the least amount of downtime.

Cluster Virtual Server Properties

This section describes how OTG has configured the cluster virtual server in the server cluster. The cluster

virtual server is the owner of the cluster quorum resource. The quorum resource maintains the

configuration data that is necessary for recovery of the cluster. This data contains details of all of the

changes that have been applied to the cluster database.

The following setting appears on the General tab that is available when you configure or view properties

for the cluster virtual server.


Preferredowners:

Pr

imary

passive

node

This resource group will first attempt to fail over to the primary passive nodebecause it is best equipped to host the group. If this node is not available, thisgroup will fail over to a randomly selected node (this also depends on thepossible owner settings of its resources).

The following settings appear on the Failover tab that is available when you configure or view the

properties for the cluster virtual server.


Threshold set This resource group is allowed to fail 10 times within 6 hours before it is left in a


19/40

to 10

Period set to 6

failed state. The 11th time a resource in the group fails, the Cluster service fails allother resources in the group and leaves the entire group offline instead of failingover the group.

The following settings appear on the Failback tab that is available when you configure or view the

properties for the cluster virtual server.



This resource group will not fail back when a failed node returns to operation.Instead, it will continue to run on the alternate node after the failed node comesback online. This requires administrator intervention to move the group back to theoriginal, preferred node. OTG prefers this method because the administrator hascomplete control over when the failback occurs and can coordinate this to make surethat users experience the least amount of downtime.

Resource Settings

This section describes how each of the resources used in the server cluster is configured.

All resource settings in the server cluster are the default values with the following exceptions. These

exceptions appear on the Advanced tab that is available when you configure or view the properties for

the resource:

Restart is not enabled for the Exchange Information Store and the Exchange System Attendant

resources.

Restart is enabled for all other resources (default) but the Affect the Group option is disabled.

For more information regarding why these changes were made, see each specific setting below

For more information about cluster resources, see the information about standard resource types at the

Microsoft Web site (http://go.microsoft.com/fwlink/?LinkId=19502).

For more information about Exchange resources, see Product Documentation for Exchange at the

Microsoft Web site (http://go.microsoft.com/fwlink/?LinkId=116208).

Note

Each resource of a specific type is configured identically unless otherwise noted.

IP Address Resource

The following settings appear on the General tab that is available when you configure or view the IPAddress resource properties.

http://go.microsoft.com/fwlink/?linkid=19502http://go.microsoft.com/fwlink/?linkid=116208http://go.microsoft.com/fwlink/?linkid=19502http://go.microsoft.com/fwlink/?linkid=116208


20/40

All nodes are possibleowners .

This resource can run on any of the available nodes, which allows failoverto occur. During failover, this resource will fail over to a node based on thepreferred owner settings of the resource group it belongs to.

Run as a separateResource Monitor is not

checked.

This resource does not run on a separate memory space.

The following setting appears on the Dependencies tab that is available when you configure or view the

IP Address resource properties.


This resource has noResource Dependencies .

The Cluster service uses the dependencies list when bringing resourcesonline and offline. This resource does not require another resource tooperate.

The following settings appear on the Advanced tab that is available when you configure or view the IP

Address resource properties.


Restart is not enabled.

The Affect the group checkbox is not selected (with athreshold of 3 and a period of 900 seconds).

This resource restarts after a resource failure.

The entire resource group will not be failed over when this resourcefails. When this resource is in a pending state, the Cluster service willattempt to restart the resource 3 times within 900 seconds before theCluster service assigns the resource to Offline or Failed status.

Important

All resources that restart after a resource failure are notconfigured to Affect the group . This is not a default setting.This allows OTG to track and fix any problems within the cluster.If a resource failure occurs, this setting allows OTGadministrators to notify the Microsoft developers of possible bugsin the code.

This setting also inhibits a cluster resource group from alternatingbetween nodes when a resource fails, which would result in significantinstability. This problem can result when a failed disk has defaultproperties that cause a series of bus resets. The resource group thenattempts to acquire reservation on a disk that does not exist.

This configuration is only suitable for users with strong monitoringsystems who want to control the location of their groups.

Looks Alive poll interval:

resource

type value

The Cluster service will perform a cursory check the default number of times to determine if a resource is in the Online state.

Is Alive poll interval: The Cluster service will perform an in-depth check the default number


21/40

resource

type value

of times to determine if the resource is in the Online state.

Pending timeout is set at180 seconds.

This resource has 180 seconds when in a pending state (Online Pendingor Offline Pending) to resolve its status, or at least report to the

Resource Monitor that it is making progress, before the Cluster servicewill put the resource in Failed status.

The following settings appear on the Parameters tab that is available when you configure or view the IP

Address resource properties.


Address:

XXX.XX.XX.XXX

This parameter specifies the unique IP address. This address is a static IPaddress that is not already in use on the network. This was obtained bythe network administrator.

Subnet mask:

XXX.XXX.XXX.XXX

This parameter denotes the subnet mask for the IP address specified. Thissubnet mask is the same subnet mask as the one for the associated clusternetwork. This was obtained by the network administrator.

Network :

Internal + Public

The mixed communication adapter is being used. For more information,see Network Adapters.

Enable NetBIOS for thisaddress is checked.

The NetBIOS is enabled for the specified IP address. This allows adependent network name to be published using NetBIOS.

Network Name Resource


Network Name resource properties.




Run as a separateResource Monitor is notchecked.





22/40


ResourceDependencies:

I

P Address

Resource

The Cluster service uses the dependencies list when bringing resources online andoffline. A dependent resource is one that requires another resource to operate.

The network name resource is dependent on the IP Address that you want tocorrespond to the name.

The following settings appear on the Advanced tab that is available when you configure or view the







Important





resource

type value


Is Alive poll interval:

resource

type value

The Cluster service will perform an in-depth check the default numberof times to determine if the resource is in the Online state.


This resource has 180 seconds when in a pending state (Online Pendingor Offline Pending) to resolve its status, or at least report to theResource Monitor that it is making progress, before the Cluster service


23/40

will put the resource in Failed status.

The following settings appear on the Parameters tab that is available when you configure or view the



DNS Registrationmust succeed.

Exchange virtualserver:

Is not checked.

Domain Name System registration failures are tolerated for thisNetwork Name resource.

Backup virtualserver:

Is not checked.


Cluster virtualserver:

Is not checked.


Enable KerberosAuthentication


Is checked.

Clients connecting to the network name represented by thisresource can use Kerberos Authentication if they chose to. TheSystem Attendant will not come online unless the Kerberos isenabled in the Network Name for the Exchange virtual server.


Is not checked.

Clients connecting to the network name represented by thisresource can only use NTLM authentication. This is the default

configuration.


Is not checked.

Clients connecting to the network name represented by thisresource can only use NTLM authentication. This is the defaultconfiguration.

Physical Disk Resource

The following settings appear on the General tab that is available when you configure or view the Physical

Disk resource properties.




Run as a separateResource Monitor is not



24/40

checked.

The following settings appear on the Dependencies tab that is available when you configure or view the

Physical Disk resource properties.


ResourceDependencies


Dependenciesvary.

The Cluster service uses the dependencies list when bringingresources online and offline. This resource can have varyingdependencies based on the following:

Mount point disks are dependent on their

parent drive.

Log drives are dependent on the data drives.

Data drives have no dependencies.


Dependenciesvary.

The Cluster service uses the dependencies list when bringingresources online and offline. This resource can have varyingdependencies, which enable the transfer of data during backup.


Has nodependencies.

The Cluster service uses the dependencies list when bringingresources online and offline. This resource does not requireanother resource to operate.


Physical Disk resource properties.






Important


This setting also inhibits a cluster resource group from alternating


25/40

between nodes when a resource fails, which would result in significantinstability. This problem can result when a failed disk has defaultproperties that cause a series of bus resets. The resource group thenattempts to acquire reservation on a disk that does not exist.



resource

type value



resource

type value



This resource has 180 seconds when in a pending state (Online Pendingor Offline Pending) to resolve its status, or at least report to theResource Monitor that it is making progress, before the Cluster servicewill put the resource in Failed status.

File Share Resource

The following settings appear on the General tab that is available when you configure or view the File

Share resource properties.







File Share resource properties.


ResourceDependencies vary.

The Cluster service uses the dependencies list when bringing resources onlineand offline. This resource can have varying dependencies which enable thetransfer of data during backup.

The following settings appear on the Advanced tab that is available when you configure or view the File



26/40



The Affect the group check

box is not selected (with athreshold of 3 and a period of 900 seconds).


The entire resource group will not be failed over when this resource

fails. When this resource is in a pending state, the Cluster service willattempt to restart the resource 3 times within 900 seconds before theCluster service assigns the resource to Offline or Failed status.

Important


This setting also inhibits a cluster resource group from alternatingbetween nodes when a resource fails, which would result in significant

instability. This problem can result when a failed disk has defaultproperties that cause a series of bus resets. The resource group thenattempts to acquire reservation on a disk that does not exist.

This configuration is only suitable for users with strong monitoringsystems who want to control the location of their resource groups.


resource

type value



resource

type value




The following setting appears on the Parameters tab that is available when you configure or view the File



The maximum User Limit is allowed. OTG has not limited the number of simultaneous users.

Majority Node Set Resource


Majority Node Set resource properties.


27/40









This resource has no

Resource Dependencies .

The Cluster service uses the dependencies list when bringing resources

online and offline. This resource does not require another resource tooperate.








Important




Looks Alive poll interval: The Cluster service will perform a cursory check the default number of times to determine if a resource is in the Online state.


28/40

resource

type value


resource

type value



This resource has 180 seconds when in a pending state (Online Pendingor Offline Pending) to resolve its status, or at least report to theResource Monitor that it is making progress, before the Cluster servicewill put the resource in Offline status.

Distribution Transaction Coordinator Resource


Distribution Transaction Coordinator resource properties.

Note

OTG has implemented a Distribution Transaction Coordinator resource because Exchange 2003requires this resource in order to operate.










Net

work Name

Quo

rum Physical

Disk

The Cluster service uses the dependencies list when bringing resources onlineand offline. This resource requires the Network Name resource and the Quorumphysical disk resource in order to operate.


29/40





The Affect the group checkbox is not selected (with athreshold of 3 and a periodof 900 seconds).


The entire resource group will not be failed over when this resource fails.When this resource is in a pending state, the Cluster service will attemptto restart the resource 3 times within 900 seconds before the Clusterservice assigns the resource to Offline or Failed status.

Important

All resources that restart after a resource failure are not configuredto Affect the group . This is not a default setting. This allows OTGto track and fix any problems within the cluster. If a resourcefailure occurs, this setting allows OTG administrators to notify theMicrosoft developers of possible bugs in the code.




resource type value



resource type value

The Cluster service will perform an in-depth check the default number of times to determine if the resource is in the Online state.



Exchange Resources

This section describes how OTG has configured the following Exchange resource properties. The Exchange

System Attendant resource and Exchange Information Store resource have different configurations, which

are described at the end of this section.

Exchange Message Transfer Agent instance

Exchange Routing Service instance

Exchange SMTP virtual server instance

Exchange HTTP virtual server instance

Exchange IMAP4 virtual server instance


30/40

Exchange POP3 virtual server instance

Search Service instance

Note

After the Exchange System Attendant resource is created, the cluster automatically generates all otherExchange resources.

The following settings appear on the General tab that is available when you configure or view the various

Exchange resource properties.





These resources do not run on a separate memory space.


various Exchange resource properties.


Resource

Dependencies:

Sy

stem

Attendant

The Cluster service uses the dependencies list when bringing resources online

and offline. These resources require the System Attendant resource in order tooperate.

The following settings appear on the Advanced tab that is available when you configure or view any of

the various Exchange resource properties.






Important


31/40





resource

type value



resource

type value





Exchange System Attendant resource properties.


Possible Owners:

Active

node 1

Primar

y passive node

Altern

ate passivenode 1

Altern

ate passive

node 2

This resource drives the failover path for the Exchange virtual serversbecause this resource can only run on these specified nodes. If the activenode fails, the resource group will fail over to the primary passive node. If theprimary passive node is unavailable, the group will fail over to the alternatepassive nodes. This ensures that this resource group can only become activeon its corresponding active nodes and all passive nodes.

Run as a separateResource Monitor is



32/40

not checked.





5

physical

disks

N

etworkName

The Cluster service uses the dependencies list when bringing resources online andoffline. This System Attendant resource is dependent on five mount drive devices(which are dependent on their parent drives) and the Network name resource. TheSystem Attendant resource requires these resources in order to operate.




Restart is not enabled. This resource does not restart after a resource failure. This allows OTG totrack and fix any problems within the cluster. If a resource failure occurs, thissetting allows OTG administrators to notify the Microsoft developers of possible bugs in the code.

This setting is specific to OTG as it allows for more precise fault isolation thatcan then be reported to the server cluster developers. It is not recommendedfor typical customer deployments.

Looks Alive pollinterval:

resou

rce type value

The Cluster service will perform a cursory check the default number of t imesto determine if this resource is in the Online state.


resource type value

The Cluster service will perform an in-depth check the default number of timesto determine if this resource is in the Online state.

Pending timeout isset at 180 seconds.

This resource has 180 seconds when in a pending state (Online Pending orOffline Pending) to resolve its status, or at least report to the ResourceMonitor that it is making progress, before the Cluster service will put theresource in Failed status.


Exchange Information Store resource properties.


33/40


34/40

Diagram 4 illustrates the process that the Operations and Technology Group (OTG) uses to back up the

mailbox data in the Exchange databases. This backup strategy starts with a defined Service Level

Agreement (SLA) that requires OTG to maintain a one-hour backup restore window per database.

Backup is divided into two stages. Stage 1 is a disk-to-disk backup, and stage 2 is a disk-to-tape backup.

Stage 1 is a t raditional full backup using the Exchange Backup API and is performed daily. The active

cluster backs up data to locally mounted backup disks. Each storage group on an active node backs up itsindividual databases in sequential order, starting with database A and finishing with E. The logs get

truncated daily after OTG completes a successful backup.

In Stage 2, the backup disks are failed over to the alternate passive node(s) and then backed up to tape

through fibre attached tape devices. This backup can run at any period of the day without impacting

server performance.

For more information about how the Exchange data storage is structured, see Storage Components. For

additional information about the storage configuration for each virtual server, see Appendix A.

Diagram 4 Backup Process for the Server Cluster

How OTG Monitors the Server Cluster Updated: September 1, 2003This section briefly discusses the tools that the administrators in the Operations and Technology Group

(OTG) use to monitor the condition and configuration of the Exchange server cluster. Proactive monitoring

has several key components that include managing performance, configuration, and fault tolerance. The

primary monitoring tool OTG uses is Microsoft Operations Manager (MOM).

Microsoft Operations Manager


35/40

OTG relies on Microsoft Operations Manager (MOM) as a proactive tool to monitor the health of its

Exchange environment. MOM consolidates event performance and alert data in a centralized database.

Cluster administrators can use this data to anticipate and react to server conditions.

MOM includes predefined downloadable solutions called Management Pack modules. Management Pack

modules provide immediate monitoring of specific application and environments such as Exchange 2003.

The Exchange module includes built-in knowledge of common problems. In addition, documentation is

included to begin the steps to resolve issues when MOM identifies problems.

With the Exchange module, OTG can proactively manage and report on Exchange 2003 installations, which

helps to avoid outages. For example, the Management Pack Exchange module performs the following

tasks to alert OTG of possible critical conditions:

Monitors vital performance data, which can indicate that the server is running low on resources.

Collects important warning and error events from Exchange 2003 servers and alerts operators to

problems.

Monitors disk capacity and alerts operators when disk capacity is running low. Provides knowledge about which Exchange files are on the affected drives.

Monitors the Exchange services that are expected to be running on a specific server.

Monitors whether an Exchange database can actually be reached by a MAPI client logon, which

verifies both the Exchange database and the Active Directory functionality.

Monitors high queue lengths that are caused when e-mail cannot be sent to a destination server.

Monitors and alerts operators to a high number of simultaneous connections, which can indicate a

denial-of-service attack.

Monitors configuration errors or resource shortages affecting service levels.

In addition, OTG uses MOM to monitor backup and restore of all Exchange servers from a central console.

Other Monitoring Tools

OTG uses a variety of other tools besides MOM to monitor and report the overall performance of servers

and Exchange 2003.

HP Storage Management Appliance (SMA) . The SMA is a centralized, appliance-based

monitoring and management server for the storage area network (SAN) fabric. The SMA performs

management functions without involving the host servers and allows data transfers to proceed

independently between servers and storage devices whether the appliance is operating or not.OTG primarily uses the SMA to remotely manage regional and local SANs.

HP Open SAN Manager . This tool provides a Web-based aggregation and entry point (or portal)

for all SMA software applications. With this tool, administrators can organize, visualize, configure,

and monitor storage from a single navigation point on the SAN. It also provides a centralized

launch site for a variety of external management applications (that is, fabric management).


36/40

Command View EVA (HSV Element Manager) . This is a graphical storage monitoring and

configuration tool for the HP StorageWorks Enterprise Virtual Array (EVA). Administrators can use

it to remotely manage HP SANs.

HP StorageWorks Secure Path . Secure Path is multipath software that manages and maintains

continuous data access to HP storage systems to ensure that no single point of failure occurs

from server to storage. Secure Path is host-resident software that monitors the data pathsbetween server and storage to increase availability of information.

Secure Path Manager 4.0x . This tool provides simplified management of secure paths through

a Web-based GUI that can integrate with the Storage Management Appliance. This tool can host

and store views of preferred and available paths.

Best Practices for Server ClustersUpdated: September 1, 2003

This section describes best practices for deploying a server cluster.

Planning Your Cluster

Before designing your cluster, analyze how your messaging system is used and design your cluster in a

way that will accommodate peak usage periods. For example, the entire configuration of the server cluster

that is discussed in this document (hardware, software and cluster configuration) is designed to meet the

requirements for peak usage. At Microsoft, Monday mornings are the busiest time for e-mail, as

employees attempt to catch up from the weekend. Every server cluster configuration should accommodate

the peak usage periods.

Planning the Hardware for Your Cluster

Use identical network adapters in all cluster nodes. Each adapter should be the same make,

model, and firmware version.

Reserve one network exclusively for internal communication between cluster nodes. This is the

private network. Use other networks for communication with clients. These are public networks.

Having two networks ensures that the network will be available for internal cluster

communications.

Do not use teaming network adapters on the private network. Teaming network adapters, which

are network adapters that are grouped on multiple ports to a single physical network segment,

are used to provide fault tolerance.

Create a standard naming system for your resources, nodes, disks, IP Addresses, and backup

sets. This will increase clarity for anyone working with the cluster.

Configuring the Network Adapters

Use static IP addresses for all nodes on the private network.

Use static IP addresses for all nodes on all public networks. We do not recommend using dynamic

configuration with Dynamic Host Configuration Protocol (DHCP). Failure to renew a lease of an IP

Address could disrupt cluster operation.


37/40

Do not configure DNS servers, WINS servers, or a default gateway on the private network

adapters.

Configure WINS or DNS servers on the public network adapters. If network name resources will

be deployed on the public network, then the DNS servers should support dynamic updates.

In Cluster Administrator, set the network connection order for each node to be:

Private network highest priority

Public network(s)

Remote access connections lowest priority

Change the default name for each network connection to clearly identify the use of each network.

For example, you might change the name of the private network connection from Local Area

Connection (x) to Private Cluster Network.

Isolate the internal network from public network communication. Only nodes that are part of the

cluster should be connected to the private subnet. Where there are several clusters, using the

same subnet for the private network for all of the clusters is reasonable.

Configuring the Cluster Service

Set the private network role to Internal cluster communications only. Verify that the role for each

public network is set to All communications, which is the default value. These settings are

available in Cluster Administrator.

In Cluster Administrator, configure the private network to be the highest priority for internal

cluster communication. In addition, make sure the Public network is first in the TCP/IP Binding

Order.

More Information

For more server cluster best practices, see Best Practices for Configuring and Operating Server

Clusters at the Microsoft Web site (http://go.microsoft.com/fwlink/?LinkId=19500).

For more information about best practices for Enterprise Virtual Arrays, see EVA Best Practices -

Cost, Performance, and Availability and other related white papers in the Storage Array Systems

White Papers section of the Hewlett-Packard Web site (http://go.microsoft.com/fwlink/?

LinkId=19716).

Summary (Using Clustering with Exchange 2003: An

Example)Updated: September 1, 2003The Operation and Technology Group (OTG) at Microsoft uses Windows Server 2003 server cluster

technology in a typical way, with a few changes that maximize the performance of the hardware. Because

OTG selected these configurations (both server and storage) for deployment at Microsoft, they might not

meet your specific deployment needs. IT professionals should use the Microsoft implementation as an

example when configuring their own Exchange server cluster. As always, careful analysis, planning,

testing, and piloting should precede any Exchange deployment and cluster design decisions. Ensure that
http://go.microsoft.com/fwlink/?LinkId=19500http://go.microsoft.com/fwlink/?LinkId=19716http://go.microsoft.com/fwlink/?LinkId=19500http://go.microsoft.com/fwlink/?LinkId=19716


38/40

you apply all applicable best practices and recommendations when making your own design decisions. In

addition, ensure that you have properly characterized your environment before server configuration

choices are made and hardware is purchased. When properly used and implemented, server cluster

technology can create more reliable and available messaging

Related Topics (Using Clustering with Exchange 2003: AnExample)Updated: September 1, 2003

See the following resources for more information:

For more information about server clusters, search the Microsoft Knowledge Base

(http://go.microsoft.com/fwlink/?LinkId=4441) using the key words server clusters and

clustering.

You can also see Availability and Scalability/Windows Clustering in Help and Support Center on

any product in the Windows Server 2003 family.

To view Help and Support Center in the Windows Server 2003 family, click Start , and then click

Help and Support .

For more information about deploying server clusters, see Planning for High Availability and

Scalability at the Windows Server 2003 Deployment Kit Web site

(http://go.microsoft.com/fwlink/?LinkId=4742).

For information about the Windows Server 2003 family, see Product Documentation for Windows

Server 2003 at the Microsoft Web site (http://go.microsoft.com/fwlink/?LinkId=12908).

Appendix A (Using Clustering with Exchange 2003: AnExample)Updated: September 1, 2003

The following table shows the disk drives, data types, disk groups, Enterprise Virtual Arrays, sizes, Vraid

types, and Logical Unit Numbers (LUNs) that correspond to each Exchange virtual server in the Exchange

server cluster that is described in this document.

The server cluster contains four Exchange virtual servers (EVS), four backup virtual servers, and one

cluster virtual server (CVS). The Exchange virtual servers include all of the resources that are required to

form an Exchange server.

Hostname Disk drive letters Data type Disk group EVA Size/ GB RAID type LUN

EVS1 E:\Exchsrvr (Mount point) SMTP DG1 EVA-1 50 1 1

E$ SG1-Data DG1 EVA-1 350 1 2

F$ SG2-Data DG1 EVA-1 350 1 3

G$ SG3-Data DG1 EVA-1 350 1 4
http://go.microsoft.com/fwlink/?LinkId=4441http://go.microsoft.com/fwlink/?LinkId=4742http://go.microsoft.com/fwlink/?LinkId=4742http://go.microsoft.com/fwlink/?LinkId=12908http://go.microsoft.com/fwlink/?LinkId=4441http://go.microsoft.com/fwlink/?LinkId=4742http://go.microsoft.com/fwlink/?LinkId=12908


39/40


40/40

O$ SG3-Data DG1 EVA-2 350 1 22

P$ SG4-Data DG1 EVA-2 350 1 23

W$ SG1-Backup DG4 EVA-2 350 5 45

W:\Dump2 (Mount point) SG2-Backup DG4 EVA-2 350 5 46



M:\Log (Mount point) SG1-Log DG3 EVA-2 40 1 24

N:\Log (Mount point) SG2-Log DG3 EVA-2 40 1 25

O:\Log (Mount point) SG3-Log DG3 EVA-2 40 1 26

P:\Log (Mount point) SG4-Log DG3 EVA-2 40 1 28

EVS4 Q:\Exchsrvr (Mountpoint)

SMTP DG2 EVA-2 50 1 19

Q$ SG1-Data DG2 EVA-2 350 1 29

R$ SG2-Data DG2 EVA-2 350 1 30

S$ SG3-Data DG2 EVA-2 350 1 31

T$ SG4-Data DG2 EVA-2 350 5 49

X$ SG1-Backup DG5 EVA-2 350 5 50

X$:\Dump2 (Mount point) SG2-Backup DG5 EVA-2 350 5 51



Q:\Log (Mount point) SG1-Log DG3 EVA-2 40 1 33

R:\Log (Mount point) SG2-Log DG3 EVA-2 40 1 34

S:\Log (Mount point) SG3-Log DG3 EVA-2 40 1 35

T:\Log (Mount point) SG4-Log DG3 EVA-2 40 1 36

CVS Y$ Quorum Quorum DG3 EVA-1 10 1 53

Using Clustering With Exchange 2003

Documents

Transcript of Using Clustering With Exchange 2003