Hp Best Practices For Microsoft Exchange Server 2000 And 2003 Cluster Deployments

HP best practices for Microsoft Exchange Server 2000 and 2003 Cluster deployments

Executive summary............................................................................................................................... 4 Introduction......................................................................................................................................... 4 Overview of Microsoft Exchange Server clusters ...................................................................................... 5

Clusters explained............................................................................................................................ 5 Microsoft cluster server ..................................................................................................................... 5 Shared-nothing cluster model............................................................................................................. 5 Resources and resource groups.......................................................................................................... 6 Key cluster terminology..................................................................................................................... 8 Microsoft cluster service components .................................................................................................. 9

The cluster service ...................................................................................................................... 10 The resource monitor .................................................................................................................. 10 The resource DLL ........................................................................................................................ 10

Cluster failover .............................................................................................................................. 11 Cluster-aware vs. non cluster-aware applications ............................................................................... 11

Active/active vs. active/passive................................................................................................... 11 Exchange virtual servers.................................................................................................................. 12 Clustering support for Microsoft Exchange Server components............................................................. 12

Key component: Exchange resource DLL (EXRES.DLL) ...................................................................... 14 Key component: Exchange cluster admin DLL (EXCLUADM.DLL)........................................................ 15 Microsoft Exchange Server cluster resource dependencies............................................................... 15 Microsoft Exchange virtual servers................................................................................................ 18

What’s new in Microsoft Exchange Server 2003................................................................................... 19 Planning your Microsoft Exchange 2003 cluster deployment................................................................... 20

Best practices for hardware............................................................................................................. 20 Check for hardware compatibility................................................................................................. 20 Choose standardized configurations............................................................................................. 20 Hardware redundancy................................................................................................................ 21

Best practices for cluster configuration .............................................................................................. 22 Understanding resource dependency ............................................................................................ 22 Planning cluster groups ............................................................................................................... 22 Planning quorum type ................................................................................................................. 22

Storage planning ........................................................................................................................... 23

Storage design is critical ............................................................................................................. 23 Placement of Microsoft Windows and Microsoft Exchange components ............................................ 24 Microsoft Windows Server .......................................................................................................... 24 Microsoft Exchange Server 2003 binaries..................................................................................... 25 Storage groups, databases, and transaction logs ........................................................................... 25 Assigning drive-letters and labels.................................................................................................. 26 Choosing RAID levels for Microsoft Exchange components .............................................................. 29 Allocate dedicated spindles to each Microsoft Exchange virtual server.............................................. 30 Using mount points ..................................................................................................................... 30 Use a consistent naming standard for folders and databases ........................................................... 31 Obtain TCP/IP addresses prior to installation................................................................................. 33 Cluster naming conventions ......................................................................................................... 33

Microsoft Exchange Server roles ...................................................................................................... 34 Implement dedicated Microsoft Exchange clusters........................................................................... 34 Assign specific roles to clusters for enterprise deployments .............................................................. 34 First server and site replication services (SRS) server ....................................................................... 35 Implement clusters as mailbox servers and public folder servers ....................................................... 35 Clusters and active/active scalability limits .................................................................................... 35

What’s new in Microsoft Windows Server 2003 ................................................................................... 36 Best practices for server configuration .................................................................................................. 36

Build redundancy into your Microsoft Windows Server infrastructure ................................................ 37 Cluster service account best practices ............................................................................................... 37

Create one cluster service account per cluster ................................................................................ 37 Use a consistent naming convention for cluster service accounts....................................................... 37 Do not use the cluster service account to logon interactively............................................................. 38 Delegate Microsoft Exchange permissions ..................................................................................... 38

Upgrade to the latest service pack on each node............................................................................... 38 Change the temp folder used by the cluster service account............................................................. 38

Network configuration.................................................................................................................... 40 Separate private and public networks ........................................................................................... 40 Do not use DHCP for cluster network connections ........................................................................... 40 Label network connections........................................................................................................... 41 Modify the binding order on network connections .......................................................................... 41 Disable unnecessary protocols, services on the cluster interconnect connection .................................. 42 Set correct settings for cluster communications................................................................................ 43 NIC teaming configuration .......................................................................................................... 44 IPSec ........................................................................................................................................ 45

Geographically-dispersed clusters .................................................................................................... 45 Set staggered boot delays on each node .......................................................................................... 45 OS tuning for large memory............................................................................................................ 46

More than 1GB of RAM in Microsoft Windows 2000 Advanced Server ........................................... 46 More than 1GB of RAM in Microsoft Windows 2003 Server........................................................... 46

Cluster validation testing ................................................................................................................. 47 Planned failover ......................................................................................................................... 47 Unplanned failover ..................................................................................................................... 47 Power outage Test ...................................................................................................................... 47

Best practices for Microsoft Exchange Server installation..................................................................... 47 Prerequisites .............................................................................................................................. 47 Document your cluster installation................................................................................................. 48 Front-end / back-end architectures................................................................................................ 49

Upgrading Microsoft Exchange 5.5 clusters ...................................................................................... 50 Upgrading Microsoft Exchange 2000 clusters ................................................................................... 50

Removing Microsoft Exchange 2000 Tuning Parameters ................................................................. 51 Before upgrading or refreshing hardware ......................................................................................... 51

Best practices for systems management ................................................................................................ 51 Active Directory ............................................................................................................................. 52

Create a separate OU for virtual servers and cluster network names................................................. 52 Cluster configuration ...................................................................................................................... 52

Capture configuration information ................................................................................................ 52

Non-critical cluster resources........................................................................................................ 52 Majority node set ....................................................................................................................... 53

Training and expertise .................................................................................................................... 53 Microsoft Windows Server resource kits............................................................................................ 54 Securing your cluster ...................................................................................................................... 54

Preventing viral attacks................................................................................................................ 55 Standardizing configuration ............................................................................................................ 56

Hardware configuration .............................................................................................................. 57 Device drivers ............................................................................................................................ 57 Using QFECHECK to verify hotfix installations................................................................................ 59

Microsoft Exchange Server service packs .......................................................................................... 59 Upgrade to the latest Microsoft Exchange Server service pack ......................................................... 59 Recommended procedure for Microsoft Exchange service packs ...................................................... 59 How to avoid reboots during a rolling upgrade ............................................................................. 60

Third-party products........................................................................................................................ 60 Use cluster administrator to perform cluster operations........................................................................ 60 Use system manager to change Microsoft Exchange virtual servers ...................................................... 61 Failovers ....................................................................................................................................... 62

Planned failovers........................................................................................................................ 62 Tips for reducing planned failover time ......................................................................................... 62 Unplanned failovers.................................................................................................................... 63 Tips for reducing unplanned failover time...................................................................................... 64 Managing storage...................................................................................................................... 64

Performance monitoring.................................................................................................................. 65 Operations checks...................................................................................................................... 65 Event logs.................................................................................................................................. 65 Monitoring mail queues............................................................................................................... 66 Monitoring virtual memory........................................................................................................... 68

Best practices for disaster recovery ...................................................................................................... 69 What should I backup? ............................................................................................................... 70 Checking backups ...................................................................................................................... 71 Microsoft Cluster Tool ................................................................................................................. 72 Restoring Quorum ...................................................................................................................... 73

Conclusion........................................................................................................................................ 74 For more information.......................................................................................................................... 75

Utilities...................................................................................................................................... 75 Microsoft Knowledge base articles ............................................................................................... 75

4

Executive summary Microsoft® Exchange Server cluster deployments have additional complexity over single server implementations. Clusters depend on complex interaction between hardware, operating system, applications, and administrators. Attention to detail is needed to ensure that the Microsoft Exchange Server cluster implementation is properly planned, installed, and configured. The aim of this whitepaper is to present HP Best Practices for Microsoft Exchange Server cluster deployments. The following areas are addressed:

• Overview of Microsoft Exchange Server clusters • What’s new in Microsoft Exchange Server 2003 • Planning your Microsoft Exchange Server cluster deployment • What’s new in Microsoft Windows® Server 2003 • Best Practices for Microsoft Windows Server configuration • Best Practices for Systems Management • Best Practices for Disaster Recovery

Introduction This document provides best practices for successful Microsoft Exchange Server 2003 cluster deployments. These best practices have been derived from the experience of HP engineers in deployments across a wide range of customer environments. This paper is an update to the original whitepaper on best practices for Microsoft Exchange Server 2000 deployments, and there are references in this document to both Microsoft Exchange Server 2003 and 2000. The original document is available on HP ActiveAnswers, http://h71019.www7.hp.com/1-6-100-225-1-00.htm, so if you are only considering a Microsoft Exchange 2000 deployment, you should consult that document at http://activeanswers.compaq.com/aa_downloads/6/100/225/1/42311.pdf. However, if you wish to understand the differences and the advantages of a Microsoft Exchange 2003 cluster, then this document will cover those differences. A primary consideration is whether you plan to run Microsoft Windows Server 2003. Microsoft Exchange 2000 can not run on Microsoft Windows Server 2003, yet Microsoft Exchange 2003 can run on either Microsoft Windows Server 2000 or 2003.

This paper addresses the following areas: • Overview of Microsoft Exchange Server clusters: Covers concepts such as Microsoft Cluster Server,

cluster resource groups, cluster terminology, cluster failover and failback operations, and how Microsoft Exchange Server interacts with Microsoft Cluster Server.

• What’s new in Microsoft Exchange Server 2003: focusing on the features and changes in the new product that affect clustering.

• Planning your Microsoft Exchange Server cluster: Best practices for choosing hardware for your cluster, designing storage for Microsoft Exchange, cluster naming conventions and recommendations for your Microsoft Windows Server infrastructure.

• What’s new in Microsoft Windows Server 2003: Specifically, what features and changes are in the new product and how they affect clustering.

• Best Practices for Microsoft Windows Server configuration: This section looks at configuring Microsoft Windows Server optimally for Microsoft Exchange Server 2003 clusters. Detailed recommendations on network configuration, failover testing, and securing your Microsoft Exchange Server 2003 cluster from viruses are presented.

http://h71019.www7.hp.com/1-6-100-225-1-00.htm

http://activeanswers.compaq.com/aa_downloads/6/100/225/1/42311.pdf

5

• Best Practices for Systems Management: Covers areas such as System Manager training, best practices for Service Pack upgrades, tips for reducing downtime during failover/failback operations, and performance monitoring

• Best Practices for Disaster Recovery: Covers cluster specific disaster recovery concepts. How to use Resource Kit tools such as the cluster recovery utility and the cluster diagnostic tools.

Overview of Microsoft Exchange Server clusters

Clusters explained A cluster server consists of a group of servers that are configured to work together to provide a single system with high availability to data and applications. When an application or hardware component fails on a standalone server, service cannot be restored until the problem is resolved. When a failure occurs on a server within a cluster, resources and applications can be redirected to another server in the cluster, which takes over the workload.

Microsoft cluster server Microsoft introduced clustering support with the release of Microsoft Windows NT 4.0 Enterprise Edition in 1997. Microsoft Cluster Server (MSCS) could be used to connect two NT 4.0 servers and provide high availability to file and print services and applications. Note the following differences and progression:

• Microsoft Cluster Server in Microsoft Windows NT 4.0 is limited to a maximum of two servers (nodes) in a cluster.

• Microsoft Cluster Server in Microsoft Windows 2000 Advanced Server was also limited to two nodes per cluster; however, each node could host an active Microsoft Exchange Virtual Server (more detail on the design limitations of doing so later)

• Microsoft Windows 2000 Server Datacenter Edition supports a maximum of four nodes per cluster and could host N+I active virtual servers, for example 3 active and one passive.

• Microsoft Windows Server 2003 Enterprise Edition supports a maximum of eight nodes per cluster and could host N+I active virtual servers, for example 6 active and 2 passive.

• Microsoft Windows Server 2003 Datacenter Edition also supports a maximum of eight nodes per cluster.

See http://www.microsoft.com/windowsserver2003/evaluation/features/compareeditions.mspx for a Microsoft Windows Server 2003 product comparison.

For a thorough discussion of what Microsoft cluster server can and cannot do, see: http://www.microsoft.com/technet/prodtechnol/windowsserver2003/support/SerCsFAQ.asp.

Shared-nothing cluster model Microsoft Cluster Server uses the shared-nothing cluster model. With shared-nothing, each server owns and manages local devices as specific cluster resources. Devices that are common to the cluster and physically available to all nodes are owned and managed by only one node at a time. For resources to change ownership, a complex reservation and contention protocol is followed and implemented by cluster services running on each node. The shared-nothing model dictates that while several nodes in the cluster may have access to a device or resource, the resource is owned and managed by only one system at a time. (In a Microsoft Cluster Server cluster, a resource is defined as any physical or logical component that can be brought online and taken offline, managed in a cluster, hosted by only one node at a time, and moved between nodes.)

http://www.microsoft.com/windowsserver2003/evaluation/features/compareeditions.mspx

http://www.microsoft.com/technet/prodtechnol/windowsserver2003/support/SerCsFAQ.asp

6

Each node has its own memory, system disk, operating system, and subset of the cluster’s resources. If a node fails, the other node takes ownership of the failed node’s resources (this process is known as failover). Microsoft Cluster Server then registers the network address for the resource on the new node so that client traffic is routed to the system that is available and now owns the resource. When the failed node is later brought back online, Microsoft Cluster Server can be configured to redistribute resources and client requests appropriately (this process is known as failback).

Resources and resource groups The basic unit of management in a Microsoft cluster is the resource. Resources are logical or physical entities or units of management within a cluster system that can be changed in state from online to offline, are manageable by the cluster services, and are owned by one cluster node at a time. Cluster resources include entities such as hardware devices like network interface cards and disks or logical entities like server name, network name, IP address, and services. Resources are defined within the cluster manager and selected from a list of pre-defined choices. Microsoft Windows Server 2003 provides a set of resource DLLs (such as File and print shares and Generic services or applications) and cluster-aware applications provide their own resource DLLs and resource monitors.

In a cluster, there are both physical and logical resources typically configured. Within the Microsoft Cluster Server framework, shown in Figure 1, resources are grouped into logical units of management and dependency called resource groups. A resource group is usually comprised of both logical and physical resources such as virtual server name, IP address, and disk resources. Resource groups can also just include cluster-specific resources used for managing the cluster itself. The key in the shared-nothing model is that a resource can only be owned by one node at a time. Furthermore, the resources that are part of the resource group owned by a cluster node must exist on that node only.

7

Figure 1. Basic 2-Node Microsoft Cluster Server Configuration

System files on System files on local drive

System files on System files on local drivelocal drive

ClientsClients

Node A Node A Node BNode B

Data and executable files Data and executable files on shared cluster drives on shared cluster drives

System files on System files on local drive local drive

System files on System files on local drivelocal drive

ClientsClients

Node A Node A Node BNode B

Data and executable files Data and executable files on shared cluster drives on shared cluster drives

The shared-nothing model prevents different nodes within the cluster from simultaneous ownership of resource groups or resources within a resource group. As mentioned earlier, resource groups also maintain a dependency relationship between different resources contained within each group. This is because resources in a cluster very often depend upon the existence of other resources in order to function or start. For example, a virtual server or network name must have a valid IP Address in order for clients to access that resource. Therefore, in order for the network name or virtual server to start (or not fail), the IP Address it depends upon must be available. This is known as resource dependency. Within a resource group, the dependencies among resources can be quite simple or very complex.

Resource dependencies are maintained in the properties of each resource and allow the cluster service to manage how resources are taken offline and brought on line. Also, resource dependencies cannot extend beyond the context of the resource group to which they belong. For example, a virtual server cannot have a dependency upon an IP Address that exists within a resource group other than its own resource group.

This restriction is due to the fact that resource groups within a cluster can be brought online and offline and moved from node to node independently of one another. Each resource group also maintains a policy that is available cluster-wide that specifies the Preferred Owner (the node in the cluster it prefers to run on) and the Possible Owners (the node that it can fail over to in the event of a failure condition).

8

Resources are the fundamental unit of management within a Microsoft Windows Cluster. As such, it is important that you have an understanding of how they function and operate. Resource Groups are used to logically organize these resources and enable easier management of failover between nodes.

Key cluster terminology There has been much confusion in the world of cluster technology not over just basic architectural implementations but also terminologies. Table 1 below highlights some key terminology for Microsoft Cluster Server.

Table 1. Microsoft Cluster Server Key Terminology

Term Definition/Description

Resource Smallest unit that can be defined, monitored and maintained by the cluster. Examples are Physical Disk, IP Address, Network Name, File Share, Print Spool, Generic Service and Application. Resources are grouped together into a Resource group. The cluster uses the state of each resource to determine whether a failover is needed.

Resource Group A collection of resources that logically represents a client/server function. The smallest unit that can failover between nodes.

Resource Dependency A resource property defining its relationship to other resources. A resource is brought online after any resource that it depends on. A resource is taken offline before any resources that it depends on. All dependent resources must failover together.

Quorum Resource Stores the cluster log data and application data from the registry used to transfer state information between clusters. Used by the cluster service to determine which node can continue running when nodes cannot communicate. Note that the type of quorum resource, and therefore the type of the cluster (either a quorum-device cluster or a majority-of-nodes cluster), is established during cluster setup and can not be changed later.

Active/Passive Term used for Service Failover mode where service is defined as a resource using the generic resource DLL. Failover Manager limits application operation to only one node at a time.

Active/Active More comprehensive failover capability also known as Resource Failover mode. Utilizes ISV-developed resource DLLs that are “cluster aware”. Allows for operation of service on multiple nodes and individual resource instances failover instead of entire service.

Membership Term used to describe the orderly addition and removal of active nodes to and from the cluster.

Global Update Global update propagates cluster configuration changes to all members. The cluster registry is maintained through this mechanism and all activities are atomic, ordered, and tolerant to failures.

Cluster Registry Separate from the NT registry, the cluster registry maintains configuration updates on members, resources, parameters and other configuration information and is stored on the cluster quorum disk. Loaded under HKLM\Cluster

Virtual Server The network resource used by client for the Exchange cluster resource group – a combination or collection of configuration information and resources such as network name and IP address resources. Can refer to a Microsoft Cluster Server virtual server or a logical set of services provided by Internet Information Server (IIS).

Physical Machine The physical hardware device that comprises an individual cluster node. Can host multiple virtual servers and resources

9

Microsoft cluster service components Microsoft Cluster Service is implemented as a set of independent and somewhat isolated group of components (device drivers and services). This set of components layers on top of the Microsoft Windows Server operating system and acts as a service. By using this design approach, Microsoft avoided many complexities that may have been encountered in other design approaches such as system scheduling and processing dependencies between the cluster service and the operating system. When layered on Microsoft Windows Server, the cluster service provides some basic functions that the operating system needs in order to support clustering. These basic functions include dynamic network resource support, file system support for disk mounting and un-mounting, and shared resource support for the I/O subsystem. Table 2 below provides a brief overview of each of these components.

Table 2. Microsoft Cluster Service Components

Component Role/Function

Node Manager Maintains resource group ownership of cluster nodes based on resource group node preferences and the availability of cluster nodes.

Resource Monitor Utilizes the cluster resource API and RPCs to maintain communication with the resource DLLs. Each monitor runs as a separate process.

Failover Manager Works in conjunction with the resource monitors to manage resource functions within the cluster such as failovers and restarts.

Checkpoint Manager Maintains and updates application states and registry keys on the cluster quorum resource.

Configuration Database Manager Maintains and ensures coherency of the cluster database on each cluster node that includes important cluster information such as node membership, resources, resource groups, and resource types.

Event Processor Processes events relating to state changes and requests from cluster resources and applications.

Membership Manager Manages cluster node membership and polls cluster nodes to determine state.

Event Log Replication Manager Replicate system event log entries across all cluster nodes.

Global Update Manager Provides updates to the Configuration Database Manager to ensure cluster configuration integrity and consistency.

Object Manager Provides management of all cluster service objects and the interface for cluster administration.

Log Manager Works with the Checkpoint Manager to ensure that the recovery log on the cluster quorum disk is current and consistent.

10

For discussions pertaining to Microsoft Exchange Server, there are three key components in Microsoft Cluster Service to consider:

The cluster service The Cluster Service (which is actually a group of components consisting of the Event Processor, the Failover Manager/Resource Manager, the Global Update Manager, and so forth) is the core component of Microsoft Cluster Server. The Cluster Service controls cluster activities and performs such tasks as coordinating event notification, facilitating communication between cluster components, handling failover operations, and managing the configuration. Each cluster node runs its own Cluster Service.

The resource monitor The Resource Monitor is an interface between the Cluster Service and the cluster resources, and runs as an independent process. The Cluster Service uses the Resource Monitor to communicate with the resource DLLs. The DLL handles all communication with the resource, thus shielding the Cluster Service from resources that misbehave or stop functioning. Multiple copies of the Resource Monitor can be running on a single node, thereby providing a means by which unpredictable resources can be isolated from other resources.

The resource DLL The third key Microsoft Cluster Server component is the resource DLL. The Resource Monitor and resource DLL communicate using the Microsoft Cluster Server Cluster Resource API, which is a collection of entry points, callback functions, and related structures and macros used to manage resources. Applications that implement their own resource DLLs to communicate with the Cluster Service and that use the Cluster API to request and update cluster information are defined as cluster-aware applications. Applications and services that do not use the Cluster or Resource APIs and cluster control code functions are unaware of clustering and have no knowledge that Microsoft Cluster Server is running. These cluster-unaware applications are generally managed as generic applications or services. Both cluster-aware and cluster-unaware applications run on a cluster node and can be managed as cluster resources. However, only cluster-aware applications can take advantage of features offered by Microsoft Cluster Server through the Cluster API. Cluster-aware applications can report status upon request to the Resource Monitor, respond to requests to be brought online or to be taken offline gracefully, respond more accurately to IsAlive and LooksAlive requests issued by the cluster services. Cluster-aware applications should also implement Cluster Administrator extension DLLs, which contain implementations of interfaces from the Cluster Administrator extension API. A Cluster Administrator extension DLL allows an application to be configured into the Cluster Administrator tool (CluAdmin.exe). Implementing custom resource and Cluster Administrator extension DLLs allows for specialized management of the application and its related resources, and enables the system administrator to install and configure the application more easily.

As discussed earlier, to the Cluster Service, a resource is any physical or logical component that can be managed. Examples of resources are disks, network names, IP addresses, databases, IIS Web roots, application programs, and any other entity that can be brought online and taken offline. Resources are organized by type. Resource types include physical hardware (such as disk drives) and logical items (such as IP addresses, file shares, and generic applications). Every resource uses a resource DLL, a largely passive translation layer between the Resource Monitor and the resource. The Resource Monitor calls the entry point functions of the resource DLL to check the status of the resource and to bring the resource online and offline. The resource DLL is responsible for communicating with its resource through any convenient IPC mechanism to implement these methods. Applications or services that do not provide their own resource DLLs can still be configured into the cluster environment. Microsoft Cluster Server includes a generic resource DLL, and the Cluster Service treats these applications or services as generic, cluster-unaware applications or services. However, if an

11

application or service needs to take full advantage of a clustered environment, it must implement a custom resource DLL that can interact with the Cluster Service and take full advantage of the full set of features provided by Microsoft Cluster Service.

Cluster failover With Microsoft Cluster Server, two types of failover are supported: Resource and Service Failover. Both allow for increased system availability. More comprehensive in capabilities, the Resource Failover mode takes advantage of cluster APIs that enable applications to be “cluster aware.” This is provided via a Resource DLL that can be configured to allow customizable failover of the application. Resource DLLs provide a means for Microsoft Cluster Server to manage resources. They define resource abstractions, interfaces, and management. In a resource failover mode of operation, it is assumed that the service is running on both nodes of the Microsoft Cluster Server cluster (also known as “Active/Active”) and that a specific resource – such as a database, virtual server, or an IP address – fails over, not the entire service. Many applications, from independent software vendors as well as those from Microsoft, do not have resource DLLs available that enable them to be cluster aware. To offset this, Microsoft has provided a generic service resource DLL, which provides basic functionality to these applications running on Microsoft Cluster Service. The generic resource DLL provides for the Service Failover mode and limits the application to running on one node only (also known as “Active/Passive”). In a Service Failover mode, a service is defined to Microsoft Cluster Server as a resource. Once defined, the Microsoft Cluster Server Failover Manager ensures that the service is running on only one node of the cluster at any given time. The service is part of a resource group that uses a common name throughout the cluster. As such, all services running in the resource group are available to any network clients using the common name.

Cluster-aware vs. non cluster-aware applications Cluster-aware applications provide the high levels of functionality and availability in a Microsoft Cluster Server environment. The applications and the cluster services can provide feedback that facilitates optimal operation. In this scenario, as much application state as possible is preserved during failover. Examples of cluster-aware applications are Microsoft SQL Server, SAP/R3, Baan, PeopleSoft, Oracle, and Microsoft Exchange Server 2003 Enterprise Server. Non cluster-aware applications have several limitations discussed previously. The application and the cluster software cannot communicate with each other. Any communication that occurs is limited to that provided by the generic resource DLL provided with Microsoft Cluster Server. Examples of non cluster-aware applications are file and print services, Microsoft Internet Information Server, and Microsoft Exchange Server 5.5 Enterprise Edition. The cluster software has no application awareness and simply understands that a generic service or group of services and resources must be treated as a failover unit.

Active/active vs. active/passive When deploying cluster solutions with Microsoft Windows Server, the level of functionality and flexibility that an application can enjoy in a clustered environment directly relates to whether it supports active/passive or active/active configuration. Active/active means that an application provides functionality on all nodes in the cluster at the same time. This means that the application services are running and servicing users from each node in the cluster. To do this, an application must have support for communicating with the cluster services via its own resource DLL (cluster-aware). Also, the application must be architected in such a way that specific resource units can be treated independently and failed over to other nodes. Per the discussions above, this requires specific support from the application vendor (whether Microsoft or third-party vendors) in order for the application to run in an active/active cluster configuration. For active/passive configurations, the application is either limited architecturally (can not be run on two active notes), has no specific resource DLL support, or both. In an active/passive 2-node configuration, the application runs only on one cluster

12

node at a time. The application may or may not be cluster aware (see above). In some applications capable of active/active, such as Microsoft Exchange Server 2000 and 2003, there may be limitations on running in active/active versus active/passive (covered later).

The discussions above on the failover types (Service and Resource Failover) as well as the differences between cluster-aware and non cluster-aware applications are distinct from active/active and active/passive configurations. These terms are often confused, but an active/passive cluster application can be cluster aware, with application services running on every node, even those that are not actively providing end-user functionality. However, an active/active application must be cluster aware and a non cluster-aware application must be deployed in an active/passive configuration.

Exchange virtual servers From an administrative perspective, all components required to provide services and a unit of failover are grouped into a Microsoft Exchange Virtual Server (EVS). One or more Microsoft Exchange Virtual Servers can exist in the cluster, and each virtual server runs on one of the nodes in the cluster. Microsoft Exchange Server 2000/2003 can support multiple virtual servers on a single node, although this may be less than desirable – see the discussion of active/active design and failover.

A Microsoft Exchange Virtual Server, at a minimum, will include a storage group, and required protocols. From the viewpoint of Microsoft Cluster Server, a Microsoft Exchange Virtual Server exists as a resource in each cluster resource group. If you have multiple Microsoft Exchange Virtual Servers that shared the same physical disk resource (i.e. each has a storage group that resides on the same disk device), they must all exist within the same resource group and cannot be split into separate resource groups. This is done to ensure that the resources and virtual servers all failover as a single unit and is an administrative restriction that ensures that resource group integrity is maintained. Clients connect to the virtual servers the same way that they would connect to a stand-alone server. The cluster service monitors the virtual servers in the cluster. In the event of a failure, the cluster service restarts or moves the affected virtual servers to a healthy node. For planned outages, the administrator can manually move the virtual servers to other nodes. In either event, the client will see an interruption of service only during the brief time that the virtual server is in an online/offline pending state.

Clustering support for Microsoft Exchange Server components Clustering support differs according to the component. All components of Microsoft Exchange Server 2003 are not currently supported in a clustered environment. The following table details which components are supported, and in some cases, the type of clustering they are capable of supporting:

Table 3. Microsoft Exchange Server Component Cluster Support

Source: Deploying Exchange Server Clusters, and What's New in Exchange Server 2003 -- Microsoft

Microsoft Exchange Server Component

Cluster Functionality

Microsoft Exchange 2000 Microsoft Exchange 2003

Exchange System Attendant

Active/Active Each Microsoft Exchange Virtual Server is created by the System Attendant resource when configured.

Same as Microsoft Exchange 2000

Information Store Active/Active Each cluster node is limited to 4 Storage Groups.


Message Transfer Agent (MTA)

Active/Passive

The MTA will be in only one cluster group. One MTA instance per cluster.


13

Microsoft Exchange Server Component

Cluster Functionality

Microsoft Exchange 2000 Microsoft Exchange 2003

POP3 Protocol Active/Active Multiple virtual servers per node.


IMAP Protocol Active/Active Multiple virtual servers per node.


SMTP Protocol Active/Active Multiple virtual servers per node.


HTTP DAV Protocol Active/Active Multiple virtual servers per node.


NNTP Protocol Active/Active Multiple virtual servers per node.


MS Search Server Active/Active One Instance per virtual server


Site Replication Service Active/Passive Not Supported in a cluster Same as Microsoft Exchange 2000

MSMail Connector Active/Passive Not Supported in a cluster Microsoft Exchange 2000 only

cc: Mail Connector Active/Passive Not Supported in a cluster Microsoft Exchange 2000 only

Lotus Notes Connector Active/Passive Not Supported in a cluster Same as Microsoft Exchange 2000

Novell GroupWise Connector

Active/Passive Not Supported in a cluster Same as Microsoft Exchange 2000

SNADS Connector Active/Passive Not Supported in a cluster Microsoft Exchange 2000 only

PROFS Connector Active/Passive Not Supported in a cluster Microsoft Exchange 2000 only

Active Directory Connector

Active/Passive Not Supported in a cluster Same as Microsoft Exchange 2000

Key Management Service

Active/Passive Not Supported in a cluster Microsoft Exchange 2000 only

Chat Service Active/Passive Not Supported in a cluster Microsoft Exchange 2000 only

Conferencing Manager Services


Video Conferencing Service


Microsoft Exchange Server provides its core features and support for Microsoft Cluster Server via two key components as shown in Figure 2. These are the Microsoft Exchange Cluster Administration DLL and the Exchange Resource DLL.

14

Figure 2. How Microsoft Exchange interfaces with Microsoft Cluster Services

Key component: Exchange resource DLL (EXRES.DLL) If you recall the earlier discussion of cluster-aware versus non cluster-aware applications, you remember that it is the existence of an application-specific resource DLL that is the key differentiator for cluster-aware applications. Also remember that Microsoft Exchange 5.5 did not provide its own resource DLL and made use of the generic resource DLL that is provided with Microsoft Cluster Server. For Microsoft Exchange Server 2003, Microsoft developers took the extra time and effort to guarantee full-cluster functionality. The result of that effort is the Microsoft Exchange Resource DLL called EXRES.DLL. This DLL for Microsoft Exchange Server 2003 is installed when the setup application realizes that it is operating in a clustered environment. EXRES.DLL acts as a direct resource monitor interface between the cluster services and Microsoft Exchange Server 2003 by implementing the Microsoft Cluster Services API set. Table 4 below shows the typical interactions and indications that EXRES.DLL will provide between Microsoft Exchange resources and cluster services.

Table 4. Microsoft Exchange Resource DLL (EXRES.DLL) Interactions and Functions

Interaction/Indicator Function

Online/Offline Microsoft Exchange Virtual Server Resource is running, stopped, or in an idle state.

Online/Offline Pending In the process or service is in the state of starting or shutting down.

Looks Alive/Is Alive Resource polling functions to determine whether resource should be restarted or failed. Can be configured in cluster administrator on a per-resource basis.

Failed The resource failed on the Is Alive call and was not able to be restarted (restart failed).

Restart Resource has failed on Is Alive call and it directed to attempt to restart.

15

Key component: Exchange cluster admin DLL (EXCLUADM.DLL) In order for Microsoft Exchange resources to be configured and controlled by the Cluster Administrator, there must be an enabler for Microsoft Exchange services to communicate with the Cluster Administrator and for the Cluster Administrator program to provide Microsoft Exchange-specific configuration parameters and screens. The Microsoft Exchange Cluster Administration DLL or EXCLUADM.DLL provides this support. The Microsoft Exchange Cluster Admin DLL provides the necessary wizard screens when configuring Microsoft Exchange resources in Cluster Administrator and presents Microsoft Exchange resources that can be added as resources in the cluster such as the Microsoft Exchange System Attendant. The cluster administration DLL is a key component in configuration and management of Microsoft Exchange services in the cluster. It is not required for resource monitoring and restart or failover actions. The Microsoft Exchange Resource DLL (EXRES.DLL) performs this role.

Microsoft Exchange Server cluster resource dependencies Figure 3 illustrates a tree structure of Microsoft Exchange 2000 cluster resource dependencies and Figure 4 shows the newer Microsoft Exchange Server 2003 cluster resource dependencies. Microsoft Exchange services must have certain resources as predecessors before they can be brought online as a cluster resource. By default, Microsoft Exchange Server 2003 installs nine resources in the form of virtual servers into a cluster resource group that is being configured in the Cluster Administrator. Table 5 provides a brief description of each resource and its function.

16

Figure 3. Microsoft Exchange Server 2000 Cluster Resource Dependencies

Figure 4. Microsoft Exchange Server 2003 Cluster Resource Dependencies.

17

When you make a change to Storage Group configuration (including Recovery Storage Groups, covered later in this document), you will see the prompt in System Manager shown in Figure 5 below to ensure proper disk resource dependencies.

Figure 5. Prompt in System Manager to Ensure Proper Disk Resource Dependencies.

Note: As shown in Table 5 below, Microsoft Exchange Server 2003 changes the Resource dependencies so that the Internet protocols shown above are dependent directly on the System Attendant instead of the Store.

Table 5. Microsoft Exchange Server 2000 and 2003 Cluster Resources

Resource Role

System Attendant Foundation Microsoft Exchange resource that must exist prior other resources being added to the cluster.

Resource Dependency: Network Name, All Physical Disks for that Virtual Server

Information Store Virtual Server Instance for the STORE.EXE process and its presentation to MAPI clients and other services.

Resource Dependency: System Attendant

Routing Service Microsoft Exchange Server 2003 Routing Service virtual server instance.


MTA Service Message Transfer Agent virtual server. Exists only on one cluster node (active/passive). Provided for legacy messaging connector support and routing to Microsoft Exchange 5.5 environments.


18

Resource Role

MS Search Service Microsoft Search Engine virtual server instance. Provides Microsoft Exchange content indexing service for clients. Can be removed if that protocol is not needed.

Resource Dependency: Microsoft Exchange 2003 - System Attendant

Microsoft Exchange 2000 - Information Store

SMTP Service SMTP virtual server instance. Provides Internet protocol email sending functionality and message routing within Microsoft Exchange 2000/2003.



HTTP Virtual Server HTTP-DAV protocol virtual server. Provides Web/Browser-based client access to information store.



POP3 Service POP3 protocol virtual server. Provides POP3 client access to information store. Can be removed if that protocol is not needed.


Microsoft Exchange 2000 – Information Store

IMAP Service IMAP protocol virtual server. Provides IMAP client access to information store. Can be removed if that protocol is not needed.



When configuring cluster resources for Microsoft Exchange, four prerequisites must be satisfied, each of these will be explained in more detail:

1. Microsoft Exchange Server 2003 must be installed on all cluster nodes where Microsoft Exchange Virtual Servers will run.

2. The Microsoft Distributed Transaction Coordinator (MSDTC) is required as a cluster resource and must be configured before Microsoft Exchange 2003 can be installed or Microsoft Exchange 2000 can be upgraded.

3. The network name must be created. As this resource is dependent upon the IP address, the IP address must be assigned and the resource created first.

4. The final step that must be accomplished before creating the Microsoft Exchange System Attendant is to create the physical disk resources that are required by the virtual server you are configuring. At a minimum, there must be at least one physical disk resource configured for Microsoft Exchange virtual servers to be added to the cluster configuration.

When Microsoft Exchange cluster resources start and stop (change states), they must do so in order of resource dependency. This means that on startup, resources start in forward order of resource dependence (bottom to top in Figure 5). When resources are stopped (or a resource group is taken offline), resources are shutdown in reverse order of dependence (top to bottom in Figure 5). When configuring cluster resources, having an understanding of the resource dependencies for Microsoft Exchange Server clustering makes the task simpler.

Microsoft Exchange virtual servers The virtual server is the key unit of client access for Microsoft Exchange Server 2003 services running in a clustered environment. Virtual servers exist for several different Microsoft Exchange Server services (as shown in Table 5). The virtual server is the name by which clients, resources, and other services access individual Microsoft Exchange services within the cluster. In Microsoft Exchange

19

Server 2003, a virtual server contains resources for Internet client protocols (SMTP, IMAP, POP3, and HTTP-DAV), the Information Store (for MAPI clients), the MTA service, and the Microsoft Search Service. The virtual server takes on the name property of the network name resource that is configured prior to configuring Microsoft Exchange resources. For example, if you configure the network name resource as “EXVS1,” each Exchange virtual server resource configured in the cluster resource group for that virtual server will respond to that virtual server name when used by clients and other services and resources.

Microsoft Exchange virtual server resources will all failover in the cluster as a managed unit. This means that the entire cluster resource group containing the virtual server resources will be failed over together from one node in the cluster to another. One or more Microsoft Exchange information storage groups can be configured as part of a Microsoft Exchange Server 2003 virtual server. However, a storage group can only belong to one virtual server. When deploying Microsoft Exchange Server 2003 clusters, ensure that you familiarize yourself with virtual servers and how they are used.

Microsoft Exchange Server 2000/2003 can support multiple virtual servers on a single node. In an active/active design, one node must be capable of taking on the workload of the other in the event of a failover. This limits the workload of each node (in a 2-node cluster), as you must leave ample headroom for the failover to occur. In light of this, Microsoft places restrictions on active/active design workloads. See Table 8 and the section preceding it for the restrictions. For clusters greater than two nodes, they must be designed in an N+I active/passive configuration.

What’s new in Microsoft Exchange Server 2003 From a clustering standpoint, the most relevant changes are:

• Support for Microsoft Windows 2003 features such as 8 nodes, volume mount points, and new quorum resource models.

• Additional prerequisite checking during Microsoft Exchange installation • Flattening of the resource dependency tree allowing for greater flexibility and shorter resource

failover times • The Microsoft Exchange 2003 permissions model has changed, for greater security

– IPSec support for front-end to back-end server connections – Cluster service account requires no Microsoft Exchange-specific permissions – Support for Kerberos (enabled by default)

• Many Microsoft Exchange 2000 Tuning Parameters no longer necessary, as Microsoft Exchange 2003 provides more accurate tuning in large memory and multi-processor servers

• The virtual memory management process is improved, to more efficiently reuse blocks of memory and thus reduce fragmentation

• On failover, Microsoft Exchange services are stopped on the passive node, which frees up memory and prevents fragmentation

• IMAP4 and POP3 resources are not added automatically when you create a Microsoft Exchange Virtual Server (but are upgraded if existing in Microsoft Exchange 2000)

• Change in the Setup routine so that it no longer prompts that a cluster has been detected (this would halt the setup until acknowledged)

See the Microsoft document What's New in Microsoft Exchange Server 2003, http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/newex03.mspx for a more comprehensive list.

http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/newex03.mspx

20

See also HP’s paper Exchange Server 2003 Clusters on Windows Server 2003 http://h71028.www7.hp.com/aa_downloads/6/100/225/1/65234.pdf

Planning your Microsoft Exchange 2003 cluster deployment Planning is critical for successful Microsoft Exchange Server 2003 cluster deployments. If you do not plan your implementation carefully, you may have to tear down your cluster implementation and re-implement it, including backup and restoration of critical data. The following areas should be addressed:

• Server Hardware • Storage Planning • Naming Standards • Microsoft Exchange Server roles • Microsoft Windows Server infrastructure • Cluster Service account

In addition to the following extensive list you may also wish to see the checklist: Preparation for installing a cluster at http://go.microsoft.com/fwlink/?LinkId=16302.

Best practices for hardware Check for hardware compatibility The hardware used in a cluster must be listed on the Microsoft Windows Catalogs from Microsoft. The Windows Catalogs are replacing the Hardware Compatibility List (HCL). The catalogs and HCL can be found at http://www.microsoft.com/whdc/hcl/default.mspx. If you implement hardware that is not in the catalog, Microsoft will not support your cluster configuration.

Choose standardized configurations You should configure all nodes in a cluster with identical specifications for memory, disks, CPUs, etc. Implementing nodes of varying specifications can lead to inconsistent performance levels as Microsoft Exchange Virtual Servers are moved between nodes. HP makes it easier to choose hardware by offering a wide selection of supported cluster configurations (see http://www.hp.com/servers/proliant/highavailability).

You may hear references to deployments of non-standardized configurations; however, be certain of the reasons for using varying hardware platforms. For example, within Microsoft’s internal clustering deployment, several passive nodes are used for failover targets during rolling upgrades. These are desirable mainly due to the frequency of applying new builds (since Microsoft frequently tests the newest versions; applying them into production during the development release cycle). In addition, varying hardware platforms can be used in a TEST cluster for evaluating new technology such as the difference between 4-way and 6-way platforms of varying processor speeds and cache sizes etc.

In addition to standardized hardware (from the Microsoft Windows Catalogs), Microsoft specifies that many configuration settings must be the same. For example, all network adapters on the same cluster network must use identical communication settings such as speed, duplex mode, flow control, and media type. The adapters connected to the same switch must have identical port settings on the switch. Ideally, the NICs for each network should be identical hardware models and firmware revision level in each server.

http://h71028.www7.hp.com/aa_downloads/6/100/225/1/65234.pdf

http://go.microsoft.com/fwlink/?LinkId=16302

http://www.microsoft.com/whdc/hcl/default.mspx

http://www.hp.com/servers/proliant/highavailability

21

In addition, each cluster network must be configured as a single IP subnet whose subnet is distinct from those of the other cluster network. The addresses may be assigned by DHCP, but manual configuration with static addresses is preferred, and the use of Automatic Private IP Addressing (APIPA) is not supported.

Hardware redundancy The key design principle of any cluster deployment is to provide high availability. In the event of a hardware failure, a failover operation will move resources from the failed node to another node in the cluster. During failovers of Microsoft Exchange, users will not be able to access e-mail folders for a short time as resources are taken offline on the failed node and brought online on the other node. For each node in the cluster, implement redundant hardware components in order to reduce the impact of a hardware failure and thus avoid a failover. Examples of components where redundancy can be implemented are as follows:

• Redundant Networks - Two or more independent networks must be used to connect the nodes of a cluster. A cluster connected by only one network is not a supported configuration. In addition, each network must fail independently of every other. No single component failure, neither a Network Interface Card (NIC) nor switch failure, should cause both cluster networks to fail. This also eliminates the use of a single Network Interface Card (NIC) with multiple ports for both cluster networks.

• Network Adapter or Network Interface Card (NIC) Teaming - NIC Teaming is a feature enabled on HP ProLiant servers by the addition of Support Pack software. Teaming allows multiple network cards to act as a single virtual card which is configured as if it were a single card. This allows for the failure of a single NIC card, cable or switch port without any interruption of service. When split across multiple network switches, switch failure can be handled. There are three types of teaming: Network Fault Tolerant (NFT), which handles faults but provides no additional throughput, Transmit Load Balancing (TLB), which can provide multiple outbound connections, and Switch-Assisted Load Balancing (SALB), which requires specific network switches, such as Cisco Fast EtherChannel. Use NFT teaming on the public (client) network in a cluster; however do not use NIC teaming for the private (heartbeat) network in a cluster as it is not supported by Microsoft (see Q254101). For more information on NIC Teaming, see HP.com - ProLiant networking - teaming, at: http://h18000.www1.hp.com/products/servers/networking/teaming.html.

• Power supplies and fans – Connect redundant power supplies to separate Power Distribution Units (PDU). If both power supplies are connected to the same PDU, and the PDU fails, power will be lost.

• Host Bus Adapter/Array Controllers – If you have implemented a Storage Area Network (SAN) with your cluster, implement redundancy into the connections between your nodes and the SAN, so that failures of the HBA, Fibre Channel connections, and SAN switches can be handled without the need to induce cluster node failover. HBA redundancy is controlled by HP StorageWorks Secure Path. For more information see: HP StorageWorks Secure Path for Windows: business continuity software - overview & features, at http://h18006.www1.hp.com/products/sanworks/secure-path/spwin.html.

http://h18000.www1.hp.com/products/servers/networking/teaming.html


http://h18006.www1.hp.com/products/sanworks/secure-path/spwin.html



22

Note: The latest version of HP StorageWorks Secure Path adds distributed port balancing across HBAs in a cluster. Previous versions did not load-balance, and required manually setting the paths (which was not persistent across reboots). If you are using an older version or are not sure, check the SAN throughput at the switch ports to see that they are both active.

Best practices for cluster configuration Understanding resource dependency In the earlier section on resource dependencies, it was made clear that there is a specific sequence in which resources are brought online or taken offline, as dictated by the resource dependency. It is critical in your design to understand that the Microsoft Exchange System Attendant Resource is dependent on all physical disks for that virtual server. This means that an entire Microsoft Exchange Server can be affected by failure of any disk resource, be it database or log file drives. For very large servers in a SAN environment, this exposes all users on that server to downtime for any disk resource failure, which is quite undesirable. It is essential that all drives be protected from any failure through RAID and redundant paths and that the number of disk resources is limited in order to reduce the risk of a drive failure impacting the Microsoft Exchange virtual server. For example, a very large Microsoft Exchange server with disks for four (4) Storage Groups, log files and separate disks for the SMTP mailroot, and each of twenty (20) databases would perhaps be best split into 2 or more virtual servers.

Planning cluster groups Create a cluster group to contain only the quorum disk resource, cluster name, and cluster IP address. The benefit of separating the quorum resource from the Microsoft Exchange cluster groups is that the cluster will remain on-line if one of the resources fails in a Microsoft Exchange cluster group. The cluster resource group owning the quorum can be run on the passive node, which isolates it from the load of the active nodes. If the active node experiences a hard fault resulting in node failover, the failover may be quicker because the cluster resource group is already online on the surviving node.

Planning quorum type Deciding on the quorum type is important, as the type of quorum resource determines the type of cluster. This decision between one of three types must be made before cluster installation. The cluster type is either a quorum-device cluster (using either a local quorum or a single quorum device on a shared SCSI bus or Fibre Channel) or a majority-of-nodes cluster. A quorum-device cluster is defined by a quorum using either the Physical Disk resource type or local quorum resource type (in either Microsoft Windows 2000 or 2003). The local quorum resource is often used for the purpose of setting up a single-node cluster for testing and development. In Microsoft Windows Server 2003, the Cluster service automatically creates the local quorum resource if it does not detect a shared disk (you do not need to specify a special switch as in the Microsoft Windows 2000 procedure). You may also create the local quorum resource, after installation of a quorum on a single quorum device (e.g., a multi-node cluster) in Microsoft Windows Server 2003. This can be beneficial if you need to replace or make other repairs on the shared disk resource subsystem (e.g., rebuild the drive array in a manner that is data destructive).

23

Figure 6. Selecting Quorum type during Cluster Setup

A majority-of-nodes cluster is only available in Microsoft Windows 2003, and is defined by a majority node set quorum, where each node maintains its own copy of the cluster configuration data. Determination of which node owns the quorum and whether the cluster is even operational, is done through access to file shares. This design helps in geographically dispersed clusters and addresses the ‘split-brain’ issue (where more than one node could gain exclusive access to a quorum device, because that device is replicated to more than one location). Since this design is much more complex, it is not advised unless you work carefully with the team or engineer providing the storage architecture. Microsoft states that you should use a majority node set cluster only in targeted scenarios with a cluster solution offered by your Original Equipment Manufacturer (OEM), Independent Software Vendor (ISV), or Independent Hardware Vendor (IHV)1.

Storage planning Storage design is critical Before building a cluster, you should try to estimate storage requirements. Microsoft Exchange databases expand over time and you may start to run out of disk space if you have not implemented sufficient capacity. Having spare capacity is also useful for performing database maintenance and disaster recovery exercises. Adding additional disk space to a cluster may require future downtime if your array controller does not support dynamic volume expansion. Before you can size mailbox stores, you must answer the following questions:

• How many users need mailboxes? • What is standard mailbox quota? • How many servers will be deployed? • How many Storage Groups will be deployed per server? • How many Databases per Storage Group? • Will the mailbox stores contain legacy e-mail and if so how much? • What is the acceptable time to restore a database in a Disaster Recovery Scenario? • What is the classification of mail usage (light, medium, heavy)? • How long is deleted mail retained for?

1

http://www.microsoft.com/technet/prodtechnol/windowsserver2003/proddocs/entserver/sag_mscs2planning_32.asp

24

• What mail clients are used: POP, IMAP, HTTP, or MAPI? (This affects whether mail will be primarily stored in the EDB or STM files).

To assist customers in sizing storage, HP has developed a Microsoft Exchange Storage Planning Calculator which is available for download from http://www.hp.com/solutions/activeanswers, specifically HP ProLiant Storage Planning Calculator for Microsoft Exchange 2000, http://h71019.www7.hp.com/1%2C1027%2C2400-6-100-225-1%2C00.htm.

Placement of Microsoft Windows and Microsoft Exchange components As part of the storage design process, you will need to consider the placement of the following Microsoft Windows and Microsoft Exchange components, listed in Table 6, below.

Table 6. Placement of Microsoft Windows and Microsoft Exchange components.

Component Default Location (disk\path)

Microsoft Windows Server binaries %systemroot%

Microsoft Windows Server pagefile Typically C:\

Quorum disk (for clusters) Administrator defined

Microsoft Exchange Server 2003 binaries %ProgramFiles%\Exchsrvr typically on C:

Microsoft Exchange Server Storage Groups Administrator defined – default for cluster is on ‘Data Directory’

● Microsoft Exchange Server mail stores (EDB and STM files)

Administrator defined – default for cluster is on ‘Data Directory’

● Microsoft Exchange Server transaction logs Administrator defined – default for cluster is on ‘Data Directory’

● Public Folder stores (EDB and STM files) See ‘Data Directory’ below

Microsoft Exchange Server 2003 SMTP mailroot folders (pickup, queue and badmail)

See ‘Data Directory’ below

Microsoft Exchange Server 2003 Message Transfer Agent (MTA) work directory

See ‘Data Directory’ below

Microsoft Exchange Server 2003 message tracking logs

Administrator defined – see FTI whitepaper on HP ActiveAnswers

Full text Indexing (FTI) files Administrator defined

Legacy e-mail connectors Default Location (disk\path)

Microsoft Windows Server Microsoft Windows Server should be implemented on a mirrored (RAID1) drive. For clusters holding large numbers of mailboxes, Microsoft recommends deploying a dedicated mirrored drive for the pagefile to achieve improved performance. A write-caching controller can help improvement performance of these disks (and it should be battery-backed for protection against data loss during power disruptions).

http://www.hp.com/solutions/activeanswers

http://h71019.www7.hp.com/1,1027,2400-6-100-225-1,00.htm

http://h71019.www7.hp.com/1%2C1027%2C2400-6-100-225-1%2C00.htm

25

Microsoft Exchange Server 2003 binaries Microsoft Exchange Server 2003 can be installed on to the same mirrored drive as Microsoft Windows Server with little impact to performance. A separate logical volume or disk array can be used for convenience if desired.

Storage groups, databases, and transaction logs A storage group is a Microsoft Exchange management entity within the Store process that controls a number of databases that use a common set of transaction logs. Microsoft Exchange databases are composed of:

• EDB files (*.EDB). The header and property information of all messages is held in the EDB file. • Streaming Database files (*.STM) – are the primary store for Internet clients/protocols such as

Microsoft Outlook Express/IMAP. Content in the streaming file is stored by the Microsoft Exchange Installable File System (IFS) in native format so messages with attachments in audio/video format do not require conversion when stored and accessed with IMAP clients. Automatic format conversion to the EDB file takes place when MAPI (Microsoft Outlook) clients try to access content information in the streaming file.

• Transaction Logs (*.LOG) – All changes to the database are first written to the transaction log, then to database pages cached in memory (IS buffers), and finally (asynchronously), to the database files on disk.

• Checkpoint files (*.CHK) – Checkpoint files keep track of which transactions have been committed to disk.

Figure 7 shows two storage groups. Storage Group 1 has two sets of EDB and Streaming databases. Storage Group 2 has three sets of EDB and Streaming Database. As mentioned previously, there is one set of transaction logs per storage group.

26

Figure 7. Storage Groups

Assigning drive-letters and labels Microsoft Knowledge Base article Q318534 describes some best practices for assigning drive letters on a cluster server. Microsoft recommends using Q as the quorum drive, and use letters R through to Z as drives in the shared storage. This has some advantages:

• Additional disks can be added to storage on the local node and use drive letters E and F (by convention drives C and D are used in local nodes for Microsoft Windows Server/Microsoft Exchange Server 2003 binaries)

• It reduces the risk of the administrator mapping a drive on a cluster node and causing a drive letter conflict when a failover occurs.

Another good tip from Q318534: label the drives to match the drive letter. For example, for the V drive, label it as DRIVEV, shown in Figure 8. In a disaster recovery situation the drive letter information might get erased. Using meaningful labels makes it easy to determine that DRIVEV was originally assigned the drive letter V.

http://support.microsoft.com/default.aspx?scid=kb;en-us;Q318534


27

Figure 8. Setting a drive label to match the drive letter.

Assign Q:\ to the quorum drive

By convention, the quorum resource is usually placed on a partition with the drive-letter Q. You should assign a separate physical disk in the shared storage to the quorum. This practice makes it easier to manage the cluster and your storage. If you share the physical drive with another server/application and you want to perform maintenance on the drive, you will have to take the cluster offline and move the quorum resource.

M:\ drive

In Microsoft Exchange 2000, the Microsoft Exchange Installable File System (ExIFS) created an M:\ drive as part of the installation. It was important in Microsoft Exchange 2000 that you did not assign the drive letter M:\ to any drives in the shared storage. If a partition has been assigned the letter M:\ or a network drive has been mapped using M:\, it would cause the Microsoft Exchange 2000 installation to fail. A new installation of Microsoft Exchange 2003 does not explicitly mount the ExIFS as drive M: but can be exposed via a registry key2. Servers upgraded from Microsoft Exchange 2000, however, may still expose the ExIFS as an M:\ drive.

Assigning drive-letters to storage groups and databases

In some scenarios, Microsoft Exchange cluster administrators have exhausted all available drive letters, especially on clusters with multiple storage groups, databases, and virtual servers. For example, an administrator decides to allocate a drive letter to each Microsoft Exchange database on a virtual server with four storage groups. In each storage group, there are six databases. In this scenario, 24 drive-letters will have to be assigned. Drives A, C, Q, and M may already be allocated

2 Requires creating a string key ‘DriveLetter’ in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EXIFS\Parameters.

28

to the local storage, Quorum disk, and IFS respectively so there are insufficient drive letters available in this scheme.

Microsoft Windows Server 2003 lifts the restrictions on drive letters by providing support for mount points in clusters. Mount points are used as an alternative to drive letters when formatting a partition; instead of selecting a drive letter, a folder in which to mount the drive is selected on a ‘parent’ drive. The new drive is then accessed via this folder. For example, in Figure 9 below, Disk 25 is mounted as the folder S:\Logs (where S: is Disk 21 in logical disk manager). Writes to S:\Logs will go directly to Disk 25, as shown in logical disk manager below.

Figure 9. Sample Disk Management view showing Disk 25 as Mount Point

See the section on mount points for more detailed information on proper configuration.

Another way to avoid drive letter restrictions is as follows: do not dedicate volumes to individual Microsoft Exchange databases. Create partitions large enough to hold multiple Microsoft Exchange databases and group all databases from the same storage group on the same partitions. If a partition contains databases from multiple storage groups and the disk resource goes offline, it will dismount databases in all of the storage groups on that disk partition.

29

Choosing RAID levels for Microsoft Exchange components All EDB and streaming databases in the same Storage Group should be placed on the same drive. If files are placed across multiple volumes and one volume fails, the other databases would still be affected. At a minimum, the Stores will go offline and then come back online if possible. Choose RAID 0+1 (or RAID5 only if assigning sufficient physical spindles for both storage space and performance) for EDB and streaming databases and keep the STM and EDB files on the same partition. This makes it easier to locate and manage database files. Also, if you place STM and EDB files on different partitions and lose one or the other, they must both be restored. Use the HP storage planning calculator for Microsoft Exchange whenever possible to ensure that the design includes enough physical spindles.

Transaction logs should be placed on a mirrored (RAID 1 drive). Transaction logs allow you to replay transactions that have taken place since the last backup. They should be placed on drives separate to the databases to facilitate recovery and for performance reasons.

SMTP mailroot, message transfer agent, message tracking logs

The SMTP folders, Message Transfer Agent, and Message Tracking Logs should be placed on a mirrored drive, separate from the database and transaction logs. The SMTP virtual server can achieve better performance when spread over multiple disks using RAID 1 (2 disks) or RAID 0+1 (4 or more disks in an even number). The HP Enterprise Virtual Array can spread the virtual disk over many physical spindles for high performance, and also provide RAID protection as vraid1 with hot sparing.

Keeping these components separate from the databases also makes it easier to evaluate the performance of the database and queue drives. Note that the SMTP mailroot is placed on the designated data drive in a cluster (which is external storage by definition), and that the steps for relocating the SMTP mailroot for Microsoft Exchange 2000 described in Q318230, 318230 - XCON: How to Change the Exchange 2000 SMTP Mailroot Directory Location, were not supported in a cluster. Microsoft Exchange 2003 exposes the location for the SMTP Badmail directory and the Queue directory in the Microsoft Exchange System Manager as shown in Figure10 below. These are properties of each SMTP virtual server, and will be grayed-out unless you run the Microsoft Exchange System Manager directly on that server.



30

Figure 10. Microsoft Exchange 2003 System Manager SMTP property for SMTP Badmail and Queue directories

Allocate dedicated spindles to each Microsoft Exchange virtual server Each disk in the SAN should be allocated to one Microsoft Exchange virtual server and one storage group. Do not split spindles across multiple Microsoft Exchange virtual servers and cluster groups. If you need to take a disk offline for maintenance, only one Microsoft Exchange virtual server would be impacted.

Using mount points As stated earlier, mount points are used as an alternative to drive letters when formatting a partition; instead of selecting a drive letter, a folder in which to mount the drive is selected on a ‘parent’ drive. The new drive is then accessed via this folder, for example S:\Logs (where S: is disk 21 in logical disk manager) will write directly to disk 25, as shown in logical disk manager in Figure 11. Microsoft Windows Server 2003 eases restrictions on assigning drive letters by providing support for mount points in clusters.

Resource dependencies

Mount points are a physical disk resource type and should be created as dependent on the parent resource (the drive which is assigned a letter). If the parent resource goes offline then the junction point for the volume mount point (VMP), which is a folder on parent resource, is no longer available and writes to the VMP will fail. Thus, it is critical that the mount point be gracefully taken offline first, forcing all outstanding writes.

31

Recovering drives with mount points

If it is necessary to replace or recover the parent drive resource, the mount point must be re-associated with the folder on the parent drive. This is done by selecting the mount point volume in Disk Manager (Diskmgmt.msc) and selecting Change Drive Letter and Paths… Figure 11 below shows browsing for the folder to associate a volume mount point.

Figure 11. Re-associating an Existing Volume Mount Point with a Folder in Disk Manager

Warning: If using a Volume Mount Point for Microsoft Exchange Storage Group Logs and the parent drive resource has been replaced, DO NOT bring the Information Store resource online or it will create the Log folder on the drive (as a folder and not a mount point). This folder will have newly created Microsoft Exchange transaction logs, which must be removed.

Use a consistent naming standard for folders and databases You should use a consistent naming standard for Microsoft Exchange folders and databases. It makes it easier to determine which Storage Group a database belongs to. A suggested naming convention is shown in Table 7. In this example, one could determine from the name of the file that the mailbox store EDB file V:\exchsrvr\SG2_MDBDATA\SG2Mailstore3.EDB is mailbox store 3 and is owned by Storage Group 2.

32

On clusters with multiple Microsoft Exchange virtual servers you should extend the naming standard to include the virtual server. For example, the use of VS1 in the filename V:\exchsrvr\SG2_VS1_MDBDATA\SG2VS1Mailstore3.EDB denotes that this database belongs to virtual server 1.

All Microsoft Exchange components should be placed in folder trees that have ‘Exchsrvr’ or ‘Exchange’ as the root folder name. This makes it easier for the administrator to determine that Microsoft Exchange components are stored in the folders.

Table 7. Example Naming standard for Microsoft Exchange folders

Component Folder Name

Microsoft Exchange Binaries D:\exchsrvr\BIN

Message Transfer Agent R:\exchsrvr\MTADATA

Message Transfer Agent Work Directory R:\exchsrvr\MTADATA

SMTP Mailroot R:\exchsrvr\Mailroot

Message Tracking Logs R:\exchsrvr\Yourservername.log

Storage Group 1

SG1 Transaction Logs S:\exchsrvr\SG1_TRANSLOGS\

Database folder T:\exchsrvr\SG1_MDBDATA

SG1Mailstore1 T:\exchsrvr\SG1_MDBDATA\SG1Mailstore1.EDB

T:\exchsrvr\SG1_MDBDATA\SG1Mailstore1.STM







Storage Group 2

SG2 Transaction Logs U:\exchsrvr\SG2_TRANSLOGS\

Database folder V:\exchsrvr\SG2_MDBDATA

SG2Mailstore1 V:\exchsrvr\SG2_MDBDATA\SG2Mailstore1.EDB

V:\exchsrvr\SG2_MDBDATA\SG2Mailstore1.STM

33

Component Folder Name







Obtain TCP/IP addresses prior to installation For a two node Microsoft Exchange Server 2003 Active/Passive cluster, you will need to assign IP addresses for the following network resources:

• Node 1 (Public Network) • Node 2 (Public Network) • Microsoft Exchange Virtual Server • Cluster IP Address • Node 1 (Private Network) • Node 2 (Private Network)

In Active/Active deployments, you will need to assign an IP address for a second virtual server. If you plan to implement HP ProLiant Integrated Lights-Out (iLO) or Remote Insight Lights-Out Edition boards for remote management of the cluster nodes, be sure to allocate an additional IP address for each card.

Cluster naming conventions Given the additional complexity and terminology of clusters, it is a good idea to choose a consistent naming convention to aid understanding. In a two node Microsoft Exchange cluster, you will need to assign names to cluster groups, node names, and virtual servers, as shown in Cluster Administrator in Figure 12. Here is a suggested naming convention:

• Cluster Node 1 XXXCLNODE1 • Cluster Node 2 XXXCLNODE2 • Cluster Network Name XXXCLUS1 • Microsoft Exchange Group Name XXXEXCGRP1 • Microsoft Exchange Virtual Server Name XXXEXCVS1

XXX – represents a site or company code to match your naming convention CL – represents a cluster node/name EXCGRP – represents a Microsoft Exchange cluster group EXCVS – represents a Microsoft Exchange virtual server name

34

Figure 12. Cluster entities viewed from Cluster Administrator

Microsoft Exchange Server roles Implement dedicated Microsoft Exchange clusters Microsoft Exchange clusters should not be implemented to double up as File and Print Servers, Microsoft SQL Servers, or infrastructure servers (DHCP/WINS/DNS). If a server has multiple roles, it makes it more difficult to manage, troubleshoot performance issues, and also adds additional complexity to disaster recovery.

Assign specific roles to clusters for enterprise deployments Deploying servers with dedicated Microsoft Exchange roles is a best practice for Microsoft Exchange 5.5 and earlier but also holds true for Microsoft Exchange Server 2003. Microsoft Exchange Server 2003 Server roles can be described as follows:

• Mailbox servers • Public Folder servers • Bridgehead servers • third-party connector servers • Instant Messaging servers • Conferencing Servers • Site Replication Services (SRS) Servers • Active Directory Connector (ADC) Servers • Front-end Servers (OWA, SMTP, IMAP, POP)

35

First server and site replication services (SRS) server When installing Microsoft Exchange Server 2000 or 2003 into a mixed-mode environment (to interoperate with Microsoft Exchange 5.5), the first server installed will host the Site Replication Services (SRS). The SRS is used to communicate with the Microsoft Exchange 5.5 Directory Service. The SRS is not supported on a cluster, so a standalone server needs to be installed before the cluster.

Implement clusters as mailbox servers and public folder servers Clusters should be used as mailbox servers and Public Folder servers only. Here are some reasons to keep Connectors and Bridgeheads on separate machines:

• The I/O patterns of message transfer are different than of end-user usage. Separate machines allows for better tuning.

• Putting connectors on mailbox servers reduces performance of the latter by 20-30%. • Some connectors only run on one node of the cluster. • Some third-party connectors may not be supported on a cluster. • As mentioned previously, many Microsoft Exchange components are not supported in a cluster and

must be implemented on standalone servers. Examples of such components are Instant Messaging and NNTP. Legacy connectors such as the Microsoft Exchange connector for Lotus cc: Mail and Novell GroupWise are not supported on clusters.

For a full list of the components supported on clusters, consult Microsoft Knowledge Base Article Q259197. When planning your cluster deployment, be sure to allocate standalone Microsoft Exchange servers to meet the requirements for your environment.

Clusters and active/active scalability limits Microsoft’s recommended cluster configuration for Microsoft Exchange Server 2000 and 2003 is active/passive. The scalability limits of Microsoft Exchange Server 2000 active/active clusters are covered in Microsoft’s Deployment Guide for Exchange Server 2000 Service Pack 3, as shown in Table 8.

Table 8. Microsoft’s active/active cluster restrictions

Microsoft Exchange 2000 version

Maximum number of concurrent connections

RTM 1,000

SP1 1,500

SP2 1,900

SP3 1,900

All Microsoft Exchange Server 2003 clustering recommendations from Microsoft are for active/passive cluster configurations. Active/active clustering is only supported on two nodes, and the scalability limits of Microsoft Exchange 2003 active/active clusters should be considered the same as the last release of Microsoft Exchange 2000 (SP3) unless stated otherwise.


http://www.microsoft.com/technet/prodtechnol/exchange/exchange2000/proddocs/library/e2ksp3.asp

36

What’s new in Microsoft Windows Server 2003 See http://www.microsoft.com/windowsserver2003/evaluation/overview/technologies/default.mspx from Microsoft for what’s new in Microsoft Windows Server 2003.

Of notable interest for clustering:

• Configurations of up to 8 nodes • New Cluster Server Wizard to assist in installation, including multiple simultaneous nodes • Inclusion of the cluster component in Microsoft Windows (removing the need to reboot after adding

it) • Similarly, removing a node from a cluster by evicting it does not require a reboot • Improved installation routine with:

– software and hardware configuration error detection and – multiple node simultaneous deployment

• Better support for geographically dispersed cluster configurations, especially using Majority Node Set quorums

• Virtual servers exist as objects in Active Directory, meaning that they can be managed within AD and use Kerberos authentication (introduced in Microsoft Windows 2000 Service Pack 3, see Microsoft Knowledge Base Article - 235529 Kerberos Support on Windows 2000-Based Server Clusters)

• Improved storage support:

– Support for volume mount points – Support for online dynamic volume growth (using DiskPart)

• Improved network state detection and response • Better support for large-memory environments (in the 4GB to 8GB range) and applications that can

use this memory • Almost all functionality of Cluster Administrator tool is available in the cluster.exe command-– it can

be used to create new clusters, add servers to existing clusters and remove servers from clusters. • WMI provider that allows a Server cluster to be managed and gain access to cluster events (such as

server up, server down, resource online, resource offline, resource failed, etc.) • Improvements to Backup and Restore: Disk Signatures can be restored with a new Automated

System Restore Wizard. The cluster quorum can now be restored with NTBACKUP (Clusrest.exe is no longer required).

This is a primary consideration: whether you will run Microsoft Windows Server 2000 or 2003, as Microsoft Exchange 2000 can not run on Microsoft Windows Server 2003, yet Microsoft Exchange 2003 can run on either Microsoft Windows 2000 or 2003.

Best practices for server configuration This section describes how to configure Microsoft Windows Server 2000 and 2003 for optimal Microsoft Exchange Server 2003 performance. Ideally these steps should be performed before Microsoft Exchange Server is installed on your cluster, but they may also be performed after Microsoft Exchange Server has been installed.

http://www.microsoft.com/windowsserver2003/evaluation/overview/technologies/default.mspx

http://support.microsoft.com/support/kb/articles/q235/5/29.asp

37

Build redundancy into your Microsoft Windows Server infrastructure A critical element to the success of any Microsoft Exchange Server 2003 deployment is a stable and resilient Microsoft Windows Server infrastructure. Microsoft Exchange Server 2003 stores much of its configuration information in Active Directory Domain Controllers (DCs) and Global Catalog Servers (GCs). DCs hold a complete copy of all the objects belonging to a domain plus a copy of objects replicated in the forest-wide configuration naming context (NC). GCs hold a complete copy of all objects in their own domain plus partial copies of objects from all other domains within the forest. Redundancy can be implemented into the Microsoft Windows Server infrastructure as follows:

• Implement two GCs in the same Microsoft Windows Server site and LAN as your Microsoft Exchange clusters. DSAccess is the Microsoft Exchange component responsible for locating and retrieving configuration information. If DSAccess is unable to contact a GC, the System Attendant will fail and a failover will take place. Implementing two GCs mitigates the impact of a GC going offline. If there is another GC available in the site, Microsoft Exchange will use DSAccess to locate it. If there is no GC available in the site, Microsoft Exchange will attempt to use GCs in other sites and this will result in downtime as DSAccess attempts to locate a GC. GCs are also used by Outlook clients to query and retrieve the Global Address List. If a GC goes offline, Microsoft Outlook sessions are adversely impacted.

• Microsoft Exchange 2003 has specific a requirement for GCs\DCs to be running on Microsoft Windows 2000 Service Pack 3 or later or on Microsoft Windows 2003. Microsoft Exchange 2003 cannot bind to a GC running Microsoft Windows 2000 RTM, SP1, or SP2. Before deploying your Microsoft Exchange 2003 cluster, ensure GCs\DCs are running at the required revision levels.

• Implement multiple DNS Servers. DNS (Domain Name Services) is used by Microsoft Windows Server to resolve server names to TCP/IP addresses and locate resources. If a DNS server goes offline, and no secondary DNS server is available, Microsoft Exchange may start to experience issues such as DSAccess errors, non-delivery of mail, and authentication problems. These problems would arise because Microsoft Exchange would be unable to resolve server names to TCP/IP addresses.

• Implement multiple WINS Servers. WINS (Windows Internet Naming Service) servers are used to resolve NetBIOS names to IP addresses. WINS is used in Microsoft Windows networks for name resolution so it will be required if your Microsoft Exchange cluster is running in a mixed Microsoft Exchange Server 2003/5.5 Organization.

Cluster service account best practices The cluster service account is used by the cluster service to log on and join a cluster. It has ‘logon as service’ rights on each node and is a member of the local administrators group. Do not use the default Administrator account as the cluster service account. If the administrator account gets disabled, it will cause the cluster service to fail.

Create one cluster service account per cluster This reduces the impact if the cluster service account gets locked out or disabled. Instead of affecting all your clusters, only one cluster will be impacted. This is especially helpful if different persons or teams manage different clusters.

Use a consistent naming convention for cluster service accounts. Examples of meaningful account names are XXX-clus1-svc, XXX-clus2-svc. XXX indicates a site code, clus1 represents the cluster name, and svc indicates the account is a service account.

38

Do not use the cluster service account to logon interactively To reduce the risk of the service account becoming disabled, do not use the service account to log on to a cluster – other than during the initial installation and configuration of Microsoft Exchange. Instead, create a separate security group for cluster administrators. Add the users who manage clusters to this security group, and add this security group to the local administrators group on each cluster.

Delegate Microsoft Exchange permissions Use care when configuring Microsoft Exchange Admin accounts and privileges. In Microsoft Exchange 2000, cluster service accounts require Microsoft Exchange Full Administrator rights to create and manage a Microsoft Exchange Virtual Server. Microsoft Knowledge Base Article Q289811 describes how you can use the Microsoft Exchange Delegation Wizard to grant these rights.

Microsoft has changed the permissions model in Microsoft Exchange 2003. To install or upgrade the first Virtual Server in an organization requires Microsoft Exchange Full Administrator role for the organization. Similarly, to install or upgrade the first Virtual Server in an Administrative Group requires Microsoft Exchange Full Administrator role for the organization. Subsequent upgrades or installations require only Microsoft Exchange Full Administrator role at the Administrative Group level. See the Microsoft Exchange 2003 Deployment Guide for more information: http://www.microsoft.com/technet/prodtechnol/exchange/Exchange2003/proddocs/library/DepGuide.asp.

Upgrade to the latest service pack on each node Before forming the cluster apply the latest Microsoft Windows Service Pack and hotfixes. Take care to ensure that all nodes in the cluster are at the same revision level.

Change the temp folder used by the cluster service account Under heavy load, a Microsoft Exchange 2000 Server (the Installable File System component) may create temporary files in the …\<service account profile>\Local Settings\Temp folder. These files are created with filenames such as LB10.tmp, LB11.tmp, as shown in Figure 13. These files are created when a large message or file is streamed into the store and the streaming file is too fragmented to handle the entire object one at a time. These files are created and deleted by the Store process and do not have to be manually deleted. However, on large clusters there is a risk that these temporary files can consume all available disk space. This behavior is documented in Microsoft Knowledge Base Article Q294462. This article references Microsoft Exchange 2000 only, and not Microsoft Exchange 2003. In a Microsoft Exchange 2003 cluster, the LB*.tmp files are written to the System TEMP and TMP variable location (e.g. %systemroot%\temp).


http://www.microsoft.com/technet/prodtechnol/exchange/Exchange2003/proddocs/library/DepGuide.asp

http://www.microsoft.com/technet/prodtechnol/exchange/Exchange2003/proddocs/library/DepGuide.asp

http://support.microsoft.com/default.aspx?scid=kb;en-us;294462

39

Figure 13. Temporary files created by the Microsoft Exchange IFS

To change the location for these temporary files to a disk with sufficient disk space and fast access, change the Temp and Tmp environment variables used by the cluster service account. To change the temp folder:

• Right click My Computer, select Properties • Select Advanced, click Environment Variables • Change the value of TEMP and TMP folders, shown in Figure 14 • Make this change on each node in the cluster • Reboot the each node in the cluster

40

Figure 14. Environment variables used by the cluster service account

Network configuration A key step in the successful implementation of a cluster is the setup of the network connections. Clusters have two networks. An internal network, called the cluster interconnect or heartbeat connection, enables each node to see if the other node is online. The heartbeat connection usually consists of a crossover-cable connection or a dedicated hub between nodes. A public network is the clusters connection to the LAN, and can also be used for heartbeat communication.

Separate private and public networks As discussed earlier, you should identify separate private and public networks and reserve the private network for heartbeat traffic only (Internal Cluster Communications Only). The private network should be the highest priority for internal cluster communications. The following sections will identify additional changes to the private network, such as not using NIC teaming, DHCP, DNS, WINS, or a default gateway on the private NICs. For two-node clusters you may use a crossover cable for the private network; however, this is not possible for more than two nodes.

Do not use DHCP for cluster network connections Do not use DHCP for private or public connections. An exception to this rule – you can assign a DHCP reservation to cluster network connections. If a TCP/IP address changes on a network connection, it can cause the cluster to fail an ‘isalive’ check and trigger a failover. Changing the IP address on a cluster is not straightforward. When an IP address is changed, it can take some time to replicate to the cluster metabase. The ‘isalive’ check can fail and cluster resources will not come online. See Microsoft Knowledge Base Article Q230356 for more information.


41

Label network connections On each node, label network connections so that they have meaningful names. Label the cluster interconnect as Heartbeat, and Public facing connection as ‘LAN’ or ‘Public, as shown in Figure 15. Labeling the network connections in this way makes it easier to troubleshoot and configure network connections.

Figure 15. Labeled Network Connections

Modify the binding order on network connections Occasionally, Microsoft Windows clusters can be built with the cluster heartbeat set higher in the binding order than the public-facing LAN. Modify the binding order so that the Public-facing connection is highest, followed by the heartbeat, as shown in Figure 16. To check the binding order, go to Network Connections, Advanced, Advanced Settings, Adapters and Bindings. Standardize the binding order on each node.

42

Figure 16. Correct Binding Order For Network Connections

Disable unnecessary protocols, services on the cluster interconnect connection The following changes should be made on the Private Cluster NIC on both nodes in the cluster:

• Disable Client for Microsoft Networks and File and Printer Sharing • Disable register this connection with DNS

• Disable NetBIOS

These are enabled by default as part of the Microsoft Windows Server installation. The ‘Client for Microsoft Networks’ and ‘File and Printer Sharing’ components are not required by the cluster interconnect. The cluster interconnect is used solely for intra-cluster node communications; it should not try and register with DNS. The correct settings are shown in Figure 17:

43

Figure 17. Component and DNS Settings for the cluster inter-connect

Set correct settings for cluster communications As shown in Figure 18 below, use Cluster Administrator to set communications on the Private network to Internal only and set properties in the Public network to All communications (mixed network). Setting the Public network to ‘All communications’ allows the cluster heartbeat to function over the Public network in the event of a failure on the Private network. Incorrect cluster communication settings can cause failovers to malfunction. Microsoft Knowledge Base article 258750 contains guidelines on the correct network configuration for a cluster.

http://support.microsoft.com/default.aspx?scid=kb;EN-US;258750

44

Figure 18. Cluster Communication settings

NIC teaming configuration The HP Network Teaming and Configuration Utility combines two Network Interface Cards (NICs) or the two ports of a dual-port NIC into a single resource, creating a fault tolerant configuration. The utility is available in the HP Support Pack for Microsoft Windows Server (NTCSP) on the HP SmartStart CD or at http://h18023.www1.hp.com/support/files/server/us/smartstartinfo.html; however, Datacenter customers need to ensure that they are using the correct revision of HP SmartStart. The Datacenter program provides a certification process specific to each HP ProLiant server platform, and NIC teaming is validated for each HP ProLiant server model (for example, DL760) used in the Datacenter program.

HP NIC Teaming operates in one of two modes, selected in the Configuration Utility. See the earlier section on “Best practices for hardware” for an explanation of the differences between Network Fault Tolerant (NFT) and Load Balancing modes.

Further information on HP NIC Teaming can be found in a white paper at HP.com - ProLiant networking - teaming,3 http://h18004.www1.hp.com/products/servers/networking/teaming.html. According to this white paper, Microsoft Windows Server provides a NIC monitoring resource as part of the Cluster Service. If the NIC fails, the Cluster Service will recognize this and fail over the applications and services to the other node. For maximum availability, a redundant NIC team is recommended.

A Microsoft Knowledge Base article on Network Adapter Teaming and Server Clustering (Q254101), states that the use of NIC teaming on the private cluster interconnect is not supported because of possible delays in the transmission and receipt of heartbeat packets. Using NIC teaming on the public or client network is acceptable. If problems arise, Microsoft Product Support may require that NIC teaming be disabled during troubleshooting. If it is determined that the problem directly involves the NIC teaming, PSS will redirect the support request to HP.

3 Direct link to the whitepaper: ftp://ftp.compaq.com/pub/partners/microsoft/infolib/high-availability/147q-0101a-wwen.pdf

http://h18023.www1.hp.com/support/files/server/us/smartstartinfo.html




45

IPSec While it is now possible to use Internet Protocol security (IPSec) in a Microsoft Exchange 2003 cluster for front-end to back-end connections, Microsoft discourages the usage of IPSec unless you acknowledge and are willing to tolerate outages of five minutes or possibly more. The reason for the five minute interval is that the default time-out or lifetime for the Security Association Idle Timer is five minutes, and clients may not be able to reestablish connections until at least five minutes after all resources are brought back online after a failover. See Microsoft Knowledge Base Article – 306677.

Geographically-dispersed clusters Geographically-dispersed clusters (often referred to as stretch- or distance-clusters) add additional requirements from Microsoft – such as: the private and public networks between cluster nodes must appear as a single, non-routed LAN, which may require using virtual LANs (VLANs). As with LANS, each VLAN must fail independently of all other cluster networks. The round-trip latency between any pair of cluster nodes must be no more than 500 milliseconds. In order to avoid the split-brain situation (where more than one node thinks it owns the quorum resource), geographically-dispersed clusters may require use of the majority node set for the quorum (covered later in this document).

Set staggered boot delays on each node When power is restored after a power failure, each node in the cluster will attempt to access the shared storage at the same time. To avoid this happening, set the boot delay on the boot menu for your preferred passive node to a longer period compared to the active node. The boot time can be increased or reduced under properties of My Computer, Advanced, Startup and Recovery (shown in Figure 19). Set node 1 to 20 seconds and node 2 to 60 seconds. Alternatively, you can manually edit the boot.ini file, as you may be performing that step in the next section.

http://support.microsoft.com/?kbid=306677

46

Figure 19. Setting the boot menu delay

OS tuning for large memory More than 1GB of RAM in Microsoft Windows 2000 Advanced Server If your Microsoft Windows 2000 server has more than 1 gigabyte (GB) of RAM you need to add the switch /3GB to the startup line (boot.ini). See Microsoft Knowledge Base article 266096 for detailed instructions for modifying the boot.ini file. This change is required so that Virtual Memory is handled properly on servers with more that 1GB of RAM.

Caution: For servers with more than 1GB but less than 4GB of RAM, the /3GB switch can negatively affect operating system performance. Test and monitor the impact before deploying on production systems. Most notably, monitor SystemPTEs.

More than 1GB of RAM in Microsoft Windows 2003 Server Both Microsoft Windows 2003 Standard and Enterprise editions support the use of the switch /3GB in the startup line (boot.ini). However, they both add a new switch: /userva which allows a custom environment size for the application virtual address space. For Microsoft Exchange Server 2003, it is recommended that only /userva=3030 be used.


47

Cluster validation testing Before installing Microsoft Exchange on to your cluster verify that failover is functioning by performing the following tests:

Planned failover The first test tests planned failover where the active node shuts down gracefully.

• Move the cluster group and Microsoft Exchange group from node 1 to node 2 • Move the cluster group and Microsoft Exchange group from node 2 to node 1 • Shut down node 1 • Verify cluster group and Microsoft Exchange group failover correctly to node 2 • Power down node 2 • Power up node 1 • Verify cluster fails over correctly to node 2 • Verify Fibre Channel path failover (for example using Secure Path)

Unplanned failover This test simulates an unscheduled power loss or hardware failure to one of the nodes in the cluster.

• Move the cluster group and Microsoft Exchange group from node 1 • Remove power to node 1 • Verify cluster group and Microsoft Exchange group failover correctly to node 2 • Power up node 1 • Remove power to node 2 • Verify cluster fails over correctly to node 1

Power outage Test This test verifies that the cluster can boot up correctly from an unscheduled power down of all nodes and the storage subsystem.

• All nodes in the cluster are online • Remove power to the shared storage, and all nodes in the cluster • Power up the shared storage, and all nodes in the cluster • Verify each node comes on line and that the cluster is formed correctly

Best practices for Microsoft Exchange Server installation Detailed procedures and best practices for Microsoft Exchange Server cluster installation can be found in the HP whitepapers listed in the section titled For more information.

Prerequisites The requirements that must be met before upgrading or installing a Microsoft Exchange cluster include those for a stand-alone (non-clustered) server and a few additional prerequisites. For example, be sure to check application compatibility for any add-ins running on the server (such as special purpose connectors or anti-virus).

48

Another prerequisite for installing a Microsoft Exchange cluster is that you must first create a resource for the Microsoft Distributed Transaction Coordinator (MSDTC) in the cluster. This can be done using Cluster Administrator or the cluster command, such as:

cluster <clustername> res "MSDTC Resource" /CREATE /GROUP:"MSDTC Group" /TYPE:"Distributed Transaction Coordinator"

The Microsoft Exchange CDO component has a dependency on the MSDTC for installation; however, once Microsoft Exchange is installed, you may remove the MSDTC resource. If you do so, be sure to change the MSDTC Service startup from automatic to disabled. If you keep the MSDTC resource, you may consider placing it in its own group that contains a Physical Disk, Network Name, and an IP Address, as it will require dependencies on these. Another option is merely to remove the property “affect the group” so that if the MSDTC resource fails it will not affect any other resources in that group.

See Microsoft Knowledge Base Article - 301600 How to Configure Microsoft Distributed Transaction Coordinator on a Windows Server 2003 Cluster. MSDTC requires a disk resource, an IP address resource, and a network name resource to be created before an MSDTC resource can be installed.

See also Deploying a New Exchange 2003 Cluster or Upgrading an Exchange 2000 Cluster to Exchange 2003 in the Microsoft Exchange Server 2003 Deployment Guide at http://www.microsoft.com/exchange/library.

Document your cluster installation When your cluster installation is complete, take time to document your cluster configuration. This information may be critical in a Disaster Recovery scenario and it is also useful background information for other System Managers who will be managing the cluster. Table 9 shows a sample checklist for this documentation.

Table 9. Sample 2-Node Cluster Configuration Template

Node 1 Node 2 Virtual Server 1 Cluster

Administrative Group Name

N/A N/A Geo1 N/A

Service Account Details

NETWORK CONFIGURATION

Network Name EXCLNODE1 EXCLNODE2 EXCVS1 EXCLUS

TCP/IP Addresses

Public Network x.x.x.x x.x.x.x x.x.x.x x.x.x.x

Gateway x.x.x.x x.x.x.x

DNS Servers x.x.x.x

y.y.y.y

x.x.x.x

y.y.y.y

WINS Servers x.x.x.x x.x.x.x

Subnet Mask x.x.x.x x.x.x.x

Private Network 10.0.0.1 10.0.0.2 N/A N/A

Remote Insight Board x.x.x.x x.x.x.x N/A N/A

http://support.microsoft.com/default.aspx?kbid=301600

http://www.microsoft.com/exchange/library

49

Node 1 Node 2 Virtual Server 1 Cluster

CLUSTER CONFIGURATION

Cluster Group Name N/A N/A EXCGRP1 Cluster Group

Preferred Owner EXCLNODE1

Disk Configuration

Boot disks C:\ - 18GB Mirrored (W2K Binaries)

D:\ - 18GB Mirrored (E2K Binaries + Swap File)

C:\ - 18GB Mirrored (W2K Binaries)

D:\ - 18GB Mirrored (E2K Binaries + Swap File)

N/A

Shared Storage F:\ 90GB RAID 0+1 (Storage Group 1 DB’s)

G:\ 90GB RAID 0+1 (Storage Group 2 DB’s)

F:\Logs (Mount Point) 18GB Mirrored (Storage Group 1 Transaction Logs)

G:\Logs (Mount Point) 18GB Mirrored (Storage Group 2 Transaction Logs)

F:\ExData (Mount Point) 18GB Mirrored (SMTP Mailroot, MTA, Message Tracking Logs)

Q: - 9GB Mirrored (Quorum database)

Folder Paths C:\WINNT – Microsoft Windows Server

D:\Exchsrvr – Microsoft Exchange Server 2003

C:\WINNT – Microsoft Windows Server

D:\Exchsrvr – Microsoft Exchange Server 2003

F:\Exchsrvr\SG1DB

G:\Exchsrvr\SG2DB

F:\Exchsrvr\ExData

Disk Signatures

Front-end / back-end architectures For full explanation of front-end / back-end architectures, see Microsoft Knowledge Base Article - 246739 Exchange Server Front-end/Back-end Terminology and Implementation.

Host headers

Typically, on a clustered back-end server, you must enter a host header for every front-end virtual server. As an alternative, remove the host name and use a blank header. However, this contradicts the statement “Exchange Server Cluster Services requires that a specific virtual server name be in the host header field” from Microsoft Knowledge Base Article - 287726 - XCCC: How to Configure Host Header and Authentication Information in Exchange 2000 Outlook Web Access on a Windows 2000 Cluster Server,

http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q246/7/39.ASP&NoWebContent=1




50

http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/Q287/7/26.ASP&NoWebContent=1 .

Upgrading Microsoft Exchange 5.5 clusters Microsoft Exchange 5.5 Servers and clusters can not be upgraded in place. The recommended procedure is to deploy a new Microsoft Exchange 2003 clustered server and move the mailboxes to it. This does not allow for re-using the existing server name, unfortunately. If using existing hardware for the cluster, one Microsoft Exchange 5.5 node can be evicted and the Microsoft Exchange 2003 cluster built using one node. After all mailboxes are moved, the original node can be destroyed and rebuilt to join the new cluster. Note that the resource properties will need to be set to include the second node once it has joined the cluster.

Upgrading Microsoft Exchange 2000 clusters A Microsoft Exchange 2000 (SP3 or later) to Microsoft Exchange 2003 upgrade can be done in place. It is performed on one node at a time, on the inactive (passive) node. Be sure to check application compatibility for all add-ins running on the server (such as special purpose connectors or anti-virus). Third-party products for backup and antivirus should be tested and if necessary upgraded so that they work with Microsoft Exchange 2003. Microsoft Windows 2000 SP4 or Microsoft Windows 2000 SP3 with hotfix 329938 is required for the cluster nodes. Apply any operating system upgrades several days before upgrading Microsoft Exchange, just in case. If Microsoft Windows and Microsoft Exchange are upgraded at the same time, it is more difficult to roll back the changes if there are difficulties.

The JET file format used by the Store in Microsoft Exchange 2003 is the same as Microsoft Exchange 2000 SP3, so the installation procedure does not need to upgrade any databases. However, when Microsoft Exchange 2003 mounts the databases for the first time, it stamps them with information to indicate that the database has been accessed by a new version of ESE, the Extensible Storage Engine that connects the logic in the Store with the underlying JET database. The net effect is that you cannot restore backups taken with Microsoft Exchange 2000 to a Microsoft Exchange 2003 server. Before you upgrade, take full backups of your server including the databases, system state, and system drive. Use tapes outside your normal backup cycle and keep them safe in case you need to restore them at some point in the future.

You will also need to follow the requirements for upgrading the Microsoft Exchange 2000 organization to Microsoft Exchange 2003. For example, all domain controllers, global catalog servers, and Active Directory Connector (ADC) servers must be upgraded to Microsoft Windows 2000 Server Service Pack 3 (SP3) or later, or Microsoft Windows Server 2003. See Microsoft Knowledge Base Article - 822942 – Considerations When You Upgrade to Exchange Server 2003.

Use an account with Microsoft Exchange Full Administrator role at the organization level if the Microsoft Exchange Virtual Server is the first server to be upgraded in either the domain or organization. If this is not the first server to be upgraded in the domain or organization, you can use an account with Microsoft Exchange Full Administrator role at the Administrative Group level.

If the server is running full-text indexing (a catalog has been created and populated), manually pause the index; then, after the upgrade, resume when you are ready. The upgrade will disable the search property (making it unavailable to clients) and the index population is not paused. Be sure to re-enable the checkbox "This index is currently available for searching..." in either the cluster or standalone configuration. It may be necessary to delete the catalogs and rebuild the full-text indices after the upgrade.

Once Microsoft Exchange 2003 is installed on one node, stop the System Attendant and fail the Virtual Server over to the other node. To perform the upgrade, right-click on the Virtual Server and select Upgrade Virtual Server, as shown in Figure 20 below. Alternatively, as a validation test before

http://support.microsoft.com/default.aspx?kbid=822942

51

upgrading the production Virtual Server, create a test Virtual Server with one mailbox store and upgrade this Virtual Server first. This allows you to validate the upgrade and inter-related components such as Kerberos, Dynamic DNS, etc. Remove this test Microsoft Exchange virtual server when all virtual servers are successfully upgraded.

Figure 20. Upgrading the Microsoft Exchange Virtual Server to Microsoft Exchange 2003

Removing Microsoft Exchange 2000 Tuning Parameters See the section “Removing Exchange 2000 Tuning Parameters” in the Microsoft document What's New in Exchange Server 2003 for a comprehensive list.

Before upgrading or refreshing hardware For clustered server hardware being upgraded, it is important to check all firmware, software and driver versions and bring them to the appropriate levels. This may include:

• Drive and Array Controller Firmware • Fibre Channel Switches • Host Bus Adapters (Driver and Firmware) • Multi-path Software (e.g. Secure Path) • Storage management software (e.g. HP OpenView Storage Virtual Replicator – formerly known as

SWVR or SANworks Virtual Replicator) • Server BIOS Firmware

Best practices for systems management The following section covers configuration procedures and administrative best practices that can be performed after the cluster is initially built.

http://www.microsoft.com/technet/prodtechnol/exchange/Exchange2003/proddocs/library/NewEx03.asp

http://www.microsoft.com/technet/prodtechnol/exchange/Exchange2003/proddocs/library/NewEx03.asp

52

Active Directory Create a separate OU for virtual servers and cluster network names Virtual Servers and Cluster Network Names should be created in a separate Organizational Unit (OU) in the Active Directory. As they are virtual network names, not physical servers, group policy should not be applied to them. The policy will only be applied to the server currently running the virtual server, and policy settings may be lost when the virtual server fails over. Placing them in a separate OU makes it easier to exclude them from Group Policy.

Cluster configuration Capture configuration information Use the Microsoft Windows 2003 resource kit utilities (listed below) to capture the cluster configuration as early as possible, during the build stages. The procedure to use this captured information is described in the Disaster Recovery section of this paper.

Non-critical cluster resources If a specific resource is not critical to the functionality of your Microsoft Exchange Server, consider removing it. For example, if you provide no IMAP or POP usage, then delete these resources. See Table 5 for other resources that can be removed. As an alternative, if you wish to leave the resource but do not want the Virtual Server to be affected by resource failure, modify the resource properties so that Affect the group is unchecked. This means that resource failure will not cause Virtual Server failover. You will need to ensure that you have a separate means to monitor and are notified of that resource failure.

53

Figure 21. Resource Properties for MS Search resource, showing Affect the group unchecked.

Majority node set The majority node set cluster is a new type of configuration in Microsoft Windows Server 2003 where each node maintains a copy of the cluster configuration data. Ownership of the determinant cluster configuration information is determined by majority of the nodes and access to a network share. The nodes do not use shared storage (e.g., on the SAN) for the quorum resource. In a majority node set cluster, if more than half of the nodes fail, then the entire cluster fails. This negates the use of four or fewer nodes for a majority node set cluster, as the failure of more than 1 node will fail the entire cluster.

This model should only be deployed in specific scenarios, such as when using an HP solution for geographic clusters (which may also require replication of data at the storage level).

Training and expertise It is important to provide specialized Cluster training to System Managers who are responsible for managing clusters. Clusters are more complex than single server deployments, and System Managers should be familiar with concepts such as the quorum, failover/fail back operations, and using Cluster Administrator. Clusters depend on a complex interaction between hardware, operating system, applications, and humans. Attention to detail is needed to ensure that the cluster is installed and configured properly before any application is introduced into the equation. Applications must be installed in a proper manner and then configured to work on a cluster. It is not enough to assume that Microsoft Windows and Microsoft Exchange behave only slightly differently on a cluster – practice makes perfect! However, because clusters are expensive, it is often difficult for administrators to get the necessary experience on clusters before they are deployed. Few test environments incorporate a fully configured production-quality cluster. Additional information and training resources can be found here: http://www.hp.com/education/sections/microsoft.html and http://www.hp.com/education/courses/h7080s.html .

http://www.hp.com/education/sections/microsoft.html

http://www.hp.com/education/courses/h7080s.html

54

Administrators must understand how Microsoft has implemented cluster services for Microsoft Windows Server and then how applications are modified to support those services. For example, you can only install Microsoft Exchange Server 2003 on a cluster in a mixed mode site if another Microsoft Exchange Server 2003 server is already present. Administrators should understand the differences between standard servers and clusters and understand how to manage and troubleshoot both environments, including backup and restore and various disaster recovery scenarios.

Operational procedures developed for standard servers need to be modified for clusters. The software used to monitor servers and applications may not support clusters and require a change or replacement. At a minimum, understand whether the software should be configured using the cluster virtual server name or the actual machine name of the node. Some third-party software may not be supported and force you to change the operational procedures to accommodate a different package. In addition, clusters are sensitive to change, so upgrades and installations of new software must be carefully planned and tested before changes are made to production environments.

Microsoft Windows Server resource kits The Microsoft Windows Server Resource Kit contains valuable tools for cluster administrators. It can be purchased for Microsoft Windows Server 2000 or downloaded directly for Microsoft Windows Server 2003. It contains approximately 300 utilities that aid system management of Active Directory and Microsoft Windows servers. Several of the utilities are specific to clusters.

Among the most important for Microsoft Windows 2000 are:

• DUMPCFG.EXE – manages and records disk signature information (both Microsoft Windows 2000 and 2003) – see Microsoft KB article 280425.

• Microsoft Cluster Tool – configuration backup and restore wizards, as well as a wizard for NT to 2000 resource migrations, which migrates resources (file shares and shared printers) from a stand-alone Microsoft Windows 2000 or NT Server to a cluster.

• CLUSREST.EXC – restores the quorum resource

The Resource Kit for Microsoft Windows Server 2003 includes new and improved cluster utilities such as:

• ClusterRecovery.exe – used for restoring resource checkpoint files, replacing a failed disk or recovering from disk signature changes, and migrating cluster data to a different disk in the cluster

• The Cluster Diagnostics Tool – provides diagnostic tests to verify the functionality of a cluster and assists in reading the cluster log files

Usage of these utilities is described in the Disaster Recovery section of this whitepaper.

Securing your cluster The following are recommendations for securing your Microsoft Windows Server operating systems and may not be specific to clustering. In addition, you may wish to use "Best practices for securing server clusters"

http://go.microsoft.com/fwlink/?LinkId=18173.

http://www.microsoft.com/windows2000/techinfo/reskit/default.asp

http://www.microsoft.com/downloads/details.aspx?FamilyID=9d467a69-57ff-4ae7-96ee-b18c4790cffd&DisplayLang=en




http://go.microsoft.com/fwlink/?LinkId=18173

55

Preventing viral attacks You should lockdown your Microsoft Exchange cluster to prevent spread of network-based virus attacks. Viruses can infect systems via the following:

• File Shares • Web browsing • E-mail • Weak local passwords

Your anti-virus strategy should address each of these areas to eliminate vulnerabilities. For more comprehensive information see the HP whitepaper “Anti-Virus Solutions for Microsoft Exchange 2000 Server” at Microsoft Exchange Server, http://h71028.www7.hp.com/HP/render/4821-6-100-225-1-00.htm.

Caution: When deploying file-based anti-virus scanning, ensure that the Microsoft Exchange database files and transient files (MTA and mailroot folders) are excluded from scanning. Ideally, the file-based anti-virus should allow pre-defining these exclusions before installation, to ensure that it is never incorrectly deployed.

Secure open shares

To prevent the spread of viruses via file shares, you should set the permissions on all non-hidden file shares to ‘read-only’. In Microsoft Exchange 2000, when you create a virtual server, a shared log folder for the message tracking logs is created on the cluster disk resource as \Exchsrvr\<virtual_server_name.log> folder. By default, the Microsoft Exchange 2003 installation program assigns the ‘Everyone’ security group, which has full write access, to this share. Microsoft Knowledge Base Article Q312465 describes how the permissions can be changed on the Tracking Log share from the Everyone group having Full Control and Change to the Everyone group having only Read permissions. Microsoft has addressed this in Microsoft Exchange 2003 by setting the permissions on Microsoft Exchange shares to read only.

As an added precaution, you should not implement any additional non-hidden shared folders on your Microsoft Exchange cluster. To hide a share, append the ‘$’ symbol to the share name.

Implement latest security patches for Internet Explorer and IIS

Use the Microsoft Baseline Security Analyzer (MBSA) to audit the security settings on your cluster. The MBSA is a free tool available for download from Microsoft. See http://www.microsoft.com/technet/security/tools/mbsahome.mspx for more information on MBSA. MBSA provides practical advice on how security issues can be resolved. For example, clicking on the list of missing Microsoft Windows Security Updates presents the user with a list of hotfixes. One of the best features of MBSA is that it reduces the time it takes to locate and download security fixes. Each hotfix has the URL for its download location on the Microsoft site which helps save you time.

The latest security rollup patches from Microsoft include protection from viruses. They are available from http://www.microsoft.com/windows2000/downloads/critical or http://www.microsoft.com/windowsserver2003/downloads/updates/default.mspx.

http://h71028.www7.hp.com/HP/render/4821-6-100-225-1-00.htm


http://www.microsoft.com/technet/security/tools/mbsahome.mspx

http://www.microsoft.com/windows2000/downloads/critical

http://www.microsoft.com/windowsserver2003/downloads/updates/default.mspx

56

The software update service should be used to download and manage the latest security patches: http://www.microsoft.com/windowsserversystem/sus/default.mspx.

Preventing the spread of viruses through e-mail

You should implement an antivirus solution to protect your cluster from e-mail viruses. Third-party antivirus products for Microsoft Exchange can scan mailboxes in real-time for viruses such as ‘Melissa’, and ‘Iloveyou’. When you have implemented your antivirus solution, be sure to schedule regular updates of virus pattern files from your vendor. Microsoft introduced VSAPI (Virus Scanning Application Programming Interface) in Microsoft Exchange Server 2003. VSAPI enables Anti-virus vendors to develop software that can scan Microsoft Exchange components such as databases, SMTP queues, and the Message Transfer Agent. You should choose an antivirus solution that is VSAPI compliant.

Choose antivirus products that are ‘Cluster Aware’. Many third-party products are not cluster aware, and to install them, you need to take the virtual server offline and perform the installation on the active node.

Antivirus signature/pattern files contain information about the latest viruses. You should update your antivirus signatures/pattern files daily. If your antivirus patterns are out of date, your cluster will be vulnerable to new viruses.

Upgrade to Microsoft Outlook 2003 or deploy Microsoft Outlook security update

By default, Microsoft Outlook 2002 and Microsoft Outlook 2003 block e-mail attachments associated with unsafe files including .EXE and .VBS files. They also warn if a program attempts to access your address book. If your users are running Microsoft Outlook 2000 or Microsoft Outlook 98, you should consider deploying the Outlook E-Mail Security Update which provides similar security features to Outlook 2002/2003. Similarly, if users in your organization use Microsoft Outlook Express, upgrading to version 6 will provide attachment security and address book protection like that in Microsoft Outlook 2002/2003.

Microsoft Exchange 2003 Outlook Web Access (OWA) provides attachment blocking features similar to Microsoft Outlook 2002/2003.

Standardizing configuration One of the challenges for a cluster System Manager is to maintain homogeneity between nodes in a cluster. The configurations of each node in a cluster should be identical to ensure smooth operation and maximize system availability. The following areas should be considered:

• Hardware configuration • System BIOS • Firmware on drives, array controllers • Microsoft Windows Server versions (Service Packs, Hotfixes) • Microsoft Exchange versions (Service Packs, Hotfixes) • Layered products (OpenView, Anti-virus, Monitoring) • Microsoft Windows Server configuration • Network configuration – (IP Address, DNS, WINS servers) • Drive letters • Performance settings (Paging file size and location on each node) • Registry changes • Local User Account Database

http://www.microsoft.com/windowsserversystem/sus/default.mspx

http://www.microsoft.com/office/outlook/evaluation/security.asp

57

• Manual registry settings • Microsoft Exchange Server 2003 configuration • Manual registry changes

Hardware configuration The hardware specification on each node in the cluster should be identical. When you upgrade components such as memory or processor, be sure to upgrade each node in the cluster. Running nodes with differing specifications makes it difficult to quantify performance.

Device drivers HP provides a convenient method of upgrading device drivers. The HP ProLiant Remote Deployment Utility, shown in Figure 22, can be used to analyze the current versions of all drivers and automatically upgrade them.

Figure 22. The HP ProLiant Remote Deployment Utility

You can use checklist to record when hotfixes and Service Packs are applied. Table 10 lists a suggested format to keep track of Service Pack and hotfix installations, as well as keep a record of

58

downtime for failovers/reboots. The data is useful for comparing uptime against Service Level Agreements.

Table 10. Change Control Checklist

Node 1 Date Applied

Node 1 Installed (Y/N)

Node 2 Date Applied


Comments Down-time (min)

Microsoft Windows Server

Y Y 0

Service Pack Y Y Reboot required

15

SP Hotfixes

Q12222 Y Y Reboot required

10

Q12334 Reboot required

10

Q22344 Reboot required

10

Service Pack Y Y Reboot required

15

SPx Hotfixes Reboot required

Security Patch Y Y Reboot required

IE Patch Y Y Reboot required

Microsoft Exchange Server

Y Y 0

Service Pack Y Y Virtual Server failover

10

SP Hotfixes

59

Node 1 Date Applied


Node 2 Date Applied


Comments Down-time (min)

Using QFECHECK to verify hotfix installations Microsoft provides a utility called QFECHECK to audit installation of Microsoft Windows 2000 hotfixes. QFECHECK should be performed on each node and the results compared to verify that each node in a cluster has identical sets of fixes applied. To obtain QFECHECK, check Microsoft Knowledge Base article Q282784.

Microsoft Exchange Server service packs Upgrade to the latest Microsoft Exchange Server service pack For example, Microsoft Exchange Server 2000 Service Pack 3 is available for download from http://www.microsoft.com/exchange/downloads/2000/sp3/default.asp. It incorporates over 400 bug fixes. Virtual Memory fragmentation is a major problem for large Microsoft Exchange installations, especially servers that host large numbers of MAPI clients. Service Pack 3 includes an updated version of STORE.EXE to help address the virtual memory problem.

Perform full Microsoft Exchange database backups

It is important to take full backups of your Microsoft Exchange server before and after installation of a Microsoft Exchange service pack. Set aside tapes that are outside your normal backup cycle. Microsoft Exchange service packs usually include a new version of STORE.EXE. The databases are upgraded to work with the new Store binary when they are mounted after the service pack is applied. This means that backups taken of SP2 servers cannot be restored to SP3 servers. You should update your Disaster Recovery servers and procedures to reflect the changes. Microsoft Knowledge Base Article Q316794 contains some background information on Store version mismatch scenarios.

Verify permissions

See the earlier section on necessary permissions and ensure that the requirements are met.

Create a cluster test environment for service pack upgrades

It is a good idea to create a parallel test environment so that Service Packs, hotfixes, and third-party products can be tested before deployment in production. Many third-party products are not ‘cluster aware’ and require some customization to work properly on clusters. It may not be cost-effective to exactly replicate your production configuration, but you should consider a low cost clustering solution such as the HP ProLiant DL380 G2 server, http://h18004.www1.hp.com/solutions/enterprise/highavailability/microsoft/win2003.html.

Recommended procedure for Microsoft Exchange service packs One of the benefits of using clusters with Microsoft Exchange is the ability to perform ‘rolling’ upgrades of service packs, incurring very little loss of availability or downtime to the end-users. A ‘rolling’ upgrade entails moving all Microsoft Exchange virtual servers to one node and performing the installation on the passive node. You should only upgrade one node at a time in order to minimize downtime. For example:

• NODE 1 is the active node with one virtual server EVS1


http://www.microsoft.com/exchange/downloads/2000/sp3/default.asp


http://h18004.www1.hp.com/solutions/enterprise/highavailability/microsoft/win2003.html

60

• NODE 2 is the passive node with no active cluster resources • Apply Microsoft Exchange Service Pack to NODE2 • Move Microsoft Exchange Virtual Server EVS1 to NODE2 • Check the Application Event Log for errors • Verify Microsoft Exchange starts correctly on NODE2 • Apply Microsoft Exchange Service Pack to NODE1 • Move Microsoft Exchange Virtual Server EVS1 to NODE1 • Check the Application Event Log for errors • Verify Microsoft Exchange starts correctly on NODE1

How to avoid reboots during a rolling upgrade It has been found that application of a Microsoft Exchange Service pack through a Terminal Services session may end with the need to reboot the server. A reboot of the node may be avoided by logging directly on to the console to perform the installation. In addition, close down any Microsoft Exchange System Manager connections to the virtual server.

Third-party products Choose third-party products that are supported in a cluster and if possible, are ‘Cluster Aware’. ‘Cluster Aware’ means they work properly and understand the difference between the physical servers and the virtual servers. Also, they should support rolling upgrades and retain their configuration between the nodes. Many third-party products are not cluster aware, and to install them, you need to take the virtual server offline and perform the installation on the active node. In addition, you may need to configure each node separately and must be careful whether to install to the virtual server or physical server name.

Use cluster administrator to perform cluster operations Always use Cluster Administrator, shown in Figure 23, to take Microsoft Exchange Virtual Servers offline and to perform failover/fail back operations. Do not manually start or stop Microsoft Exchange services (e.g., using NET STOP command line or Control Panel) such as the Microsoft Exchange System Attendant. Microsoft Exchange services in a cluster are controlled by EXRES.DLL. Manually stopping/starting services can cause a cluster to go into an inconsistent state.

61

Figure 23. Cluster Administrator

Use system manager to change Microsoft Exchange virtual servers Microsoft Exchange Server 2003 is tightly integrated with Microsoft Internet Information Services (IIS). Some configuration information for Microsoft Exchange virtual servers, such as the virtual server IP address, and IFS information is stored in the Microsoft IIS metabase. All changes to virtual servers should be performed through Cluster Administrator and ESM. Internet Services Manager, shown in Figure 24, should not be used. If you use Internet Services Manager to make changes to a virtual server, the changes will be overwritten with the settings in ESM the next time the virtual server is started.

62

Figure 24. Internet Services Manager – not used for Exchange Virtual Server management.

Failovers Failover occurs when a Microsoft Exchange cluster group is moved from one node to another. Microsoft Outlook clients cannot access Microsoft Exchange during a failover, so it is important to minimize failover times to provide high availability and meet agreed Service Level Agreements. There are two types of failover, planned and unplanned.

Planned failovers Planned failovers usually take place as part of scheduled system maintenance tasks, such as a Microsoft Exchange Service Pack rolling upgrade (described earlier). In a two-node cluster, both nodes will be online throughout a planned failover. The Cluster Administrator program is used to perform the failover. The following events take place in a planned failover:

• Using Cluster Administrator, the administrator selects the Microsoft Exchange Group to be failed over, right-clicks and selects Move Group

• EXRES.DLL dismounts the shared storage being used by the virtual server, takes the Microsoft Exchange components in the Microsoft Exchange Cluster Group offline, dismounts the storage groups, and stops the protocols such as IMAP, POP, and HTTP etc.

• On the other node in the cluster, EXRES.DLL remounts the disks, brings the Microsoft Exchange resources online, and mounts the storage group.

• The quorum is updated

Tips for reducing planned failover time Planned Failover time can be reduced by performing the failover at the following times:

• Just after a full online database backup. On a heavily loaded Microsoft Exchange server, as much as 500MB of transaction logs can be created daily. At the end of a full database backup, all transaction logs created since the last backup are purged. When the Microsoft Exchange Information Store mounts databases at start-up, it replays transaction logs. Transaction log replay will take less time if there are fewer transaction logs.

63

• Outside normal working hours. On a heavily loaded Microsoft Exchange server, there could be as many as a 1,000 MAPI connections. Performing failovers at such times takes longer because each connection must be terminated. Outside working hours, the numbers of connections decreases and so the cluster has less network connections to disconnect.

Planned failovers can take longer after an upgrade of a Microsoft Windows Server service pack. After a Microsoft Window 2000 service pack upgrade, the Microsoft Exchange Store process rebuilds indexes. This process can add several minutes to your normal failover time. The delay depends on the size of your Microsoft Exchange databases. Event ID 611 in the Application Event Log, shown in Figure 25, indicates an Index rebuild is taking place.

Figure 25. Index rebuild message

Unplanned failovers Unplanned failovers occur when the Active node hosting a Microsoft Exchange Virtual server crashes or loses power. The following takes place:

• The node hosting the virtual server goes down • The cluster service detects that the active node is no longer available as the heartbeat connection is

no longer available. • The disks belonging to the Microsoft Exchange Virtual Server are brought online in the other node. • The Microsoft Exchange cluster resources are brought online by EXRES.DLL. The Microsoft Exchange

Store mounts the storage groups and performs recovery tasks. • The quorum is updated.

64

Tips for reducing unplanned failover time There is not much that can be done to reduce failover time in unplanned failovers. However, if you have designated a node in your cluster as the preferred owner, you may want to schedule failback operations to occur outside working hours in order to reduce downtime during working hours. In an unplanned failover scenario, two failover operations occur. The first failover occurs when the active node shuts down unexpectedly, and the Microsoft Exchange virtual server is failed over to the second node. When the preferred owner node comes back on line, a second failover operation (called failback) occurs as the Microsoft Exchange virtual server is moved back to the preferred node.

To reduce downtime during working hours, you should schedule failback outside of normal business hours. Failback can be set by right clicking the Microsoft Exchange Group in Cluster Administrator, selecting Properties, and choosing the Failback tab. In Figure 26, failbacks are configured to take place between 21:00 and 22:00.

You should choose a time that does not coincide with out-of-hours events such as nightly backups and online database defragmentation. The failback process would disrupt these tasks because the Microsoft Exchange databases are dismounted and remounted during a failback operation.

Figure 26. Options for scheduling failback

Managing storage Before making any changes to the storage, perform a full backup of your cluster and Microsoft Exchange databases. Use DUMPCFG to record the disk signature information of the disks in your shared storage. The procedure for running DUMPCFG is described in the Disaster Recovery section of this whitepaper. If you add new storage and the disk signature information changes, your cluster will not come online after the storage is added.

65

Performance monitoring Operations checks As part of your weekly operations procedures you should regularly check the following on your Microsoft Exchange clusters.

• Backups • Disk space • Event Logs • Mail queues • Microsoft Windows Critical Updates • Antivirus updates

Many of these tasks can be done manually, but if you have a large number of servers to manage, you should consider deploying a monitoring application such as HP OpenView. A monitoring application will help you save time as it can help automate and streamline these tasks.

Event logs To check the Event Logs on many Microsoft Exchange servers would take a long time. By default, the Microsoft Exchange components log a large amount of data in the event logs. If you activate any of the diagnostic logging settings, the amount of data can quickly become overwhelming. Attention should be focused on the error and warning category events. These categories include some events that can generally be ignored, such as the System Attendant’s event 5008 warning about the deletion of a message-tracking log file.

Monitoring applications allow you to generate a consolidated view of the Event Logs. These packages provide mechanisms that allow you to generate consolidated reports, filter out the events that are superfluous, and focus on those that really need attention and research time. They also provide the added benefit of allowing notifications to be triggered as soon as something of significance, such as the failure to locate a DC or GC is logged. Table 11 lists some of the events that can be excluded from the daily review.

Table 11. Application events that can be “ignored” or filtered out during monitoring

Event Source ID Description

MSExchangeSA 9095 MAD Monitoring thread initializing

MSExchangeSA 9096 MAD Monitoring thread initialized

SceCli 1704 Group policy successfully applied

MSExchangeIS Public 1221 White space report

MSExchangeIS Private 1221 White space report

ESE 700 Start of on-line defragmentation

66

Event Source ID Description

ESE 701 End of on-line defragmentation

MSExchangeFBPublish 8274 Free/Busy publish/cleanup complete / report

MSExchangeFBPublish 8273 Free/Busy publish/cleanup start

MSExchangeIS Mailbox 9535 Deletion of expired mailboxes (retention period expired) complete / report

MSExchangeIS Mailbox 9531 Start of Deletion of expired mailboxes

MSExchangeIS Mailbox 1207 End purge of deleted items recovery / report

MSExchangeIS Mailbox 1206 Start purge of deleted items recovery

MSExchangeSA 5008 Purge Message Tracking Log

Monitoring mail queues You can create an MMC console to view and monitor queues for a selected group of Microsoft Exchange clusters using the Microsoft Windows Performance Monitor (Perfmon.exe). To open Performance Monitor, click Start, Programs, Administrative Tools, and Performance. This tool works by fetching information from performance “counters” published by applications such as Microsoft Exchange. A small application may only have a single performance object, or something that you can monitor. Large applications divide performance reporting across a set of objects. For example, SMTP activity is monitored by selecting the SMTP Server performance object. However, since a Microsoft Exchange virtual server can host multiple SMTP virtual servers that actually do the work to process messages, there is a further level of granularity to deal with. You have to select an “instance” – in this case, the SMTP virtual server – that you wish to monitor. Therefore, to monitor queue activity for the default SMTP virtual server, you select the Performance Object SMTP Server and the instance SMTP 1, and then the performance counters that you wish to monitor, as shown in Figure 27. The default SMTP server corresponds to instance “SMTP 1”. If there are multiple SMTP servers on your server they will be numbered SMTP 2, SMTP 3 etc.

67

Figure 27. SMTP Counters

The following counters provide a good overview of messaging activity on a server, so add them to the console:

• Categorizer Queue Length • Local Queue Length • Local Retry Queue Length • Current Messages in Local Delivery • Remote Queue Length • Remote Retry Queue Length

If the Microsoft Exchange virtual server is a mixed mode site, the MTA handles traffic to the legacy Microsoft Exchange servers, so you need to monitor the MTA queue lengths. These can be found by selecting the ‘MSExchangeMTA Connections’ Performance Object, and the ‘Associations’ counter (shown in Figure 28).

68

Figure 28. MTA Performance Counters

Monitoring virtual memory On clusters with over 1,500 mailboxes, you should proactively monitor for memory fragmentation. Some additional Microsoft Exchange performance counters were added to allow monitoring of virtual memory fragmentation. These are shown in Table 12.

Table 12. Microsoft Exchange performance counters for monitoring of virtual memory fragmentation

Performance Object Performance Counter Description

MSExchangeIS VM Largest Block Size Size in bytes of the largest free virtual memory block

MSExchangeIS VM Total Free Blocks Total number of free virtual memory blocks

MSExchangeIS VM Total 16MB Free Blocks Total number of free virtual memory blocks larger than or equal to 16 MB.

MSExchangeIS VM Total 16MB Free Block Bytes

Total number of bytes in free virtual memory blocks larger than or equal to 16MB

On a server experiencing virtual memory fragmentation, event ID 9582 will be written to the application log. The description of error 9582: “The virtual memory necessary to run your Microsoft Exchange server is fragmented in such a way that normal operation may begin to fail. It is highly recommended that you restart all Microsoft Exchange services to correct this issue.” Note that some third-party products – particularly virus checkers – can impact virtual memory use. Monitor the situation with the product enabled and disabled to see if it makes a difference.

69

Figure 29. Monitoring Virtual Memory

Microsoft Knowledge Base article Q296073 describes how to use Performance Monitor, shown in Figure 29, to detect virtual memory fragmentation.

Best practices for disaster recovery Disaster recovery for Microsoft Exchange Server is complex even in a single server environment and clusters add further complexity to Disaster Recovery. Exchange Administrators should familiarize themselves with the following Disaster Recovery whitepapers:

Microsoft Exchange Server Disaster Recovery

http://www.microsoft.com/technet/prodtechnol/exchange/exchange2003/proddocs/library/default.asp

Microsoft Exchange Server 2000 and 2003 Backup and Restore using HP Technology, http://activeanswers.compaq.com/ActiveAnswers/Render/1%2C1027%2C6535-6-100-225-1%2C00.htm.

Microsoft Exchange Server 2003 Recovery Storage Group and HP ProLiant Server Technologies, at Microsoft Exchange Server 2003 White Papers, http://h71019.www7.hp.com/6183-6-100-225-1-00.htm

Active Directory Disaster Recovery: http://h71019.www7.hp.com/activeanswers/2472-6-100-225-1.htm

These papers contain valuable information on tried and tested techniques for backing up and recovering information in the event of a disaster. Using these procedures, develop and regularly test a disaster recovery plan.

Logistical difficulties often hamper disaster recovery efforts. You should put together a Disaster Recovery Kit. This kit should be kept offsite and could include the following:

• Microsoft Windows Server CDs



http://activeanswers.compaq.com/ActiveAnswers/Render/1,1027,6535-6-100-225-1,00.htm.


http://h71019.www7.hp.com/activeanswers/2472-6-100-225-1.htm

70

• Microsoft Exchange Server 2003 CDs • Microsoft Windows Server Resource Kit • Microsoft Windows and Microsoft Exchange Service Pack CDs • Third-party Product CDs and Installation procedures • Microsoft Windows and Microsoft Exchange Server Build Documentation • Disaster Recovery Procedures • SAN Operations Procedures • Key Contact details – for example Technical Account Managers, Microsoft Product Support, HP etc.

Document the procedures for escalating issues to Microsoft and other vendors. All the relevant information such as telephone numbers, account details, and serial numbers should be readily available. Perform a ‘dry run’ of your escalation procedures so they can be refined. In the event of a disaster, time should not be wasted trying to locate this information.

What should I backup? Microsoft’s whitepaper on Microsoft Exchange Disaster Recovery contains extensive information on planning a backup strategy for your Microsoft Exchange Server 2003 clusters and stand-alone servers. Backup procedures for clusters are slightly different to backup procedures for stand-alone servers.

In addition to the Microsoft Exchange databases, the cluster quorum database and cluster configuration on each node in the cluster must be backed up.

On an Active Node:

• Local drives – Microsoft Windows and Microsoft Exchange Binaries • System State – this includes the boot files, Cluster Quorum database, and the Registry as shown in

Figure 30) • Full Online Backup of Microsoft Exchange Databases

On a Passive Node:

• Local drives – Microsoft Windows and Microsoft Exchange Binaries • System State, shown in Figure 30

71

Figure 30. Choosing the System State from NTBACKUP

When performing file-level backups of the drives in the shared storage on clusters, be sure to exclude folders holding the Microsoft Exchange databases and transaction logs (*.EDB, *.STM, *.CHK, and *.LOG). These files should be backed up using ‘Exchange Aware’ Backup utilities. Microsoft Exchange provides a backup API that allows backup software to work with the Extensible Storage Engine (ESE) to backup the database. This API allows the Microsoft Exchange databases to be backed up without taking Microsoft Exchange services offline. When Microsoft Exchange Server 2003 is installed, it updates the NTBACKUP utility to allow ESE backups to take place. If you want to perform Microsoft Exchange database backups over the network with NTBACKUP, install ESM on the server or follow the procedures listed in Knowledge Base Q275876 to update NTBACKUP. NTBACKUP does not support direct Fibre Channel backups; to do this, you must purchase another backup product.

Note: Do not backup the Installable File System (IFS) drive -- the M:\ drive in Microsoft Exchange 2000. Backing up this drive can cause problems with attachments.

Checking backups You will need to determine the appropriate codes for the backup application that you choose to use. The codes in Table 13 describe the events for the backup application NTBACKUP.


72

Table 13. Backup Related Events

Event Source Event ID Meaning

Ntbackup 8000 Start of backup

Ntbackup 8001 End of backup

Ntbackup 8008 Start of verify

Ntbackup 8009 End of verify

ESE 210 Start of full Information Store backup

ESE 211 Start of incremental Information Store backup

ESE 213 Successful completion of Information Store backup

ESE 220 Beginning the backup of an STM or EDB file

ESE 221 Ending the backup of an STM or EDB file

ESE 223 Start backup of log files / Start of differential backup

ESE 224 Purging Log files

The 8000 and 8001 events from NTBACKUP signal the start and end of backup from the backup utility’s perspective, while the 210 and 213 events signal the start and end of a normal backup from the ESE perspective. What you should also notice is that NTBACKUP also logs events 8008 and 8009 to signal the start and end of the backup verification process. During the verification process, the backup utility reads the data and associated checksums from the tape to confirm that the backup is useable.

Microsoft Cluster Tool Cluster Tool, shown in Figure 31is part of the Microsoft Windows 2000 Server Resource Kit. It can be used to backup the cluster configuration, but only for Microsoft Windows 2000 and not Microsoft Windows Server 2003. Also, be advised that it only backs up the cluster configuration information, so do not use it as an alternative to backups. Use the Cluster Tool to backup your cluster database before you make changes to the cluster configuration such as creating new groups/changing failback options. If you need to restore the cluster configuration, you can use the backup created by the Cluster Tool.

The Cluster Tool is not installed during typical Microsoft Windows 2000 Resource Kit installation. You must run the setup.exe located in the ..\apps\clustool directory of your Microsoft Windows 2000 Resource Kit CD.

73

Figure 31. Microsoft Cluster Tool

Restoring Quorum For a cluster running on Microsoft Windows 2000, you can use CLUSREST.EXE and NTBACKUP to restore the cluster quorum database as a two-step operation (see below). With Microsoft Windows 2003, NTBACKUP can restore the cluster quorum database without having to use CLUSREST.

To restore the cluster quorum on Microsoft Windows 2000, you must restore the System State first. Before starting the restore, you should bring one node online and power down other nodes in the cluster. The nodes that are offline can be brought back online when the restore process is complete. NTBACKUP places the quorum information in a folder in the local storage on the node performing the restore. The second step involves using the CLUSREST.EXE utility to move the restored quorum database from the node’s storage to the quorum disk in the shared storage. When CLUSREST.EXE (Figure 32) has completed, reboot the node and the cluster will be formed. At this stage, other nodes can be started and can join the cluster.

74

Figure 32. CLUSREST Tool

Conclusion In conclusion, the lessons learned from successful Microsoft Exchange Server 2003 cluster deployments are:

• Planning is critical: Use the guidelines in this whitepaper to design your storage, decide on naming conventions, and to implement a redundant Microsoft Windows Server infrastructure.

• Microsoft Windows Server installation: Correct configuration of Microsoft Windows Server is essential for smooth operation of Microsoft Exchange Server 2003 clusters. Pay careful attention to the network configuration and test failover operations before you install Microsoft Exchange Server 2003.

• System Management: Specialized training for cluster System Managers is essential. Document your cluster installation and take care to ensure that each node in your cluster has an identical configuration. Record downtime during failover operations for Service Level Reporting.

• Disaster Recovery: Cluster Administrators must familiarize themselves with Disaster Recovery procedures from Microsoft/HP and cluster-specific tools in the Microsoft Windows Server Resource Kit.

• Microsoft Exchange 2003 and Microsoft Windows 2003 offer significant advantages in configuration options, ease of cluster deployment, and supported features.

For more information Microsoft Exchange Server 2003 http://www.microsoft.com/exchange/techinfo/productdoc/default.asp http://www.hp.com/solutions/exchange Microsoft Exchange Server Disaster Recovery http://www.microsoft.com/technet/prodtechnol/exchange/exchange2003/proddocs/library/default.asp Microsoft Exchange Server 2000 and 2003 Backup and Restore using HP Technology, http://h71019.www7.hp.com/6535-6-100-225-1-00.htm Microsoft Exchange Server 2003 Recovery Storage Group and HP ProLiant server technologies http://h71028.www7.hp.com/aa_downloads/6/100/225/1/64974.pdf Clusters Clustering Technologies: http://www.microsoft.com/windowsserver2003/technologies/clustering/default.mspx High availability clusters from HP: http://www.hp.com/servers/proliant/highavailability Deploying Microsoft Exchange 2000 Clusters: http://h18004.www1.hp.com/products/servers/consolidation/exchange2000.html Microsoft Exchange Server 2003 Clusters: http://h71028.www7.hp.com/aa_downloads/6/100/225/1/65234.pdf HP ProLiant clustering whitepapers: http://www.compaq.com/solutions/enterprise/highavailability/whitepapers.html Utilities Microsoft Windows Server 2003 Resource Kit http://download.microsoft.com/download/8/e/c/8ec3a7d8-05b4-440a-a71e-ca3ee25fe057/rktools.exe http://www.microsoft.com/windowsserver2003/techinfo/reskit/resourcekit.mspx Microsoft Windows Server 2003 Deployment Kit http://www.microsoft.com/windowsserver2003/techinfo/reskit/deploykit.mspx Clusrest.exe: Cluster Quorum Restore Utility from Microsoft: http://www.microsoft.com/windows2000/techinfo/reskit/tools/existing/clusrest-o.asp Microsoft Knowledge Base Articles Best Practices for Drive-Letter Assignments on a Server Cluster http://support.microsoft.com/default.aspx?scid=kb;en-us;Q318534 Status of Exchange Server 2003 Server Components on Cluster Server http://support.microsoft.com/default.aspx?scid=kb;en-us;Q259197 Minimum Permissions Necessary to Perform Exchange-Related Tasks http://support.microsoft.com/default.aspx?scid=kb;en-us;316792 Exchange 2000 Server Role Permissions http://support.microsoft.com/default.aspx?scid=kb;en-us;289811

http://www.microsoft.com/exchange/techinfo/productdoc/default.asp

http://www.hp.com/solutions/exchange




http://www.microsoft.com/windowsserver2003/technologies/clustering/default.mspx

http://www.hp.com/servers/proliant/highavailability

http://h18004.www1.hp.com/products/servers/consolidation/exchange2000.html


http://www.compaq.com/solutions/enterprise/highavailability/whitepapers.html

http://download.microsoft.com/download/8/e/c/8ec3a7d8-05b4-440a-a71e-ca3ee25fe057/rktools.exe

http://www.microsoft.com/windowsserver2003/techinfo/reskit/resourcekit.mspx

http://www.microsoft.com/windowsserver2003/techinfo/reskit/deploykit.mspx

http://www.microsoft.com/windows2000/techinfo/reskit/tools/existing/clusrest-o.asp





Lb*.tmp Files are displayed in the Temp Folder http://support.microsoft.com/default.aspx?scid=kb;en-us;Q294462 Changing the IP address of Network Adapters in Cluster Server http://support.microsoft.com/default.aspx?scid=kb;en-us;Q230356 Recommended Private “Heartbeat” Configuration on a Cluster Server http://support.microsoft.com/default.aspx?scid=kb;en-us;q258750 Network Adapter Teaming and Server Clustering http://support.microsoft.com/default.aspx?scid=kb;en-us;Q254101 Exchange Server 2003 requires /3GB switch with more than 1 Gigabyte of Physical RAM http://support.microsoft.com/default.aspx?scid=kb;en-us;Q266096 The NIMDA virus may infect the files in log folders on New Exchange Server Virtual Servers in a cluster http://support.microsoft.com/default.aspx?scid=kb;en-us;Q312465 QFECHECK verifies the installation of Windows 2000 and Windows XP hotfixes http://support.microsoft.com/default.aspx?scid=kb;en-us;Q282784 Exchange Server 2000 SP2 does not allow you to restore Exchange Server 2000 or Exchange Server 2000 SP1 http://support.microsoft.com/default.aspx?scid=kb;en-us;Q316794 Monitoring for Exchange Server 2000 Memory Fragmentation http://support.microsoft.com/default.aspx?scid=kb;en-us;Q296073 How to use NTBackup from a Non-Exchange Server 2000 computer http://support.microsoft.com/default.aspx?scid=kb;en-us;Q275876 Event ID 1034 for MSCS Shared Disk after disk replacement http://support.microsoft.com/default.aspx?scid=kb;en-us;Q243195

© 2004 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Microsoft and Windows are U.S. registered trademarks of Microsoft Corp.

[03/2004]-1



http://support.microsoft.com/default.aspx?scid=kb;en-us;q258750









Hp Best Practices For Microsoft Exchange Server 2000 And 2003 Cluster Deployments

Technology

Transcript of Hp Best Practices For Microsoft Exchange Server 2000 And 2003 Cluster Deployments