Implementing SAP R3 4.5B Using Microsoft

Implementing SAP R/3 4.5B Using MicrosoftCluster Server on IBM Netfinity Servers

David Watts, Matthew Cali, Edward Charles, Olivier De Lampugnani, David Dariouch,Mauro Gatti, Bill Sadek

International Technical Support Organization

SG24-5170-01

http://www.redbooks.ibm.com


International Technical Support Organization SG24-5170-01

Implementing SAP R/3 4.5B Using MicrosoftCluster Server on IBM Netfinity Servers

November 1999

© Copyright International Business Machines Corporation 1998, 1999. All rights reserved.Note to U.S Government Users - Documentation related to restricted rights - Use, duplication or disclosure is subject to restrictionsset forth in GSA ADP Schedule Contract with IBM Corp.

Second Edition (November 1999)

This edition applies to SAP R/3, Release 4.5B for Microsoft Windows NT 4.0.

Comments may be addressed to:IBM Corporation, International Technical Support OrganizationDept. HZ8 Building 678P.O. Box 12195Research Triangle Park, NC 27709-2195

When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any wayit believes appropriate without incurring any obligation to you.

Before using this information and the product it supports, be sure to read the general information in Appendix A,“Special notices” on page 181.

Take Note!

© Copyright IBM Corp. 1998, 1999 iii

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Chapter 1. Introduction to high availability computing . . . . . . . . . . . . . . . .11.1 Clustering — a means to an end . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11.2 What clustering provides . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11.3 Business data — to replicate or not?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21.4 Disk sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21.5 MSCS-based solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

1.5.1 Shared-disk configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .31.6 What the user sees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .41.7 SAP R/3 support for Microsoft clustering . . . . . . . . . . . . . . . . . . . . . . . . . . .41.8 IBM Netfinity clustering solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4

Chapter 2. SAP R/3 and high availability . . . . . . . . . . . . . . . . . . . . . . . . . . . .52.1 SAP R/3 components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52.2 Using MSCS with SAP R/3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .72.3 Alternate approaches to SAP R/3 high availability . . . . . . . . . . . . . . . . . . . .9

2.3.1 Cold standby server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .102.3.2 Replicated database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112.3.3 Replicated DBMS server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .142.3.4 Multiplatform solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

2.4 Microsoft Cluster Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .162.4.1 Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182.4.2 Resource Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .192.4.3 Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .192.4.4 Resource types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .202.4.5 Resource states . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .222.4.6 Resource groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .222.4.7 Quorum resource . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .242.4.8 TCP/IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .242.4.9 Additional comments about networking with MSCS . . . . . . . . . . . . . .252.4.10 Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .272.4.11 Failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .272.4.12 Failback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .292.4.13 LooksAlive and IsAlive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29

2.5 Microsoft Cluster Service and SAP R/3 . . . . . . . . . . . . . . . . . . . . . . . . . . .302.5.1 SAP R/3 extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .302.5.2 Cluster Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .302.5.3 Group SAP-R/3 <SID> . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .322.5.4 Database resource group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32

2.6 Backup and recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .332.6.1 Complete backup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .342.6.2 Tape drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .342.6.3 Unique file identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .352.6.4 Backup scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .362.6.5 Offline backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .372.6.6 MSCS-special files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .37

Chapter 3. Hardware configuration and MSCS planning . . . . . . . . . . . . . .393.1 Checklist for SAP MSCS installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .40

3.1.1 Minimum cluster hardware requirements . . . . . . . . . . . . . . . . . . . . . .40

iv Implementing SAP R/3 4.5B Using Microsoft Cluster Server on IBM Netfinity Servers

3.1.2 Minimum cluster software requirements . . . . . . . . . . . . . . . . . . . . . . 413.2 Certification and validation of hardware . . . . . . . . . . . . . . . . . . . . . . . . . . 42

3.2.1 iXOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433.2.2 Microsoft . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443.2.3 Certification of SAP R/3 and Windows 2000 . . . . . . . . . . . . . . . . . . 45

3.3 Netfinity sizing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463.3.1 Terms and definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463.3.2 What is to be sized? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473.3.3 Sizing methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

3.4 Disk layouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.4.1 Operating system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.4.2 Shared external drives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

3.5 ServeRAID SCSI configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623.5.1 Netfinity EXP15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 623.5.2 Netfinity EXP200 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643.5.3 Ultra2 and LVDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643.5.4 SAP R/3 disk configurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 653.5.5 Cluster configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 663.5.6 SCSI tuning recommendations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

3.6 Fibre Channel configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 683.6.1 IBM Netfinity Fibre Channel components . . . . . . . . . . . . . . . . . . . . . 683.6.2 Cluster configurations with Fibre Channel components . . . . . . . . . . 723.6.3 Fibre Channel tuning recommendations. . . . . . . . . . . . . . . . . . . . . . 76

3.7 Network configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 763.7.1 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 773.7.2 Different name resolution methods . . . . . . . . . . . . . . . . . . . . . . . . . 793.7.3 Resolve IP address to the correct host name. . . . . . . . . . . . . . . . . . 803.7.4 Redundancy in the network path . . . . . . . . . . . . . . . . . . . . . . . . . . . 813.7.5 Auto-sensing network adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 813.7.6 Redundant adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 823.7.7 Load balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

Chapter 4. Installation and verification . . . . . . . . . . . . . . . . . . . . . . . . . . . 854.1 General overview of the installation process . . . . . . . . . . . . . . . . . . . . . . 854.2 Setting up security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

4.2.1 Windows NT security planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 874.2.2 SAP R/3 security planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 874.2.3 DBMS security planning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

4.3 Hardware configuration and installation . . . . . . . . . . . . . . . . . . . . . . . . . . 894.4 Windows NT installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 894.5 MSCS pre-installation testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 964.6 Basic Windows NT tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

4.6.1 Server service tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 974.6.2 Page file tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 984.6.3 4 GB tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 994.6.4 Remove unnecessary drivers and protocols . . . . . . . . . . . . . . . . . . 100

4.7 Microsoft Cluster Server installation . . . . . . . . . . . . . . . . . . . . . . . . . . . 1014.8 Service pack and post-SP installation steps . . . . . . . . . . . . . . . . . . . . . 1024.9 MSCS verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

4.9.1 Checking the mapping of host names . . . . . . . . . . . . . . . . . . . . . . 1064.9.2 Test the failover process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

4.10 Create the installation user account . . . . . . . . . . . . . . . . . . . . . . . . . . . 1074.11 SAP and DBMS installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

Contents v

4.12 SAP verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1104.12.1 Connection test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1104.12.2 SAP system check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1104.12.3 System log check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1104.12.4 Profiles check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1114.12.5 Processes check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1124.12.6 SAPWNTCHK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .113

4.13 DBMS verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1144.13.1 Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1144.13.2 DB2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1144.13.3 SQL Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .114

4.14 Backbone configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1144.15 SAP cluster verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1174.16 Tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .119

4.16.1 Advanced Windows NT tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . .1194.16.2 General SAP tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1194.16.3 SAP tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1224.16.4 SAP tuning documentation and publications . . . . . . . . . . . . . . . . .1264.16.5 DB tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .127

4.17 Configuring a remote shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .127

Chapter 5. Installation using Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1295.1 Preliminary work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1305.2 Oracle installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1305.3 Installation of the R/3 setup tool on Node A . . . . . . . . . . . . . . . . . . . . . .1335.4 Installation of the R/3 Central Instance on Node A . . . . . . . . . . . . . . . . .1335.5 Install the R3Setup files for cluster conversion . . . . . . . . . . . . . . . . . . . .1355.6 SAP cluster conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1365.7 Oracle cluster conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1395.8 Completing the migration to an MSCS . . . . . . . . . . . . . . . . . . . . . . . . . .143

Chapter 6. Installation using DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1456.1 Preliminary work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1466.2 DB2 installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1466.3 Install the R3Setup tool on Node A . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1496.4 Install the Central Instance and Database Instance. . . . . . . . . . . . . . . . .1516.5 DB2 cluster conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1526.6 Install R3SETUP files for cluster conversion . . . . . . . . . . . . . . . . . . . . . .1536.7 SAP cluster conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1546.8 Complete the Migration to MSCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .156

Chapter 7. Installation using SQL Server . . . . . . . . . . . . . . . . . . . . . . . . .1597.1 Preliminary work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1607.2 Install SQL Server and SAP R/3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1607.3 Install the R/3 Setup tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1617.4 Install the SAP R/3 Central Instance . . . . . . . . . . . . . . . . . . . . . . . . . . . .1627.5 Convert the database to cluster operation . . . . . . . . . . . . . . . . . . . . . . . .1647.6 Install the cluster conversion tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1647.7 SAP cluster conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1657.8 Complete the MSCS migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1667.9 Removal of unused resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .167

Chapter 8. Verifying the installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1698.1 How to troubleshoot the system at the end of the installation . . . . . . . . .169

vi Implementing SAP R/3 4.5B Using Microsoft Cluster Server on IBM Netfinity Servers

8.2 Log files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1708.2.1 MSCS log file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1708.2.2 SAP log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

8.3 Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1738.3.1 Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1738.3.2 DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1748.3.3 SQL Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

8.4 Accounts and users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1758.4.1 Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1768.4.2 DB2 users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1778.4.3 SQL Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

8.5 R3Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

Appendix A. Special notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .181

Appendix B. Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .183B.1 International Technical Support Organization publications . . . . . . . . . . . . . .183B.2 Redbooks on CD-ROMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .183B.3 Related Web sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .183

B.3.1 Netfinity technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .184B.3.2 Windows NT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .184B.3.3 Microsoft Cluster Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .184B.3.4 SAP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .184B.3.5 Oracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .185B.3.6 DB2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .185B.3.7 SQL Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .185

B.4 Downloadable documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .185B.4.1 Microsoft Windows NT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .185B.4.2 Microsoft Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .185B.4.3 SAP R/3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .185

B.5 SAPSERV FTP site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .186B.6 OSS notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .187B.7 Knowledge Base articles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .188B.8 Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .191

How to get ITSO redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193

List of abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

ITSO redbook evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205

© Copyright IBM Corp. 1998, 1999 vii

Preface

Many of today's corporations run their businesses on the SAP R/3 applicationsuite. High availability of these applications to the user community is essential.Downtime means lost sales, profits and worse. The combination of SAP R/3 andMicrosoft Cluster Server is an important element in providing the required accessto key business applications.

This book will help you plan and install SAP R/3 4.5B in a Microsoft ClusterServer environment running Microsoft Windows NT 4.0 on Netfinity servers.Installation procedures complement the existing SAP documentation and coverthe integration of the three major database management systems: Oracle, DB2and SQL Server.

The first two chapters introduce the concepts of clustering and high availability inSAP R/3 environments. Chapter 3 describes planning needed beforeimplementing a clustered SAP R/3 configuration. Discussions include certificationof hardware and software components, server sizing, disk layout and networkconfigurations. SAP R/3 and Windows 2000 are also discussed. The installationpart of the book is a step-by-step set of instructions to lead the reader through theprocess of installing SAP R/3, Microsoft Cluster Server and the particulardatabase you choose to install, Oracle, DB2 or SQL Server. Finally, Chapter 8offers tips on how to verify your installation and where to look if you haveproblems.

This book should be especially helpful to people who are involved with planningor installing SAP R/3 in a Microsoft Cluster Server environment. Such people willoften be technical planners and IT managers in user organizations who need tounderstand the requirements of such an installation, IT professionals orconsultants who need to install SAP R/3 in an MSCS environment, andexperienced SAP R/3 professionals who need to understand the ramifications ofclustering on SAP R/3 and how to implement such a solution.

The team that wrote this redbook

This redbook was produced by a team of specialists from around the worldworking at the International Technical Support Organization, Raleigh Center.

David Watts is an Advisory Specialist for Netfinity Servers at the ITSO Center inRaleigh. He manages residencies and produces redbooks on IBM NetfinityServers. He has authored over a dozen publications, his most recent being thethird edition of Implementing Netfinity Disk Subsystems and the second edition ofNetfinity Server Management. He has a Bachelor of Engineering degree from theUniversity of Queensland (Australia) and has worked for IBM for over 10 years.He is an IBM Professional Server Specialist.

Matthew Cali is the Technical Alliance Manager for SAP in the Netfinity Partnersin Development organization within IBM. He has 12 years of experience in IBMincluding eight years with the TJ Watson Research Laboratory as a developmentengineer involved in semiconductor and microelectronics design, prototyping andprocess formulation. He was also the IBM Global Services project manager forthe IBM fixed assets SAP implementation used throughout the company. Hecurrently holds five patents, four of which are associated with mainframe

viii Implementing SAP R/3 4.5B Using Microsoft Cluster Server on IBM Netfinity Servers

electronic circuit repair and the other is with a prototype of the IBM TrackPointstrain gauge design. He has a masters degree in Mechanical Engineering fromManhattan College, New York.

Edward Charles is a Senior IT Specialist in IBM Global Services, NetworkIntegration from Fort Lauderdale in Florida. He has over 10 years of experiencedesigning enterprise networks. He holds a BA in Management InformationSystems and an MBA. He is a certified SAP R/3 consultant on UNIX/OracleRelease 4.x. Edward is also a certified Cisco Network Associate, CNE and MCP.For the past five years he has been consulting and teaching networking for IBM.

David Dariouch is a Systems Engineer in France. He has one year of experiencewithin IBM, as an Advisory IT specialist supporting Netfinity technologies andnetwork operating systems implementation on the Netfinity Presales TechnicalSupport team. His area of expertise includes Windows NT-based ERP solutions,Netfinity hardware and software engineering. He is part of the IBM SAP TechnicalFocus Group in EMEA. David holds an Engineering Degree in Computer Sciencefrom the Ecole Speciale de Mecanique et d’Electricite school of engineering inParis.

Olivier De Lampugnani is an IT Specialist working for IBM Global Services in theService Delivery EMEA West group, located in La Gaude, France. He has beenworking with IBM for three years. His areas of expertise include all MicrosoftWindows NT products and Netfinity hardware. Olivier is a Microsoft CertifiedProfessional in Windows NT Server and Workstation.

Mauro Gatti is an Advisory IT Specialist in the Netfinity Presales TechnicalSupport in Italy. He is a Microsoft Certified System Engineer. He is responsiblefor supporting the main SAP installations on Netfinity servers in Italy. He is part ofthe IBM SAP Technical Focus Group in EMEA. Before joining IBM a year ago,Mauro was a trainer and consultant for Microsoft, Hewlett Packard, IBM and othercompanies. Mauro holds a degree in Physics and a PhD in Theoretical Physics.

Bill Sadek is an Advisory Specialist for SAP R/3 at the International TechnicalSupport Organization, Raleigh Center. He is a certified SAP R/3 Applicationconsultant in Sales and Distribution. He writes extensively on SAP R/3 andWindows NT. Before joining the ITSO, Bill worked in IBM Global Services as anSAP R/3 Architect and SAP R/3 Sales and Distribution Consultant. Bill has 23years of experience with IBM.

Preface ix

Figure 1. The team (L-R): Matthew, Mauro, David Watts, Bill, Ed, David Dariouch (inset: Olivier)

This book is based on High Availability for SAP R/3 on IBM Netfinity Servers,SG24-5170-00. Thanks to the authors:

Bill SadekPeter DejaegereGuy HendrickxFabiano MatassaTorsten Rothenwaldt

Thanks to the following people from the ITSO Center in Raleigh:

Gail ChristensenTate RennerLinda RobinsonShawn Walsh

Thanks to the following people from IBM for their invaluable contributions to theproject:

Steve Britner, PC Institute, RaleighAndrew Castillo, North America ERP Solution Sales, San JosePeter Dejaegere, IBM SAP International Competency Center, WalldorfRainer Goetzmann, IBM SAP International Competency Center, WalldorfAndreas Groth, Second Level EMEA Technical Support, GreenockThomas Knueppel, Netfinity SAP Sales, StuttgartGregg McKnight, Netfinity Development Performance, RaleighSalvatore Morsello, Netfinity Presales Technical Support, MilanKiron Rakkar, MQSeries Early Programs, RaleighTorsten Rothenwaldt, Netfinity Technical Support, Frankfurt

x Implementing SAP R/3 4.5B Using Microsoft Cluster Server on IBM Netfinity Servers

Ralf Schmidt-Dannert, Advanced Technical Support for SAP R/3, Foster CitySiegfried Wurst

Thanks to the following people from SAP AG for their invaluable contributions tothe project:

Frank HeineReiner Hille-Doering

Comments welcome

Your comments are important to us!

We want our redbooks to be as helpful as possible. Please send us yourcomments about this or other redbooks in one of the following ways:

• Fax the evaluation form found in “ITSO redbook evaluation” on page 205 to thefax number shown on the form.

• Use the online evaluation form found at http://www.redbooks.ibm.com/

• Send your comments in an Internet note to [email protected]

http://www.redbooks.ibm.com/contacts.html

http://www.redbooks.ibm.com/contacts.html

© Copyright IBM Corp. 1998, 1999 1

Chapter 1. Introduction to high availability computing

High availability in computing is defined as making a business application setavailable to the users as high a percentage of the time as possible.

This simple statement covers a lot of ground. It can be as simple as planning forthe loss of electrical power for a piece of equipment, or as disastrous as a fire andearthquake which can cause the loss of the entire computing site.

Murphy’s Law says anything that can go wrong will. The space shuttle is built onthis premise. All critical systems have one or more redundant backup systems. Incase of a failure, the backup system automatically takes over the function of thefailed primary system. Fault tolerance is the concept that a system can surviveany failure in a hardware or software component and continue to function. Thisobviously will help to maintain the high availability we are looking for in ourbusiness applications.

This same idea is used in designing high availability into computer systemssupporting business-critical applications. For example, a mail order businesscomes to a screeching halt if the order entry application becomes unavailable tothe users for any reason.

1.1 Clustering — a means to an end

Clustering is the use of two or more computers or nodes for a common set oftasks. If one computer fails, the others will take up the slack. This design supportsthe idea of fault tolerance. A second computer can be used as the redundantbackup for the first computer.

Clustering can be used to increase the computing power of the entire computerinstallation. This also allows a system to be scalable. Adding more computersincreases the power and hence the design can support more users. In our case,we use clustering for application availability with increasing processing power asa fortunate side effect.

1.2 What clustering provides

The current range of clustering solutions allows for a level of fault tolerance. Thedegree to which failures can be tolerated depends largely on two things:

1. The location of the failure

If the failure is within the cluster (for example, failed hardware in a node or atrapped operating system), then the high availability software will probably beable to recover and continue servicing its users. If the failure is outside thecluster, it is less likely that the high availability software will be able to maintainservice. Failures such as power distribution failures, complete networkoutages, and data corruption due to user error are examples of faults thatcannot be contained by products such as Microsoft Cluster Server.

2. The ability of applications to cope with the failure

Microsoft Cluster Server, for example, allows applications and resources thatwere running on a failed system to be restarted on the surviving server. For“generic” applications, this is simply a matter of restarting the program. For

2 Implementing SAP R/3 4.5B Using Microsoft Cluster Server on IBM Netfinity Servers

more complicated applications (for example, SAP R/3 servers), there must bea certain sequence to the restart. Certain resources, such as shared disks andTCP/IP addresses, must be transferred and started on the surviving serverbefore the application can be restarted. Beyond that, other applications (forexample database servers) must have clustering awareness built into them sothat transactions can be rolled back and logs can be parsed to ensure thatdata integrity is maintained.

Microsoft Cluster Server provides high availability only. The Microsoft solutiondoes not as yet address scalability, load balancing of processes nor near-100%up time. These can currently be achieved only through more mature clustering,such as that which is implemented in RS/6000 SPs.

Microsoft also offers its Windows Load Balancing Service, part of Windows NT4.0 Enterprise Edition. It installs as a standard Windows NT networking driverand runs on an existing LAN. Under normal operations, Windows Load BalancingService automatically balances the networking traffic between the clusteredcomputers.

1.3 Business data — to replicate or not?

Adding computers, networks and even cloning entire computer centers for faulttolerance purposes does not solve all the problems. If the business is relativelyunchanging, the databases can be replicated along with the rest of the system.Any updates can be accomplished for all the replicated copies on a scheduledbasis. An example of this is a Web server offering product information tocustomers. If one Web server is down the other servers can serve the customer.In this example the cluster is used for both performance and availability.

On-line Transaction Processing (OLTP) applications have different datarequirements. As the name implies the data is always changing based onbusiness transactions. A customer places an order for a product. Inventory isallocated to the order and is then shipped from the warehouse to the customer. Ifthis data is replicated, there would be the possibility of promising the same itemto two different customers. Somebody would be unhappy.

1.4 Disk sharing

Shared disks is one of the cluster architectures in the industry today. It may be usedfor scalability as well as for high availability purposes. In a typical two-node highavailability cluster, both nodes can access the same storage devices, but only oneserver at a time controls the storage devices shared by both servers. If one serverfails, the remaining server automatically assumes control of the resources thatthe failed server was using, while still controlling its own resources at the sametime. The failed server can then be repaired offline without the loss of time orwork efficiency, because access to that server's data and applications is stillavailable.

The key point is that only one server has control of the storage devices at anypoint in time. There is only one copy of the data, so data accuracy is maintained.

For an overview of cluster architectures, see In Search of Clusters, by Gregory F.Pfister.

Chapter 1. Introduction to high availability computing 3

1.5 MSCS-based solutions

As part of the early adopter’s agreement with Microsoft, IBM has announcedvalidated solutions of hardware and software to enable customers to runMicrosoft Cluster Server (MSCS) in a shared disk environment.

For managing Microsoft Cluster Server configurations, IBM has developed IBMCluster Systems Management (ICSM). It provides portable, generic clustersystems management services that integrate into existing Systems Managementtools such as IBM Netfinity Manager, Intel LANDesk, and Microsoft SMS.

ICSM offers enhancements to the manageability of MSCS in three distinctcategories:

1. Ease-of-use

2. Productivity

3. Event/Problem notification

1.5.1 Shared-disk configurationsShared-disk configurations, such as those implemented with Microsoft ClusterServer, use an external disk enclosure to house drives containing data, as shownin Figure 2. The external disks can be connected to both servers usingServeRAID, SSA, or Netfinity Fibre Channel hardware.

Figure 2. Typical MSCS cluster configuration

The external resources (that is, the drives and the data on the drives) arecontrolled (or owned) by one of the two servers. Should that server fail, theownership is transferred to the other server. The external resources can bedivided into groups so that each server can own and control a subset of them.The shared-disk configuration is also known as a swing-disk configuration.

Server A

5170

-01

MSCS cluster

Server B

Private interconnect (heartbeat)

Externalshareddisks

Virtual clusterTCP/IP address


This configuration offers the advantages of lower cost and quick recovery. Onlyone set of disks is required to hold the data used by both servers. The externaldisk enclosure should be configured for RAID to ensure data redundancy. When afailure does occur, since there is only one copy of the data, the time to recoverand bring the failed system back online is minimal.

1.6 What the user sees

The clustering solution offers high availability to a certain degree but does notoffer 100% error-free non-stop system availability.

Each configuration offers certain benefits over the others but in each there aretrade-offs to be made: cost versus redundancy, capacity, bandwidth and thedegree to which systems are isolated are some examples.

Within the specified service level and the specified level of fault tolerance, usersshould expect little or no interruption to service. The level of interruption dependslargely on the applications running on the servers at the time of failure. Forapplications such as word processors where the server does not maintain statusinformation about clients (stateless connections), the interruption is quite short.

For transaction-based applications such as databases or SAP R/3, there may bea relogin requirement once the application has been restarted. All uncommittedtransactions are rolled back during restart and may have to be reentered.

Typically, the failover and restart process involves three steps:

• Failure detection• Resource relocation• Application restart

The restarting of applications typically takes the longest time. For a simpleapplication, the failover process can take as little as 30 seconds, but largeapplications such as databases may take several minutes to recover from suchfailures.

1.7 SAP R/3 support for Microsoft clustering

In Windows NT 4.0, Microsoft has chosen the shared storage approach to itstwo-node clustering solution. SAP R/3 supports this natively in its kernel.Although other solutions may work, SAP only supports the Microsoft clusteringsolution. More information can be obtained at the SAP Web site:

http://www.sap.com

1.8 IBM Netfinity clustering solutions

For the latest information on Netfinity cluster solutions check the following Websites:

• IBM Netfinity cluster solutions:http://www.pc.ibm.com/us/netfinity/clustering.html

• IBM/SAP Alliance Home: http://www.ibm.com/erp/sap/

http://www.sap.com

http://www.pc.ibm.com/us/netfinity/clustering.html

http://www.ibm.com/erp/sap/


Chapter 2. SAP R/3 and high availability

In this chapter, we consider the special issues with SAP R/3, including thefollowing:

• Critical elements of SAP R/3 systems• How Microsoft Cluster Server (MSCS) can be used with SAP R/3• Alternate approaches to high availability

Because the focus of this redbook is on MSCS, the alternative solutions will notbe covered in detail. For the following discussion we assume that you are familiarwith the basic components of R/3 systems as described in the following books:

• Liane Will, SAP R/3 System Administration, Sybex, ISBN 0782124267l

• Hartwig Brand, SAP R/3 Implementation with ASAP, Sybex, ISBN 0782124725

• Diane Bullock et al., DB2 Universal Database and SAP R/3 Version 4, IBM,SC09-2801

• Robert E. Parkinson et al., Basis Administration for SAP, Prima Tech, ISBN0761518878

• Thomas Schneider, SAP R/3 Performance Optimization, Sybex, ISBN0782125638

• Optimizing IBM Netfinity Servers for SAP R/3 and Windows NT, SG24-5219.

2.1 SAP R/3 components

SAP R/3 has features to improve system availability:

• Automatic reconnecting• Automatic restart of processes• Redundancy by multiple instances

Most R/3 services have the ability to reconnect dynamically if the correspondingprocess is restarted. This means automatic recovery is possible from manyproblems that affect only individual processes or connections between them.

For example, if the SAP graphical user interface (SAPGUI) loses connection to itscorresponding dialog process on the application server because of a networkoutage, it asks the user to reconnect. The user context is not preserved on theapplication server. Thus all transactions which were not committed are cancelled.Data changes from committed transactions remain persistent in the database.

This feature will not help if the process at the other site fails. But most R/3processes are restarted automatically by the dispatcher after process failure. Thisincludes all work processes (dialog, update, enqueue, batch, spool) and the R/3gateway process. Only the processes of the SAPGUI, message and dispatcherservices are not restarted automatically. The SAPGUI can be recovered manuallyby simply restarting it on the workstation.

Most of the R/3 application services (such as dialog, update, batch, gateway andspool) can all be configured redundantly on multiple servers to provide improvedavailability.


There are three critical R/3 system components that cannot be made redundantby configuring multiple instances of them on different machines:

• DBMS• Enqueue service• Message service

Thus these three components are single points of failure in the SAP R/3architecture.

Thus the R/3 components can be divided into two groups, depending on theirimportance for system availability:

• Services that are R/3 system-wide single points of failure

The enqueue and the message processes, and the DBMS cannot beconfigured redundantly and require human intervention for recovery. Theseservices should be centralized to reduce the number of servers requiringadditional protection against failure.

• Services that are not single points of failure

These services may be configured redundantly and should be distributed onseveral servers.

The following table summarizes the elements of an R/3 system and their recoverycharacteristics:

Table 1. Redundancy and recovery of SAP R/3 services

A second group of critical points comes from the network environment. The twofile shares SAPLOC and SAPMNT must be available. Some installations use an

R/3 services Number per R/3 system Recovery

SAPGUI Manual restart by user; reconnect toprevious context may be used

Dispatcher Manual restart of application service bysystem administrator

Dialog Automatic restart by dispatcher

Batch Automatic restart by dispatcher

Update Automatic restart by dispatcher

Enqueue Automatic restart by dispatcher

Message Manual restart by system administrator

Gateway Automatic restart by dispatcher

Spool Automatic restart by dispatcher

saprouter Manual restart by system administrator

atp server n=1

Database Manual restart (and recovery, ifnecessary) by system administrator

Note: Automatic restart by the dispatcher assumes that the dispatcher itself is not affectedby the failure. It is possible that locks held by one user may be simultaneously granted toanother user.

n 1≥

n 1≥

n 1≥

n 1≥

n 1≥

n 1=

n 1=

n 1≥

n 0≥

n 0≥

n 1=

Chapter 2. SAP R/3 and high availability 7

Internet Domain Name Service (DNS) or Windows Internet Name Service (WINS)for name resolution. DNS or WINS are also single points of failure. Additionally,the Windows NT domain controller is needed to log on to the R/3 serviceaccounts.

To summarize, the following components must be protected to make the R/3system highly available:

• DBMS• R/3 application service on the enqueue/message host (central instance)• File shares SAPLOC and SAPMNT• Name resolution service (DNS or WINS)• Windows NT domain controller

The domain controller will be made redundant by configuring multiple domaincontrollers as primary and backup controllers. In a similar fashion, there are waysto provide redundant DNS or WINS. The details are out of the scope of thisredbook. How to protect the R/3 central instance (services and file shares) andthe database is discussed in the remainder of this chapter.

2.2 Using MSCS with SAP R/3

With the release of Microsoft Cluster Server, SAP—for the first time in its historysupporting different operating systems—has begun to support a clusterintegration for R/3, called R/3 Cluster for Windows NT. This is the cluster-awareversion of the R/3 software on Windows NT. The R/3 system (mainly the R/3kernel and the installation procedure) has been extended with components thatintegrate R/3 with MSCS. The R/3 processes, services, disks, addresses, andshares are configured as cluster resources and monitored by MSCS. Additionally,the R/3 kernel handles the reconnect issues in the case of failover.

SAP supports a configuration based on a two-node cluster in an active–activemode as shown in Figure 3. The database runs on one node and the centralinstance of the R/3 system runs on the other node. In the event of failure, thesurviving node continues to provide both services: database and central instance.MSCS monitors the R/3 and database resources for failure and switches overservices in the event of a failure. Thus the critical R/3 components, as shown inTable 1 on page 6, are made highly available. This default configuration is shownin Figure 3. A situation where both are services running on the same node maybe considered as a special case which happens after a failover, during plannedmaintenance, or for other reasons.


Figure 3. Typical MSCS configuration for SAP R/3

A supported SAP R/3 cluster configuration has to fulfill the following conditions:

• The hardware must be IBM Server Proven. For a complete list of IBM testedNetfinity MSCS solutions, go to the Web site:

http://www.pc.ibm.com/us/netfinity/cluster_server.html

• The hardware must be in the Microsoft Hardware Compatibility list. Note thatsuch approval includes:

– Both servers– The shared disk subsystem between them (controllers and storage

expansions)– The cluster-private network connection (only for long distance

configurations)

For a complete list of Microsoft certified IBM Netfinity MSCS solutions , go tothe Web site:

http://www.microsoft.com/hwtest/hcl/

• The hardware must be certified for SAP R/3, as for every R/3 system. For a listof certified IBM Netfinity configurations, see the Internet web site:

http://www.r3onnt.de/

ApplicationServer

DatabaseServer

CentralInstance

5170

-01

MSCS cluster

This cluster configuration (database server and central instance of the sameR/3 system on two cluster nodes with mutual failover) is the only one supportedby SAP. There is no support for other possible combinations (for example,failover of a central production system to a test server, or failover of the centralinstance to another application server, or running an additional applicationserver on the DBMS machine).

A special cluster combination may work. But the R/3 installation procedure isnot prepared to handle it, and there will be no support from SAP for dailyoperations or release upgrades.

Supported MSCS configuration

See 3.2, “Certification and validation of hardware” on page 42 for moredetails on the certification processes.

Note

http://www.r3onnt.de/



http://www.microsoft.com/hwtest/hcl/


• The Operating system release, the Windows NT Service Pack release, theDBMS release and the SAP R/3 release are SAP supported. To check if yourSAP R/3 version supports MSCS you should obtain the latest update of theOSS Note 0106275, Availability of R/3 on Microsoft Cluster Server. At the timeof writing, the following support was available or planned with Intel-basedWindows NT servers:

Table 2. MSCS support for Intel-based Windows NT servers with different SAP R/3 releases

Note: Refer to OSS Note 0106275, Availability of R/3 on Microsoft Cluster Serverfor the latest support matrix.

2.3 Alternate approaches to SAP R/3 high availability

Using MSCS may not be the only possible solution in your business case toachieve high availability for SAP R/3. Here, we discuss the advantages anddisadvantages of some other approaches. For a more complete discussion seethe excellent book Windows NT Microsoft Cluster Server by Richard R. Lee.

There are several other failover solutions available for Windows NT, for example:

• Vinca CoStandbyServer for Windows NT• Veritas First Watch• NCR LifeKeeper• Legato/Qualix FullTime Cluster• CompaQ/Digital Cluster Plus for Windows NT• CompaQ/Tandem NSK-Lite• Legato Octopus HA+• NSI Software Double-Take

Note: This is not a complete list. It does not intend to express the significance ofthe products mentioned, nor their compliance with a particular R/3 release testedby IBM or SAP.

Even if the description of the product may lead to the assumption that it workswith MSCS, this does not say anything about the functionality with SAP R/3.Failover products are inherently complex. Setting up R/3 and the DBMS to failover correctly is a tremendous task including the handling of registry keys, DLLs,environment variables, failover scripts, and other objects that may beundocumented and changed with the next release of the R/3 or DBMS software.Thus you should always ensure that you get appropriate support, comparable

DBMS R/3 V3.1I R/3 V4.0B R/3 V4.5A R/3 V4.5B

Oracle 7.3 Supported No No No

Oracle 8.0.4 Supported Supported Supported No

Oracle 8.0.5 Supported Supported Supported Supported

Microsoft SQL 6.5 Supported No No No

Microsoft SQL 7.0 Supported Supported Supported Supported

Informix Supported Supported Supported Supported

Adabas No No Supported Supported

DB2 No No No Supported


with that provided by SAP for MSCS, from the vendor over the full lifetime of yoursystem.

2.3.1 Cold standby serverThe idea to improve availability with a so-called cold standby server is tocompletely exchange the production server if it fails. For that purpose, you needto configure an additional server identical to the production machine. This meansthat all components (especially I/O adapters) should be of the same type and inthe same slots in both machines. However, the standby server does not containany disks with an operating system or data. When the production server fails, anoperator moves all disks from the production server to the standby server.Because of the identical configuration, the copy of the operating system from theproduction machine will boot without problems on the standby server too. Thenthe R/3 instance or DBMS can be started. After doing some checks in the R/3system, the standby machine can be made available to the users with the sameaddresses and services as the production machine.

To move the disks quickly, all disks should be installed in an external storageexpansion enclosure. Then moving the disks means only recabling the storageexpansion. For smaller configurations it may be sufficient to have the disks ashot-swap drives in the production machine, without an additional storageexpansion enclosure.

This looks primitive. But despite its simplicity, there are some advantagesespecially for a small business:

• The solution can be provided by almost any vendor with arbitrary hardware,which does not need to be approved for a cluster solution like MSCS. The onlyrestriction about server models, disk technology, and other components is thecertification for SAP R/3.

• No special cluster support is required from the vendor or SAP. All R/3 andDBMS releases work in the usual way.

• There is no need for splitting the production system into separate servers forthe DBMS and the R/3 central instance. The additional server may act as coldstandby for multiple production servers (of the same hardware configuration)from different R/3 systems, or a test machine may be used.

• There are no cluster-related restrictions for using hardware or software RAID.

The disadvantages are obvious:

• Human intervention is necessary to recover from a server failure. This requiresmonitoring of the system and an appropriate reaction time. Moving the disks,and starting and checking the system should be done by trained operators. Ifthe system is used only during normal office hours, that may be acceptable.But batch jobs during the night or weekend or any other kind of 24x7 usagerequires automatic reaction to system outages.

• Only hardware-related failures are covered. If there is a problem with theoperating system on the production machine (for example, driver failures orwrong network setup), then this will happen on the standby machine also.

• The same hardware configuration of the machines must always be kept, in alldetails. When the standby machine cannot be used otherwise, maintaining alarge machine only as a big spare part may be too expensive.


Thus a cold standby server is worth considering if the sizing of your SAP R/3systems leads to a central installation with production R/3 services and DBMS onthe same machine, and if the restrictions about service availability are acceptablein your business.

2.3.2 Replicated databaseThis section discusses ways to achieve high availability by replicating thedatabase using DBMS features. Note that this technique differs from RAID andfrom replicated DBMS servers:

• Replicating the database is maintaining a (not necessarily complete or actual)copy of data that is under the control of the DBMS.

• Replicating the DBMS server means running an additional, fully functionalDBMS in parallel. This is discussed in 2.3.3, “Replicated DBMS server” onpage 14.

• RAID protects data (inside or outside the database) against disk failures.Server or DBMS failures are not covered.

Replicated databases are sometimes referred to as shadow databases. There areseveral solutions available from the DBMS vendors:

• Oracle Standby Database• Symmetric Replication from Oracle• High-availability Data Replication (HDR) from Informix• SQL Server Replication from Microsoft• Replicated Standby Database for DB2/UDB from IBM

The idea is to have a second DBMS in standby mode on a different node thatcontinuously receives information about database changes made on theproduction node. Then the standby DBMS can apply (replicate) these changes. Ifthe initial data were identical on both nodes, and if no change information hasbeen lost, then the standby DBMS maintains an exact copy of the productiondatabase. When the production node fails then the R/3 system can be connectedto the standby node.

The following are among the issues that you should consider when selecting astrategy for database replication:

• Synchronous versus asynchronous replication

Synchronous replication considers a transaction as completed only when theremote DBMS also committed successful execution of all changes.Asynchronous replication performs the remote updates at a later time. Thusasynchronous replication schemes may suffer transaction loss in the case offailure, while synchronous replication schemes by definition guarantee notransaction loss.

• Log-based versus statement-based replication

Log-based replication evaluates or copies the redo logs of the DBMS to getthe change information for the remote node. Log-based implementations tendto be asynchronous, while statement-based implementations are mostlysynchronous.


• Level of replication

Some solutions provide replication at the schema or table level, while othersperform replication at the database level. Schema level replication is moreflexible but requires extra effort in that the system administrator has to definethe list of tables, either whole tables or subsets (rows or columns), to bereplicated.

• BLOB handling

There may be limitations for the replication of binary large objects (BLOBs).

From the solutions mentioned above, Oracle Standby Database, ReplicatedStandby Database for DB2/UDB (both at the database level), and Microsoft SQLServer Replication (at the schema or table level) implement asynchronouslog-based replication. Symmetric Replication from Oracle is statement based andmay be synchronous or asynchronous at the schema or table level. InformixHigh-availability Data Replication (HDR) offers asynchronous and synchronouslog-based replication at the database level.

There are different ways to use data replication for R/3 high availability, includingthe following:

1. Maintain complete, hot standby database

The R/3 system switches to the standby database and continues to functionwithout any loss of transactions. Synchronous replication is required. From theproducts above, only Informix HDR in synchronous mode may be used for thispurpose. This can be considered as an alternative to remote mirroring of thedatabase disks.

2. Maintain an incomplete, warm standby database

If it is sufficient to have a warm standby database (loss of some transactions isacceptable) rather than a hot standby, then any of the log-based replicationschemes may be used.

As an example of a warm standby database, we outline the configuration ofOracle Standby Database. The hardware architecture and the operating systemmust be the same on both nodes. The version of the Oracle software on thestandby node must be the same or higher as on the production node.

1. Take a complete backup of the data files of the production server.

2. Create a standby database control file.

3. If the backup was not an offline backup, then archive the current online redolog of the production database.

4. Transfer the backed up data files, the control file, and all archived redo log filesto the standby machine.

5. To begin with actual replication, start the DBMS on the standby machine,mount the database in standby mode, and let the DBMS operate in recoverymode.

6. To keep the update gap between the production and the standby database assmall as possible, you have to ensure that the archived redo log files from theproduction system are transferred to the standby system as soon as possible.


Figure 4. Oracle standby database

The greatest advantage of this approach, compared with clustering, is the moredistant separation of production and the standby system. Even geographicdistribution is simple because only a network connection with small bandwidth isneeded for redo log file transfer. There are no cluster-related restrictions on whichkind of hardware to use.

The greatest disadvantage is that only the database is protected. Failures of theR/3 central instance services or file shares are not covered. If you plan to preparea standby central instance on the standby database node also, keep in mind thatchanges in non-database files (like R/3 profiles) are not replicated. Otherdisadvantages are:

• The standby database is in recovery mode and always lags slightly behind theproduction database. The loss of transactions during the failover may includeall changes stored in up to three redo log files:

1. One because it was the last active file2. A second because of incomplete archiving3. A third because of incomplete transfer to the standby server

After work resumes using the standby database, all users have to be madeaware that transactions may be lost and that they have to check whether thelast changes they made are still available.

• Oracle does not provide anything to transfer the archived redo logs from theprimary node to the standby node. However, Libelle Informatik of Germanyprovides a third-party tool. See:

http://www.libelle.de

• The remote database is mounted in standby mode and cannot be used in anyway other than for recovery.

• An Oracle database mounted in standby mode cannot be opened in thestandard way. This prevents an accidental opening, which would invalidate thestandby state of the database. But if the production database becomesunavailable, the standby database has to be activated, shut down, and thenopened for normal use. This procedure should be performed by anexperienced system administrator. Automating this procedure may giveunexpected results if, for example, only a network outage or a reboot of theproduction server suspends a log file transfer.

Redo LogFiles

DataFiles

ControlFile

Redo LogFiles

DataFiles

ControlFile

PermanentRecovery

Archiving File Transfer

Production Database Standby Database

http://www.libelle.de


• It is complicated and time-consuming to fail back to the original productiondatabase because of the risk of losing transactions. Basically the sameprocedure as for setting up the standby database is needed for failback.

• Structural database changes such as adding or dropping data files, ormodifying parameters stored in the INIT.ORA file, are not propagated to thestandby database.

• If there are failures on the standby system, then inconsistencies may result,and tablespaces may be lost. In this situation, the only way is to set up thestandby database from scratch again.

In addition, you have to consider the effects on the R/3 application serversconnected to the database, when failover to the standby database is performed.Because the standby node must have its own network addresses for receiving thelog files, the R/3 processes cannot reconnect to the address known as the DBMSservice. This may be solved by a restart using different R/3 profiles to connect tothe standby database or by using an alias name for the DBMS node andremapping the alias with DNS.

There are third-party products available which resolve problems in dailyoperations of a replicated database. For example, the Libelle Database Mirror (forOracle only, see http://www.libelle.de/) helps create the replica, controls thetransfer of the archived redolog files, detects and replicates structural databasechanges. Without such DBMS-specific tools, a replicated database should beconsidered only if the system administrator has detailed knowledge andexperience with the DBMS used. Of course, such tools don't change the basicarchitectural principles of the database replication (long distance, hot standbyversus warm standby with time delay and possible loss of transactions) which arethe criteria to decide if such a solution is appropriate (or may be combined withMSCS).

2.3.3 Replicated DBMS serverAs mentioned in 2.3.2, “Replicated database” on page 11, replicating thecomplete DBMS server means running an additional DBMS in parallel with theproduction system. This includes replication of the database itself and increasesthe availability of the R/3 system. As an example, we consider Oracle ParallelServer (OPS).

An operational Oracle DBMS used by an application such as SAP R/3 consists ofseveral main components. There are files that form the database (data files, redolog files, control files), other objects such as the system global area (SGA, thebuffer area in memory), and a set of database background processes (log writerLGWR, database writer DBWR, and others). SGA and background processes arecalled the Oracle instance. Each instance has a private set of online redo logfiles. A normal installation has only one instance per database. With OPS it ispossible to have multiple Oracle instances on different machines providingaccess to the same database.

This provides protection against DBMS failures. If one instance fails, a survivinginstance performs the recovery. Transactions open at the time of the instancefailure are rolled back, and committed transactions are reapplied. The survivinginstance continues to provide database access. Application processes connectedto the failed instance can reconnect to any surviving instance. OPS can also be

http://www.libelle.de/


used to enhance the DBMS performance, using multiple active instancesconcurrently. OPS is implemented on the shared workload cluster model, incontrast to MSCS which is based on the partitioned workload cluster model. WithOPS, the failure of one database server is only a special case in permanent loadbalancing between instances.

Figure 5. Architecture of Oracle Parallel Server on Windows NT

Oracle Parallel Server is available on Windows NT beginning with Version 7.3.3on certified hardware platforms with between two and eight DBMS nodes. Formore detailed information about the OPS implementation for Windows NT and thehardware, see the Internet at:

http://www.oracle.com/nt/clusters/

The availability of the R/3 system with OPS depends on the hardware platformand R/3 release. SAP supports OPS only for approved installations. You mustensure that you have this approval before going into production. Only the setupwith an idle instance is currently supported. Two instances are configured, all R/3processes connect to the same instance. The other instance is idle and acts as ahot standby. If the production instance fails, the R/3 processes reconnect to thestandby instance.

Compared with a replicated database, this approach has some advantages:

• In the event of failure, no committed transactions are lost.

• There is no operator intervention necessary to fail over the DBMS.

• Setup and failback are easier because no backup and restore are necessary.

Some of the disadvantages of OPS when used with SAP R/3 are the same as forreplicated databases:

OPS protects against DBMS failures only. Problems of the SAP R/3 centralinstance services or file shares are not covered. OPS imposes more restrictionson the hardware that may be used. Because all instances must have direct(shared) access to all database disks, a network connection is not sufficient.There must be a shared disk subsystem between all database servers. Thus theoptions for the separation of the nodes are limited by the shared storagetechnology. Another disadvantage is that OPS for Windows NT is incompatiblewith MSCS. Both products cannot be on the same nodes.

Data Files

Control File

Redo LogFiles

Redo LogFiles

SGA

LGWR DBWR

SGA

DBWR LGWR

Instance A Instance B

http://www.oracle.com/nt/clusters/


2.3.4 Multiplatform solutionsWith SAP R/3, the DBMS and the R/3 application processes may reside onmachines with different operating systems. The R/3 processes need only TCP/IPconnections to a DBMS with a compatible version according to the requirementsfor that R/3 release. If the DBMS is configured as highly available on anotheroperating system platform, then only the critical R/3 processes of the centralinstance and the file shares must be protected under Windows NT. Other highavailability platforms are:

• HACMP (High Availability Cluster Multi-Processing) for AIX• HAGEO (High Availability Geographic Cluster) for AIX• Oracle Parallel Server on HACMP for AIX• OS/390 Parallel Sysplex (with DB2)

For more information see the following redbooks available fromhttp://www.redbooks.ibm.com:

• Disaster Recovery with HAGEO: An Installer’s Companion, SG24-2018

• Bullet-Proofing Your Oracle Database with HACMP: A Guide to ImplementingAIX Databases with HACMP, SG24-4788

• Oracle Cluster POWERsolution Guide, SG24-2019

• High Availability Considerations: SAP R/3 on DB2 for OS/390, SG24-2003

To protect the Windows NT server with the R/3 central instance, an MSCSconfiguration, a cold standby server, or an alternate failover solution can be used.When using MSCS, keep in mind the restrictions of R/3 support for MSCS (see2.2, “Using MSCS with SAP R/3” on page 7). There are plans for clusterconfigurations with the central instance on one node and another applicationserver of the same R/3 system (with a different system number) on the other node(no failover of this second instance), but currently the failover of an R/3 instanceto a server with other R/3 services is not supported by SAP.

This approach has advantages for large installations. If a cluster configuration forAIX or mainframe systems already exists, then the R/3 database may beintegrated easily and fast. Additional infrastructure for high availability (networkenvironment, appropriate operating service, disaster recovery planning) isavailable immediately. Because the DBMS exploits a cluster implementation moremature than MSCS, there are more choices for the configuration, and the clustercan also be used to boost database performance (Oracle Parallel Server for AIXor DB2 Parallel Edition).

If there is no AIX or mainframe cluster available, then a multiplatform approachmay be more expensive than a pure Windows NT environment. The costs includehardware, software and training. Because of the variety of platforms,maintenance is more complicated and requires more specialists. Multiplatformsolutions are favored primarily for large installations with strong performancerequirements.

2.4 Microsoft Cluster Server

Microsoft Cluster Server (MSCS), as part of the Microsoft Windows NT ServerEnterprise Edition, is Microsoft's first major push into the enterprise computingarena. Providing the capability to link servers together to form a single computing




resource is one way Microsoft is positioning Windows NT as a viable alternativeto UNIX in large-scale business and technical environments.

MSCS is particularly important as it provides an industry-standard clusteringplatform for Windows NT and it is tightly integrated into the base operatingsystem. This provides the benefits of a consistent application programminginterface (API) and a software development kit (SDK) that allow applicationvendors to create cluster-aware applications which are relatively simple to install.

The first release of MSCS, also referred to as Phase 1, links two servers togetherto allow system redundancy. Even before the release of MSCS, hardwaremanufacturers such as IBM provided redundancy for many server components,including power supplies, disks, and memory. This, however, would only protectyou from component failure and not application failure.

Providing system redundancy means that a complete server can fail and clientaccess to server resources will remain largely unaffected. MSCS extends this byalso allowing for software failures at both operating system and application levels.If the operating system fails, all applications and services can be restarted on theother server. Failure of a single application is managed by MSCS individually.This, in effect, means that a failure can occur, but the cluster as a whole remainsintact, still servicing its users’ requests.

MSCS achieves this by continually monitoring services and applications. Anyprogram that crashes or hangs can be immediately restarted on the same serveror on the other server in the cluster.

If a failure does occur, the process of restarting the application on the otherserver is called failover. Failover can occur either automatically, such as when anapplication or a whole server crashes, or manually. By issuing a manual failover,the administrator is able to move all applications and resources onto one serverand bring the first server down for maintenance. When the downed server isbrought back online, applications can be transferred back to their original servereither manually or automatically. Returning resources to their original server isoften referred to as failback.

The current MSCS allows only two configured servers, or nodes, to be connectedtogether to form a cluster (Microsoft has stated that future versions will supportmore nodes). The nodes are made available to client workstations through LANconnections. An additional independent network connection is used for internalhousekeeping within the cluster. Both nodes have access to a common disksubsystem. Figure 6 shows the basic hardware configuration to support MSCS:


Figure 6. Basic cluster configuration

In MSCS terminology, the applications, data files, disks, IP addresses, and anyother items known to the cluster are called resources. Cluster resources areorganized into groups. A group can reside on either node, but on only one node atany time, and it is the smallest unit that MSCS can fail over.

2.4.1 ResourcesResources are the applications, services, or other elements under the control ofMSCS. The status of resources is supervised by a Resource Monitor.Communication between the Resource Monitor and the resources is handled byresource dynamic link library (DLL) files. These resource DLLs, or resourcemodules, detect any change in state of their respective resources and notify theResource Monitor, which, in turn, provides the information to the Cluster Service.Figure 7 shows this flow of status data between the Cluster Service and acluster’s resources:

Cluster Interconnect

External Shared Disk

Public LAN

Two common cluster topologies are Shared Disk and Shared Nothing. From apurely hardware point of view MSCS is a Shared Disk clustering technology.But this can be misleading because from a cluster point of view MSCS is aShared Nothing technology. Indeed disks are only shared in that they areaccessible by both systems at a hardware level. MSCS allocates ownership ofthe disks to one server or the other. In normal operation, each disk is accessedonly by its owning machine. A system can access a disk belonging to thesecond system only after MSCS has transferred ownership of the disk to thefirst machine.

Common versus shared disk


Figure 7. Communication between the Cluster Service and the resources

2.4.2 Resource MonitorA Resource Monitor watches its assigned resources and notifies the ClusterService if there is any change to their state. We have shown only a singleResource Monitor in Figure 7, but in fact, each node may run more than one ofthem. By default, the Cluster Service will start one Resource Monitor to serviceall resources on a node. You can choose to run a resource under its own separateResource Monitor when the resource is defined during the cluster installationprocess. You would normally do this only for resources that are being debuggedor if conflicts with other resources have occurred.

The Resource Monitor is separated from the Cluster Service to provide an extralevel of security. The resource DLLs are running in the address space of theapplications themselves. If the applications fail, the resource DLLs maymalfunction, causing the Resource Monitor to fail as well. In these circumstances,the Cluster Service, however, should remain available to the cluster.

2.4.3 DependenciesDependencies are used within Microsoft Cluster Server to define how differentresources relate to each other. Resource interdependencies control thesequence in which MSCS brings those resources online and takes them offline.As an example, we will look at a file share for Microsoft Internet InformationServer (IIS).

A file share resource requires a physical disk drive to accommodate the dataavailable through the share. To bind related resources together, they are placedwithin an MSCS Group. Before the share can be made available to users, thephysical disk must be available. However, physical disks and file shares initiallyare independent resources within the cluster and would both be brought onlinesimultaneously. To make sure that resources in a group are brought online in thecorrect sequence, dependencies are assigned as part of the group definition.

ResourceDLL

Group 1

Resource Resource

ResourceDLL

ResourceMonitor

Cluster Service

Node A


Other items are required to make a fully functional file share, such as an IPaddress and a network name. These are included in the group, and so is anInternet Information Server (IIS) Virtual Root (see 2.4.4, “Resource types” onpage 20 for more information on this resource. The group structure is shown inFigure 8.

Figure 8. Resource dependencies

This diagram shows the hierarchy of dependencies within the group as a treestructure, where an arrow points from one resource to another resource uponwhich it depends. We see that the IIS Virtual Root is dependent on two otherresources:

• A File Share resource that is itself dependent on a Physical Disk resource

• A Network Name resource that is itself dependent on an IP Address resource

When Microsoft Cluster Server is requested to bring the IIS directory online, itnow knows that it must use the following sequence of steps:

1. Bring the Physical Disk and the IP Address resources online.

2. When the Physical Disk becomes available bring the File Share online andsimultaneously,

3. When the IP Address becomes available bring the Network Name online, then

4. When both the File Share and the Network Name become available, bring theIIS Virtual Root online.

As described in 2.4.6, “Resource groups” on page 22, all dependent resourcesmust be placed together in a single group and a resource can only belong to onegroup.

2.4.4 Resource typesMSCS defines 12 standard resource types. Other resource types can be offeredby third-parties if they create suitable resource DLLs using the applicationprogramming interface (API) in the Microsoft Platform Software Development Kit(SDK).

IIS Virtual Rootresource

5170

-01 File Share

resource

Physical Diskresource

Network Nameresource

IP Addressresource


• DHCP Server

• Distributed Transaction Coordinator

This resource type allows you to use Microsoft Distributed TransactionCoordinator (MSDTC) in MSCS. Two dependencies are required for thisresource: a Physical Disk resource and a Network Name resource.

• File Share

The File Share resource type lets you share a directory on one of theclustered disks in your configuration to give access to that directory to networkclients. You will be asked to enter the name of the share, the network path, acomment, and the maximum number of users that can connect to the share atthe same time.

The configuration of a File Share resource type is identical to the configurationof a file share in Windows NT Explorer. File Shares require a Physical Diskresource and a Network Name resource.

• Generic Application

The Generic Application resource type allows existing applications that areotherwise not cluster-aware to operate under the control of MSCS. Theseapplications can then fail over and be restarted if a problem occurs. There areno mandatory resource dependencies.

Microsoft Cluster Server is often demonstrated using the Windows NT clockprogram. To do so, the clock.exe program is defined to the cluster as aGeneric Application.

• Generic Service

This resource type can be used for services running on Windows NT. You mustenter the exact name of the service at the creation of the resource. Just as forGeneric Applications, the Generic Service resource does not have anyresource dependencies.

• IIS Virtual Root

The IIS Virtual Root resource type provides failover capabilities for MicrosoftInternet Information Server Version 3.0 or later. It has three resourcedependencies: an IP Address resource, a Physical Disk resource, and aNetwork Name resource.

• IP Address

An IP Address resource type can be used to assign a static IP address andsubnetmask to the network interface selected in the Network to Use optionduring the definition of the resource. IP Addresses do not have anydependencies.

• Microsoft Message Queue Server

This resource type supports clustered installations of Microsoft MessageQueue Server (MSMQ) and is dependent on a Distributed Transaction

Although support for this resource type is discussed in the Microsoft ClusterServer Administrator's Guide, a DHCP server resource is not supported.Refer to Microsoft Knowledge Base article Q178273.

Documentation error


Coordinator resource, a Physical Disk resource, and a Network Nameresource.

• Network Name

The Network Name resource type gives an identity to a group, allowing clientworkstations to see the group as a single server. The only dependency for aNetwork Name is an IP Address resource.

For example, if you create a group with a Network Name resource calledFORTRESS1 and you have a File Share resource with the name UTIL, youcan access it from a client desktop entering the path \\FORTRESS1\UTIL.This will give access to the directory on the share regardless of which clusternode actually owns the disk at the time.

• Physical Disk

When you first install MSCS on your nodes, you are asked to select theavailable disks on the common subsystem. Each disk will be configured as aPhysical Disk resource. If you find it necessary to add more disks after theinstallation, you would use the Physical Disk resource. This resource does nothave any dependencies.

• Print Spooler

The Print Spooler resource type allows you to create a directory on a commonstorage disk in which print jobs will be spooled. Two resources are needed tocreate a Print Spooler resource: a Physical Disk resource and a NetworkName resource.

• Time Service

This is a special resource type that maintains date and time consistencybetween the two nodes. It does not have any dependencies. The cluster mustnot have more than one Time Service resource.

2.4.5 Resource statesResources can exist in one of five states:

1. Offline - the resource is not available for use by any other resource or client.

2. Offline Pending - this is a transitional state; the resource is being taken offline.

3. Online - the resource is available.

4. Online Pending - the resource is being brought online.

5. Failed - there is a problem with the resource that MSCS cannot resolve. Youcan specify the amount of time that MSCS allows for specific resources to goonline or offline. If the resource cannot be brought online or offline within thistime, the resource will go into the failed state.

2.4.6 Resource groupsThe smallest unit that MSCS can fail over from one node to the other is a group.Related resources that have been defined from the palette of available resourcetypes are collected together into groups. Dependencies between the resources ina group are then assigned as already described in 2.4.3, “Dependencies” onpage 19.


Any cluster operation on a group is performed on all resources within that group.For example, if a resource needs to be moved from node A to node B, all otherresources defined in the same group will be moved. Figure 9 depicts how MSCSgroups might be distributed between the nodes:

Figure 9. Example of MSCS Groups

Group statesA resource group can be in any one of the following states:

• Online - all resources in the group are online.

• Offline - all resources in the group are offline.

• Partially Online - some resources in the group are offline and some are online.

Virtual serversVirtual server is a cluster group which contains an IP address and a NetworkName resource and optionally a disk and other resources.

Groups that contain at least an IP Address resource and a Network Nameresource appear on the network as servers. They appear in NetworkNeighborhood on Windows clients and are indistinguishable from real servers asfar as a client is concerned. These groups are, therefore, sometimes referred toas virtual servers.

To gain the benefits of clustering, your network clients must connect to virtualservers and not the physical node servers. For example, if you create a groupwith a Network Name resource called IIS_Server and then browse your network,you will see an entry (a virtual server) called IIS_Server in the same domain asthe physical servers. Although you can browse for the physical server names, you

Dependent resources must be grouped together.

When one resource is listed as a dependency for another resource, then thetwo resources must be placed in the same group. If all resources are ultimatelydependent on the one resource (for example, a single physical disk), then allresources must be in the same group. This means that all cluster resourceswould have to be on a single node, which is not ideal.

Note

Group 1 Group 2

Resource A

Resource B

Resource C

Resource D

Node B

Group 3 Group 4

Resource E Resource F

Resource G

SharedExternalDisks

Node A


should not use them for connections as this will circumvent the cluster failoverfunctionality.

2.4.7 Quorum resourceOne physical disk that is accessible from both nodes is used to store informationabout the cluster and is called the quorum disk. This resource maintains loggeddata that is essential to maintain cluster integrity and to keep both nodes insynchronization, particularly when the nodes fail to communicate with each other.The quorum disk can be owned by only one node at a time and is used todetermine which node will take ownership of cluster resources in certainsituations. For example, if the nodes lose contact with each other, the node thatcannot contact the quorum disk will withdraw from the cluster while the othernode assumes ownership of all cluster resources.

You specify an initial quorum resource when installing the first MSCS node. Itmust be located on a drive in the common disk subsystem. Therefore, thephysical drive used when defining the quorum logical drive must reside in astorage expansion enclosure with the other drives. The drive containing thequorum resource may also contain other applications and data, but this is notrecommended.

Note: In the case of IBM’s ServeRAID adapters, the quorum resource istemporarily defined on a local disk during installation. By completion, however,the quorum resource has been migrated to one of the common subsystem drives.

2.4.8 TCP/IPMSCS uses TCP/IP to communicate with network applications and resources.Cluster IP addresses cannot be assigned from a Dynamic Host ConfigurationProtocol (DHCP) server. These include IP Address resources, the clusteradministration address (registered at the installation of MSCS), and theaddresses used by the nodes themselves for intracluster communication.

Note that each node will usually have at least two network adapter cardsinstalled. Although a single network connection can be used, Microsoftrecommends using a private network for cluster traffic. One adapter is used toallow communication over the external network for administration andmanagement of the cluster and for user access to cluster resources. The physicalserver IP addresses assigned to these adapters could be obtained throughDHCP, but it is important that users attach to clustered addresses. Werecommend the use of static IP addresses for all adapters in your cluster;otherwise, if a DHCP leased address expires and cannot be renewed, the abilityto access the cluster may be compromised (see Microsoft Knowledge Basearticle Q170771).

The second adapter in each machine is for intracluster communication, and willtypically have one of the TCP/IP addresses that conform to those reserved forprivate intranets. Table 3 shows the allocated ranges for private IP addresses:


Table 3. Private IP address ranges

For more information refer to TCP/IP Tutorial and Technical Overview,GG24-3376, and Chapter 3 of Microsoft Cluster Server Administration Guide.

2.4.9 Additional comments about networking with MSCS1. Microsoft SNA Server, Proxy Server and WINS server currently have their own

capabilities for high availability and do not use MSCS.

2. Clustered servers can be connected to multiple subnets. MSCS additionallysupports multihoming configurations using multiple network cards. If we havetwo subnets connected by a router, however, there is no way to fail over an IPAddress resource.

3. MSCS cannot use a second network card as a hot backup for the clientaccess. This means that the card may be a critical failure point.

The problem of redundancy for the client network access must be solved in anetwork layer below the address assignment by Windows NT. We recommendthat you use redundant network adapters. IBM offers fault tolerant token-ring,fast Ethernet and Gigabit Ethernet adapters.

Cluster problems may arise due to the priority of network adapters in amultihomed Windows NT server. Because each cluster node has adaptersconnected to at least two different networks (the cluster’s private link and thepublic LAN), each cluster node is also a multihomed host. On such systems, thequestion, “What is my IP address?” is answered by a list of IP addressesassigned to all network cards installed in the machine. The Windows Sockets APIcall gethostbyname() is used to obtain these addresses.

Some cluster applications (for example, Oracle FailSafe and SAP R/3) aresensitive about the IP address order in the list returned by gethostbyname(). Theyrequire that the IP address of the adapter to which their cluster virtual addresswill be bound appears on top. This means that the address of the adapterconnected to the public LAN must be listed before the address of the cluster’sprivate link adapter. If not, it may be impossible to connect to the applicationusing the virtual address after a failover.

To avoid such problems, you should check the order of assigned IP addresses inthe address list before you install MSCS (to ensure that the network assignmentsare right from the beginning). The two simplest ways to do this are:

• Ping each node from itself. For example, on Node_A you type in a commandwindow:

IP address range Description

10.0.0.0 through 10.255.255.255 A single Class A network

172.16.0.0 through 172.31.255.255 16 contiguous Class B Networks

192.168.0.0 through 192.168.255.255 256 contiguous Class C Networks

You must have TCP/IP installed on both servers in order to use MSCS.Applications that use only NetBEUI or IPX will not work with the failover abilityof MSCS. However, NetBIOS over TCP/IP will work.

Note


ping node_a

The address from which the ping is answered must be the address assigned tothe adapter card in the public LAN.

• The Windows NT utility ipconfig shows all addresses in the order of thegethostbyname() list.

If the result you get from any of the above methods is in the wrong order, correctit before installing MSCS. When the network adapters are of the same type, thenit is sufficient to simply exchange their outgoing cable connections. If you havedifferent adapter types (for example, 10/100 EtherJet for the cluster-private linkand redundant FDDI for the public LAN), then you need a way to control theinternal IP address order.

Under Windows NT 3.51, the IP address list is in the same order as the TCP/IPnetwork card binding order; therefore, altering the TCP/IP network card bindingorder (Control Panel > Network > Bindings) will change the IP address orderreturned by gethostbyname(). Unfortunately, under Windows NT 4.0, the bindingorder does not influence the IP address order; using the Move Up or Move Downbuttons in the Bindings tab of the network control will not work. This isdocumented in the Microsoft Knowledge Base article number Q171320. To solvethe problem, you have to manually add a registry value, DependOnService, to changethe IP address order.

Assuming that your two network adapter cards have the driver names Netcard1and Netcard2 (the IBM Netfinity 10/100 EtherJet Adapter has the driver nameIBMFE), ping and ipconfig show you Netcard1 first, but your public LAN is onNetcard2. To change the IP address order so that Netcard2's address is listedfirst, you must edit the Windows NT Registry as follows (remember that this canhave serious consequences if not done carefully):

1. Start regedt32.exe (Regedit.exe will not work since it cannot add complex valuetypes) and select the following subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netcard1

2. Add a new value with the following specifications:

Value Name: DependOnService

Value Type: REG_MULTI_SZ

Data: Netcard2

3. Exit the Registry Editor and reboot the machine. ping and ipconfig should nowshow you the addresses in the required order.

As documented in Microsoft Knowledge Base article number Q164023, thisproblem has been resolved for Windows NT Server 4.0 in Service Pack 4.

SP4 fixes the problem

Note that the new registry value, DependOnService, will be deleted wheneverWindows NT rebuilds the network bindings. Thus after each modification ofnetwork parameters you should verify that the order is still correct. If youchange the IP settings frequently, you will save time by exporting the value to a.REG file for convenient registry merging.

Tip


2.4.10 DomainsThe following information specifies the criteria for clustered MSCS servers inregard to domains:

1. The two servers must be members of the same domain.

2. A server can only be a member of one cluster.

3. The following are the only valid domain relationships between cluster nodes:

• A primary domain controller and a backup domain controller• Two backup domain controllers• Two stand-alone servers

In general, we recommend that the nodes are set up as stand-alone servers. Thiswill remove the additional workload generated by the authentication chores andthe domain master browser role performed by domain controllers. However, thereare situations, such as when domain size is small, when it may be appropriate fornodes also to be domain controllers.

2.4.11 FailoverFailover is the relocation of resources from a failed node to the surviving node.The Resource Monitor assigned to a resource is responsible for detecting itsfailure. When a resource failure occurs, the Resource Monitor notifies the ClusterService, which then triggers the actions defined in the failover policy for thatresource. Although individual resource failures are detected, remember that onlywhole groups can fail over.

Failovers occur in three different circumstances: manually (that is, at the requestof an administrator), automatically, or at a specific time as set by IBM ClusterSystem Manager.

Automatic failovers have three phases:

1. Failure detection2. Resource relocation3. Application restart (usually the longest part of the failover process)

An automatic failover is triggered when the group failover threshold is reachedwithin the group failover period. These are configuration settings, defined by theadministrator.

1. Resource failover settings

Failover threshold is the number of times in the specified period that MSCSallows the resource to be restarted on the same node. If the threshold count isexceeded, the resource and all other resources in that group will fail over tothe other node in the cluster.

For SAP, the servers must be configured as stand-alone.

Note

Both groups and resources have failover threshold and period propertiesassociated with them. The functions these properties control, however, dependon whether they are associated with a group or a resource.

Group and resource failover properties


Failover period is the time (in seconds) during which the specified number ofattempts to restart the resource must occur before the group fails over.

After exceeding the threshold count of restart attempts, MSCS fails over thegroup that contains the failing resource and every resource in that group willbe brought online according to the startup sequence defined by thedependencies.

2. Group failover settings

Failover threshold is the maximum number of times that the group is allowed tofail over within the specified period. If the group exceeds this number offailovers in the period, MSCS will leave it offline or partially online, dependingon the state of the resources in the group.

Failover period is the length of time (in hours) during which the group will beallowed to fail over only the number of times specified in Threshold.

For example, consider an application clock.exe in group CLOCKGROUP. Otherresources in the group include a File Share resource and a Physical Diskresource, as shown in Figure 10.

Figure 10. Failover example

The administrator who set up this cluster has assigned a failover threshold of 3with a failover period of 60 seconds to the clock.exe resource and a failoverthreshold of 5 with a failover period of 1 hour to the CLOCKGROUP group.

Consider now the situation when clock.exe continually fails. The program (aGeneric Application resource type) will be restarted on Node A three times. Onthe fourth failure within one minute, it and its group, CLOCKGROUP, will fail overto Node B. This counts as one CLOCKGROUP failover. When CLOCK fails fourtimes (that is, one more than the resource threshold) on Node B, it will fail over toNode A. This counts as the second CLOCKGROUP failover.

After the fifth CLOCKGROUP failover within one hour (Node A->B->A->B->A->B),MSCS will not attempt a restart of CLOCK, nor will it fail over CLOCKGROUP.Instead, it will leave CLOCK in the failed state and CLOCKGROUP will be placedin the partially online state. The other resources in the group will be placed in the

Failover

Node A

clock.exeThreshold = 3Period = 60 seconds

CLOCKGROUPThreshold = 5Period = 1 hour

Node B

File Share

PhysicalDisk

clock.exe

CLOCKGROUP


failed state if they are dependent on CLOCK; they will remain online if they arenot dependent on CLOCK.

2.4.12 FailbackFailback is a special case of failover and is the process of moving back some orall groups to their preferred owner after a failover has occurred.

A group’s preferred owner is the node in the cluster that you have declared as theone upon which you prefer the group of resources to run. If the preferred ownerfails, all of its clustered resources will be transferred to the surviving node. Whenthe failed node comes back online, groups that have the restored node as theirpreferred owner will automatically transfer back to it. Groups that have nopreferred owner defined will remain where they are.

You can use the preferred owner settings to set up a simple load-balancingconfiguration. When both servers are running with failback enabled, theapplications and resources will move to their preferred owner, thereby balancingout the workload on the cluster according to your specifications.

When you create a group, its default failback policy is set to disabled. In otherwords, when a failover occurs, the resources will be transferred to the other nodeand will remain there, regardless of whether the preferred node is online. If youwant failback to occur automatically, you have the choice of setting the group tofailback as soon as its preferred node becomes available, or you can set limits sothe failback occurs during a specific period, such as outside of business hours.

2.4.13 LooksAlive and IsAliveTo determine if resources are available, the Resource Monitor polls (requestsstatus information from) the resource DLLs for which it is responsible. Two levelsof polling are supported by the Resource Monitor and you can adjust howfrequently each of them occurs for each resource. The two levels are calledLooksAlive and IsAlive polling. They are defined as follows:

1. LooksAlive polling:

The Resource Monitor makes a superficial check to determine if the resourceis available.

If a resource fails to respond to a LooksAlive poll, then the Resource Monitorwill notify the Cluster Service. When you create a new resource, you definethe interval (in milliseconds) between LooksAlive polling requests. The defaultinterval is 5,000 milliseconds.

2. IsAlive polling:

The Resource Monitor performs a complete check of the resource to verifythat it is fully operational. If a failure is returned, the Cluster Service isimmediately notified and, depending on the configuration defined for theresource, the Resource Manager will either terminate the resource or try tobring it back online on the same node or on the other node (as part of a groupfailover). The default interval is 60,000 milliseconds.

Consider again the Windows NT clock example in Figure 10 on page 28.Assuming that the clock.exe program’s Generic Application resource is createdwith the default parameters, the Resource Monitor calls the LooksAlive function inthe resource DLL every five seconds to make a cursory check that the clock is


functioning. Every 60 seconds, the IsAlive function is called to perform a morerigorous test to check that the clock is operating correctly.

2.5 Microsoft Cluster Service and SAP R/3

In order to give a general idea on how SAP R/3 works with Microsoft Clusterservice we shall analyze the general structure of the cluster groups for the SAPR/3 4.5B installation. This section only aims to provide a general description ofthe extensions introduced in SAP R/3 in order to allow it to work in a MicrosoftCluster environment. More details can be found in the SAP document SAP R/3 ina Microsoft Cluster Environment Technical Overview.

2.5.1 SAP R/3 extensionsTwo DLLs have been developed in order to make SAP R/3 a cluster-awareapplication: SAPRC.DLL and SAPRCEX.DLL. The first DLL allows the MicrosoftCluster Server Resource Monitor to check the status of the SAP R/3 system andto start and stop the SAP services in case of failover. The second DLL isnecessary in order to allow the Cluster Administrator to manage the SAPresource.

The SAP R/3 resource DLL implements the following functions (as described inSAP R/3 in a Microsoft Cluster Environment Technical Overview):

• SaprcOnline — Start the R/3 service and work processes.

• SaprcOffline — Stop the R/3 service and work processes.

• SaprcTerminate — Stop the R3 service without stopping R/3 beforehand. Thisfunction is only for testing purposes.

• SaprcIsAlive — IsAlive status check by R/3.

• SaprcLooksAlive — LooksAlive status check by R/3, that is identical to IsAlive.

Other specific extesions are:

• The program INSAPRCT.EXE has been added in order to create the SAP R/3resource type

• The program CRCLGRP.EXE has been added in order to create the clustergroup SAP R/3 <SID>

• The program COCLGRP.EXE has been added in order to create the resourceSAP R/3<SID> and put it in its own group.

2.5.2 Cluster Group

Exactly what constitutes a superficial check or a complete check in thedescriptions above is determined by the programmer who wrote theresource DLL. For generic applications such as the clock program, the twotests may be identical. More sophisticated resources such as databaseelements will usually implement a different test for each entry point.

Superficial versus complete checks


Figure 11 shows an example of the dependencies that relate the resources in theCluster Group at the end of an SAP R/3 installation with Oracle DBMS:

Figure 11. Oracle cluster group

A slightly different group is created during the DB2/UDB installation as shown inFigure 12:

Figure 12. DB2 cluster group

Figure 13 shows the group created during the SQL Server installation:

Figure 13. SQL Server cluster group

The MSDTC dependencies must be removed and then the resource must bedeleted as described in 7.9, “Removal of unused resources” on page 167.

Time Serviceresource

Oracle Fail Saferesource

Quorum Diskresource

Cluster NetworkName resource

Cluster IP Addressresource

5170

-01


Quorum Diskresource



5170

-01


Quorum Diskresource



5170

-01

MSDTC resource


2.5.3 Group SAP-R/3 <SID>The SAP MSCS resource group is database-independent. Thus the group for theR/3 instance has the same structure with an SQL Server database as withOracle. Figure 14 shows the dependencies that relate the resources in theSAP-R/3 ITS group.

Figure 14. SAP-R/3 ITS group dependencies

2.5.4 Database resource groupIn order to make the Oracle database cluster aware, Oracle has implementedsome extensions. The main extension is the creation of the FSODBS.DLL.

Figure 15 shows the dependencies that relate the resources for the Oracle ITSgroup.

Figure 15. Oracle cluster dependencies

In order to make the DB2/UDB DBMS cluster aware, IBM has implemented someextensions. One of the main extensions is the creation of the DB2WOLF.DLL DLL.

5170

-01

SAP-R/3 <SID>

SAPMNT SAPLOC SAP-R/3 Netname

<shared-disk>:\usr\sap

SAP-R/3 IPAddress

ITS.WORLD

Disk L:

Disk N:Disk K:

ORACLEITS IP Address

IP Addressresource

(CLUSRES.DLL)

ORACLEITSNetwork Name


(CLUSRES.DLL)Disk M:

Disk P:

Disk O:

Physical Diskresources(CLUSRES.DLL)

OracleTNSListener

80Fslitsora

Oracle Agent80ITSora

Generic Serviceresource

(CLUSRES.DLL)

Oracle Databaseresource(FSODBS.DLL)

5170

-01


(CLUSRES.DLL)


Figure 16 on page 33 shows the dependencies in the DB2 group:

Figure 16. DB2 cluster dependencies

Many extensions have been introduced in SQL Server 7.0 Enterprise Edition inorder to make the DBMS cluster aware. The main extension are the creation ofthe SQAGTRES.DLL and SQLSRVRES.DLL.

Figure 17 shows the dependencies in the SQL Server group.

Figure 17. SQL Server cluster dependencies

2.6 Backup and recovery

Backup and recovery in a cluster are slightly different compared to stand-alonesystems. The same strategies may be used, but some details and restrictionshave to be considered.

DB2ITS

Disk L: Disk M:Disk K:

Physical Disk resources(CLUSRES.DLL)

DB2IP Address

ITS

IP Address resource(CLUSRES.DLL)

DB2Network Name

ITS

Network Nameresource(CLUSRES.DLL)

DB2 resource(DB2WOLF.DLL)

5170

-01

Disk K:

Disk L:

Disk M:

Disk N:

Disk O:

ITSSQLIP Address

ITS SQLNetwork Name

ITSSQLVserver

ITSSQLSQL Server 7.0

ITSSQLServer Agent 7.0


IP Addressresource(CLUSRES.DLL)

Generic Serviceresource(CLUSRES.DLL)


SQL Server 7.0resource(SQLSRVRES.DLL)

SQL Server Agent 7.0resource(SQAGTRES.DLL)

5170

-01


The MSCS model of partitioned disk access causes some problems for backupand recovery which we will discuss in this section:

• A backup program running on a cluster node may not be able to access allshared disks.

• Tape devices cannot be shared cluster resources.

• Files on the shared disks need to be identified in a unique fashion.

• Backup scheduling has to take into account the cluster resource situation.

• Bringing databases and applications offline requires special care.

• Additional MSCS files may not be handled by backup software.

Here we give only a short overview of what to consider for cluster backup. Theredbook Using Tivoli Storage Management in a Clustered NT Environment,SG24-5742 discusses these cluster problems in detail. A future SAP R/3 redbookwill cover implementing backup solutions for SAP R/3 on IBM Netfinity servers indetail.

2.6.1 Complete backupThe complete backup of a cluster includes the operating system files and otherlocal drives on each node as well as all shared disks. In an MSCS cluster duringnormal operation, only a subset of the shared disks is accessible from each nodeat any time. Thus a backup from one node will be incomplete. There are two waysto get a complete backup:

• Move all resources to one node before beginning the backup.

• Start backup jobs on both nodes.

Moving all resources automatically before and after each backup operationrequires that the backup program can launch pre- and post-backup procedurescontaining CLUSTER commands (alternatively, Netfinity Manager may be used toschedule cluster actions). This may be an option if the availability requirementsare moderate and the service interruption is acceptable. Keep in mind, however,that moving R/3 or DBMS resources aborts transactions and batch jobs.

Starting backup jobs on both nodes is the more cluster-oriented way and seemsto be obvious because the local disks also need to be backed up regularly. Whenusing local tape drives, each server must write to its own drives. This imposes onthe system manager the additional task of keeping track of which set of tapesfrom all drives together form a full cluster backup at a specific time. Otherwise, acentralized network backup gives the problem of unique file identification onshared disks.

2.6.2 Tape drivesAs for any other Windows NT system, you can use local tape drives or a centralbackup server. Tape drives cannot be defined as MSCS cluster resources forcoordinated access from both nodes. Thus if you decide on local tape drives, youshould atttach the same number and type to each server. With one tape drive inthe cluster you would be forced to move cluster resources before and after eachbackup operation, as described above. Additionally, you should take precautionsin case your default backup server fails. The other server should also be prepared


to perform the backup. This is another reason to attach tape drives to both clusternodes.

2.6.3 Unique file identificationNetwork backup software, consisting of backup clients on each node and acentral backup server, catalogs saved files with the client’s network name and thepath in the client’s disk and file system. In this manner, the file H:\DIR_X\FILE_Yat node A is different from H:\DIR_X\FILE_Y at node B. This still remains valid ifH: is on a shared disk between node A and node B. In a cluster it is possible thatthe backup of H:\DIR_X\FILE_Y is made today from node A and tomorrow fromnode B. These two copies of the same file would be cataloged as backups ofdifferent objects because of the different backup client names. This is not justinconvenient if you are looking for all copies of a specific file or for a completebackup listing of a specific day. The major problem is that incremental backupschemes like the one used by ADSM fail. The two copies of H:\DIR_X\FILE_Y arenot considered to be subsequent backup generations of the same file. Theincremental schema gets out of synchronization.

Figure 18. Same file from shared disk but cataloged differently

There are two approaches to guarantee consistency:

• Alias definitions at the backup server

This means defining at the server that files from node A’s disk H: are to beconsidered as identical with files from node B’s disk H: ("A\\H: is an alias forB\\H:"). Aliases have to be defined at the level of individual disks because a

Backup Server Catalog

1998/08/12 Node_A H: DIR_X\FILE_Y1998/08/10 Node_A H: DIR_X\FILE_Y

1998/08/14 Node_B H: DIR_X\FILE_Y1998/08/13 Node_B H: DIR_X\FILE_Y1998/08/11 Node_B H: DIR_X\FILE_Y…

Available backups of object 1

…

Available backups of object 2

H:\DIR_X\FILE_Y

Node_A Node_B

Backup made on:1998/08/101998/08/12

Backup made on:1998/08/111998/08/131998/08/14


general alias valid for all disks (“node A is an alias for node B”) would alsomerge local disk backups. This approach requires prerequisites in the backupserver software.

• Using virtual names for backup clients

With this method, we assume that each shared cluster disk belongs to aresource group with at least one virtual network name in that group. In an R/3cluster, each disk belongs either to the database group (with virtual databasename) or to the R/3 group (with virtual R/3 name). The quorum disk may bemoved manually into any group (the cluster group with the cluster virtual nameis the natural choice). Now we configure three backup clients on each clusternode:

– Client using the normal node name to identify itself– Client using the virtual database name– Client using the virtual R/3 name

The backup procedures are implemented in such a way that each of theseclients performs the backup only for these disks which belong to thecorresponding resource group. Because the backup client which saves the fileH:\DIR_X\FILE_Y is known to the backup server by a virtual name, all copiesof H:\DIR_X\FILE_Y are kept under this name. Thus all files on shared disksare cataloged on the backup server in a unique manner, independent of whichcluster node sent the data.

The Tivoli Storage Manager uses a special option for the backup client toidentify cluster disks and then implements the second approach using thecluster name in the backup file catalog (see redbook Using Tivoli StorageManagement in a Clustered NT Environment, SG24-5742).

2.6.4 Backup schedulingVirtual backup clients also help to solve the job scheduling problem. Any backupjob has to be started in such a way that its result doesn’t depend on the clusterresource situation. The job must not fail because the disks needed are not onlineon the expected node. If different backup policies are used (for example, foroperating system and database) then the correct policy should be appliedindependent of which node serves the backup. We schedule the backup jobs tothe virtual clients, which are configured as services in the corresponding clusterresource group. Then the job is always started on that node which has (or mayget) access to the disks needed. Backup policies are not bound to a specificcluster node. Instead policies are bound to a logical set of data as formed by aresource group.

A backup of shared disks may be made without active backup client software bysimply defining file shares with virtual names. As explained in Micosoft ClusterServer Administrator's Guide, Microsoft document X0327902, hiddenadministrative shares are useful for that purpose. For example, you might use theNew Resource wizard in the NT Cluster Administrator to create a<virtual_name>\\HBACKUP$ file share for the root directory of partition H:. Thisshare would not appear in the Windows NT browse list and could be configured toallow access only to members of the Backup Operators group. Then the backupserver connects to the share and performs a network drive backup, independentof the current resource situation in the cluster.


In an R/3 cluster you may use this technique to back up files of the R/3 disk(directory \USR\SAP). But you cannot make an online database backup in thesame way because the database files are open, and database operations wouldinterfere with backup access, leading to an inconsistent backup. Offline databasebackups in a cluster require additional considerations.

When using the Tivoli Storage Manager for cluster backup, backup clients areadded as generic services to the R/3 resource group as well as the databasegroup. To schedule backups from the R/3 CCMS (transaction DB13), you have toset up a remote function call environment. This is described in the SAP OSS note0114287.

2.6.5 Offline backupsThe database services are monitored cluster resources. Stopping the database ina pre-backup procedure with the usual commands (for example, with Oracle usingSVRMGR23 or SVRMGR30) is not possible in a cluster. The resource monitordetects loss of the database processes and tries to restart them immediately onthe same or on the other cluster node. The offline backup fails. Instead of thedatabase management utility, the appropriate CLUSTER command has to beissued. The CLUSTER command should affect only the database service, not theentire database resource group, because the disks must stay online for thebackup.

Again see the SAP OSS note 0114287 on how to start offline backups from R/3CCMS.

2.6.6 MSCS-special filesCurrently the backup process for the cluster configuration itself needs to behandled with a workaround. MSCS adds files to the system that containinformation about cluster configuration and resource status:

• Several files in directory \MSCS on the quorum disk• Two files CLUSDB and CLUSDB.LOG in the directory %WINDIR%\CLUSTER

These files are permanently open. Currently we don’t have a solution to applynormal backup procedures to them. The cluster information on the quorum diskmay be restored if the configuration information on the nodes itself (in theCLUSDB file) is correct. Procedures to recover from a failed quorum resource orcorrupted quorum log are described in Microsoft Cluster Server Administrator'sGuide, Microsoft document X0327902, and in Microsoft Knowledge Base, articlesQ172951 and Q172944. Thus you should maintain at least an emergency copy ofCLUSDB and CLUSDB.LOG. To get such a copy, there are two workarounds:

• Using the second copy of Windows NT (emergency system), you can do acomplete offline backup of the production NT, including the special files.Because the emergency system should not have an MSCS installation (toavoid any trouble in the case of emergency restore), this requires you toseparate the cluster node from the shared bus (or to shut down the othernode).

• The two files contain the registry part below the top level keyHKEY_LOCAL_MACHINE\Cluster. You can export this tree using one of theregistry utility programs.


Chapter 3. Hardware configuration and MSCS planning

In this chapter, we show how the specific cluster requirements for SAP R/3 can befulfilled using IBM Netfinity components. The examples explained in the previouschapter at the operating system level will be extended to complete configurationswith the following two shared storage technologies:

• SCSI using the ServeRAID II Ultra-SCSI Adapter or ServeRAID-3L/3H/3HBUltra2 SCSI Adapter

• Fibre Channel using the Fibre Channel PCI Adapter

Only sample configurations are described in this book. Complete installationinstructions are given in the redbook Implementing Netfinity Disk Subsystems:ServeRAID SCSI, Fibre Channel and SSA, SG24-2098. The Fibre Channelinstallation procedure is described in 3.6, “Fibre Channel configurations” on page68.

For configuring an R/3 cluster, you should be aware of general guidelines toincrease server availability, especially:

• An SAP system may only be installed on hardware with certification. iXOS R/3NT Competency Center (R/3 NTC) certifies hardware platforms for SAP onMicrosoft Windows NT. For a list of certified platforms and the rules applied forthe certification process, see:

http://www.r3onnt.com

• A Windows NT server may only be installed on hardware with certification.Microsoft evaluates hardware compatibility using the Windows NT HardwareCompatibility Test (HCT). Each hardware that you use must be included in theWindows NT Hardware Compatibility List (HCL). To access the HCL, see:

http://www.microsoft.com/hcl/

• Hardware that passes testing is included on the Windows NT HardwareCompatibility List (HCL).

For IBM employees on the internal IBM network, the letters from the MicrosoftWHQL to IBM identifying particular Netfinity cluster configurations that havepassed the Microsoft Cluster HCT can be found at:

http://w3.kirkland.ibm.com/Certification/prevcert.asp

• Install an uninterruptible power supply (UPS) system

• Add redundant power and cooling options to the Netfinity servers.

• Configure the Advanced System Management adapter or processor in yourserver.

• Install a second copy of the Windows NT Operating System so that you canquickly boot to the second version if the production copy fails for some reason.

R/3 configurations for MSCS require careful planning because of complexityand cluster-specific restrictions. These restrictions are different for each type ofshared storage subsystem and should be checked before the components areordered.

Note

http://www.r3onnt.com




• Get the latest information and drivers from IBM Web sites. Go tohttp://www.pc.ibm.com/support.

• For ServeRAID installation, review IBM Netfinity High Availability ClusterSolutions for IBM ServeRAID-3H, ServeRAID-3HB and ServeRAID-3LInstallation and User's Guide, available from http://www.pc.ibm.com/support:

Select Server from Select a brandSelect Clustering from Select your familySelect Online publicationsSelect Installation Guides

• Review the IBM Cluster Checklist, available fromhttp://www.pc.ibm.com/us/searchfiles.html:

Enter keywords “cluster checklist”

• Check for the latest drivers and BIOS level for your hardware.

3.1 Checklist for SAP MSCS installation

In this section, we discuss what you need and everything you should be aware ofbefore the installation of SAP R/3. All the specific information will be explained indetail later in this chapter.

3.1.1 Minimum cluster hardware requirementsTo provide high availability to your SAP R/3 system you must determine yourhardware with precision. You need two identical Netfinity servers certified for useon Windows NT. For details regarding the process certification and how to chooseyour hardware see 3.2, “Certification and validation of hardware” on page 42.Keep in mind that your platform must be certified to be sure that you will havesupport from IBM, Microsoft and SAP in case you experience trouble.

• RAM. Each server must have at least 256 MB, although for clusteringconfigurations, this minimum is higher. You will also need significant diskspace for paging files (4-5 times the amount of RAM installed, as described in4.6.2, “Page file tuning” on page 98).

• ServeRAID or Fibre Channel for the external shared disks. For Fibre Channelconfigurations, you need at least one Fibre Channel adapter on both machinesconnected via hubs to a Fibre Channel RAID controller. The external disks arethen connected to the RAID controller.

• ServeRAID for internal non-shared disks. If you also use ServeRAID for theexternal shared disks, then you can use one of the internal channels forconnection rather than using a separate ServeRAID adapter. With ServeRAIDFirmware v3.5, the SCSI heartbeat cable is no longer required for clustering.

• Network. For the network configuration, you need four network adapters. Twoare dedicated for the private network (called the “heartbeat”). This is usually acrossover Ethernet connection. The other network connections are for thepublic network, used for data communication with clients. It is better toconfigure a new network segment dedicated to server communications (calleda backbone).

Table 4 lists a summary of the minimum hardware requirements:

http://www.pc.ibm.com/support


http://www.pc.ibm.com/us/searchfiles.html

Chapter 3. Hardware configuration and MSCS planning 41

Table 4. Minimum hardware requirements

3.1.2 Minimum cluster software requirementsFrom the software point of view, the requirements for the system are common toboth the SCSI and the Fibre Channel configuration. Microsoft Windows NT 4.0Enterprise Edition Service Pack 4 plus the component needed for Y2Kcompliance (Microsoft Internet Explorer 4.01 Service Pack 1 and Microsoft DataAccess components 2.0 Service Pack 1) are required, but it is better to install thelatest Service Pack from Microsoft. For SAP R/3, which Service Pack you installdepends on which database you want to use. Refer to OSS note 30478 fordetails.

Refer to SAP OSS Note 0030478 to get latest information about Service Packs andSAP R/3. Currently, the situation with MSCS is as follows: for R/3 systems usingMSCS, SP4 may only be used if the Microsoft hot fixes are imported according toOSS Note 0144310 (for MS SQL Server 7.0 only hotfix "RNR20.DLL" is required). Ifpossible, you should immediately implement Service Pack 5. For the application ofService Packs in cluster systems, you must read Note 0144310. The use of ServicePacks is also described in 4.8, “Service pack and post-SP installation steps” on page102.

The minimum space requirements on the local disks and the shared disks aresummarized in Table 5:

Configuration Minimum requirements

Devices • Two identical Netfinity servers, certified for clustering

Memory • At least 256 MB of RAM• 4xRAM for virtual memory or 1200 MB whichever is larger

SCSI • Two IBM ServeRAID-3L/H adapters with the latest BIOS and firmwarelevels for internal disk connectivity

• Two IBM ServeRAID-3H adapters with the latest BIOS and firmwarelevels for external disk connectivity

• External disk enclosure

Fibre Channel • Two IBM ServeRAID-3L/H adapters with the latest BIOS and firmwarelevels for internal disk connectivity

• Two Fibre Channel host adapters• One Fibre Channel RAID controller• One HUB Fibre Channel hub with at least four GBICs• External disk enclosure

Network • Two network cards for private network communications• Two network cards for public network communications• Four physical IP addresses for the network cards• Three virtual IP addresses for the cluster

If you plan to install Windows NT Service Pack 5 (SP5) without installingService Pack 4 (SP4) first, you should remember to install the additionalcomponents in SP4 for Y2K compliance. These include Microsoft InternetExplorer 4.01 Service Pack 1 and Microsoft Data Access Components 2.0Service Pack 1. Alternatively, install SP4 first, then SP5.

Y2K compliance


Table 5. Minimum space requirements

3.2 Certification and validation of hardware

All hardware vendors providing Windows NT server hardware must obtaincertification and validation of hardware configurations by iXOS Software AG(iXOS) and Microsoft. The IBM Netfinity ServerProven and ClusterProvenprograms will not be addressed in this redbook as SAP does not currentlyparticipate in these IBM membership programs.

As an independent assessor, iXOS performs certification of all Windows NTserver platforms offered by hardware vendors competing for shares of the SAPR/3 market. SAP will not support an R/3 installation on a hardware platform notcertified by iXOS.

Microsoft maintains a validation program for testing the compatibility of vendorserver hardware and associated drivers with Windows NT. As part of its hardwarevalidation program, Microsoft maintains a program specifically for validatingvendor server hardware in a clustered configuration using MSCS. Microsoft willnot support an MSCS installation on a hardware platform that is not validated byMicrosoft.

Component Minimum space requirement

Internal drivescapacity oneach node

• 500 MB for Microsoft Windows NT 4.0 Enterprise Edition, ServicePack 4, Microsoft Internet Explorer 4.01 Service Pack 1, andMicrosoft Data Access Component 2.0 Service Pack 1

• 3 MB for MSCS• 10 MB for SAP cluster files• 1200 MB for Windows NT page file• Oracle: 600 MB for Oracle server software RDBMS 8.0.5, 10 MB for

Oracle FailSafe software V 2.1.3• DB2: 100 MB for DB2/CS Software V5.2• SQL: 65 MB for MS SQL Server 7.0 Enterprise Edition program files

Oracle: sharedvolumesrequirements(externalenclosure)

• 100 KB for Cluster quorum resource• 1 GB for SAP R/3 executable files• 10 GB initially for SAP R/3 data files• 120 MB for Online redo logs set A• 120 MB for Online redo logs set B• 120 MB for Mirrored online redo logs set A• 120 MB for Mirrored online redo logs set B• 6 GB for the backup of online redo logs• 2 GB for SAPDBA directories

DB2: sharedvolumesrequirements(externalenclosure)

• 100 KB for Cluster quorum resource• 1000 MB initially for SAP R/3 data files• 100 MB for DB2 database files• 10 GB initially for SAP R/3 data files• 1 GB at least for DB2 log files• 1 GB at least for DB2 archived log files

Microsoft SQL:sharedvolumesrequirements(externalenclosure)

• 100 KB for Cluster quorum resource• 6 GB initially for R/3 data files• 45 MB for SQL server master DB• 300 MB for SQL server temp DB• 1 GB minimum for the transaction log files (Log Device)


3.2.1 iXOSiXOS Software AG (iXOS), in close cooperation with SAP, has developed a serverhardware certification process for the purpose of investigating the performanceand stability of SAP R/3 on Windows NT platforms. This certification is essentialfor all hardware manufacturers. Since 1993, the R/3 NT Competency Center (R3NTC) of iXOS has exclusively performed the certification process, as anindependent assessor, for a large number of Windows NT platform hardwarevendors.

Note that SAP supports the operation of certified R/3 systems only.

There are five different hardware certification categories in place as developed byiXOS in conjunction with SAP. The hardware certification testing is executed byiXOS:

1. Initial certification: The first, and only the first, Windows NT server offeringfrom a hardware vendor must undergo the initial certification. A very detailedtest sequence is performed.

2. Ongoing certification: All server offerings for R/3 on Windows NT currentlyoffered to the market are subject to the process of the ongoing certification.Twice a year, the ongoing certification tests are repeated on each serveroffering. The ongoing certification is executed to validate the operation of thesystem in conjunction with a new release of Windows NT or an upgrade ofhardware or firmware by the hardware vendor.

3. Ongoing controller certification: This level of certification allows for a set ofreduced tests to be performed in order to certify I/O controllers offered byhardware vendors. Once an I/O controller is certified, by iXOS, in oneWindows NT server product offered by a hardware vendor, it is certified for usein all the hardware vendor’s Windows NT server products that have beencertified by iXOS.

4. S/390 certification: This certification requires reduced tests to be performedagainst an already certified NT platform that is to have access to a DB2database on an S/390. For this certification category, it is the connectivity typethat is certified. Each connectivity type (for example, FDDI, Fast Ethernet,ESCON) must be certified once per hardware vendor.

5. Outgoing certification: Hardware platforms no longer being sold for R/3 onWindows NT, but still used by customers in a productive environment aresubject to an outgoing certification.

To enter the SAP R/3 on Windows NT market, a hardware vendor must secure aninitial certification for its server platform. The first server offering and allsubsequent server offerings a hardware vendor may supply to the market aresubjected to the ongoing certification until they are no longer offered for SAP R/3on Windows NT.

3.2.1.1 Hardware componentsAccording to the iXOS certification process a hardware platform consists of threedifferent types of components:

1. Critical components

– The chip set: A platform is defined by its chip set and corresponding chipset extensions that enable the data transfer between processor, memory,


and I/O. Changing the chip set requires that a Windows NT server systemmust undergo an ongoing certification.

– The I/O controller: A particular I/O controller must be certified once foreach hardware vendor. If an I/O controller has been successfully testedwith one Windows NT server offering supplied by a hardware vendor, it iscertified with all other iXOS certified Windows NT server offerings suppliedby the vendor (if supported by the vendor).

The same applies to the certification of S/390 connectivity types.

2. Peripheral components

The hardware vendor is obligated to provide a list of all peripheral componentsassociated with the server system to be used in support of SAP R/3 onWindows NT. The hardware vendor guarantees function and support of thecomponents listed. If any of the peripheral components are replaced the list isto be updated by the hardware vendor and no new certification is necessary.Peripheral components are:

– Hard disks– Memory– Network adapter– Backup device

3. Non-critical components

All components that are not defined as critical components or peripheralcomponents are defined as non-critical components. Changing non-criticalcomponents does not affect the certification of the platform. Non-criticalcomponents are, for example:

– Monitor– Graphic adapter– Mouse– Others

3.2.2 MicrosoftMicrosoft evaluates hardware compatibility using the Windows NT HardwareCompatibility Tests (HCTs). The HCTs are run for the purpose of testing theinteraction between device drivers and hardware. These tests issue the full rangeof commands available to applications and operating systems software, and aredesigned to stress hardware beyond the level of most real-world situations. At theWindows Hardware Quality Labs (WHQL), Microsoft personnel run the HCTs andreport results to the hardware manufacturer. Hardware that passes testing isincluded on the Windows NT Hardware Compatibility List (HCL). The HCL may beviewed by visiting http://www.microsoft.com/hcl/. A validated clusterconfiguration can potentially include any server that is on the Microsoft HCL forWindows NT server. For validating hardware in a cluster configuration, theMicrosoft Cluster HCT is executed.

SAP R/3 certification rules do not require the certification of I/O subsystems.However, iXOS offers, to vendors of I/O subsystems, special tests that validatethe stability and measure the I/O performance of the storage solution.

I/O subsystems



The most important criteria for MSCS hardware is that it be included in avalidated cluster configuration on the Microsoft HCL, indicating it has passed theMicrosoft Cluster HCT. Microsoft will only support MSCS when used on avalidated cluster configuration. Only complete configurations are validated, notindividual components. The complete cluster configuration consists of twoservers and a storage solution.

Microsoft allows hardware manufacturers to run the Microsoft Cluster HCT attheir own facilities. The result of a successful test is an encrypted file that isreturned to Microsoft for validation. Validated cluster configurations may beviewed by selecting the Cluster category at:


IBM currently tests MSCS solutions with the Netfinity 10/100 Ethernet Adapter(34L0901) and a crossover cable as cluster interconnect. This is the configurationsubmitted to Microsoft for certification, and it is recommended that you follow thispractice. The MSCS certification rules allow replacement of the interconnectcards with another 100% compliant NDIS PCI card listed on the Windows NTHCL (see http://www.microsoft.com/hwtest/sysdocs/wolfpackdoc.htm). Consultwith the Netfinity ServerProven list to be sure that the alternate card is listed asone that is tested and supported on Netfinity servers.

Additionally, for IBM employees who can access the internal IBM network,information regarding IBM Netfinity cluster configurations, validated by theWHQL, can be viewed at the IBM Center for Microsoft Technologies (CMT) Website at http://w3.kirkland.ibm.com/Certification/prevcert.asp. The IBM CMTWeb site posts letters from the WHQL to IBM identifying particular Netfinitycluster configurations that have passed the Microsoft Cluster HCT.

3.2.3 Certification of SAP R/3 and Windows 2000SAP is planning to release R/3 for Windows 2000 at the launch time of Windows2000. Currently, Microsoft is running R/3 tests with every major Windows 2000build.

Because of this situation the following rules for hardware certification for R/3 onWindows 2000 will apply:

• Customers and partners can install R/3 on Windows 2000 if the hardware:

– Has been certified by iXOS for Windows NT (is listed on iXOS Web page)

The Microsoft HCL also has the categories Cluster Fibre Channel Adapter,Cluster/SCSI Adapter, Cluster/RAID, Cluster RAID Controller, and ClusterRAID System.

It is important to note that inclusion of these components on the HCL does notqualify a component for MSCS support services unless the component wasincluded in a validated cluster configuration. Make sure you consult the Clustercategory on the HCL to view the validated configuration with included storageadapter. These other cluster categories are intended for vendors, systemintegrators, and test labs that are validating complete cluster configurations.

Cluster component candidates



http://www.microsoft.com/hwtest/sysdocs/wolfpackdoc.htm


– Is also listed on the Microsoft Hardware Compatibility List (HCL) forWindows 2000 (on the Microsoft Web page)

– Is officially supported for SAP installations on Windows 2000 by thehardware partner.

• The hardware partner must test and support all peripherals (including networkadapter, tape device, graphic adapter and so on) that will be used by SAPcustomers for Windows 2000.

• New hardware must be certified by iXOS as before. Hardware partners mustnotify SAP and iXOS as to which hardware will be supported Windows 2000.iXOS will add a Windows 2000 list to the Web page with all qualifyinghardware platforms.

3.3 Netfinity sizing

This section will help you understand the general approach to sizing Netfinityservers for use with SAP R/3 with Microsoft Cluster Server. It explains themechanism currently used to provide an accurate hardware configuration.

3.3.1 Terms and definitionsFirst of all, you should be aware that the configuration resulting from an SAPsizing process is a matter of definitions. The customer and the hardware vendorhave to agree on the definition of the words used and on the assumptionsconcerning the future configuration. This is the first step of the sizing.

• SAP Application Performance Standard (SAPS)

SAPS is a definition of throughput developed by SAP capacity planning andperformance testing personnel. It characterizes machine capability at 100%load condition. It does not reflect customer allowable response timeconditions, nor should it be used for sizing because response times as afunction of load are non-linear.

– 100 SAPS are defined as 2,000 fully business-processed order line itemsper hour in the standard SAP SD (sales/distribution) applicationbenchmark. This is equivalent to 6000 dialog steps and 2000 postings tothe database per hour.

– With SAPS, you measure the maximum (at 100% CPU utilization) numberof dialog steps on a particular model of machine. This means you canexpress the maximum throughput of each model in a total number of SAPS,independent of response time.

– SAPS is valid for a theoretical SD load only.

• Users

– Named users: The users that have an account in the R/3 system.

– Connected or logged-on users: The users that are connected or logged onto the system at one point in time. The percentage of named userssimultaneously connected to the system varies from customer to customer.

– Active users The users that are connected/logged on and putting load on tothe system. The default assumption recommended by the IBM/SAPInternational Competency Center (ISICC) is that 65% of the logged-onusers are active. Active users can further be divided into three workloadcategories (power/high, standard/medium, occasional/low).


3.3.2 What is to be sized?Here are the components that have to be sized before choosing any hardwareconfiguration:

• CPU processor capacity: Mainly depends on the version of SAP, the databasesystem chosen, and the number of active users (including multisession users).

• Main memory: Mainly depends on the software version, the number ofinstances installed on the system, and the number of logged-on users(including multisession users).

• Internal disks: Mainly depends on the operating system and the main memory(for the pagefile).

• External disks: Mainly depends on the database software, the number ofnamed users, and the level of security/performance expected.

• Backup capacity: Based on the amount of data to be stored/restored and thetime frame given for storage operations.

• Network load: Mainly depends on the number of users and the type ofnetwork.

• Client for the SAP GUI: Only depends on the SAP software version.

Figure 19 shows the major influencing factors that should be considered anddiscussed with the team in charge of hardware sizing:

Figure 19. Influencing factors

3.3.3 Sizing methodologyA sizing estimate result includes recommendations for CPU(s), memory and disksrequired to run the basic SAP R/3 application suite. Most likely, there will besoftware, server, and network requirements that are not addressed by this sizingestimate. For example, many SAP R/3 installations run software for systemmanagement, EDI translation, file transfer, help desk management, etc. Todetermine the total hardware and software configuration required for any R/3implementation, we recommend that customers work with an SAP R/3 consultantwho will help to develop the overall system architecture.

Sizing the hardware requirements for each customer’s SAP R/3 implementation isan iterative process, which may be refined and repeated a number of times. In theearly stages of planning, customers have limited information about their plannedSAP R/3 environment. In this case, the sizing estimate can be completed with

SAP / IBM

. R/3 version

. Modules

. DB version

. OS version

. Hardware

PerformanceSizing

ThroughputResponse Time

. # of users

. Reporting

. Batch

. Load profile

. Customizing

CUSTOMER


general information about the numbers of users the IBM SAP R/3 system needsto support. Further along in the R/3 implementation planning, customers willknow more about SAP R/3, the R/3 applications they plan to use, and theirpotential R/3 transaction activity. At that time, another sizing estimate should berequested based on more detailed information.

It is important to understand that the sizing estimate is a presales effort, which isbased on benchmark performance data; it should not replace capacity planningfor installed systems. The sizing estimate can be used for preinstallationplanning. However, during the process of implementing R/3, customers shouldwork with an IBM/SAP Capacity Planning consultant to monitor and predict theongoing resource requirements for the production R/3 system.

The IBM/SAP sizing methodology is continually reviewed and revised to providethe best possible estimate of the IBM hardware resources required to run SAPR/3. Guidelines for sizing R/3 come from a number of sources, including SAP,SAP R/3 benchmarks, and customer feedback. Based on information from thesesources and the sizing questionnaire completed by the customers, the IBM ERPsales team will analyze the SAP R/3 requirements and recommend an IBMhardware configuration with a target CPU utilization of 65%.

See the IBM ERP sales team for the sizing questionnaire.

There are basically two different methods for interactive sizing: user-based sizingand transaction-based sizing.

• In general, user-based sizing determines the total interactive load of an SAPcomplex by summing up the total number of mathematical operation financialtype users. The method then makes a processor recommendation based on a65% loading condition of these mathematical operation users. This allowssignificant capacity for peak interactive conditions, batch loads, reportgeneration, and other forms of complex loading.

• The transaction-based sizing methods developed and used by IBM sums thetotal amount of normalized financial (FI) transactions an SAP complex will see,and then assuming an average amount of dialogs per transaction computes atotal normalized dialog load. The new transaction-based method developed bySAP and the hardware partner council is based on measurements that havebeen achieved on a reference machine for every single transaction evaluatedat this point in time (1-3 transactions per module).

Both user-based sizing and transaction-based sizing have relative advantages anddisadvantages. In fact, we recommend that you obtain both user data and transactiondata from a customer when doing a sizing (per the sizing questionnaire). This willallow you to have the data to cros-check the user-based sizing with ransaction sizing.

The objective of the IBM/SAP sizing methodology is to estimate the hardwareresources required to support the peak hour of business processing. Our sizingphilosophy is that if we size the hardware to provide acceptable response time forthe peak application workload, then all workloads outside of the peak hour shouldalso provide acceptable response time.

The peak hour is the busiest hour of activity from an information-processingstandpoint. It is the hour in which the CPU utilization is the highest. In identifyingthe peak hour, you should consider how your processing volumes vary throughoutthe year and select a peak hour during the busiest time of the year. A survey in


the user departments of the various SAP R/3 modules may be helpful. Typically,the peak hour occurs somewhere between 8:00 a.m. and 6:00 p.m., but this canvary. In Figure 20, the thick line shows the transaction volumes for all of the SAPR/3 modules used in one organization, with the peak hour occurring from 10:00a.m. to 11:00 a.m.

Figure 20. Identifying the peak hour of processing and the potential SAP R/3 workload

The SAP R/3 functions that will be in use during that hour (refer to Figure 20)have to be determined also. For user-based sizing, it will be the R/3 modules thatwill be active during the peak hour and the numbers of users of each module. Fortransaction-based sizing, we will break down the modules by transaction type andspecify the number of transactions to be processed during the peak hour. Forexample, in user-based sizing, customers could indicate that twenty FinancialAccounting (FI) users will be active; for transaction-based sizing, customerswould specify some number of A/P Payments, A/R Invoices, GL Postings, etc. It isimportant to understand that every SAP R/3 module/transaction to be used by theorganization should not be included. Instead, only those R/3modules/transactions that will be active during the peak hour should be reported.

The batch processing workload should also be identified. For user-based sizing,the batch volume is a percentage of the total system workload. By default, werecommend that 20% of the total workload during the peak hour is batchprocessing. For transaction-based sizing, it is the batch transaction volumes forthe peak hour.

For further information, you can contact the IBM national ERP Solutions team orthe IBM/SAP International Competency Center (ISICC) in Walldorf, Germany. ForIBM employees, refer to the ISICC intranet Web site:

http://w3.isicc.de.ibm.com

Identifying the Peak Hour of Processing and the Potential SAP R/3 Workload

0

20

40

60

80

100

120

140

160

180

200

9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00

Time

Tra

nsa

ctio

ns Gen'l Ledger

A/P Payments

A/R Invoices

Asset xfers

Inventory

Prod. Orders

IM

Payroll

QM

Total

peak hour



Two electronic sizing tools are available to help IBM sales team, partners, andcustomers.

• The SAP QuickSizer is accessible from the Internet on the SAPNET Web site.(http://sapnet.sap.com, user ID, and password are required).

• The second tool is the ISICC R/3 Sizing Tool, available on the ISICC LotusNotes Server accessible only by IBM employees at the time of writing. Toaccess this server, please review the instructions on the ISICC intranet Website.

3.4 Disk layouts

One of the basic differences between a non-clustered SAP installation and aclustered SAP installation is strict separation of R/3 files and DBMS files ondifferent shared disks. MSCS uses the shared nothing architecture, which meansthat each system accesses its own disk resources (that is, they logically sharenothing at any point in time). Because DBMS and R/3 services run on differentnodes during normal operation, their disk space must not intersect. Thus moredisks are needed in non-cluster R/3 systems where DBMS and R/3 directoriesmay reside on common disks.

Throughout this redbook, the term Windows NT disk n means the disk n weobserve in the Windows NT Disk Administrator window. This may be:

• A single physical disk (not recommended)• A RAID disk array• A subdivision of a RAID disk array (so-called logical drive or logical unit)

For more details on specific advantages and limitations of SCSI or Fibre Channeldisk subsystems please refer to 3.5, “ServeRAID SCSI configurations” on page62 and 3.6, “Fibre Channel configurations” on page 68.

Only the local disks contain the operating system and page file. Data redundancyfor local disks may be achieved with hardware RAID. MSCS currently does notsupport software RAID for shared disks so a certified cluster solution withhardware RAID must be used. Shared disks are needed for:

• MSCS quorum• R/3 files• DBMS data and log files (here the details differ for the specific DBMS)

3.4.1 Operating systemMSCS requires that the operating system files (including page files) are placedon non-shared disks. We recommend you configure two separate Windows NTdisks for the following reasons:

• R/3 assumes large page files (four times the amount of RAM). Separatingpage file I/O from system file access may improve performance.

• For fast system repair and recovery, a second copy of the operating system(so-called emergency system) should be installed.

Throughout this book, you will find references to the usual conventions; <SID>denotes the three-character SAP system identification code.

Convention


http://sapnet.sap.com


As a minimum, you should use four hard disk drives in two separate RAID-1arrays:

• Because page files have a write/read ratio of 50% and more, placing them onRAID-5 arrays would decrease performance.

• To ensure fast recovery even in the case of losing more than one hard diskdrive, the second copy of Windows NT should be installed on a RAID arraydifferent from that with the production system.

Using only two large hard disk drives or combining all drives in a single RAIDarray would not reach our objectives. For these disks, hardware RAID must beused for SCSI clustering configuration and Fibre Channel configuration.

With the ServeRAID II and ServeRAID 3H/3HB, it is possible to also attach theoperating system disks to shared channels (using ServeRAID merge groups).This seems to be efficient usage of controllers and storage enclosures. However,we do not recommend this because it increases the complexity of the clusterinstallation. Some recovery and maintenance tasks are more difficult becausedisconnecting a server from the shared bus prevents booting this machine. Withrespect to the growth of typical R/3 databases, the savings would only beshort-term.

SAP recommends you create a Windows NT page file of four times the RAMamount for non-cluster configuration, and five times the amount for clusterconfiguration. However, one Windows NT page file has an upper size limit of 4GB. For performance reasons, it is recommended to install all the page files onseparate physical disks.

Therefore, when installing a large SAP configuration by using the Netfinity7000 M10 or the Netfinity 8500R, adding one or two expansion enclosures forexternal disks may be necessary. The EXP15 can be split into two SCSIchannels, each one used by a node of the cluster and connected to aServeRAID adapter that is not configured for clustering.

Windows NT page file


According to these considerations, we should have the following disk layout onboth cluster nodes (Table 6):

Table 6. Windows NT disk layout recommendations

Note: As per Knowledge Base article Q114841, the boot partition can be up to7.8 GB in size.

All the Windows NT disks described in Table 6 do not always have to be created.The configuration will depend on the Netfinity server model and RAM amountinstalled. Each model has a different number of hot-swap bays for hard diskdrives and different maximum RAM size.

Windows NT disk 0

• Partition C:

We use this primary NTFS partition for the production Windows NT 4.0Enterprise Edition operating system (called boot partition).

Disk Name andLetter

Purpose Disk Type

WindowsNT disk 0

C:\(4.5 GB)

• Windows NT 4.0 EE productionoperating system

• Oracle: complete software in \ORANT• Microsoft SQL Server: client software• DB2 software in \SQLLIB• Client for backup software• System management software

Two internal diskdrives 9.1 GB in aRAID-1 array withtwo logical drives

E:\(4 GB)

Page file #2 (up to 4 GB)

WindowsNT disk 1

D:\(4.5 GB)

• Second copy of operating system• Client for backup software

Two internal orexternal disk drives9.1 GB in a RAID-1array with two logicaldrives

F:\(4.5 GB)


WindowsNT disk 2

G:\(4 GB)


9.1 GB internal orexternal disk drivesin RAID-1 array

WindowsNT disk 3

H:\(4 GB)


WindowsNT disk 4

Additional disk for Page file #5 (up to 4 GB) if you followthe 5 times RAM recommendation.

No Name Hot Spare 9.1 GB disk drive

To avoid any confusion with the drive letter assignment for each partition,we recommend you use only primary partitions. The Windows NT 4.0installation procedure does not allow a second primary partition onWindows NT disk 0. We first have to create a primary partition (D:) onWindows NT disk 1, install Windows NT 4.0 there, boot from that one, andwith Windows NT Disk Administrator, create the second primary partition onWindows NT disk 0. We then install Windows NT 4.0 on drive C:.

Partitions


• Partition E:

This partition contains only a second page file (the first one is in priority on theF: partition). It will be used if the page file total size has to be greater than the4 GB installed on F:.

Windows NT disk 1

• Partition D:

We use this drive as an emergency (or backup) Windows NT (NTFS, primarypartition). Configuring an emergency Windows NT is a good installationpractice and recommended by SAP. To avoid as many problem sources aspossible, this system should be configured as a stand-alone server, and itshould not contain any software that is not absolutely necessary for basicrestore operations.

• Partition F:

This drive contains a page file up to 4 GB. If the page file size has to begreater, partition E: has to be used for a total size up to 8 GB. Beyond thisvalue, more disks in RAID-1 arrays have to be added.

The advantages of using two Windows NT disks are higher availability and fasterrecovery. If disk 0 fails we can boot from (the already present and configured)disk 1 (with a Windows NT formatted diskette and some necessary, basicWindows NT files). We are able to fix driver, registry, and other problems on disk0, or we immediately start a full restore from a previous, offline backup to disk 0.For more details on Windows NT availability and recovery, see the redbookWindows NT Backup and Recovery with ADSM, SG24-2231.

Intel has developed two extended server memory architectures. Thesetechnologies are known as Physical Address Extension (PAE) and Page SizeExtension (PSE). The PSE driver allows Windows NT 4.0 (Enterprise Editiononly) to use memory beyond 4 GB. At the time of writing, no information isavailable regarding the certification of this driver and the ability for the databasesoftware to run on such system.

Windows 2000 will use the PAE feature on Intel 32-bit (IA-32) servers to supportmore than 4 GB of physical memory. PAE allows up to 64 GB of physical memoryto be used as regular 4 KB pages, providing better performance than is availablethrough the Intel PSE36 driver. Therefore, you will not need the PSE36 driver asdescribed on this Web site for use with Windows 2000.

For more information about PSE and PAE, see Chapter 6 of the redbook NetfinityPerformance Tuning with Windows NT 4.0, SG24-5287.

Nevertheless, the physical memory beyond 4 GB is not pageable. Therefore, thetotal page file will never have to be greater than 20 GB, divided in several 4 GB

Windows NT 4.0 Enterprise Edition does not allow you to create a page filegreater than 4 GB, and is not able to operate more than 4 GB of RAM.

Therefore, the maximum total page file size that can be needed can neverexceed 4 GB x 5 = 20 GB (according to the SAP recommendation for runningMSCS).

Page file size


max files (or 16 GB if you intend to use only four times the RAM). Therefore, hereis a description of a sample operating system disk layout for each Netfinity server,with the maximum page file size:

Netfinity 5000

• Maximum RAM: 2 GB• Maximum pagefile: 10 GB• Maximum OS disk configuration (see Table 6):

– Windows NT disk 0: two internal disks in RAID-1– Windows NT disk 1 and 2: three internal disks in RAID-1E– No internal bay available for hot-spare disk.

Netfinity 5500

• Maximum RAM: 1 GB• Maximum page file: 5 GB• Maximum OS disk configuration (see Table 6):

– Windows NT disk 0: two internal disks in RAID-1 (with no page file)– Windows NT disk 1: three internal disks in RAID-1 (page file on drive F:)– One internal bay available for hot-spare disk

Netfinity 5500 M10


– Windows NT disk 0: two internal disks in RAID-1– Windows NT disk 1 and 2: three internal disks in RAID-1E– One internal bay available for hot-spare disk

Netfinity 5500 M20


– Windows NT disk 0: two internal disks in RAID-1– Windows NT disk 1: two internal disks in RAID-1– Windows NT disk 2: two internal disks in RAID-1– Windows NT disk 3: two external disks in RAID-1 (EXP15 in a non-cluster

configuration)– Windows NT disk 4: two external disks in RAID-1 (EXP15 in a non-cluster

configuration)– One external bay available for hot-spare disk (if the EXP15 is divided into

two separate SCSI buses, each one used by one node of the cluster)

Netfinity 7000 M10


– Windows NT disk 0: two internal disks in RAID-1– Windows NT disk 1: two internal disks in RAID-1– Windows NT disk 2, 3 and 4: four external disks in RAID-1E (EXP15 in a

non-cluster configuration)– One external bay available for hot-spare disk (if the EXP15 is divided into



Netfinity 8500 R


– Windows NT disk 0: two internal disks in RAID-1– Windows NT disk 1, 2, 3 and 4: four external disks in RAID-1 Enhanced

(EXP15 in a non-cluster configuration)– One external bay available for hot-spare disk (if the EXP15 is divided into


3.4.2 Shared external drivesThe shared disks are configured as follows:

3.4.2.1 QuorumMSCS requires a shared quorum disk to arbitrate for access in the case ofconnection loss between the cluster nodes. Quorum and log information is storedin the directory tree \MSCS. Per the SAP R/3 installation guide, SAPrecommends that no R/3 or data files should be stored on the quorum disk forsafety reasons (so as to not put the disk offline by accident and to not have spaceproblems if the data files grow too large). The ownership of the quorum diskdetermines which node is the master of the cluster and controls the clusterconfiguration.

When using a Fibre Channel RAID controller, as the quorum files are very small(100 KB), it is possible to create two logical unit (LUN) on the same RAID-1 array.The MSCS quorum file can be stored on the first LUN, and the SAP software(\USR\SAP) can installed on the other LUN without any problem. Thisconfiguration is not allowed in a ServeRAID configuration as you cannot havemultiple logical drives configured in a shared RAID array.

3.4.2.2 SAP R/3 filesThe R/3 files (system-global software and profile directories, instance-specificdirectories, and central transport directory) must reside on a shared Windows NTdisk different from the quorum and the data files disks.

The main directory on this disk is \USR\SAP. The cluster resources SAPMNT andSAPLOC of type file share, part of the SAP cluster group, point to this directory.

Whatever the database or the disk subsystem technology used, only a limitednumber of Windows NT partitions can be created, one for each of the 26 lettersof the alphabet. This can be a critical weakness for a very large database, ashalf of the letters are already in use by system files, SAP software, or databaselogs.

Therefore, the Windows NT partitioning and number of arrays for the data filesshould be created carefully, taking into consideration the limitation of bothSCSI and Fibre Channel. See 3.5, “ServeRAID SCSI configurations” on page62 and 3.6, “Fibre Channel configurations” on page 68 for detailedexplanations about these limitations.

Drive letters in Windows NT


In an MSCS environment, you cannot store any database related files on thisdisk, because this disk is a requirement for the SAP application server.

3.4.2.3 Oracle filesThe DBMS data and log files must reside on shared Windows NT disks differentfrom the quorum and the R/3 disk. There are recommendations for Oracle andSQL Server and DB2 to distribute the files over several RAID arrays for securityand performance reasons — these are discussed in the following sections.

The Oracle software home directory (default is \ORANT) will be installed locallyon each cluster node. A natural choice for placing this directory is the productionWindows NT operating system partition (in our disk layout, C: on Windows NTdisk 0). The redo log files, and the Oracle FailSafe repository are on shared disks.

Redo logs are fundamental Oracle transaction logs. Oracle organizes logs in redolog groups containing identical copies of the logs. One of the copies is saidoriginal log while the other is said mirrored log. The Oracle LGWR (Log Writer)process writes simultaneously on both files, hence these files are really specular.The purpose is to have a backup of log files to be used if there is any problemwith the original logs. Since both these log files are continuously written by theLGWR process it is fundamental to put them in different disks.

Once Oracle fills one of the log files, a log switch is performed, that is, the LGWRprocess starts to write on the log files in another log group. Simultaneously theARCH (Archiver) background process starts to copy the logs from the filled redolog group to the archive directory (SAP archive). The archiving is necessarybecause when the LGWR has exhausted all the redo log groups the next logswitch brings it back to the original redo log group and then it begins to overwritethe original log. Thus, to avoid to losing the data in the logs it is necessary toarchive them. Since the ARCH process reads the data from the log directorieswhile the LGWR is writing in them it is important to have more than one redo loggroup. In this way while ARCH is reading from one log file on one disk, the LGWRis writing in a second log file in a different disk. For the same reason, it isimportant to have a dedicated SAP archive. The ARCH can write data on thearchive disk without competing with the LGWR for the I/O resources.

During the installation, R3Setup creates four redo log groups having the structureshown in Table 7:

Table 7. Redo log groups

For the reasons explained above, it is important to have at least groups 11 and 13in a separate physical disk than the one containing groups 12 and 14.

Redo Log Group name Log files

11 ORILOGAMIRRORLOGA

12 ORILOGBMIRRORLOGB

13 ORILOGAMIRRORLOGA

14 ORILOGBMIRRORLOGB


For further details see:

• Oracle Architecture by S. Bobrowski, Chapter 10

• Oracle 8 Tuning by M. J. Corey et al., Chapter 3

The Oracle DBWR (Database Writer) process gains access to data and indexessimultaneously. Hence it is fundamental to create different tablespaces for dataand indexes and put them on different disks. Besides this basic tuning rule, asecond fundamental rule says that you must try to put most often used tables indifferent disks. SAP R/3 4.5B tries to satisfy these requirements creating 27different tablespaces scattered in the six SAPDATA directories.

If you have exactly six different disks for the SAPDATA directories you can beconfident on the correct distribution of your data and indexes. If you have stillmore space you can try to further improve this distribution. If you have less spaceyou can try to improve the R3Setup distribution of tablespaces. If large customerdata is expected, SAP recommends to store at least the four followingtablespaces in separate physical disks:

Table 8. Tablespaces

If, for performance reasons, you need a nondefault data file distribution you cancustomize the configuration modifying manually the CENTRALDB.R3S file.

For further details see:

• Oracle 8 Tuning by M. J. Corey et al., Chapter 3.

To ensure that no data is lost in a single disk or RAID array failure, the followingthree groups of Oracle files should be on different shared RAID arrays:

• Data files (tablespace files)• At least one set (original or mirrored) of online redo log files• Archived redo log files

We recommend Oracle software mirroring for the database online redo log files,even if hardware mirroring is already provided. There are a lot of possible causesfor corrupt online redo log files (disk failures, driver errors, internal Oracle errors),and not all of them are covered by hardware RAID. To avoid controversy(responsibility) and because of support implications, we advise using Oraclemirroring of online redo log files. The additional level of security outweighs thesmall loss of disk space and performance.

Another Oracle-specific directory that has to reside on a shared Windows NT diskis the Oracle FailSafe Repository (directory \ORANT\FailSafe). It is used to storeinformation (50-100 MB) about the Oracle databases in the cluster that areconfigured for failover (in an SAP MSCS cluster, this is only the R/3 database).Because of cluster resource dependencies, the Oracle FailSafe Repository mustreside on a disk that will belong to the Oracle cluster resource group. The

Tablespace Default data file

PSAPSTABD SAPDATA5

PSAPSTABI SAPDATA6

PSAPBTABD SAPDATA3

PSAPBTABI SAPDATA6


OracleAgent80<virtual_DB_name> MSCS resource depends on the OracleFailSafe Repository as well as on the Oracle <SID> Network Name resource.Thus the Oracle FailSafe Repository disk belongs to the Oracle resource group.

You may place this directory on any Windows NT disk used for Oracle data or logfiles, or configure a separate shared disk for clarity.

For more details on different configurations see the SAP manuals R/3 Installationon Windows NT-Oracle Database, Release 4.5B (SAP product number 51005499,May 1999) and “Conversion to Microsoft Cluster Server: Oracle, Release 4.0B4.5A 4.5B (SAP product number 51005504) (May 1999)”.

Recommended disk configuration

The following disk configuration is not the only possible layout, but is an optimalone based on Oracle, SAP and Microsoft recommendations for excellent level ofsecurity and performance.

Table 9. Oracle / MSCS recommended disk layout

At least two EXP15 expansion enclosures are necessary for configuring SAP andOracle in a cluster environment. 10 disk drives are used for installing the MSCSquorum, SAP files and all the online redo logs. Then, data files can be stored onas many drives as necessary, depending on the database size. Five arrays are

Disk Name andLetter

Purpose Disk Type

WindowsNT disk 4

I:\Quorum

\MSCS 2 disk drives in a RAID-1array with 1logical drive

WindowsNT disk 5

J:\SAPExe

\USR\SAP 2 disk drives in a RAID-1array with 1logical drive

WindowsNT disk 6

K:\DBLog

\ORACLE\<SID>\orilogA\ORACLE\<SID>\orilogB\ORACLE\<SID>\sapbackup\ORACLE\<SID>\sapcheck

2 disk drives in a RAID-1array with 1logical drive

WindowsNT disk 7

L:\ArchLog

\ORACLE\<SID>\mirrlogA\ORACLE\<SID>\mirrlogB\ORACLE\<SID>\sapreorg\ORACLE\<SID>\saptrace


WindowsNT disk 8

M:\SAPArch

\ORACLE\<SID>\saparch\ORANT\FailSafe


WindowsNT disk9,....etc

N:\ ,...etcData

\ORACLE\<SID>\sapdata1 Number of drives dependson the database size:RAID-1, RAID-1E, RAID-5 orRAID-5E

\ORACLE\<SID>\sapdata2





No Name Hot Spare same disk drives as thelarger one


necessary for the software installation on the shared disks. For large database,two Fibre Channel RAID controller units may be useful.

For better performance, we recommend you create only RAID-1 arrays whenusing ServeRAID. With ServeRAID, you will see a significant performance differencebetween RAID-1 and RAID-5 because ServeRAID caching must be set to writethrough for the shared drives to ensure consistency in the case of server failure. TheFibre Channel controllers have mirrored caches external to the servers, thus cacheconsistency is ensured independent of server failures, so RAID-5 performance withwrite-back caching is comparable with RAID-1 (especially when the RAID-5 array islarge).

3.4.2.4 IBM DB2 filesDB2 UDB Enterprise Edition has to be installed locally on both nodes. Thestandard directory is C:\SQLLIB. The conversion into a clustered database isdone after the regular R/3 installation. The clustered DB2 instance stored in thedirectory \DB2PROFS and the databases <SID> and <SID>adm need to beinstalled on a shared drive.

The database data files, the active database logs, and the archived database logshave to be stored on different NT disks to avoid that data is lost or at least theamount of data lost is minimized in the case of a RAID array failure.

The disk holding the active database logs is the disk with the largest number ofsynchronous disk writes. Try to connect this disk to a channel or adapter with lowload. We recommend to separate this disk at least from the one storing thearchived database logs, because every time the DB2 user exit process is started,a log file is copied from the active log files disk to the archived log files disk, whilethe database is writing new log file data to the active log disk.

In a cluster configuration, it is not possible to store the archived database logs onthe same drive as the R/3 binaries. The DB2 user exit process is always startedon the machine running the database instance. If the R/3 server is running on theother node, the DB2 user exit process has no access to the R/3 Files drive.

Figure 21 shows this configuration. The DB2 RDBMS runs on Node B andcontrols the database <SID>, the active database log files, and the invocation ofthe DB2 user exit (DB2UE). When a database log file becomes inactive, theRDBMS calls the DB2UE (1). The DB2UE in turn copies the database log filefrom the active logs disk to the archived logs disk (2), and then deletes theoriginal database log file (3).

Database data files and both sets of online redo log files should always bedistributed over different physical disks. As the online redo logs are writtensynchronously, they produce the most I/O activity of all database files.

In large configurations, this activity can become critical. The recommendationis then to add two new RAID-1 arrays to physically separate the active onlinelogs A and B, as well as the mirrored logs A and B.

Log files


Figure 21. Database log archives must be separate from R/3 binaries

For database data, you can use one RAID-5 array or several RAID-1 arrays. If youplan to use the IBM ServeRAID controller, try to use RAID-1 arrays. The reasonbehind this is, that you have to disable the write back cache for all shared diskdrives. Without this cache, write access to RAID-5 arrays is very slow becauseevery write access requires that besides the real data, the new parity be writtenback to disk in synchronous mode. If you use a different controller type inconjunction with RAID-5, make sure that it has a write back cache and that thiscache can be activated in a cluster environment.

The maximum number of RAID-1 arrays to be configured for the database datashould not exceed six for a new installation because R3SETUP offers only thedistribution of SAPDATA1-SAPDATA6 to different Windows NT disks. You can addadditional drives later as needed.

Recommended Disk Configuration

For performance reasons, the database data files should be distributed over sixWindows NT disks. One possible, optimal distribution of database data ondifferent Windows NT disks is shown in Table 10:

archivedDB logs

disk

R/3 FilesNT disk 3

Windows NTshared disks


DB2Database

<SID>


Node Bvirtual

DB serverRSDDB

DB2RDBMS

2

DB2 UserExit

activeDB logs

disk


3

Node Avirtual

R/3 serverRSDR3

R/3 WorkProcess

R/3 WorkProcess

R/3 WorkProcess

1


Table 10. DB2 UDB/MSCS recommended disk layout

Four arrays are necessary for the software installation on the shared disks. Forlarge databases, two Fibre Channel RAID controller units may be useful.

3.4.2.5 SQL Server 7.0 filesThe DBMS data and log files must reside on shared Windows NT disks differentfrom the quorum and the R/3 disk.

The SQL Server database software consists of a server and a client part. Theclient software must be installed locally on each cluster node (default directory:\MSSQL). A natural choice for placing this directory is the production WindowsNT operating system partition (in our disk layout, C: on Windows NT disk 0). Theserver part of the software (default is another \MSSQL directory), the SQL Servermaster database, and the container files with application data and logs (calledSQL Server devices) are on shared disks.

To ensure that no data is lost in a single disk or RAID array failure, the followingthree types of SQL Server devices should be on different shared RAID diskarrays:

• Data devices• Log devices• TEMP device

Disk name and letter Purpose Disk type

WindowsNTdisk 4

I:\Quorum

\MSCS Two disk drives in aRAID-1 array withone logical drive

WindowsNTdisk 5

J:\SAPExe

\USR\SAP Two disk drives in aRAID-1 array withone logical drive

WindowsNTdisk 6

K:\DBLog

\DB2\<SID>\log_dir Two disk drives in aRAID-1 array withone logical drive

WindowsNTdisk 7

L:\ArchLog

\DB2\<SID>\log_archive\DB2<SID>\saprest \DB2\<SID>\sapreorg\db2rsd\DB2PROFS\db2rsd\DB2\<SID>\saparch

Two disk drives in aRAID-1 array withone logical drive

WindowsNTdisk 8, etc.

M:\ ,etc.Data

\DB2\<SID>\sapdata1 Number of drivesdepends on thedatabase size:RAID-1, RAID-1E,RAID-5 or RAID-5E

\DB2\<SID>\sapdata2

\DB2\<SID>\sapdata3

\DB2\<SID>\sapdata4

\DB2\<SID>\sapdata5

\DB2\<SID>\sapdata6

No Name Hot Spare Same disk drivesas the larger one


Table 11. MS SQL 7/MSCS recommended disk layout

Four arrays are necessary for the software installation on the shared disks. Forlarge databases, two Fibre Channel RAID controller units may be useful, providedthat each controller unit not handle more than eight logical units.

3.5 ServeRAID SCSI configurations

This section introduces the configuration of the ServeRAID disk subsystem. Weonly provide the configuration plan. The basic installation instructions aredescribed in IBM Netfinity High-Availability Cluster Solutions Using the IBMServeRAID -3H and IBM ServeRAID-3HB Ultra2 SCSI Controllers Installation andUser’s Guide available from:

http://www.pc.ibm.com/netfinity/clustering

3.5.1 Netfinity EXP15The Netfinity EXP15 Expansion Enclosure is a rack-drawer storage unitcontaining 10 hot-swap drive bays that support both half-high and slim-line 7200RPM or 10,000 RPM SCSI disk drives.

Disk name and letter Purpose Disk type

WindowsNT disk 4

I:\Quorum

\MSCS Two disk drives in aRAID-1 array withone logical drive

WindowsNT disk 5

J:\SAPExe

\usr\sap Two disk drives in aRAID-1 array withone logical drive

WindowsNT disk 6

K:\SQLSrv

\MSSQL\TEMPDB\TEMPDB.DAT

Two disk drives in aRAID-1 array withone logical drive

WindowsNT disk 7

L:\SQLLog

\<SID>LOG<n>\<SID>LOG<n>.DAT Two disk drives in aRAID-1 array withone logical drive

WindowsNT disk8,etc.

M:\ ,etc.Data

\<SID>DATA<n>\<SID>DATA<n>.DAT

Number of drivesdepends on thedatabase size:RAID-1, RAID-1E,RAID-5 or RAID-5E






No Name Hot Spare Same disk drivesas the larger one

Do not forget that the ultimate limitation is the 26 letters used by Windows NTfor all the disks. Some letters are already in use for the floppy disk drives, theCD-ROM drives, and so on.

Windows NT limitation



The EXP15 supports Ultra SCSI data transfers of up to 40 MBps at distances ofup to 12 meters from the server using LVDS cabling.

Figure 22. EXP15

The EXP15 contains an electronics board (ESM or Environmental ServicesMonitor board) that interfaces between the external SCSI cables and hot-swapbackplanes. The ESM board provides two main functions:

• Status reporting for the subsystem through the SCSI interface• SCSI connection between the subsystem and the server

The EXP15 has two hot-swap redundant 350W power supplies. Each powersupply contains its own power cord. In addition, two hot-swap cooling unitscontaining separate dual fans provide cooling redundancy. If a failure occurs witheither of the redundant power supplies or cooling fans, an LED will light toindicate a fault and its location.

The EXP15 has two SCSI connections, both using the VHDCI .8 mm 16-bit SCSIconnector. (The EXP10 uses standard 68-pin SCSI connectors.)

There are three configurations possible with the EXP15: two independentfive-drive buses, one 10-drive bus, and a 2-node clustering configuration:

• Configuration for two independent SCSI buses

To configure both SCSI buses independent of each other, connect one cablefrom a ServeRAID channel to the bus 1 connector (as shown in Figure 23) andanother ServeRAID channel to the bus 2 connector. To separate the twobuses, set switch 1 in the switch block to on (up). In this configuration, eachbus contains five disks.

Figure 23. EXP15 Rear View

• Configuration for one bus

To configure the Netfinity EXP15 as a single 10-disk SCSI bus, attach oneexternal SCSI cable from the ServeRAID adapter to the bus 1 connector, then

Bus 1 Connector Bus 2 Connector

Option Switch Block


set switch 1 to off (down). There is no need to connect a terminator to the bus2 connector as the EXP15 will automatically terminate the bus.

• Clustering configuration

To use the EXP15 in a cluster with a maximum of 10 drives on a single SCSIbus, connect the external SCSI cable from the ServeRAID adapter in serverone to the SCSI bus 1 connector and connect another SCSI cable from aServeRAID adapter of server 2 to the SCSI bus 2 connector, then set switch 1to off (down). This configuration shares the data storage of the EXP15between two clustered servers.

3.5.2 Netfinity EXP200Similar to the EXP15, the Netfinity EXP200 Expansion Enclosure is a rack-drawerstorage unit containing 10 hot-swap drive bays that support both half-high andslim-line 7200 RPM or 10,000 RPM SCSI disk drives.

Figure 24. Netfinity EXP200

The EXP200 supports Wide Ultra2 (80 MBps) transfer speeds at up to 20 metersusing LVDS SCSI cabling. The EXP200 shares the same drive options as thenew Netfinity 8500R and Netfinity 5600 servers.

3.5.3 Ultra2 and LVDSThe EXP15 and EXP200 support low voltage differential signaling (LVDS) SCSIconnection from the ServeRAID-3H and ServeRAID-3L adapters. This means theadapter and enclosure can be connected by cables of up to 12 meters in length.

The EXP200 supports Ultra2 transfer speeds (80 MBps). The EXP15 onlysupports transfer rates of up to 40 MBps (Ultra SCSI). The electronics in theenclosure convert the incoming LVDS signal into a single-ended SCSI signal andpass it to the backplane.

Note: By definition, Ultra2 requires LVDS connectivity, but the opposite is nottrue: LVDS does not require Ultra2. Hence it is possible for the EXP15 to operatewith 12 meter LVDS cables and Ultra SCSI transfer rates.

For more information, see Chapter 3 of the redbook Implementing Netfinity DiskSubsystems: ServeRAID SCSI, Fibre Channel and SSA, SG24-2098.


3.5.4 SAP R/3 disk configurationsWith respect to the rules for R/3 cluster configuration as explained in 3.4, “Disklayouts” on page 50, usage of different disk types makes sense.

• Since SAP recommends that no R/3 or data files be stored on the MSCSquorum disk (as described in 3.4.2.1, “Quorum” on page 55), small drives areappropriate. The quorum disk has no influence on cluster performance duringnormal operation. MSCS data takes up few megabytes. Thus a RAID-1 arrayof two 4.5 GB disks is sufficient.

• For the volume containing the \USR\SAP directory tree, a 4.5 GB or 9.1 GBRAID-1 array fulfills the needs. Using fast disks (10,000 RPM) may improveperformance on R/3 systems with many context switches (rol lfile and page fileusage).

• The database online log files are not large but critical for security andresponse time. Thus these volumes should be placed on RAID-1 arrays of fastdisks. The size depends on the additional database directories placed there.

• Database archived log files are only written once in background. Disk speedwill suffer R/3 performance only if a very large amount of log data must bearchived quickly. The disk space necessary depends on the number oftransactions between scheduled backups. Heavy loaded systems without dailyfull backup require large archives. A RAID-1 array of the necessary sizeshould be configured.

• Data files may be stored on RAID-1 or RAID-5 arrays. Because of the largedatabase size in R/3 systems, 9.1 GB and 18.2 GB disk drives are preferred.

Other cluster-specific restrictions for the ServeRAID adapters that must beconsidered during planning are:

• The unit of failover is the physical array. Only one ServeRAID logical drive canbe created per shared physical array.

• The maximum number of logical drives per adapter is eight before or after afailover. A failover will not complete if this number is exceeded. Note that this limitincludes shared and nonshared logical drives. Thus the maximum number ofshared logical drives per adapter pair is eight minus the largest number ofnonshared logical drives from the adapters.

• RAID-0 and RAID-1 logical drives may failover in normal or critical state. But theServeRAID adapters do not allow failover of RAID-5 logical drives that are incritical state (that is, one drive in the RAID-5 array has failed). For this reason,you should not use RAID-5 for the quorum disk.

• Hot-spare drives are not shared between the adapters in a pair. EachServeRAID adapter must have its own hot-spare drive(s) defined. Allhot-spare drives must be connected to shared channels so that, if thehot-spare is used to replace a failed shared disk, the disk can be accessedfrom either node in the case a failover occurs later.

• Compared with a non-clustered SAP R/3 installation with ServeRAIDadapters, as a rule of thumb, in a cluster you will need two additional RAID-1arrays with two disks each, and you have to double the number of hot-sparedrives.


3.5.5 Cluster configurationFigure 25 shows a sample of an SCSI-based Microsoft Cluster Serverconfiguration:

Figure 25. SCSI-based MSCS configuration

For the ServeRAID II adapter, you need to form the SCSI heartbeat connectionbetween the Channel 3 connectors on the adapter in each server that has theMSCS quorum disk defined. This will require the Third Channel Cable Kit (part76H5400) for each of the two adapters. Because both adapters have 0.8 mm VHDCIfemale connectors as external interface, there are two possibilities to form theconnection itself:

• A Netfinity Ultra2 SCSI cable with 0.8 mm VHDCI male connectors on bothends using either a 4.3 m cable, part 03K9311 or a 2 m cable, part 03K9310.

• A combination of a 0.8 mm to 68-pin cable (4.3 m, part 01K8029 or 3 m, part01K8028) with a 68-pin to 0.8 mm adapter, part 01K8017).

Server A

500MHz

Pentium III XeonMicroprocessors


Channel 1

Channel 2 Channel 2

Channel 1

Arrays

Arrays

SCSI IDs

Crossover Cable (Heartbeat)

Public Network

A AA A

0 38 11

B BB B HSP

HSP

HSP

HSP

9 122 101 4

EXP15 Enclosure 1

D D D DC C C C

0 38 119 122 101 4

EXP15 Enclosure 2

ServeRAID 3H

ServeRAID 3H

EtherJet

SCSI ID

Option Switch 1Set to OFF


Server B

500MHz

ServeRAID 3H

ServeRAID 3H

EtherJet

EtherJet EtherJet

RAID-1 Arrays RAID-1 Arrays

Figure 25 shows the ServeRAID-3H/HB basic cluster configuration. Whenrunning Microsoft Cluster Server software, you no longer need to interconnectthe third channel of your controllers to which the quorum is connected,(formerly known as a the SCSI heartbeat connection). You can now attachinternal or external drives to the third channel of your ServeRAID 3HBcontroller.

SCSI heartbeat not needed


There are two network adapters configured in Figure 25, but there can be more,depending on your networking requirements. We discuss this in 3.7, “Networkconfigurations” on page 76.

3.5.6 SCSI tuning recommendationsThis section presents some suggested optimization parameters for the SAPcluster to achieve its optimum performance.

The ServeRAID controllers have tuning parameters that have significant impacton database server performance. SAP R/3 requires mirroring of the redo logs anddefaults to software mirroring. An array controller can implement mirroring andtherefore off-load the CPU from performing those tasks. The stripe size of thecontroller must closely match the block size of the application. For example, anOracle database can use 2 KB, 4 KB, or 8 KB blocks for the tables and would givethe best performance using 8 KB stripe size on the controller.

However, the SAP R/3 application logic may write in 64 KB blocks and would beoptimized with the array controller set to 64 KB stripes. Turning the Read Aheadfunction off on the array controller will also benefit users in the SAP R/3 writeintensive workload.

Hence, our recommendations are:

• Configure two separate ServeRAID adapters:

– One for the local disks containing the operating system and page files– One for the shared disks containing the database files

• Set the stripe sizes as follows:

– 64 KB for the local disks (best performance for the page files)– 8 KB for the shared disks (best performance for the database)

• Set Read Ahead as follows:

– On for the local disks– Off for the shared disks

• Set the cache policy as follows:

– Write-through for all shared disks– Write-back for all local disks

You should install the battery backup cache option for the ServeRAID controllerconnected to the local disks.

When you install multiple ServeRAID adapters in the same server, you mustinstall the device that will manage the startup drives also called boot drives in aPCI slot that is scanned before subsequent ServeRAID adapters. Thesespecifications are not the same between the different server. To see you serverspecific information please refer to the IBM Shared Disk Clustering HardwareReference available from:


ServeRAID order



Please refer to 4.16.5, “DB tuning” on page 127 for a discussion on how to tunethe RDBMS being used in the installed SAP R/3 environment:

3.6 Fibre Channel configurations

Fibre Channel is an emerging storage technology that overcomes many of thelimitations of SCSI. Like SSA and Enterprise Systems Connection (ESCON),S/390 channel connectivity, Fibre Channel is based on the idea of storagenetworks to which servers may attach, avoiding the problems of traditional bustopologies. Fibre Channel received great market awareness even when it was notavailable for production usage. It is considered the future solution for large-scalestorage requirements.

Fibre Channel supports data transfer rates of 100 Mbps on copper and opticalinterfaces. A copper interface is used for short connections (less than 30 meters);a multimode short-wave optical interface is used for intermediate distances (lessthan 500 meters); and a single-mode long-wave optical interface is used for longdistances (up to 10 kilometers). Three topology types are defined:

• Point-to-Point. Two Fibre Channel nodes (for example, server and storagesubsystem) are directly connected.

• Arbitrated Loop (FC-AL). Up to 126 Fibre Channel devices may be attached.

• Switched Fabric. Multiple FC-ALs may be served by a switch.

The first of Netfinity’s Fibre Channel offerings were developed with Symbios Logic(now LSI Logic) to exploit Oracle Parallel Server (with up to eight nodes usingNetfinity Cluster Enabler middleware). Now IBM offers a series of Fibre Channelcomponents designed for broader usage in Netfinity MSCS clusters. Thefollowing servers may be connected to a Fibre Channel network:

• Netfinity 5000• Netfinity 5500• Netfinity 5500 M10• Netfinity 5500 M20• Netfinity 7000• Netfinity 7000 M10• Netfinity 8500R

Review 3.2, “Certification and validation of hardware” on page 42 to check thecurrent certified hardware configurations for MSCS and SAP R/3. The NetfinityFibre Channel RAID Controller Unit and Netfinity Fibre Channel Hub aresupported only in rack installations. You may combine these components withtower model servers, but we recommend that you install the whole cluster as arack solution to protect the optical cables from damage.

We provide only a general overview of the Fibre Channel technology here. Forfurther detailed information and an installation guide, refer to the redbookImplementing Netfinity Disk Subsystems: ServeRAID SCSI, Fibre Channel andSSA, SG24-2098.

3.6.1 IBM Netfinity Fibre Channel componentsNetfinity Fibre Channel solutions consist of the following key components:

• Netfinity Fibre Channel PCI Adapter


• Netfinity Fibre Channel RAID Controller Unit with one RAID controller (asecond redundant FailSafe RAID controller is optional)

• Netfinity Fibre Channel Hub, Gigabit Interface Converter (GBIC) options(short-wave and long-wave), short-wave optical cables

• EXP15 storage expansion enclosures with up to 10 Ultra SCSI disk drives

3.6.1.1 Fibre Channel PCI AdapterThe IBM Netfinity Fibre Channel PCI Adapter, 01K7297, is a direct memoryaccess bus master, half-length, host adapter. It uses the ISP2100 chip, whichcombines a RISC processor, a fiber protocol module with gigabit transceivers,and a PCI local bus interface in a single-chip solution. The adapter supports allFibre Channel peripherals that support Private Loop Direct Attach (PLDA) andFabric Loop Attach (FLA). The external connector is an SC style that supportsshort-wave fiber optic cabling with total cable length up to 500 meters.

The adapter is necessary to connect Netfinity servers to the Fibre Channelnetwork. But in contrast to the ServeRAID SCSI, the Fibre Channel PCI Adapterdoes not provide any RAID functionality. All RAID functions are performed by theFibre Channel RAID controller.

3.6.1.2 Fibre Channel RAID Controller UnitThe IBM Netfinity Fibre Channel RAID Controller Unit, 35261RU, is a 19-inch rackmount component (four high units) which may contain one or two RAIDControllers (shipped with one controller). The controllers share a 3.3 volt batterybackup unit for protecting the cache in the event of a power failure.

Figure 26. Netfinity Fibre Channel RAID Controller Unit

The host connection for each controller is provided with Fibre Channel. Toconnect the EXP15 storage enclosure, six independent low voltage differentialsignaling (LVDS) Ultra2 SCSI buses are available (with active termination and 0.8mm VHDCI connectors). The RAID controller unit supports RAID-1, RAID-1+0

Hot-Swap RAID Controller(optional Second RedundantController)

Battery Backup Unitfor Cache

Hot-Swap RedundantFan for Controllers

Hot-SwapRedundant Fanfor PowerSuppliesHot-Swap

Redundant PowerSupplies

2098

-02a


(striped mirror sets), RAID-3, and RAID-5. Data is buffered in a 128 MB buffercache, that is protected by the internal battery.

Figure 27. Rear View

The controller unit has 9-pin RS-232 connectors for each installed RAIDcontroller, used for configuration, monitoring, and diagnostics, even when thehost is not available. Remote configuration and monitoring across a network ispossible with Ethernet cable connections for each controller. Redundant powersupplies, fans, the battery unit, and the controllers are all hot-swappable.

When you install the optional Netfinity Fibre Channel FailSafe RAID Controller(part 01K7296), both controllers can be set up as a redundant pair, either in anactive/passive configuration, or in an active/active configuration, which isrecommended for better performance.

Each RAID array is shown to the servers as one or more logical units(LUN=logical unit number). LUNs are parts of a RAID array. A LUN is, from theoperating system’s point of view, similar to a logical drive on a ServeRAID array,although the implementation at the SCSI protocol layer is different. Windows NTtreats the LUN as one Windows NT disk. But in contrast to ServeRAID, thefailover unit in MSCS is the LUN (not the RAID array). Each LUN can fail overindependently from other LUNs sharing the same RAID array.

In active/active mode, both controllers may own LUNs and serve them to thehosts. All LUNs are completely independent. Multiple LUNs residing on the sameRAID array can be owned by different controllers. This may give performancebenefits if one controller is a bottleneck. In active/passive mode, one controllerowns all LUNs, while the other operates as hot standby.

For the configuration and management of the Fibre Channel RAID Controller, theSYMplicity Storage Manager software is provided.

3.6.1.3 Netfinity Fibre Channel HubThe IBM Netfinity Fibre Channel Hub (35231RU) is a 7-port centralinterconnection for Fibre Channel Arbitrated Loops. LED indicators provide statusinformation to indicate whether a port is active or bypassed. (No remotemanagement options are provided.) Hub cascading is supported to a depth oftwo, thus up two 37 ports may be connected.

4

5

6 2

3 1FiberConnectionsfrom Hub or

PCI HostAdapter

Power Supply

LVD SCSI Connectionsto EXP10s or EXP15s

Power SupplyPower Supply Fan

Host IDSwitches

RS-232CConnections

EthernetConnections

UPS Connection

2098

-02a


Figure 28. Netfinity Fibre Channel Hub (rear view)

Each port requires a Gigabit Interface Converter (GBIC) to connect it to theattached node. The hub supports any combination of short-wave or long-waveoptical GBICs. The GBICs are hot-pluggable into the hub, which means that youcan add servers and storage modules to the Fibre Channel Arbitrated Loopdynamically without powering off the hub or any connected devices. If you removea GBIC from the hub port, then that port is automatically bypassed. Theremaining hub ports continue to operate normally. Conversely, if you plug a GBICinto a hub port, it will automatically be inserted into the loop and become a FibreChannel node if valid Fibre Channel data is received from the attached device.

Four short-wave GBICs are shipped with the hub. They support connections toNetfinity Fibre Channel PCI Adapters and Netfinity Fibre Channel RAID ControllerUnits (using 5 meter or 25 meter short-wave Netfinity Fibre Channel cables orcustomer-supplied short-wave cables up to 500 meters). There are alsolong-wave GBICs available for attaching customer-supplied long-wave opticalcables between hubs. Any combination of short-wave and long-wave GBICs issupported. Netfinity Fibre Channel PCI Adapters and Netfinity Fibre ChannelRAID Controller Units can only be connected to short-wave GBICs.

A Fibre Channel hub is required when more than two devices (also called nodes)should be connected as a Fibre Channel Arbitrated Loop (FC-AL). Without a hub,failure of one device would break the loop because there is no redundancy in anFC-AL. The hub ensures that the loop topology is maintained. Each hub portreceives serial data from an attached Fibre Channel node and retransmits thedata out of the next hub port to the next node attached in the loop. This includesdata regeneration (both signal timing and amplitude) supporting full distanceoptical links. The hub detects any Fibre Channel loop node that is missing or isinoperative and automatically routes the data to the next operational port andattached node in the loop.

5170

-01


Figure 29. Hub in a Fibre Channel Arbitrated Loop

The following table summarizes the Fibre Channel components:

Table 12. Fibre Channel components

3.6.2 Cluster configurations with Fibre Channel componentsThe basic cluster configuration using Fibre Channel consists of two servers and aRAID controller unit (with one RAID controller installed) connected to a hub, asshown in Figure 30. For simplicity, we assume bridged drive buses in the EXP15.

The cluster configuration can be expanded up to six EXP15 enclosures for eachRAID controller unit. Then the next expansion step would be attaching a secondcontroller unit to the hub, giving again six LVDS SCSI channels with the sametotal amount of storage.

Description IBM part number

IBM Netfinity Fibre Channel PCI Adapter 01K7297

IBM Netfinity Fibre Channel Hub (Note 1) 35231RU

IBM Netfinity Fibre Channel Cable 5 m 03K9306

IBM Netfinity Fibre Channel Cable 25 m 03K9305

IBM Netfinity Fibre Channel RAID Controller Unit (Notes 2 and 3) 35261RU

IBM Netfinity Fibre Channel FailSafe RAID Controller (Note 4) 01K7296

Notes:1 This part number does not apply in EMEA; use SFCH1xx instead (xx refers to a

country-specific power cord and has to be replaced by the appropriate country code). Thehub is shipped with four short-wave GBICs (part 03K9308. The part number for long-waveGBIC is 03K9307).

2 Includes one Netfinity Fibre Channel RAID controller.3 This part number does not apply in EMEA; use SFCU1xx instead (xx refers to a

country-specific power cord and has to be replaced by the appropriate country code).4 To be installed in the Fibre Channel RAID Controller Unit.

Port

Fibre ChannelDevice

5170

-01

Fibre ChannelDevice

Fibre ChannelDevice

Fibre ChannelDevice

Port

Por

t

Port


Figure 30. Simple Fibre Channel/MSCS configuration

Note: There are three single points of failure in this configuration:

• The RAID controller• The hub• The Fibre Channel cable between the hub and RAID controller

Because all access to the shared disks is operated by these components whichexist only once in the cluster, a failure of one of these components would causeboth cluster servers to lose access to the shared disks. Thus all cluster diskresources would fail, and the DBMS and the R/3 processes would fail also.

The first step to avoid such situations is adding a redundant Fibre ChannelFailSafe RAID Controller in the controller unit. Attaching this second controller tothe hub would eliminate the first single point of failure only. To avoid having thehub or any FC cable as the critical point, a second hub is needed. With two hubs,each hub forms a separate FC loop. The RAID controllers are connected todifferent loops, while the servers are connected to both loops. This gives theconfiguration as shown in Figure 31 on page 74.

Server A



Public Network

A AA A A HSP HSP

CC C CC DD DD

0 03 3

B BB B

2 21 14 4

EXP15 Enclosure 2EXP15 Enclosure 1

Arrays

SCSI ID


Server B

500MHz

Fibre Channel

EtherJet

500MHz

RAID-1Arrays

Fibre Channel

EtherJet

Fibre Channel Hub

Standard RAID Controller

Empty Slot

1 2 3 4 5 6

8 89 910 1011 1112 12

ServeRAID 3H

EtherJet EtherJet

ServeRAID 3H

RAID-1Arrays



Figure 31. Fully redundant Fibre Channel/MSCS configuration

Without special precautions, the operating system would not recognize the diskaccess paths provided through the two Fibre Channel adapters as redundantpaths to the same common set of LUNs. To ensure that the second FC adapter isconsidered an alternate path to the LUNs accessed by the first adapter (and viceversa), the SYMplicity Storage Manager software adds a redundant disk arraycontroller (RDAC) driver and a resolution daemon.

This software supports a fully redundant I/O path design (adapter, cable, andRAID controller) with host-level failure recovery, transparent to applications. TheRDAC driver is not a kind of RAID software — it does not mirror between the FCloops. It ensures only Fibre Channel link integrity in the case of failure. The failingpath or I/O initiator will be detected, and the backup RDAC will take its place.Because any mirroring is done by the Fibre Channel RAID Controllers betweenSCSI channels, this is not a method for disaster protection using FC. Themaximum distance between disk drives in the same RAID-1 array is limited byLVDS SCSI cable length.

This configuration combines large disk space, high performance and high safety.When both RAID controllers in each controller unit are set as an active/activepair, then any bottlenecks in the controllers or FC paths are avoided. If a SCSI

Server A

4 Pentium III XeonMicroprocessors


Public Network

Option Switch 1Set to ON

Server B

500MHz

Fibre Channel

Fibre Channel

EtherJet

EtherJet

ServeRAID 3H500MHz

RAID-1Arrays

Fibre Channel

Fibre Channel

EtherJet

EtherJet

ServeRAID 3H

Fibre Channel Hub

Redundant Fibre Channel Hub

Standard RAID Controller

Failsafe RAID Controller

Arrays

SCSI ID

1 2 3 4 5 6

A A C C D D E E

0 0 3

BB

2 21 1

EXP15 Enclosure 1

A A C C D D F F

0 0 3 3

BB

2 21 1 4 4

EXP15 Enclosure 2

A A C C D D

0 0 3 3

BB HSPHSP

2 21 1 4 4

EXP15 Enclosure 3



4 Pentium III XeonMicroprocessors

RAID-1Arrays


channel becomes a bottleneck, then the drive buses of the EXP15 may beattached separately (see Figure 31 on page 74).

Expanding this configuration may be done by adding more EXP15 until all sixSCSI channels of the controller unit are used. Then a second controller unit(again with two redundant RAID controllers) can be attached by way of the hubsto the FC PCI adapters, giving again six SCSI channels. For performancereasons, no more than two controllers should be attached to a single FC PCIadapter.

The Fibre Channel LUNs will be configured in three different MSCS resourcegroups which may fail over independently. The R/3 group (disk with the \USR\SAPdirectory tree), the Database group (containing all Windows NT disks with a\ORACLE\<SID> tree or \DB2\<SID> tree or the MSSQL directories, and theOracle FailSafe Repository), and the Cluster Group (containing the Cluster Name,Cluster IP Address, and the Time Service). The Oracle FailSafe Repository may belocated on any Oracle disk or on a separate partition, should not be installed onthe quorum disk, as per OSS note 0112266.

The only FC topology that guarantees that no cluster component constitutes asingle point of failure is using two independent loops with redundant RAIDcontrollers, as shown in Figure 31 on page 74.

The maximum number of disks configurable into a LUN depends on the RAIDlevel defined:

• RAID-0: 20 disks per LUN• RAID-1: 30 disks per LUN (15 usable)

The concept of LUNs allows very effective usage of disk space. We cancombine several smaller Windows NT disks on the same RAID array usingdifferent LUNs. Because all LUNs are completely independent, these WindowsNT disks may even belong to different cluster resource groups. But you shouldnot combine such Windows NT disks as LUNs of the same RAID array, thatneed to be separated for data security. Failure of an array (because of multipledisk failures) causes all LUNs of that array to fail. Thus we never provide LUNswith data files and LUNs with log files from the same array.

Logical units and MSCS

There are two limits on the number of logical units you can have:

• Host adapter: Prior to v6.16 of the PCI Fibre Channel host adapter driver,Windows NT limited the maximum number of logical units (LUNs) per RAIDcontroller unit to eight (whether the controller unit has a single controller orredundant controllers). Thus, if you had a RAID controller unit with twoactive controllers, the total number of LUNs between them could not bemore than eight. However, with the release of v6.16, up to 256 LUNs aresupported by the driver.

• RAID controller: The Fibre Channel RAID Controller supports up to 32LUNs.

The limit is therefore now 32 LUNs.

Number of LUNs


• RAID-3: 20 disks per LUN (19 usable)• RAID-5: 20 disks per LUN (19 usable)

If the system runs out of disk space, then you have two possibilities:

• You can create additional RAID-1 or RAID-5 arrays (each with one or moreLUNs) and place them into the appropriate cluster group. Be aware of thelimited number of drive letters available in Windows NT.

• You can expand the existing LUNs by adding more disks. This can be donedynamically with the SYMplicity Storage Manager software. (Note, this doesnot change the size of the Windows NT partitions. You will need a third partytool to perform this function.)

At installation, the SYMplicity Storage Manager software is limited to 16 RAIDcontrollers. This limit is determined by the System_MaxControllers parametersetting in the C:\Program Files\SYMSM\RMPARAMS file. If your system has morethan 16 RAID controllers, change the parameter in the RMPARAMS file to reflectthe actual value.

Each hub is shipped with four short-wave GBICs, and can accept up to seven.

3.6.3 Fibre Channel tuning recommendationsThe RAID controller unit has tuning parameters that have significant performanceon database servers:

• Segment size. You can configure the segment size (stripe size) from 16 KB upto 128 KB. The segment size of the controller must be matched closely to theblock size of the application. For example, an Oracle database can use 2 KB, 4KB, or 8 KB blocks for the tables and would give the best performance using16 KB stripe size on the controller. However, the SAP R/3 application logicmay write in 64 KB blocks and would be optimized with the array controller setto 64 KB stripes. Note, the segment size is set at the LUN level.

• Active/active mode. If you are using the Fibre Channel FailSafe Controller,you should set the two controllers in active/active configuration. By default,they are set up in active/passive mode, which means that the secondcontroller is used as a passive hot-spare. We recommend active/activebecause it allows you to take benefit from the load balancing between thecontroller. This feature is an advantage of the Fibre Channel technology. The128 MB cache is split in half and mirrored between the two “sides”. You canbalance the LUN ownership between controllers in order to distribute the I/Ofor better performance.

• Write cache. As R/3 is a write intensive application, the write cache usageparameter should always be activated. When using the FailSafe controller, youshould also consider activating the write cache mirroring (these parametersare the default).

3.7 Network configurations

In this section we introduce the basic elements of the network for a SAPinstallation in a Microsoft clustering on IBM Netfinity servers. This is only toprovide guidelines on the general concepts and not a box-by-box description.


For an understanding of Microsoft Cluster Server (MSCS) concepts, review thechapter "Microsoft Cluster Server Concepts" in the Microsoft Cluster ServerAdministrator’s Guide, \SUPPORT\BOOKS\MSCSADM.HLP on the Windows NT4.0 Enterprise Edition CD 1.

Figure 32. Network cluster configuration example

3.7.1 ComponentsThe components that make up the network are:

• Backbone network

In large R/3 systems the network adapters can cause bottlenecks so the useof a multiple network interface card is recommended. The backbone networkterminology is used to describe the LAN on which interserver traffic will beconcentrated. This is a very fast segment of 100 Mbps or more.

This is the preferred path for local traffic between the SAP application serversand the database server. Refer to 4.16.2.9, “Network planning” on page 121,where a more complete description can be found.

• Public network

All servers and clients have a direct connection or route to this network. Thiswill be in a routed, switched, subnetted, or bridged environment. The wholeenterprise uses this network for everything from file server access, printsevers, e-mail, etc. To maximize the bandwidth of the R/3 system trafficbetween the sapgui and dialog instances, we must be aware of the total usageand capacity of this network. In a TCP/IP network the ideal way to handle thissituation is by creating Virtual LANs (VLANs). See Switched, Fast and GigabitEthernet by Robert Breyer & Sean Riley (Macmillan Technical Publishing) fordetails on how to create a VLAN.

• Private network

BackboneNetwork

PublicNetwork

APP1Node A Node B

5170

-01

ClusterInterconnect

APP2

Clients Clients

Shared disks


The term private network is used in many different ways. This example refersto the crossover-connected (100Base-T) heartbeat (or interconnect) cable.MSCS is constantly checking the status of the two nodes. The server that is inpassive mode uses this cable to poll the other server as to the status of all theresources being managed in the cluster. Provided we’ve configured the publicnetwork to accept all communications traffic (that is, client and interconnecttraffic), if this interconnect link goes down, the cluster will use the publicnetwork to do polling instead. In this way, we have two redundant paths (publicand backbone) for the cluster heartbeat. If a resource fails on the active nodethen through the use of this cable, the cluster knows when to fail over.

All Windows NT nodes are members of the same domain, SAPDOM. For the twoWindows NT servers forming the MSCS cluster, five network names and sevenstatic IP addresses are needed as shown in Table 13. We have also mentionedtwo more static IP addresses to show the connection to the backbone network:

Table 13. Network parameters for MSCS nodes

The TCP/IP HOSTS file (in directory \WINNT\SYSTEM32\DRIVERS\ETC) wasused for naming resolution, containing entries for all five IP addresses on thepublic network.

Note that the cluster-private LAN (crossover Ethernet connection) is used for theMSCS communication only. The virtual names ITSSAP and ITSDBMS on thepublic LAN are used for user access and communication between R/3 work

Server NetBIOSname

DNS hostname

Description TCP/IP addresses

WindowsNT server 1(Node A)

SERVERA servera Preferred nodefor R/3 centralinstance (CI)

Public network: 192.168.0.1

serverai Private network: 10.0.0.1

serverab Backbone network: 172.16.0.1

WindowsNT server 2(Node B)

SERVERB serverb Preferred nodefor Oracle or SQLServer DBMS


serverbi Private network: 10.0.0.2

serverbb Backbone network: 172.16.0.2

Virtualserver

SAPCLUS sapclus Cluster alias (forMSCSadministration)


Virtualserver

ITSSAP itssap R/3 alias (forconnecting to theR/3 services)


Virtualserver

ITSDBMS itsdbms DBMS alias (forconnecting to theDBMS services)

Public network:192.168.0.52

The MSCS part of the SAP installation manuals uses the names Node A andNode B for the two MSCS nodes. It does not matter which node is called A andwhich is called B. We used the real computer names SERVERA and SERVERBfrom our test installation. This convention makes it easier to understand thescreen captures throughout this redbook.

Server names


processes and the database. Thus the public network in our example has tohandle the SAPGUI traffic (which is usually of low volume) and the databasetraffic (from R/3 and ADSM (applications server), which may be of very highvolume). A small bandwidth network will not be appropriate.

A common practice in large SAP R/3 environments is to separate the user’snetwork (small bandwidth public LAN) completely from the network for databaseaccess (large bandwidth SAP backbone). You may configure the SAP backboneas additional network, thus having three network interface cards in the clusternodes.

SAP recommends that during installation all settings be restricted so the entirecommunication runs via the normal public network. The SAP installationdocumentation is also formulated under this assumption. After the installation iscompleted, you can reconfigure the cluster so that the communication to thedatabase is carried out via the backbone network. Proceed as follows:

1. Choose the IP address for the database on the backbone network.

2. Change the name resolution for the virtual database name so that thedatabase name is displayed on the new IP number (in all HOSTS files or in theDNS server).

3. Change the IP resource in the database resource group so that it contains theIP address on the backbone network. The network to which this address isbound must also be modified.

4. Stop and start the database and R/3 resource groups.

You can simply create IP resources in the R/3 resource group, so that the R/3 systemcan also be reached via the backbone network (by the ADSM server, for example).Additional steps are not required because R/3 automatically receives the incomingdata from all existing IP addresses. You can also include additional IP resources inthe database group (for example, to be able to administrate the database via theother network). Some databases (for example, Oracle) simply connect their entryports during installation to certain addresses. In this case you must make additionalconfiguration entries so that the new IP resource can be used. With Oracle,additional entries are needed in the file LISTENER.ORA.

3.7.2 Different name resolution methodsThe three most common methods used for name resolution in a Windows NTenvironment are:

• DNS

Domain Name System (DNS) can map very complex hierarchical treestructures with a large number of host systems. This method is decentralizedin its implementation. That means the entire list of all hosts are not kept onone system, only a portion of the entries, making it feasible for a searchrequest to reply in a reasonable amount of time. A good example of thescalability of DNS is its use in the management of the Internet.

• WINS

Windows Name Service (WINS) is used to dynamically map NetBIOS names(computer names) to IP addresses. The same function can be performed inthe absence of a WINS server with LMHOSTS files, but these files are staticand do not incorporate dynamic changes. WINS is suited to a pure Microsoft


Windows environment. The advantage of WINS is its simplicity, robustness,and flexibility. The disadvantage is WINS is only suited for smaller, flatlystructured homogeneous networks.

• Hosts file

The HOSTS file is an ASCII text file that statically maps local and remote hostnames to IP addresses, located in \WINNT\SYSTEM32\DRIVERS\ETC. Thisfile is read from top to bottom and as soon as a match is found for a hostname, the file stops being read. The HOSTS file is a reliable and simplesolution, since all the information that is needed to resolve a name to an IPaddress is stored on the computer. This local administration gives a very highperformance. The cost of this performance is the need for a lot of maintenanceif the environment is constantly changing. An example is:

192.168.0.1 servera node1

where node1 is the name of the alias.

The use of a second (mirrored) WINS or DNS server is stronglyrecommended.

Hint: When using a DNS server or HOSTS file with fully qualified names (forexample, server1.company.com), you should enter the DNS domain in theDomain Suffix Search Order for the TCP/IP protocol in the Network ControlPanel applet.

3.7.3 Resolve IP address to the correct host nameYou now need to check which IP address will respond when SERVERA orSERVERB is pinged. Having two or more network adapters in the same servercan lead to address requests resolving to the wrong IP address. In our lab setup,we assigned an IP address of 192.168.0.1 to SERVERA in the HOSTS file. Whenthe command PING SERVERA is entered from a command prompt, the IP addressthat may be returned is 10.0.0.1, which is incorrect. The correct address shouldbe one from the public network.

For systems with Service Pack 4 or later installed, perform the following steps onboth nodes to correct the problem, as described in Microsoft Knowledge Baseentry Q164023:

1. Open the Network applet in the Control Panel.

2. Click the Bindings tab, select show bindings for all protocols. ExpandTCP/IP protocols. This will display all network interface cards installed.

3. The solution is to change the order so the public network interface card isdisplayed before the private network interface card. Use the Move Up andMove Down buttons to change the order of the cards.

4. Repeat this procedure for any other TCP/IP protocols installed.

We recommend that you install Service Pack 4 or later for this reason. If you haveService Pack 3 or earlier installed, upgrade to later ones.

Note: This workaround only applies to Windows NT 4.0 computers that havemultiple network adapters installed and only when it is querying the local hostname. It has no effect when querying other computers on the network.


3.7.4 Redundancy in the network pathThe nodes of an MSCS cluster must be connected by one or more physicallyindependent networks (sometimes referred to as interconnects). Although MSCSclusters can function with only one interconnect, two interconnects are stronglyrecommended and are required for the verification of MSCS OEM systems thatinclude both hardware and MSCS software.

Redundant, independent interconnects eliminate any single point of failure thatcould disrupt communication between nodes. When two nodes are unable tocommunicate, they are said to be partitioned. After the two nodes becomespartitioned, both nodes start an arbitration process for the quorum resource. Thisprocess is based on a challenge-response protocol using SCSI device reservationsfor the quorum disk. Because only one node can finally hold a SCSI reservation for adisk on the shared bus, this node will be the winner of the arbitration process. Wecannot determine in advance which node will be the winner as it depends on timingduring at the moment of partitioning. The other node will stop the cluster service (thenode is still running but doesn't bring any cluster resources online).

For example, if each node has only one network adapter, and the network cableon one of the nodes fails, each node (because it is unable to communicate withthe other) attempts to take control of the quorum resource. There is no guaranteethat the node with a functioning network connection will gain control of thequorum resource. If the node with the failed network cable gains control, theentire cluster is unavailable to network clients. Each network can have one of fourroles in a cluster. The network can support:

• Node-to-node communication, referred to as private networks• Only client-to-cluster communication• Both node-to-node communication and client-to-cluster communication• No cluster-related communication

Networks that support client-to-cluster communication (either with or withoutsupporting node-to-node communication) are referred to as public networks. Werecommend that you configure the public network to be both the node-to-nodeand the client-to-cluster communications for added network redundancy.

Before you install the MSCS software, you must configure both nodes to use theTCP/IP protocol over all interconnects. Also, each network adapter must have anassigned static IP address that is on the same network as the correspondingnetwork adapter on the other node. Therefore, there can be no routers betweentwo MSCS nodes. However, routers can be placed between the cluster and itsclients. If all interconnects must run through a hub, use separate hubs to isolateeach interconnect.

Note: MSCS does not support the use of IP addresses assigned from a DynamicHost Configuration Protocol (DHCP) server for the cluster administration address(which is associated with the cluster name) or any IP Address resources. Youshould use static IP addresses for the Windows NT network configuration oneach node.

3.7.5 Auto-sensing network adaptersThe latest types of Ethernet network adapters are now able to automaticallysense the speed of the network. The typical speed is 10/100 Mbps and operation


mode of half or full duplex. A poor cable installation can cause some ports on a10/100 switch to have different speeds.

The solution is to not use auto-sensing and to manually configure the networkinterface cards to the setting of the network, for example, 100 Mbps/full duplex.The switches and hubs must be able to support the 100 Mbps full duplex type ofconfiguration. See Knowledge Base article Q174812 for more information.

3.7.6 Redundant adaptersThe use of redundant adapters is highly recommended for the public LANconnections.

Various Fast Ethernet and token-ring adapters support adapter fault tolerance(AFT), allowing two adapters to be paired inside a server in an active/passiveconfiguration. If there is a failure of the cable, connector or primary adapter, thenthe backup adapter is automatically brought online to handle the traffic. Thisfailover typically occurs in a fraction of a second, minimizing lost data andvirtually eliminating down time due to adapter or cable failures.

This fault tolerance feature also works between an adapter and the 10/100Ethernet controller on the planar of many Netfinity servers. Note that the chip setof the two adapters in the fault-tolerant pair must be identical. We recommendyou use the Netfinity Ethernet Adapters as this allows multiple redundant pairs.The use of the Netfinity 10/100 fault tolerant adapter restricts you to just onefault-tolerant pair.

The behavior of MSCS in the case of cable or adapter failure depends on thenature of the failure. If the adapter fails completely (in Windows NT, you will nolonger see the adapter when issuing an IPCONFIG command), then the IPresources bound to that adapter also fail and will be moved to the other node,causing the group to be moved. However, disconnecting a cable doesn'tconstitute an adapter failure (see Microsoft Knowledge Base articles Q175767and Q176320) and will not force MSCS to fail over any resource. This mayproduce a situation where all cluster resources are still online but cannot bereached by the clients.

In theory, connection loss because of cable failure may lead an applicationprocess to fail, thus forcing MSCS to follow the restart/failover policy defined forthis process. But as explained in Chapter 2, “SAP R/3 and high availability” onpage 5, R/3 processes will go into reconnect status. For database processes, thisdepends on the type of the database.

The exact behavior of a R/3 cluster in the case of a cable or adapter failure isundefined. You should always configure redundant adapters for the public LANconnections.

However, we do not recommend the use of redundant adapters for the clusterheartbeat, because the cluster heartbeat can be configured at the MSCS layer tofail over to the public LAN. Redundant heartbeat adapters could confuse thecluster. When Node A fails, the redundant adapters on Node B try to reconnectboth ways. Alternately, after some timeouts one adapter will remain the primary. IfNode A restarts then the adapter on the other cable may be the primary heartbeatadapter of A. Thus both servers will not establish a connection automatically,which would require operator intervention to reset the primary/secondary adapter


definitions. Again the exact behavior depends on the adapter's chipset and driver.To avoid any problem, you should not configure redundant adapters for the clusterheartbeat, but fail over to the public LAN instead.

3.7.7 Load balancingAdapters that support load balancing enable the server to add scalablebandwidth as traffic demands grow. This feature balances outgoing server trafficacross as many as four adapters of the same type, enabling server bandwidth toscale up to 800 Mbps in a full-duplex switched environment. This allows you toadd bandwidth to your server as you need it using proven, cost-effective FastEthernet technology. The following IBM adapters support load balancing:

• Netfinity Gigabit Ethernet SX• Netfinity 10/100 Ethernet• Netfinity 10/100 Fault Tolerant Adapters

Network adapter performance can be strongly improved by using linkaggregation, that is by using more network cards as a single network path. TheNetfinity Ethernet Adapter provides two link aggregation techniques: AutomaticLoad Balancing (ALB) and Fast EtherChannel (FEC).

ALB allows you to use up to four network cards as a single virtual networkadapter. What you get is not only link aggregation but also load balancing of thetraffic. That means situations in which one card manages 90% of the traffic whilethe other only has 10% are avoided by the adapter driver. The ALB guaranteeslink aggregation and load balancing only for outgoing packets. Any Fast Ethernetswitch can be used.

FEC is a proprietary Cisco technology allowing load balancing both for incomingand outgoing traffic. Again up to four adapters can be configured as a singlevirtual card. Only Cisco FEC switches like those of the Catalyst 5000 family canbe used. Similar considerations hold for the Gigabit Ethernet adapters. They alsoprovide two link aggregation techniques: Automatic Load Balancing and GigabitEtherChannel (GEC). Similar considerations hold: up to four adapters can beconfigured as a single virtual card, but ALB only balances outgoing packets. GECis a Cisco proprietary technology requiring Cisco switches.


Chapter 4. Installation and verification

In this chapter we describe the installation of SAP 4.5B on a Microsoft clusterwith three main database management systems (DBMS): Oracle, DB2, andMicrosoft SQL Server. The installation is described using flow charts and tables.

For major references to the installation, we have used the SAP R/3 publicationsand OSS notes. The intent of this chapter is not to replace the original SAPdocumentation — rather we only want to make installation as simple as possible.In each step we will make reference to the relevant documentation.

This chapter contains a general description of the installation from the very firstplanning step to the final tuning phase. We provide, as much as possible, thedetails that are DBMS independent. DBMS-dependent installation steps arecontained in the next three chapters:

• Chapter 5, “Installation using Oracle” on page 129• Chapter 6, “Installation using DB2” on page 145• Chapter 7, “Installation using SQL Server” on page 159

4.1 General overview of the installation process

The structure of the chapter is shown in Figure 33:

Figure 33. Overall installation process

In sections 4.4 to 4.16, the installation process is further divided into 29 steps asshown in Figure 34:

4.2, “Setting up security” onpage 86

4.3, “Hardwareconfiguration and

installation” on page 89

4.4, “Windows NTinstallation” on page 89

4.5, “MSCS pre-installationtesting” on page 96

4.6, “Basic Windows NTtuning” on page 96

4.7, “Microsoft ClusterServer installation” on

page 101

4.8, “Service pack andpost-SP installation steps”

on page 102

4.9, “MSCS verification” onpage 105

4.10, “Create theinstallation user account”

on page 107

4.11, “SAP and DBMSinstallation” on page 108

4.12, “SAP verification” onpage 110

4.13, “DBMS verification”on page 114

4.14, “Backboneconfiguration” on page 114

4.15, “SAP clusterverification” on page 117

4.16, “Tuning” on page 119

Covered inthe DBMSchapters

4.17, “Configuring a remoteshell” on page 127


Figure 34. More detailed installation process

4.2 Setting up security

When a company decides to implement SAP its success strongly depends onhow the SAP system works. The database of the SAP system is the mainrepository of sensitive data about customers, sales, financial activities, and so on.To get into the SAP database means to have a privileged view of the companyactivities. This can be an invaluable tool for marketing, sales, and management,but it can also be very dangerous if adequate security policies are not defined.Security is quite an obscure and delicate subject involving these maincomponents:

• The Windows NT account databases• The Windows NT file system• The SAP security system• The DBMS security system• The network configuration

Step 1: Configure internaldisks, page 89

Step 2: Fill in generalworksheet, page 90

Step 3: Windows NTinstallation on Node A,

page 90

Step 4: Windows NTinstallation on Node B,

page 92

Step 5: Modify BOOT.INI onNode B, page 94

Step 6: Configure disks,page 94

Step 7: SYMplicity StorageManager configuration,

page 95

Step 8: MSCS pre-installverification, page 96

Step 9: Configure theServer service, page 97

Step 10: Set the page filesize, page 98

Step 11: Modify BOOT.INIfor 4 GB tuning, page 99

Step 12: Remove driversand protocols, page 100

Step 13: Install MSCS onNode A, page 101

Step 14: Install MSCS onNode B, page 102

Step 15: Installation oftroubleshooting tools,

page 102

Step 16: Windows NTService Pack installation,

page 102

Step 18: Installation of IE,ADSI, and MMC, page 105

Step 17: DNS or HOSTSfile and transport directory

configuration, page 103

Step 19: Install the latestSAP R/3 DLLs, page 105

Step 20: Apply the PostSP4 DLLs, page 105

Step 21: Correction ofNetwork Bindings,

page 106

Step 22: MSCSverification, page 107

Step 23: Create theinstallation user account,

page 107

Step 25: SAP tests,page 110

Step 26: DBMSverification, page 114

Step 27: Backboneconfiguration, page 115

Step 28: SAP tuning,page 119

Step 29: DB tuning,page 127

Step 24: Installing SAP andthe DBMS, page 108

Step 30: Configure aremote shell, page 127

Chapter 4. Installation and verification 87

This section provides general hints and tips about the subject of planning thesecurity of an SAP environment.

4.2.1 Windows NT security planningAs a general reference publication to be used to configure Windows NT in asecure environment, we recommend Windows NT Security Guidelines by SteveSutton. Updated information about security problems in Windows NT can befound at the following Web sites:

http://www.microsoft.com/securityhttp://ntsecurity.ntadvice.comhttp://www.trustedsystems.com

4.2.2 SAP R/3 security planningThe steps to configure SAP R/3 in a Windows NT environment to achieve themaximum level of security are fully described in the three volumes of SAP R/3Security Guide. As described in the second volume of the SAP R/3 SecurityGuide it is recommended you create two different Windows NT domains. In thisredbook, we call them the USERS and SAPDOM domains. The two domains areused as follows to:

• Create the user and PC accounts in the USERS domain

• Create the SAP and DBMS fundamental server accounts in the SAPDOMdomain

A broad summary of the architecture of the system with indications on networkconnections is shown in Figure 35:

Figure 35. Domain layout

Backbone178.0.0.0

10.0.0.0

Public192.168.0.0

Public192.168.0.0

PDC

PDC

BDC

BDC

APP

DB CI

SAPDOM Domain

USERS Domain

ClientsClients

5170

-01

http://www.microsoft.com/security

http://ntsecurity.ntadvice.com

http://www.trustedsystems.com


4.2.3 DBMS security planningDetailed information on how to improve the security of the DBMS is provided inChapter 2, “Database Access Protection” of the R/3 Security Guide: Volume II,R/3 Security Services in Detail. This chapter includes a specific section on Oracleand DB2 under Windows NT.

The accounts that must be created during the installation are described in thefollowing tables. Table 14 shows the accounts common to all the DBMSs.

Table 14. Accounts common to all DBMSs

Account Where Details Lab values

Cluster service account Domain controller of theSAPDOM domain

User ID name2 SAPDOM\ClusSvc

User who administersthe R/3 system

Domain controller of theSAPDOM domain

<sid>adm1,3 SAPDOM\itsadm

SAP service account Domain controller of theSAPDOM domain

SAPService<SID>1,4 (for DB2UDB, it is sapse<sid>

SAPDOM\SAPServiceITS(for DB2, sapseits)

Global group of SAPadministrators


SAP_<SID>_GlobalAdmin1 SAPDOM\SAP_ITS_GlobalAdmin

Local group of SAPadministrators

Local account on SAPapplication servers andDB server

SAP_<SID>_LocalAdmin1 SAP_ITS_LocalAdmin

CI server account Domain controller of theSAPDOM domain

NetBIOS name unique in theUSERS and SAPDOM domain5

SERVERA

DB server account Domain controller of theSAPDOM domain


SERVERB

Other SAP applicationservers account



SERVERC

Throughout this chapter and the rest of the redbook, take note of the case ofparameters such as accounts, groups, and variables. The way they are writtenis case sensitive. For example if <sid> is “its”, then:

• <SID> equates to ITS• <sid> equates to its

Failure to adhere to the case of the parameters will likely result in a brokeninstallation.

Attention: Case is important


4.3 Hardware configuration and installation

A description of how to configure and install the hardware is included in thefollowing sections:

• 3.3, “Netfinity sizing” on page 46.• 3.2, “Certification and validation of hardware” on page 42• 3.4, “Disk layouts” on page 50• 3.5, “ServeRAID SCSI configurations” on page 62• 3.6, “Fibre Channel configurations” on page 68

We recommend you also plan the distribution of the adapters in the PCI slotsbefore beginning the installation. This distribution can have a major impact on theperformance of your system. For information, see Netfinity Performance Tuningwith Windows NT 4.0, SG24-5287.

4.4 Windows NT installation

This section contains a general description of the installation process of WindowsNT for an SAP server. We strongly recommend, before beginning the installation,you review 3.5, “ServeRAID SCSI configurations” on page 62 if you are workingwith a SCSI cluster and 3.6, “Fibre Channel configurations” on page 68 if you areworking with a Fibre Channel cluster. See also the continually updated OSS note0112266 R/3+MS Cluster Server: Frequent Questions and Tips.

Step 1: Configure internal disks

Configure the internal disks as described in 3.5, “ServeRAID SCSIconfigurations” on page 62. Relevant information on how to configure the disks isalso contained in 3.4, “Disk layouts” on page 50.

Notes:1 The case of the parameter must be exactly as specified.2 The cluster service account must be a domain user and belong to the local Administrators group in both nodes of the

cluster. Moreover, the account must have the following rights: Backup files and directories, Increase quotas, Increasescheduling priority, Load and unload device drivers, Lock pages in memory, Logon as a service and Restore files anddirectories

3 The <sid>adm account must be a domain user and belong to the Domain Admins, Domain Users, andSAP_<SID>_Global Admin groups. Besides the account must have the following rights: Replace a Process LevelToken, Increase Quotas, and Act as a part of the operating system. After the installation it is recommended to removethe <sid>adm account from the Domain Admins groups to improve security.

4 The SAP service account must be a domain user and have the following rights: Logon as a service.5 We recommend to use only alphabetic and numeric characters and not to use more than eight characters.

Account Where Details Lab values

See Figure 34, on page 86 for a pictorial view of the installation stepsdescribed here.

Summary of the steps


Step 2: Fill in general worksheet

Before starting the installation it is useful to gather information that could berelevant during the setup. Table 15 should be filled out with the relevant data:

Table 15. General worksheet

Step 3: Windows NT installation on Node A

Installation begins with Node A. Before installing, fill inTable 16. At the end of theinstallation install Service Pack 3:

Table 16. Node A worksheet

Parameters Lab values Your values

Windows NT domain for SAP and DB servers SAPDOM

PDC of SAP domain NetBIOS Name SAPPDC

PDC of SAP domain IP address 192.168.0.10

PDC of SAP domain subnet mask 255.255.255.0

Windows NT domain for user accounts USERS

PDC of USERS domain NetBIOS Name USERSPDC

PDC of USERS domain IP address 192.168.0.120

PDC of USERS domain subnet mask 255.255.255.0

Parameters Restrictions and recommendations Lab values Your values

Standby OS drive The Standby operating system must beinstalled before the Active one.

D:

Active OS drive The Active operating system must beinstalled after the Standby one.

C:

Windows NT directory forStandby OS

Change the default value \WINNT only ifyou are installing the Standby operatingsystem in the same logical drive in whichthe Active operating system is installed.

\WINNT

Windows NT directory forActive OS

Use the default value \WINNT. \WINNT

Computer name The maximum is 8 characters. Only letters,numbers and the "-" symbol are allowed.The last character cannot be a "-" symbol.

SERVERA

Administrator built-inaccount password

Follow the general recommendations aboutpasswords in a secure environment.

ibm

Domain role The server can not be a PDC or a BDC. Stand-aloneserver

Components to install Leave default. Defaultcomponents

RAS installation Do not install RAS. No

Microsoft IIS Do not install IIS. No

Network protocols MSCS only supports TCP/IP. Only TCP/IP


Dynamically assignedaddress

Only static addresses should be used withMSCS.

No

IP address(private network)

Private network must be a separatesubnet. The use of the IANA reservednetwork 10.0.0.0 is good practice. Use adifferent network only when this network isalready used for other purposes.

10.0.0.11

Subnet mask (private) 255.255.255.0

Default gateway (private) Must be blank Blank

IP address (public) 192.168.0.11

Subnet mask (public) 255.255.255.0

Default gateway (public) 192.168.0.1001

IP address (backbone) 172.16.0.11

Subnet mask (backbone) 255.255.255.0

Default gateway(backbone)

Must be blank Blank

DNS domain name itso.ral.ibm.com

DNS Service search order List of DNS servers. 192.168.0.1011

DNS suffix search order List of DNS suffixes. Should contain owndomain.

None

Primary WINS server(private network)

Leave empty. None

Secondary WINS server(private network)

Leave empty. None

Primary WINS server(public)

IP address of the primary WINS server. 192.168.0.101

Secondary WINS server(public network)

IP address of the secondary WINS server. None

Primary WINS server(backbone network)

Leave blank or configure a WINS server forthe backbone network.

None

Secondary WINS server(backbone network)

Leave blank or configure a WINS server forthe backbone network.

None

Enable DNS for Windowsresolution

Checked

Enable LMHOSTS lookup If you disable LMHOSTS lookup the Hostsfile becomes useless.

Checked

Domain name Name of the SAP domain. SAPDOM

SAP Domain administratorUser ID

If it is not possible for security reasons toget the administrator account the Node Acomputer account must be created in thePDC of the SAP Domain.

Administrator



At the end of the installation when the administrator logs on for the first time theWindows NT Enterprise Edition Installer is automatically started. Do not try toconfigure the cluster now. The cluster will be installed later.

Step 4: Windows NT installation on Node B

After installing Service Pack 3 on Node A, the installation of Windows NT EE onNode B can begin. Before installing fill in Table 17. At the end of the installation ofNode B, Service Pack 3 must be installed:

Table 17. Node B worksheets

SAP Domain administratoraccount password

If it is not possible for security reasons toget the administrator account the Node Acomputer account must be created in thePDC of the SAP Domain.

ibm

Note: The IP address used in our lab is reserved for private use. However, this is not necessary for your installationexcept where noted.


Standby OS drive The Standby operating system must beinstalled before the Active one.

D:

Active OS drive The Active operating system must beinstalled after the Standby one.

C:

Windows NT directory forStandby OS

Change the default value \WINNT only ifyou are installing the Standby operatingsystem in the same logical drive inwhich the Active operating system isinstalled.

\WINNT

Windows NT directory forActive OS

Use the default value \WINNT. \WINNT

Computer name The maximum is 8 characters. Onlyletters, numbers and the "-" symbol areallowed. The last character cannot be a"-" symbol.

SERVERB

Administrator built-inaccountpassword

Follow the general recommendationsabout passwords in a secureenvironment.

ibm

Domain role The server can not be a PDC or a BDC. Stand-alone server

Components to install Leave default. Default components

RAS installation Do not install RAS. No

Microsoft IIS Do not install IIS. No

Network protocols MSCS only supports TCP/IP. Only TCP/IP

Dynamically assignedaddress

Only static addresses should be usedwith MSCS.

No



IP address (private network) Private network must be a separatesubnet. The use of the IANA reservednetwork 10.0.0.0 is good practice. Use adifferent network only when thisnetwork is already used for otherpurposes.

10.0.0.21

Subnet mask (private) 255.255.255.0

Default gateway (private) Not required for the private network. Blank

IP address (public) 192.168.0.21

Subnet mask (public) 255.255.255.0

Default gateway (public) 192.168.0.1001

IP address (backbone) 172.16.0.21

Subnet mask (backbone) 255.255.255.0

Default gateway (backbone) Not required if all the applicationservers are in the same site.

None

DNS Domain name itso.ral.ibm.com

DNS Service search order List of DNS servers. 192.168.0.1011

DNS suffix search order List of DNS suffixes. None

Primary WINS server(private)

Leave empty. None

Secondary WINS server(private)

Leave empty. None

Primary WINS server (public) IP address of the primary WINS server. 192.168.0.1011

Secondary WINS server(public)

IP address of the secondary WINSserver.

None

Primary WINS server(backbone)

Leave blank or configure a WINS serverfor the backbone network.

None

Secondary WINS server(backbone)

Leave blank or configure a WINS serverfor the backbone network.

None

Enable DNS for Windowsresolution

Checked

Enable LMHOSTS lookup Checked

Domain name Name of the SAP domain. SAPDOM

SAP Domain administratorUser ID

If it is not possible for security reasonsto get the administrator account theNode A computer account must becreated in the PDC of the SAP Domain.

Administrator

SAP Domain administratoraccount password

If it is not possible for security reasonsto get the administrator account theNode A computer account must becreated in the PDC of the SAP Domain.

ibm



At the end of the installation when the administrator logs on for the first time theWindows NT Enterprise Edition Installer is automatically started. Do not try toconfigure the cluster now. MSCS will be installed in step 14 on page 102.

At the end of the installation remember to configure either the DNS server, WINSserver or the HOSTS file. If you plan to use the HOSTS file, it is important toremember that the file is case sensitive. See Knowledge Base article Q101746“TCP/IP Hosts File Is Case Sensitive” for details. It is also important not to installService Packs 4 or 5 at this point. SP4 and SP5 are installed after MSCS isinstalled.

You should also reference the following Knowledge Base articles:

• Q195462 "WINS registration and IP address behavior for MSCS" (generalinformation how to use)

• Q193890 "Recommended WINS configuration for MSCS" (disable WINS clientfrom the private LAN)

• Q217199 "Static WINS entries cause the network name to go offline" (do notuse static entries in WINS for clustered computers)

Step 5: Modify BOOT.INI on Node B

Modify the BOOT.INI file of Node B to avoid simultaneous contention of shareddrives. This can be obtained by adding a different boot timeout in the servers:

Figure 36. BOOT.INI file

Step 6: Configure disks

Now you must create your own worksheets describing the disk layout.Thedistribution of files is DBMS dependent. Examples on how to configure the disksare provided in 3.4, “Disk layouts” on page 50:

Administrator built-inaccountpassword

Follow the general recommendationsabout passwords in a secureenvironment.

ibm

Notes:1 The IP address used in our lab is reserved for private use. However, this is not necessary for your installation except

where noted.


[boot loader]timeout=5default=multi(0)disk(0)rdisk(0)partition(1)\WINNT

[operating systems]multi(0)disk(0)rdisk(0)partition(1)\WINNT="Active Operating System" multi(0)disk(0)rdisk(0)partition(1)\WINNT="Active Operating System" /basevideo /sos multi(0)disk(1)rdisk(0)partition(1)\WINNT="Standby Operating System"multi(0)disk(1)rdisk(0)partition(1)\WINNT="Standby Operating System" /basevideo /sos

On Node B only, set thisparameter to 5 seconds


• Oracle: Table 10 on page 61• DB2: Table 10 on page 61• SQL Server: Table 12 on page 72

Format shared drives and assign meaningful disk labels to the local and shareddrives by means of Windows NT Disk administrator on both nodes.

Step 7: SYMplicity Storage Manager configuration

If the cluster exploits FC-AL technology configure SYMplicity Storage Manager asdescribed in 3.6, “Fibre Channel configurations” on page 68. See also 3.4, “Disklayouts” on page 50 for a detailed description of DBMS dependent on how toconfigure the drives.

At the end of the installation you may see the following error message in theWindows NT Event Viewer:

Event ID: 8032. Source Browser. Type Error. The browser service has failed to retrieve the backup list too many times on transport \Device\NetBT_IBMFEPCI1. The backup browser is stopping.

A simple procedure to correct this problem is to disable the browser service onthe private and backbone network. The selective disabling of the browser serviceis described in Knowledge Base article Q158487 “Browsing Across Subnets w/aMultihomed PDC in Windows NT 4.0”. In our lab configuration the CI and DBserves had three identical network adapters. Figure 37 shows the registry key,HKey_Local_Machine\System\CurrentControlSet\Services, containing theconfiguration of network adapters:

Figure 37. Registry key

Figure 37 shows four subkeys: IBMFEPCI, IBMFEPCI1, IBMFEPCI2 and IBMFEPCI3.The first key contains information about the network driver, while each of theother three corresponds to one particular network adapter. The network adaptercan be recognized looking in the \Parameters\Tcpip subkey. Figure 38 shows howwe solved the problem by adding a new value in the subkeyHKey_Local_Machine\System\CurrentControlSet\Services\Browser\Parameters:


Figure 38. Correcting the entry

4.5 MSCS pre-installation testing

Before beginning the MSCS installation it is useful to verify the hardware andoperating system installation. As described in OSS note 0154700 MSCS ClusterVerification Utility, the Cluster Verification Utility included in the last release ofWindows NT Resource Kit can be useful to test SCSI clusters.

Step 8: MSCS pre-install verification

To use the Cluster Verification Utility it is necessary to install the Supplement 3 ofthe Windows NT Resource Kit. Detailed documentation on how to install thisutility is in the Resource Kit Tools Overview Help in the Windows NT Resource KitProgram Group.

Note: The Cluster Verification Utility is really composed of two different programs,the Cluster Verification Utility and the Cluster Diagnostic Utility.

To start the Cluster Verification Utility follow the procedure explained in Help. Youneed to disconnect one of the nodes from the SCSI channel before beginning thetest on the other node.

To use the Cluster Diagnostic Utility follow these instructions:

1. On Node A, execute the command CLUSTSIM /S from the command prompt.

2. Wait until you get the message saying you can start the utility on Node B.

3. On Node B, execute the command CLUSTSIM /N:ServerA from the CommandPrompt, where ServerA is the name of Node A.

4. Examine the log files.

Further information can be found in Windows NT Microsoft Cluster Server byRichard R. Lee.

4.6 Basic Windows NT tuning

Some basic tuning steps are strongly recommended:

• Configure the Windows NT Server service.

Attention: The Cluster Diagnostic Utility will destroy the content of shareddisks.

All data is destroyed


• Configure the page file.• 4 GB tuning.• Remove unnecessary drivers and protocols.

Each of these is now discussed.

4.6.1 Server service tuningStep 9: Configure the Server service

If an application asks the operating system to write data on a disk, this data isonly collected in cache memory (write back). When the number of modified pagescollected in the cache reaches a predetermined threshold the Cache managerstarts a thread (the lazy writer) that queues up one quarter of the modified pagesand then calls the Virtual Memory Manager. The Virtual Memory Manager thenwrites the queued pages to disk. This technique allows a better use of the I/Osystem, but to have a real performance improvement it is fundamental to have acorrect value of the threshold. As described in Chapter 8 of Inside Windows NT,Second Edition by David A. Salomon, the threshold is calculated at the start ofthe system and depends on the amount of memory and on the value of theregistry key:

Hkey_Local_Machine\System\CurrentControlSet\SessionManager\MemoryManagement\LargeSystemCache

The default value is 1 and corresponds to configure the Server service (alsoknown as LanmanServer) with the optimization MaximizeThroughput for FileSharing. It is highly recommended by SAP to change the value of this parameterto 0. This corresponds to configure the Server service with the optimizationMaximizeThroughput for Network Applications as shown in Figure 39:

Figure 39. Server service — SAP R/3 recommended value

This parameter can be found also in Windows NT Workstation systems, but itcannot be changed.

Some applications can consider writing immediately to disk (write through) veryimportant. To get this effect, they need to bypass the cache memory system andso they need to manage the memory directly. For these applications the setting ofthe Server service has no relevance. An example of an application doing its ownmemory management and for which the tuning of this parameter is ignored isMicrosoft SQL Server (see Optimization and Tuning of Windows NT, Version 1.4by Scott B. Suhy).

The effect of the change is to reduce the threshold in systems having largememory. The main reason is that the DBMS and SAP also have their own caches


and the use of many levels of caches decreases performance instead ofimproving it.

4.6.2 Page file tuningStep 10: Set the page file size

A fundamental SAP memory requirement is to create a very large page file. Thisis still valid for the zero administration memory management as pointed out in theOSS note 0088416 Zero administration memory management from 4.0A/NT. Themain recommendation is that the page file size is four times that of physical RAMinstalled RAM (that is, 4xRAM). However, while the 4xRAM recommendation canbe considered adequate to face the load when the CI and the DB are on differentnodes, it is not adequate when, due to some failure or for administrativepurposes, both the CI and the DB are on the same node. For this reason, SAPrecommends the page file be 5xRAM.

In reality, the page file size depends on the specific characteristics of the SAPenvironment such as the modules used and batch activity. As a result, the SAPrecommendation can only be considered a suggestion. The page file can beunnecessarily large and also too small.

Since it is not possible to create a page file of more than 4095 MB in a singlepartition and since it is not recommended for performance reasons to put differentpage files in different partitions in the same drive, we recommend you put nomore than one page file per physical array. This is not a problem for the operatingsystem, indeed as described in Inside Windows NT, Second Edition by David A.Salomon, Windows NT supports up to 16 different paging files. See 3.4, “Disklayouts” on page 50 for details on how to configure the pagefiles for systems withlarge memory. See also Optimization and Tuning of Windows NT Ver. 1.4 by ScottB. Suhy for further information.

Windows NT allows the configuration of a minimum and a maximum size for thepage file. The system starts with the minimum size and it expands the page filesize when necessary up to the maximum size specified. Since the performance ofthe system decreases during the expansion of the page file it is recommended toset the minimum dimension equal to the maximum dimension. So the finalrecommendation is:

Minimum dimension = maximum dimension = 5 x RAM installed

If the page file is exhausted or more precisely if Memory Committed bytesapproaches Memory Commit limit (see Optimization and Tuning of Windows NTVersion 1.4 by Scott B. Suhy for details) you can find three errors in the EventViewer:

• 2020: The server was unable to allocate from the system paged pool becausethe pool was empty.

• 2001: The server was unable to perform an operation due to a shortage ofavailable resources.

• 2016: The server was unable to allocate virtual memory.

The exhaustion of the system paged pool can cause blue trap screens. Wetherefore strongly recommend you respect SAP recommendations on thedimension of the page file. A suggestion we can give to customers who cannot


afford having so many disks for the paging activity is to follow the following lessrestrictive rules (Table 18):

Table 18. Page file size recommendations

4.6.3 4 GB tuningStep 11: Modify BOOT.INI for 4 GB tuning

Windows NT is a 32-bit operating system using a linear virtual addressingmemory system. The virtual addresses are divided into 2 GB sections: user modeand system mode as shown in Figure 40. The user processes can only exploit the2 GB user mode section:

Figure 40. 4 GB tuning

While this is enough for many systems it may not be enough in large SAP andDBMS installations. To resolve these situations Windows NT Enterprise Editionintroduced a new feature known as 4 GB tuning. By specifying a /3GB parameterin the BOOT.INI file, as shown in Figure 41, this changes how the operatingsystem manages the virtual addressing: 3GB for user processes and 1GB forsystem processes. Figure 40 shows the result.

Figure 41. BOOT.INI contents for 4 GB tuning

Memory Page file

1 GB or less 5 x RAM

From 1 GB to 2 GB 4.5 x RAM

From 2 GB to 4 GB 4 x RAM

ProcessVirtual

MemoryWithout

4 GB Tuning

ProcessVirtual

MemoryWith

4 GB Tuning

FFFFFFFFh

80000000h

00000000h

c0000000h

FFFFFFFFh

00000000h

SystemAddressable

Memory(2 GB)

ApplicationAddressable

Memory(2 GB)

System AddressMemory (1 GB)

ApplicationAddressMemory(3 GB)

5170

-01

[boot loader]timeout=30default=multi(0)disk(0)rdisk(0)partition(2)\WINNT[operating systems]multi(0)disk(0)rdisk(0)partition(2)\WINNT="Windows NT Server Version 4.00" /3GBmulti(0)disk(0)rdisk(0)partition(2)\WINNT="Windows NT Server Version 4.00 [VGA mode]"/basevideo /sos

For 4 GB tuning, youshould specify /3GB here


No new API is necessary to allow applications to exploit this new feature, butexecutables need to be modified in order to see this extra space. See thefollowing references for a detailed description on how to configure the SAPsystem in order to exploit this feature:

• Knowledge Base article Q171793 Information on Application Use of 4GT RAMtuning

• OSS note 0110172 NT: Transactions with large storage requirements

As described in the Microsoft document FAQ: All You Ever Wanted To Know aboutWindows NT Server 4.0 Enterprise Edition, to be able to exploit this feature thefollowing requirements have to be satisfied:

• The application is memory intensive• The application is able to utilize more than 2 GB of memory• The server has more than 2 GB of RAM• All the other components of the system have enough computing capacity

4.6.4 Remove unnecessary drivers and protocolsStep 12: Remove drivers and protocols

Do not install unnecessary drivers and protocols. Turn off installed unnecessaryservices and uninstall unnecessary drivers. The following is a typical list ofunnecessary drivers and protocols:

• Since Microsoft Cluster only supports TCP/IP protocol for failover, othernetwork protocols are usually not installed. Not installing IPX/SPX, NetBEUI,or other network protocols reduces the overhead on the server and reducesnetwork traffic. Unused network protocols can still generate network traffic dueto various automatic broadcast mechanisms (for instance, the Browserservice). If for some reason the server needs more than one protocol, thenTCP/IP should be first in the Bindings list to reduce connection time.

• If the server is not used for printing, the Spooler service should be disabledfreeing up memory.

• Typical services that can be disabled to obtain a better performance are:

– NetDDE– Messenger– Alerter

However, check to see if your SAP R/3 system needs them first. Specificallyfor Messenger, this service will be used during SAP R/3 upgrades so it shouldbe reenabled at that time.


4.7 Microsoft Cluster Server installation

Step 13: Install MSCS on Node A

Before beginning the installation, fill the Microsoft Cluster Server worksheet tocollect all the data necessary for the installation:

Table 19. MSCS worksheet

Now, start the installation of MSCS on Node A. At the end of the installationreboot Node A. Do not try to open the Cluster Administration utility yet as it isnecessary to update the HOSTS file before doing so.

Update the HOSTS file with the following lines:

127.0.0.1 localhost192.168.0.50 sapclus


Cluster name NetBIOS names rules. SAPCLUS

Cluster Service User ID This account must be created in the SAPdomain before beginning the MSCS setup1.

ClusSvc

Cluster Service password Password rules. ibm

Windows NT SAP domain name SAPDOM

Shared cluster disks See disk layout worksheet. I, J, K, L, etc.

Quorum drive See disk layout worksheet. I:

Network name for the privatenetwork adapter

Select Enable for cluster use and Use forprivate communications.

Private

Network name for the publicnetwork adapter

Select Enable for cluster use and Use forall communications.

Public

Network name for the backbonenetwork adapter

Select Enable for cluster use and Use forall communications.

Backbone

Networks available for internalcluster communication

Arrange the order as shown in the Lab valuecolumn.

PrivateBackbonePublic

Cluster IP address In the same subnet of public networkadapters.

192.168.0.502

Cluster Subnet Mask The same public network adapters. 255.255.255.0

Network Public

Notes:1 The cluster service account must be a domain user and belong to the local Administrators group in both nodes of the

cluster. Moreover, the account must have the following rights: Backup files and directories, Increase quotas, Increasescheduling priority, Load and unload device drivers, Lock pages in memory, Logon as a service and Restore files anddirectories.

2 The IP address used in our lab is reserved for private use. However, this is not necessary for your installation exceptwhere noted.


Step 14: Install MSCS on Node B

Install the Microsoft Cluster Service on Node B. Before beginning the installationupdate the HOSTS file as described in Step 13.

Useful documentation to be used during and after the installation to troubleshootproblems are:

• MSCS Administrator’s Guide (from Microsoft TechNet)

• MSCluster Service Troubleshooting and Maintenance by Martin Lucas (fromMicrosoft TechNet)

• Deploying Microsoft Cluster Server by the Microsoft Enterprise ServicesAssets Team (from TechNet)

• Microsoft Cluster Server Release Notes (from the Windows NT 4.0 EnterpriseEdition CD)

• Windows NT Microsoft Cluster Server by Richard R. Lee

Step 15: Installation of troubleshooting tools

The most important suggestions we can give to the reader to allow in-depthtroubleshooting of cluster problems are to:

• Configure the cluster logging feature as described in Knowledge Base articleQ168801 “How to Enable Cluster Logging in Microsoft Cluster Server”

• Install a network monitoring tool such as Microsoft Network Monitor to be ableto analyze the network traffic between the cluster nodes, other SAPapplication servers, and clients. Most often cluster problems are actuallynetwork problems.

4.8 Service pack and post-SP installation steps

Step 16: Windows NT Service Pack installation

Install Service Pack 4 or Service Pack 5.

Whether you use Windows NT Service Pack 4 or Service Pack 5 the installationhas to be done only after the Microsoft Cluster configuration. The maindocuments to check before beginning the installation are:

• OSS note 0030478 Service Packs on Windows NT — this is the first notesaying which Service Pack you can install; the answer depends on the DBMS.

• OSS note 0144310 NT: Installing an NT SP on R/3 MSCS Cluster — this notedescribes the Service Pack installation procedure; it also explains that if youare using SP4 you have to install the RNR20.DLL, CLUSRES.DLL postfixes.

• OSS note 0128167 Service Pack 4 on NT MSCS with Oracle Products — thisnote is only relevant for Service Pack 4 and Oracle Fail Safe; it describes howto deal with the incompatibility between these products.

• RELNOTES.TXT file in the Service Pack CD — detailed description of theprocedure for a rolling upgrade.

The installation procedure is described in RELNOTES.TXT file.You should alwayscreate the Uninstall directory to be able to go back if there is any problem with themost recent Service Pack release.


Step 17: DNS or HOSTS file and transport directory configuration

After having completed the SP installation it is necessary to configure either theDNS server, WINS server or the HOSTS file. If the choice is to use HOSTS files itis important to remember that the TCP/IP HOSTS file is case sensitive. SeeKnowledge Base article Q101746 TCP/IP “Hosts file is Case Sensitive” fordetails.

You should also reference the following Knowledge Base articles:

• Q195462 “WINS registration and IP address behavior for MSCS” (generalinformation how to use)

• Q193890 “Recommended WINS configuration for MSCS” (disable WINS clientfrom the private LAN)

• Q217199 “Static WINS entries cause the network name to go offline” (do notuse static entries in WINS for clustered computers)

Figure 42 shows how to configure the HOSTS file during an Oracle installation.The information necessary for the configuration is taken from:

• Table 16 on page 90• Table 17 on page 92• Table 32 on page 136• Table 34 on page 141• 4.14, “Backbone configuration” on page 114

Figure 42. HOSTS file for Oracle

Figure 43 shows the HOSTS file used in our DB2 installation. The informationnecessary for the configuration is taken from:

• Table 16 on page 90• Table 17 on page 92• Table 44 on page 154• Table 45 on page 155• 4.14, “Backbone configuration” on page 114.

127.0.0.1 localhost 192.168.0.1 servera SAPTRANSHOST192.168.0.2 serverb192.168.0.50 sapclus192.168.0.51 itssap192.168.0.52 itsora10.0.0.1 serverai10.0.0.2 serverbi172.16.0.1 serverab172.16.0.2 serverbb

Note: itsora is specific to theOracle database installation.


Figure 43. HOSTS file for DB2

Figure 44 shows the HOSTS file used in our SQL Server installation. The maindifference is the introduction of the msdtc line. The information necessary for theconfiguration is taken from:

• Table 16 on page 90• Table 17 on page 92• Table 51 on page 164• Table 53 on page 165• 4.14, “Backbone configuration” on page 114

Figure 44. HOSTS file for SQL Server

We raise to your attention the alias SAPTRANSHOST in the three HOSTS filesshown above (Figure 42, Figure 43, and Figure 44). This alias allows R3Setup torecognize servera as transport host. This line must be changed at the end of theinstallation. Since the transport directory is on the shared drives, the use of thehost name is not correct, so the virtual name of the SAP system (that is <sid>sap)should be used. We are assuming that you are using the shared drives of thecluster for the transport directory. If the transport directory is on an externalsystem follow these steps:

1. On the transport host create the directory \USR\SAP\TRANS.

2. On this directory grant Everyone Full Control to the group.

3. If no instance will be installed on the transport host share this directory asSAPMNT.

4. Update the HOSTS file (or the DNS server) to make SAPTRANSHOST analias for the transport host.

127.0.0.1 localhost 192.168.0.1 servera SAPTRANSHOST192.168.0.2 serverb192.168.0.50 sapclus192.168.0.51 itssap192.168.0.52 itsdb210.0.0.1 serverai10.0.0.2 serverbi172.16.0.1 serverab172.16.0.2 serverbb

Note: itsdb2 is specific to theDB2 database installation.

127.0.0.1 localhost 192.168.0.1 servera SAPTRANSHOST192.168.0.2 serverb192.168.0.50 sapclus192.168.0.51 itssap192.168.0.52 itssql10.0.0.1 serverai10.0.0.2 serverbi172.16.0.1 serverab172.16.0.2 serverbb

Note: itssql and msdtc arespecific to the SQL Serverdatabase installation.


Step 18: Installation of IE, ADSI, and MMC

Install Microsoft Internet Explorer 4.01 (or later) on both servers by running theIE4SETUP.EXE program from the \MS_IE4\WIN95_NT\EN subdirectory of thePRESENTATION CD. Do not install the Active Desktop feature.

Install the Active Directory Service Interfaces (ADSI) component on both nodesby running the ADS.EXE program from the \NT\I386\MMC subdirectory of theSAP Kernel CD-ROM.

Install the Microsoft Management Console (MMC) on both nodes by running theIMMC11.EXE program from the \NT\I386\MMC subdirectory from the SAPKERNEL CD

Step 19: Install the latest SAP R/3 DLLs

Install the latest version of the Dynamic Link Libraries on both nodes by runningthe R3DLLINS.EXE program from the \NT\I386\NTPATCH subdirectory of theSAP Kernel CD-ROM.

Figure 45. SAP R/3 DLL installation

Step 20: Apply the Post SP4 DLLs

Note: This only applies if you installed SP4. Apply the RNR20.DLL andCLUSRES.DLL Post-SP4 Hot fixes. The first fix can be downloaded from.

ftp.microsoft.com/bussys/winnt/winnt-public/fixes/usa/nt40/hotfixes-postSP4/Rnr-fix

The CLUSRES.DLL can be downloaded directly from the Microsoft site orinstalled as part of Service Pack 5. See OSS note 0134141 for details.

4.9 MSCS verification

Now the Microsoft cluster should be working without any problem. It is veryimportant to check the cluster configuration to be sure we have made no mistakeduring the setup. This is even more important when the team is composed of


people responsible for the Microsoft Cluster installation and others responsiblefor the SAP installation. To make everyone sure that the cluster is working and iscorrectly configured it is highly recommended to test the cluster configuration inas much detail as possible.. The main recommended tests are:

• Check the HOSTS file• Test the failover process

4.9.1 Checking the mapping of host namesStep 21: Correction of Network Bindings

To be sure that either the HOSTS file or the DNS server is correctly configured,the following simple PING tests should be performed (consider using the -aparameter of the PING command):

Table 20. PING tests

The following condition must always be satisfied:

If this condition is not satisfied when pinging your own host name the SAPinstallation cannot begin and the following simple procedure should beperformed.

Note: As per Microsoft Knowledge Base article Q164023, this procedure onlyapplies to servers having SP4 or later installed:

1. Open the Network applet in the Control Panel.

2. Click Bindings > Show Bindings for all Protocols

3. Select the TCP/IP protocol.

4. Change the order in which the cards are listed. They should be in the followingorder:

1. Public network2. Backbone network3. Private network

If the above condition is not satisfied while pinging any other host name thecorrect answer can be obtained by simply correcting the HOSTS file.

Node Test

ServerA PING <NodeA_Name>

ServerA PING <NodeB_Name>

ServerA PING <Cluster_Name>

ServerB PING <NodeA_Name>

ServerB PING <NodeB_Name>

ServerB PING <Cluster_Name>

The reply to the PING commands must come from the public network. Be surethat this is the case.

PING response


4.9.2 Test the failover processStep 22: MSCS verification

If everything works it is possible to continue with the fundamental six-stepscluster test. As a preliminary step you can create a Clock group to be used as atest system. The CLOCK.EXE application does not really stress the system, ifpossible it is more significant to use a more demanding application such as theWinstone99 benchmark.

Table 21. Verification tests

If you do not have enough time to complete all the steps at least follow thisrecommendation:

4.10 Create the installation user account

Step 23: Create the installation user account

Before beginning the installation process, do the following:

1. Create an account on the primary domain controller, (we will use sapinst) andadd it to the Domain Admins group. This account will be used during theentire installation process. Once installation is complete, the account can beremoved. During the installation process, R3Setup will assign the followingrights to the installation account :

Test Node A Node B

Manual failover Clock

Manual failover Clock

Microsoft CASTEST1

Regular shutdown Clock

Regular shutdown Clock

Blue screen2 Clock

Blue screen2 Clock

Power off Clock

Power off Clock

Notes:1 The Microsoft CASTEST is contained in the \MSCS\SUPPORT directory in the

Second CD of Windows NT EE. Instructions on how to use the test are containedin the CASREAD.TXT file.

2 The blue screen can be achieved killing the WINLOGON.EXE process by meansof the KILL.EXE utility contained in the Windows NT Resource Kit.

Use the Microsoft CASTEST utility; this should be considered the official MSCStest utility. Carefully examine the log file that is produced — verify whathappens when you power off one of the nodes. Specific hardware problemsdue to the lack of power on one of the I/O adapters are not seen by theMicrosoft utility.

Basic test


– Act as a part of the Operating System– Increase Quotas– Log on as a Service– Replace a Process Level Token

2. Fill out the SAP installation worksheets in this chapter and the databasechapters.

We provide, in each of the next sections dedicated to SAP installation, a SAPinstallation worksheet that has to be filled (see Table 29 on page 133 for anOracle example). In this worksheet we collect all the information necessary forthe local SAP and DBMS installation. These worksheets are DBMS dependent.They must be filled using the recommended disk layouts provided in 3.4, “Disklayouts” on page 50 as reference.

The following Windows NT accounts and groups are created automatically byR3Setup.

• <sid>adm account on the SAPDOM domain controller with the followingadvanced user rights:

– Log on as a Service– Replace a Process Level Token– Increase Quotas– Act as a part of the Operating System

• SAPService<SID> account on the SAPDOM domain controller

• SAP_<SID>_GlobalAdmin global group on the SAPDOM domain controller

• SAP_<SID>_LocalAdmin local group on each SAP R/3 application server andDB server

• Put the SAP_<SID>_GlobalAdmin global group in theSAP_<SID>_LocalAdmin local group on each SAP R/3 application server andDB server

• Assign the advanced user right Logon as a Service to the userSAPService<SID> on each SAP R/3 application server and DB server

Once installation is complete you should do the following to improve the securityof the system:

• Delete the SAP installation account that you created in step 23 (sapinst in ourlab)

4.11 SAP and DBMS installation

Step 24: Installing SAP and the DBMS

To make the installation as clear as possible, Figure 46 shows the major steps forthe database and SAP R/3 installation for each DBMS:


Figure 46. Installation steps for each DBMS

You will note from Figure 46 that the main difference between the installationprocedures is the point when the database cluster conversion occurs. Thesesteps are highlighted.

See OSS note 0138765 Cluster Migration: Terminology and Procedure for furtherinformation.

A detailed description of installation is provided in the next three chapters.

Depending on the database you are using, proceed as follows:

• If you are installing SAP with Oracle go to Chapter 5, “Installation usingOracle” on page 129.

• If you are installing SAP with DB2 go to Chapter 6, “Installation using DB2” onpage 145.

• If you are installing SAP with SQL Server go to Chapter 7, “Installation usingSQL Server” on page 159.

Preliminary work

Oracle install

Install SAP R/3 CI&DIR3SETUP -f CENTRDB.R3S

Install R/3 setup toolR3SETUP

Install R/3 files for conversionR3SETUP -f NTCLUSCD.R3S

SAP cluster conversionR3SETUP -f NTCMIGNx.R3S

Complete MSCS migrationR3SETUP -f UPDINSTV.R3S

Oracle cluster conversion

Steps for an Oracleinstallation

Preliminary work

DB2 install






DB2 cluster conversion

Steps for a DB2installation

Preliminary work

SQL Server install






SQL Server cluster conversion

Steps for an SQL Serverinstallation

5170

-01

The whole installation must be done logged on using the same installation useraccount as described in step 23 on page 107 (the only exception is step 24.15on page 138). This account must be a member of the Domain Admins group.See OSS note 134135 4.5B R/3 Installation on Windows NT (General) foradditional information.

Must always be logged on as the installation user


4.12 SAP verification

Now the SAP R/3 and DBMS installation in a non-cluster environment iscomplete. Moving all the shared disks to one node you can test SAP andeverything should work.

Step 25: SAP tests

This is a very simple list of tests that should be performed on the SAP system:

• Connection test• SAP system check• System log analysis• Profile check• SAPNTCHK

4.12.1 Connection testInstall the SAP GUI interface and log on as one of the SAP standard users:

Table 22. Client connection tests

The purpose of the tests is to confirm that it is possible to connect to the SAPserver from a client in which the SAP GUI has been installed and exploit theserver’s services. If you cannot connect to the SAP application server you cangive up your SAP system.

4.12.2 SAP system checkThere is a simple tool to check the SAP installation. Log on to the R/3 systemusing the client 000 and then use the commands Tools > Administration >Administration > Installation Check or use transaction code SM28. Ifeverything goes well you should receive the message:

SAP System Check. No errors reported

4.12.3 System log checkIn each application server, a system log is continually updated. The log file is\USR\SAP\<sid>\<instance>\SLOG00.LOG. It is possible to examine this log froma GUI using the commands Tools > Administration > Monitor > System Log ortransaction code SM21. The screen asks you to provide the time interval youwant to consider. Figure 47 shows a log with many CPIC errors.

Client User Default password

000 SAP* 06071992

000 DDIC 19920706

001 SAP* 06071992

001 DDIC 19920706

066 EARLYWATCH SUPPORT


Figure 47. System log showing CPIC errors

These errors can be analyzed by double-clicking them.

4.12.4 Profiles checkR/3 has three profiles stored on the directory \USR\SAP\<sid>\SYS\PROFILEduring the installation:

• DEFAULT.PFL (lab value: DEFAULT.PFL)

• START_<INSTANCE><INSTANCE_NUMBER>_<SAP_RESOURCE_NAME>(lab value: START_DVEBMGS00_ITSSAP)

• <SID>_<INSTANCE><INSTANCE_NUMBER>_<SAP_RESOURCE_NAME>(lab value: ITS_DVEBMGS00_ITSSAP)

The DEFAULT profile contains settings that apply to the entire system, while theSTART and SID profiles are instance specific. The START profile determines howthe individual R/3 services or processes are started. The SID profile determineshow many work processes of each type are started.

Profiles can be imported using the commands Tool > CCMS > Configuration >Profiles Maintenance (or Transaction Code RZ10) and then choosing Utilities >Import Profiles > Of Active Servers. During the import a consistency check isperformed. You can then look in the profile log to see if any error message isdisplayed (Figure 48):


Figure 48. Profile log

4.12.5 Processes checkSelecting transaction SM51 or clicking Tool > Administration > Monitor >System Monitoring > Process Overview, a screen similar to Figure 49 appears:


Figure 49. Process overview

A healthy system has many waiting dialog process as there are in Figure 49.

4.12.6 SAPWNTCHKAs described in OSS note 0065761 Determine cOnfiguration Problems underWindows NT, it is possible to download from Sapserv<x> FTP servers (see B.3,“Related Web sites” on page 183 for details) a couple of utilities, SAPNTCHK andSAPWNTCHK, to test the SAP configuration. The only difference between theseutilities is in the graphic interface. Detailed information about the utilities iscontained in the readme files enclosed. As an example of the result, Figure 50shows the output of SAPWNTCHK:

Figure 50. Output from SAPWNTCHK


4.13 DBMS verification

Step 26: DBMS verification

These tests strongly depend on the DBMS software. This section introduces thetests for the three database products.

4.13.1 OracleOracle verification: You can log on to the DBMS using Oracle EnterpriseManager. To log on you can use the standard Oracle count:

User ID: systemPassword: managerService: <SID>.worldConnect as: normal

You are then asked if you want to create the Oracle Repository and you have toanswer OK.

Oracle Fail Safe verification: You can log on to the Oracle Fail Safe managerusing the following account:

User ID: <sid>admPassword: ibmCluster alias: SAPCLUSDomain: SAPDOM

Then you can start the Oracle Fail Safe Cluster Verification utility by clickingTroubleshooting > Verify Cluster. Look at the log to see if there is any problem.

4.13.2 DB2See 5.8 “Verification of installation” in the redpaper SAP R/3 and DB2 UDB in aMicrosoft Cluster Server Environment, available from:

http://www.redbooks.ibm.com/redpapers.html

4.13.3 SQL ServerYou can log on to the DBMS using the SQL Enterprise Manager. This utility runsunder the MMC and should be installed on both nodes, and also on anadministrative desktop, so you can always access the database systemregardless of which node is running it.

Test the SQL Server installation as follows:

• Register the virtual database by clicking the button Register Server in thetoolbar, and providing the database vitual name (ITSSQL in our installation).

• Perform simple queries (from the sample pubs database, for example).

4.14 Backbone configuration

If you intend to use a network backbone for the connection between SAPapplication servers and the DB server, you need to carry out two additional steps.

http://www.redbooks.ibm.com/redpapers.html


Step 27: Backbone configuration

• Modify the IP address resource in the DB group to use the backbone insteadof the public network.

• Modify the application and DB server HOSTS file in a coherent way.

Figure 51 shows how to modify the Oracle cluster group:

Figure 51. Oracle cluster dependencies

These are the changes in the HOSTS file (Figure 52):

Figure 52. HOSTS file changes

Figure 53 shows how to modify the DB2 cluster group.

ITS.WORLD

Disk L:

Disk N:Disk K:

ORACLEITS IP Address

IP Addressresource

(CLUSRES.DLL)

ORACLEITSNetwork Name


(CLUSRES.DLL)Disk M:

Disk P:

Disk O:

Physical Diskresources(CLUSRES.DLL)

OracleTNSListener

80Fslitsora

Oracle Agent80ITSora


(CLUSRES.DLL)

Oracle Databaseresource(FSODBS.DLL)

Backbone ConfigurationReplace the public networkaddress 196.168.0.52 with thebackbone address 172.16.0.52

5170

-01


(CLUSRES.DLL)

127.0.0.1 localhost 192.168.0.1 servera SAPTRANSHOST192.168.0.2 serverb192.168.0.50 sapclus192.168.0.51 itssap #MODIFIED LINE#192.168.0.52 itsora10.0.0.1 serverai10.0.0.2 serverbi172.16.0.1 serverab172.16.0.2 serverbb172.16.0.52 itsora #ADDED LINE


Figure 53. DB2 cluster dependencies

Figure 54 shows the changes in the HOSTS file:

Figure 54. DB2 changes in the HOSTS file

Figure 55 shows how to modify the SQL Server Cluster group:

Figure 55. SQL Server cluster dependencies

DB2ITS

Disk L: Disk M:Disk K:


DB2IP Address

ITS

IP Address resource(CLUSRES.DLL)

DB2Network Name

ITS


DB2 resource(DB2WOLF.DLL)

Backbone ConfigurationReplace the public networkaddress 196.168.0.52 with thebackbone address 172.16.0.5251

70-0

1

127.0.0.1 localhost 192.168.0.1 servera SAPTRANSHOST192.168.0.2 serverb192.168.0.50 sapclus192.168.0.51 itssap#192.168.0.52 itsdb2 #MODIFIED LINE10.0.0.1 serverai10.0.0.2 serverbi172.16.0.1 serverab172.16.0.2 serverbb172.16.0.52 itsdb2 #ADDED LINE

Disk K:

Disk L:

Disk M:

Disk N:

Disk O:

ITSSQLIP Address

ITS SQLNetwork Name

ITSSQLVserver

ITSSQLSQL Server 7.0

ITSSQLServer Agent 7.0


IP Addressresource(CLUSRES.DLL)

Generic Serviceresource(CLUSRES.DLL)


SQL Server 7.0resource(SQLSRVRES.DLL)

SQL Server Agent 7.0resource(SQAGTRES.DLL)

Backbone configuration:Replace the public networkaddress 196.168.0.52 withthe backbone address172.16.0.52

5170

-01


Figure 56. HOSTS file for SQL Server

The structure of database resource dependency trees shown in Figure 51, Figure53, and Figure 55 is unrelated to the I/O technology (SCSI or FCAL) used toaccess the shared drives, but there is an important difference on the labels nearand inside the boxes describing the disks.

With FC-AL technology, disks are seen as Physical Disk resources and MSCSsees them by means of the Microsoft DLL CLUSRES.DLL. With ServeRAID,instead of this DLL, ServeRAID adapters use the IPSHA.DLL DLL and disks areseen as IBM IPSHA ServeRAID logical disk resources.

4.15 SAP cluster verification

Now that the installation is complete, it is time to verify that everything is working.We repeat the cluster tests that were done at the end of the cluster installation,although, now, we have three different cluster groups: Cluster, SAP and DB.These groups can be all in one node or spread between the nodes. All possibleconfigurations should be tested.

Due to installation errors it is possible to have a working system in all the possibleconfigurations but one. As a result, we strongly recommend you test all possibleconfigurations. If time does not allow you to test all configurations we stronglyrecommend you test at least the more problematic ones as listed in Table 23

Table 23. Minimum recommended test configurations (bold indicates expected node changes)

# Test Action Status before test Status after test X

1 Manual failover using the ClusterAdministrator

Move ClusterGroup from NodeA to Node B

Cluster Group: Node ASAP Group: Node ADB Group: Node B

Cluster Group: Node BSAP Group: Node ADB group: Node B


Move SAP Groupfrom Node A toNode B


Cluster Group: Node ASAP Group: Node BDB group: Node B

3 Manual failover using the ClusterAdministrator1

Move DB Groupfrom Node B toNode A


All groups on Node A


Move ClusterGroup from NodeB to Node A

Cluster Group: Node BSAP Group: Node BDB Group: Node A

Cluster Group: Node ASAP Group: Node BDB group: Node A

127.0.0.1 localhost 192.168.0.1 servera SAPTRANSHOST192.168.0.2 serverb192.168.0.50 sapclus192.168.0.51 itssap#192.168.0.52 itssql #MODIFIED LINE10.0.0.1 serverai10.0.0.2 serverbi172.16.0.1 serverab172.16.0.2 serverbb172.16.0.52 itssql #ADDED LINE



Move SAP Groupfrom Node B toNode A


Cluster Group: Node BSAP Group: Node ADB group: Node A

6 Manual failover using the ClusterAdministrator1

Move DB Groupfrom Node A toNode B


All groups on Node B

7 Regular shutdown of Node A Shut downNode A



8 Regular shutdown of Node B Shut downNode B



9 Regular shutdown of Node A Shut downNode A



10 Regular shutdown of Node B Shut downNode B



11 Blue screen on Node A2 Run KILL.EXE onNode A as pernote 2



12 Blue screen on Node B2 Run KILL.EXE onNode B per Note2



13 Blue screen on Node A2 Run KILL.EXE onNode A per Note2



14 Blue screen on Node B2 Run KILL.EXE onNode B per Note2



15 Power off of Node A Power off Cluster Group: Node ASAP Group: Node ADB Group: Node B


16 Power off of Node B Power off Cluster Group: Node ASAP Group: Node ADB Group: Node B


17 Power off of Node A Power off Cluster Group: Node BSAP Group: Node BDB Group: Node A


18 Power off of Node B Power off Cluster Group: Node BSAP Group: Node BDB Group: Node A


Notes:1 DB2 currently does not support the manual failover of the DB Group using the Cluster Administrator if there are active

connections open to the database.2 The blue screen can be obtained killing the WINLOGON.EXE process by means of the KILL.EXE utility contained in

the Windows NT Resource Kit.

# Test Action Status before test Status after test X


4.16 Tuning

Many books would be necessary to describe how to tune the SAP and DBMSservers. We cannot face these problems but a few recommendations can begiven.

4.16.1 Advanced Windows NT tuningThe most basic steps in operating system tuning have been described in 4.6,“Basic Windows NT tuning” on page 96. Now that the software installation iscomplete we can analyze performance and further improve the tuning. Some ofthe best sources for Windows NT information are:

• Optimization and Tuning of Windows NT Version 1.4 by Scott B. Suhy

• Inside Windows NT, Second Edition by David A. Salomon

4.16.2 General SAP tuningStep 28: SAP tuning

The main recommendations on SAP tuning are collected in the followingsubsections:

4.16.2.1 Distribute work processesSAP R/3 has six different types of work processes:

• Dialog (DIA)• Update V1 components (UPD)• Update V2 components (UP2)• Background (BTC)• Enqueue (ENQ)• Spooler (SPO)

To alter the number of each type of work process follow these steps:

1. Click Tools > CCMS > Configuration > Profile Maintenance.

2. Select Profile > Dyn. Switching > Display parameters.

3. Select the instance profile (in our lab: ITS_DVEMGS00_ITSSAP).

4. Click Basic Maintenance > Change.

See SAP R/3 System Administration on page 380 for details. General SAPrecommendations on how to distribute work processes and how to alter thenumber of work processes are contained in 4.16.2.2, 4.16.2.3, and 4.16.2.4 inthis redbook.

4.16.2.2 Run Dialog and Update work processes on dedicated serversTo avoid resource contention between Update and Dialog work processes andalso to allow specific server tuning, it is recommended to put dialog workprocesses and update work processes in different servers.

4.16.2.3 Run Enqueue work processes on a dedicated serverThe Central Instance server is the only SAP server in which Enqueue andMessage work processes run. Since this server is a focal point for all themessage flow between nodes and also because the overall performance of the


system depends on the speed of the locking activity of the Enqueue workprocess, it is recommended to use a dedicated server for Enqueue and Messagework processes.

4.16.2.4 Keep an optimal ratio between work processesAn excessively high or low number of work processes can decrease performance.General recommended ratios between work processes are:

• One Update work process of type V1 (UPD) is able to write the data comingfrom four Dialog work processes in the DB.

• One Update work process of type V2 (UP2) is able to write statistical datacoming from 12 Dialog work processes.

• One Background work process (BTC) is able to serve four Dialog workprocesses.

4.16.2.5 Distribute the users between the application serversUsers can be automatically distributed between application servers during thelogon phase, achieving a logon load balancing. This dynamic balancing can beobtained by accessing logon groups as described in Chapter 14, SAP R/3 SystemAdministration. Besides this, users should use SAPLOGON or SAP SessionManager. The SAPGUI does not allow you to exploit the logon groups. Here is ashort description of how logon load balancing works:

1. The user logs onto the SAP System by SAPLOGON or SAP Session Manager.

2. The user request is directed to the Message Server on the Central Instancenode.

3. The Message Server listens on the TCP port defined on the Services file inthe line containing the string sapms<SID>.

4. The Message Server logs the user on the SAP application server having thelowest load. To determine the load of the server, the Message Server uses twoparameters: the response time and the maximum number of users. Theseparameters can be configured by the SAP administrator (transaction codeSMLG).

If any R/3 instance belonging to the logon group has exceeded the maximum loadlimits, the limits are simply ignored.

The distribution of users by means of logon groups can improve performance butcan also worsen performance if you do not really need this distribution. Indeed,SAP application servers need to have ABAP preprocessed code in their buffers toavoid the preprocessing being done on a per-user basis. Moving users from oneserver to another can mean a move from a server with optimal buffers to anotherhaving nonoptimal buffers. For this reason, it is recommended you create logongroups on a per-module basis. For instance, if you have enough servers youcould create one logon group for FI/CO users and a second logon group for SDusers.

4.16.2.6 Operations modesOften, daily and night activities are different in an SAP system. The first one ismainly stressing Dialog work processes while the second mainly concerns Batchwork processes. To optimize the system for these different configurations you canuse operation modes. This technique allows you to configure the system in two


different ways and schedule the switch between these ways at predeterminedhours. This means you can have two operation modes with more dialogprocesses and more background processes and you can switch from the daymode to night mode at predetermined hours. A detailed description of how toconfigure operation modes is contained in Chapter 14 of SAP R/3 SystemAdministration.

4.16.2.7 Page file stripingImproving the page file I/O increases the overall SAP performance, so it isrecommended to stripe the page file as much as possible. You can either createup to 16 page files in different disks or use hardware technology like RAID-1enhanced and RAID-10 to obtain the striping. It is also important to create largepage files because this is the basis of zero administration memory management"as described in OSS note 0088416.

4.16.2.8 Memory managementAs of release 4.0, SAP exploits a new technique to manage the memory knownas zero administration memory management. This technique aims to make thetuning of memory parameters automatic. If you need to tune these parametersmanually see OSS note 0088416.

4.16.2.9 Network planningSplitting the network in two (a public one used by clients to connect to theapplication servers, and a backbone one used by application servers tocommunicate to the DB server) can improve performance. As shown in Figure 57,the backbone allows you to separate the dialog traffic between the SAPGUI, theMessage, Enqueue, and Dialog work processes from the update traffic betweenUpdate work processes and the DB server:

Figure 57. Message flow

PDC

PDC

APP1

DB CI

SAPDOMDomain

USERSDomain

Clients Clients[Mary]

5170

-01

SAPLOGON

DIA

UPD

Public network

Backbone network

Public network

MSG

Messagepath

Dialogpath

Updatepath

APP2


With the usual configuration exploiting logon load balancing, the traffic consists ofthe following main steps:

1. User Mary opens the connection using either SAPLOGON or SAP SessionManager.

2. The connection request is sent along the public network to the CentralInstance host.

3. The Message Service on the CI node receives the requests and sends theuser to the application server having the lowest load (in Figure 57).

4. The Dispatcher service of APP1 queues up the request until one of the Dialogwork processes becomes ready to serve a new request

5. When a Dialog work process becomes free the dispatcher retrieves therequest from the queue and assigns the request.

6. The Dialog work process rolls in the data, completes the request, and thenrolls out out the data.

7. The next user dialog step is still served by APP1, but not necessarily by thesame Dialog work process.

8. When the business transaction is complete the dialog service transfers thecontrol of the transaction to the update service by means of the ABAPstatement Commit Work.

9. The Update service selects one Update work process and transfers it to theupdate record containing the update request.

10.The update work process using the backbone network connects to the DB andpasses the information to the DBMS.

See 3.7, “Network configurations” on page 76 for further information. Theintroduction of a backbone network requires further steps after the clusterinstallation as described in 4.14, “Backbone configuration” on page 114. See alsoSAP document Network Integration of R/3 Servers (document number51006371).

4.16.3 SAP tuningSAP R/3 provides many tools allowing you to monitor and tune the SAP system,the most important of which is the Computing Center Control system. Thissection provides information on how to monitor the system to find out how toimprove performance.


4.16.3.1 Workload analysisWorkload analysis can be started from the SAPGUI by clicking Tools > CCMS >Control/Monitoring > Performance Menu > Workload> Analysis.

By clicking Oracle (or SQL or DB2, depending on the DBMS used, you cananalyze the DBMS. This corresponds to Transaction code ST04. Figure 58appears:

Figure 58. DBMS analysis (Oracle)


Alternatively, you can click ITSSAP to analyze the SAP application server. Thiscorresponds to dTransaction code ST03 as shown in Figure 59:

Figure 59. ST03 Workload analysis

The Av. wait time has to be no more than 1 percent of the average total responsetime. If this parameter is higher, either the number of work processes isinadequate or there is something blocking their activity.


4.16.3.2 Buffer Cache quality analysisBuffer analysis can be started by clicking Tools > CCMS > Control/Monitoring >Performance Menu > Setup/Buffers > Buffers. This corresponds to Transactioncode ST02 (see Figure 60):

Figure 60. ST02 Buffer analysis


4.16.3.3 Database reorganizationDatabase reorganization can be started by clicking Tools > CCMS > DBAdministration > DBA Planning Calendar or Transaction code DB13. Then it isnecessary to double-click the actual day. This produces Figure 61:

Figure 61. Database reorganization

You must then select the start time, period, calendar, and type of reorganization.

4.16.4 SAP tuning documentation and publicationsPrimary information sources for SAP tuning in the environment analyzed in thisredbook are:

• OSS note 0103747 Performance 4.0/4.5: Parameter Recommendations, anote collecting the recommendations on how to set SAP parameters as ofrelease 4.0A.

• OSS note 0088416 Zero aDministration Memory Management from 4.0A/N .

• OSS note 0110172 NT: Transactions with Large Storage Requirementsdescribing when and how to exploit the 3 GB feature of Windows NT EE.

For a general description of how SAP manages memory see:

• Memory Management as of R/3 Release 3.0C (SAP TechNet Web site). Thispaper has to be read together with OSS note 0088416.

Useful non-SAP documents about how to tune SAP systems are:

• Tuning SAP R/3 for Intel Pentium Pro Processor-based Servers RunningWindows NT Server 3.51 and Oracle7 Version 7.2.

• Netfinity Performance Tuning with Windows NT 4.0, SG24-5287.


4.16.5 DB tuningStep 29: DB tuning

For detailed information on how to tune the DBMS see:

• Oracle Architecture by S. Bobrowski, Chapter 10

• Oracle 8 Tuning by M. J. Corey et al, Chapter 3

• DB2 Universal Database and SAP R/3 Version 4 by Diane Bullock et al,Chapters 8 and 9.

• SAP R/3 Performance Tuning Guide for Microsoft SQL Server 7.0. fromhttp://support.microsoft.com

4.17 Configuring a remote shell

Step 30: Configure a remote shell

In an MSCS environment the SAP R/3 server and the database are runningseparately on each node of the cluster. Several R/3 transactions require thefunctionality to start an external program remotely on the database server. Astandard Windows NT environment does not support this remote commandexecution. Therefore, you have to use third party tools to provide this functionality.One recommended program is Ataman TCP Remote Logon Services. See theirWeb site:

http://www.ataman.com

Note: The Ataman software has to be installed on a local Windows NT disk.

See the following OSS notes:

• 0114287: SAPDBA in a Microsoft Cluster Server environment

• 0126985: Configuration of Ataman Remote Shell for DB2CS/NT

• 0132738: Using SAPDBA in MSCS or distributed environment

Tip: After setting up everything correctly on Node A perform the following steps tocopy the configuration to Node B:

1. Open command prompt on Node A.2. Change into Ataman directory.3. Execute the command auseradm dump >uconfig.cmd.4. Open command prompt on Node B after installing the Ataman software.5. Copy uconfig.cmd to Node B into the Ataman directory.6. Execute the command: uconfig.cmd.

For more details about configuring a remote shell for DB2, see 5.7.1 “Setting upremote function call” of the redpaper SAP R/3 and DB2 UDB in a MicrosoftCluster Server Environment available from http://www.redbooks.ibm.com.

http://support.microsoft.com

http://support.microsoft.com

http://www.ataman.com



Chapter 5. Installation using Oracle

In this chapter we describe the installation of SAP R/3 Release 4.5B on Oracle8.0.5.x. This material forms part of the overall installation as described in Chapter4, “Installation and verification” on page 85. Specifically, this chapter isreferenced in step 24 on page 108.

Figure 62. Installation process for SAP R/3 on SQL Server 7.0

A overall description of the installation process is shown in Figure 62. You willnote that some steps are to be done on Node A, some on Node B, and some areto be carried out on both nodes.

Before beginning the installation we recommend you read the followingdocuments:

• The continually updated OSS note 0134135 4.5 B R/3 Installation on WindowsNT (general).

• OSS note note 0114287 SAPDBA in a Microsoft Cluster Server environment.

Substeps described in this chapter:

Step 24.1: Fill in the installation worksheets, page 130

Step 24.2: Install Oracle on Node A, page 130Step 24.3: Install Oracle patch 8.0.5.1.1 on Node A, page 131Step 24.4: Install Oracle on Node B, page 131Step 24.5: Install Oracle Patch 8.0.5.1.1 on Node B, page 131Step 24.6: Install Oracle Fail Safe V2.1.3 on Node A, page 131Step 24.7: Install Oracle Fail Safe Patch 2.1.3.1 on Node A, page 132Step 24.8: Install Oracle Fail Safe V2.1.3 on Node B, page 132Step 24.9: Install Oracle Fail Safe Patch 2.1.3.1 on Node B, page 133

Step 24.18: Completing the migration to MSCS on Node A, page 143Step 24.19: Completing the migration to MSCS on Node B, page 144

Step 24.16: Converting the R/3 database to Fail Safe on A, page 139Step 24.17: Converting the R/3 database to Fail Safe on B, page 141

Step 24.14: Converting Node A for operation in the cluster, page 136Step 24.15: Converting Node B for operation in the cluster, page 138

Step 24.12: Install the cluster conversion files on Node A, page 135Step 24.13: Install the cluster conversion files on Node B, page 136

Step 24.10: Install R/3 setup tool on Node A, page 133

Step 24.11: Install SAP R/3 CI on Node A, page 133

B

A

A

B

Perform the operation on Node APerform the operation on Node B

A

B

A

A

B

B

A

A

A

B

A

B

A

B

A

B

Preliminary work

Oracle install






Oracle cluster conversion

Steps for an Oracleinstallation


• The document Checklist - Installation Requirements: Windows NT. The mainpoints of the checklist are discussed in 3.1, “Checklist for SAP MSCSinstallation” on page 40.

• Oracle: Conversion to Microsoft Cluster Server: Oracle 4.0B 4.5A 4.5B (doc.51005504).

• The continually updated OSS note 0134070 4.5 B Installation on Windows NT:Oracle containing a description of the known problems.

• The installation guide Oracle DBMS: R/3 Installation on Windows NT: OracleDatabase Release 4.5B (51004599).

5.1 Preliminary work

Step 24.1: Fill in the installation worksheets

Before you begin, you should fill in all the SAP installation worksheets in thischapter.

5.2 Oracle installation

Step 24.2: Install Oracle on Node A

To install Oracle on Node A, do the following:

1. Power on both Node A and Node B.

2. Move all cluster resources to Node A using Microsoft Cluster Administrator.

3. Log on to Node A as the installation user (in our lab, sapinst).

4. Insert in Node A the ORACLE RDBMS CD-ROM and start the Oracle Installerby running ORAINST.EXE from the \NT\I386\WIN32\INSTALL directory.

5. You are asked to provide many values we have collected in Table 24:

Table 24. Oracle installation


Company name IBM

Oracle Home: Name DEFAULT_HOME

Oracle Home: Location C:\orant

Oracle Home: Language English_SAP

Path update Leave path unchanged

The steps in this database chapter are substeps of step 24 on page 108 inChapter 4.

Step 24

Throughout the entire installation process, make sure you are always loggedon as the installation user (in our lab, sapinst). This user must be a domainadministrator as described in step 23.

Log on as the installation user (in our lab, sapinst)

Chapter 5. Installation using Oracle 131

At the end of the installation reboot the system.

Note: If you realize you have used the Administrator account instead of thesapinst one, you can correct the error with this simple procedure: log on assapinst and repeat all the installation steps of this subsection.

Step 24.3: Install Oracle patch 8.0.5.1.1 on Node A

To install the Oracle patch on Node A, do the following:


2. Move all cluster resources to Node A using Microsoft Cluster Administrator

3. From the Services applet in the Control Panel, stop all Oracle Services (itshould be necessary to stop only the OracleTNSListener80 service).

4. Insert the Oracle RDBMS CD-ROM and start the Oracle Installer program byrunning SETUP.EXE in the \NT\I386\PATCHES\8.0.5.1.1.\WIN32\INSTALLdirectory.

5. You are asked to provide many values we have collected in Table 25.

Table 25. Oracle patch installation

6. Ignore the request to run the scripts.

7. At the end of the installation reboot the system.

Step 24.4: Install Oracle on Node B

Repeat step 24.2 on page 130 for Node B.

Step 24.5: Install Oracle Patch 8.0.5.1.1 on Node B


Step 24.6: Install Oracle Fail Safe V2.1.3 on Node A

To install OFS on Node A, do the following:



Type of installation Oracle8 Enterprise Edition


Company name IBM




Software Asset Manager Select only Oracle8Server Patch 8.0.5.1.1

Oracle 8 Server Patchcomponents

Select all the components



3. From the Services applet in the Control Panel, stop all Oracle Services (itshould be necessary to stop only the OracleTNSListener80 service).

4. Insert the Oracle Fail Safe CD and start the Oracle installer program by doubleclicking the program ORAINST.EXE in the \NT\I386\WIN32\INSTALL directory.


Table 26. Oracle Fail Safe installation

6. If you are installing on Node B, ignore the Oracle Fail Safe discovery errormessage.

7. Continue on to step 24.7 without rebooting the system.

Step 24.7: Install Oracle Fail Safe Patch 2.1.3.1 on Node A

To install the OFS patch on Node A, do the following:

1. Insert the Oracle Fail Safe CD and start the Oracle installer program byrunning ORAINST.EXE in the \NT\I386\2131\WIN32\INSTALL directory.


Table 27. Oracle Fail Safe patch

3. At the end of the installation exit the Oracle installer and reboot the node.

Step 24.8: Install Oracle Fail Safe V2.1.3 on Node B



Company name IBM




Path update Accept path change

Software Asset Manager Select only Oracle Fail Safe Manager2.1.3.0.0 and Oracle Fail Safe Server2.1.3.0.0 under Available Products

Oracle Fail Safe docs Select all


Company name IBM




Path update Leave path unchanged

Software Asset Manager Select only Oracle Fail Safe Server2.1.3.1.0 from the list of availableproducts


Step 24.9: Install Oracle Fail Safe Patch 2.1.3.1 on Node B


5.3 Installation of the R/3 setup tool on Node A

Step 24.10: Install R/3 setup tool on Node A

To install R3SETUP, do the following:



3. Start the R3SETUP program from \NT\COMMON on the SAP KernelCD-ROM.

4. You are asked to provide many values you can take from Table 28:

Table 28. R/3 Setup tool installation

5.4 Installation of the R/3 Central Instance on Node A

Step 24.11: Install SAP R/3 CI on Node A

To install the Central Instance on Node A, do the following:


2. Start the program SAP R/3 Setup - Central & Database Instance from theSAP R/3 program menu. This is equivalent to using the command:

R3SETUP.EXE - f CENTRDB.R3S

3. You are asked to provide many values you can take from Table 29. Whileinserting drive letters you must always use uppercase followed by the colonsymbol, for example "J:".

Table 29. R/3 Central Instance installation

Values Restrictions andrecommendations

Lab values Your values

SAP system name(SAPSYSTEMNAME)

See R/3 Installation onWindows NT: OracleDatabase Release 4.5B. page 4-5

ITS

Installation directorypath (INSTALLPATH)

Leave default c:\Users\itsadm\install

Do you want to logoff? (EXIT)

Yes

Parameters Restrictions andrecommendations


SAP System Name(SAPSYSTEMNAME)

See R/3 Installation on WindowsNT: Oracle Database Release 4.5B. page 4-5. Use uppercasecharacters.

ITS


Number of the central system(SAPSYSNR)

Any two-digit number between 00and 97

00

Drive of the \usr\sap directory(SAPLOC)

• On the shared disks• Not on the Quorum disks• Not on Oracle disks

J:

Windows NT domain name(SAPNTDOMAIN)

SAPDOM

Central transport host(SAPTRANSHOST)

SAPTRANSHOST

Character set settings(NLS_CHARACTERSET)

WE8DEC

Default \Oracle<SID> driveSAPDATA_HOME

Home directory for SAPDATA<x>files

N:

SAPDATA1 • On the shared disks• Not on the Quorum disks• Not on the SAPLOC disk• Not on the OrigLog disks• Not on the MirrorLog disks• Not on the Archive disks

N:

SAPDATA2 Same as for SAPDATA1 O:

SAPDATA3 Same as for SAPDATA1 P:

SAPDATA4 Same as for SAPDATA1 P:

SAPDATA5 Same as for SAPDATA1 N:

SAPDATA6 Same as for SAPDATA1 O:

OrigLogA Same as for SAPDATA1 K:

OrigLogB Same as for SAPDATA1 K:

MirrorLogA Same as for SAPDATA1 L:

MirrorLogB Same as for SAPDATA1 L:

SapArch Same as for SAPDATA1 M:

SapBackup K:

SapCheck K:

SapReorg L:

SapTrace L:

sapr3 account password ibm

RAM that is reserved to the R/3system (RAM_INSTANCE)

Change the default only if you installmultiple R/3 Systems in a singlehost

default

Empty directory to which theExport1-CD is to be copied

Leave default Leave default




4. Continue on to step 24.12 without rebooting the system.

5.5 Install the R3Setup files for cluster conversion

Step 24.12: Install the cluster conversion files on Node A

To install the R3Setup files required for cluster conversion, do the following. Whenyou begin these steps you should still be logged on Node A as sapinst.

1. Start the cluster conversion program by running NTCLUST.BAT in the\NT\COMMON directory on the CD ROM drive. Essentially the batch fileexecutes the following command:

R3SETUP - NTCLUSCD.R3S


Table 30. R3Setup files for the cluster conversion: Node A

3. Log on to Node B as the installation user (in our lab, sapinst)

4. Start the cluster conversion program by running NTCLUST.BAT in the\NT\COMMON directory on the CD-ROM drive.

Empty directory to which theExport2-CD is to be copied


Port number of the messageserver (PORT)

Leave default 3600 (default)

<sid>adm password ibm

SAPService<SID> password ibm

Do you want to use the SAPGateway? (R2_CONNECTION)

NO

Number of R3load processes(PROCESSES)

For Oracle, set this to 1 regardlessof the number of CPUs

5

Operating system platform forwhich the report loads will beimported

Must be Windows NT Windows NT




See R/3 Installation onWindows NT: OracleDatabase. Release 4.5 B.page 4-5. Use uppercasecharacters.

ITS




Yes





Table 31. R/3 setup files for cluster conversion: Node B

6. Continue on to step 24.13 without rebooting Node A.

Step 24.13: Install the cluster conversion files on Node B

Repeat step 24.12 for Node B.

5.6 SAP cluster conversion

Step 24.14: Converting Node A for operation in the cluster

Do the following on Node A:


2. Click Start > SAP R/3 Setup for ITS > SAP R3 Setup - Configuring Node Afor a MSCS.

This is the same as the command:

R3SETUP.EXE -f NTCMIGNA.R3S

3. You are asked to provide many values we have collected in Table 32:

Table 32. Conversion of Node A for cluster operation




See R/3 Installation onWindows NT: OracleDatabase Release 4.5B. page 4-5. Useuppercase characters.

ITS




Yes



Virtual host name of the R/3Cluster Group (NETWORKNAME)

ITSSAP

Virtual IP address of the R/3Cluster Group (IPADDRESS)

If the Hosts file has beenconfigured the proposed value isthe correct one.

192.168.0.51

Subnet mask for the virtual IPAddress for the R/3 Cluster group(SUBNETMASK)

255.255.255.0

Name of the public network usedfor the R/3 Cluster group(NETWORKTOUSE)

Name of the network to which thevirtual IP Address belongs asdefined in MSCS

Public



See R/3 Installation on WindowsNT: Oracle Database Release 4.5B. page 4-5. Use uppercasecharacters.

ITS


See Note 1 00

Drive of the \usr\sap directory(SAPLOC)

See Note 1 J:


See Note 1 SAPDOM

Virtual host name of the R/3Oracle Group (DBHOSTNAME)

Attention! Do not use the virtualhost name of the Oracle grouphere. Instead use the local hostname of the Node A.

servera

Character set settings(NLS_CHARACTERSET)

See Note 1 WE8DEC

Default \Oracle<SID> driveSAPDATA_HOME

See Note 1 N:

SAPDATA1 See Note 1 N:

SAPDATA2 See Note 1 O:

SAPDATA3 See Note 1 P:

SAPDATA4 See Note 1 P:

SAPDATA5 See Note 1 N:

SAPDATA6 See Note 1 O:

OrigLogA See Note 1 K:

OrigLogB See Note 1 K:

MirrorLogA See Note 1 L:

MirrorLogB See Note 1 L:

SapArch See Note 1 M:

SapBackup See Note 1 K:

SapCheck See Note 1 K:

SapReorg See Note 1 L:

SapTrace See Note 1 L:

sapr3 account password ibm


Change the default only if youinstall multiple R/3 Systems in asingle host.

default


Leave the default value. 3600 (default)




4. Exit from R3Setup.

5. Start R3Setup a second time.

6. Provide the password values as required.

Step 24.15: Converting Node B for operation in the cluster

1. When R3Setup has finished, log off and log on again to Node A as <sid>adm(in our lab itsadm), do not use sapinst here. (Note: You are logging on to NodeA here, not Node B).

2. Connect to the Oracle Instance Manager by clicking Start > OracleEnterprise Manager > Instance Manager. Use the following information:

User name: internalPassword: oracleService: ITS.WORLDConnect as: Normal

3. Shut down the instance as shown in Figure 63.


SAPService<sid> password ibm

Do you want to use the SAPGateway? (R2_CONNECTION)

No

Note 1: Value previously used.



The R3Setup program may stop at this point. You may see the error messageshown in Figure 78 on page 179. If this happen proceed as described in 8.5,“R3Setup” on page 179.

Possible error


Figure 63. Shutting down the instance

4. At the warning message click Yes.

5. Select Immediate in the Shutdown Mode window.

6. Move the Cluster Group and the Disk groups to Node B using MicrosoftCluster Administrator.

7. Take the SAP-R/3 ITS group offline and move it to Node B.

8. On Node B bring all the resources in the SAP-R/3 ITS group online except theSAP-R/3 ITS resource. You can do this by starting from the bottom of thedependency tree and following the dependencies up the tree.

9. Log on to Node B as itsadm.

10.Click Start > SAP R/3 Setup for ITS > SAP R3 Setup - Configuring Node Bfor a MSCS, which corresponds to launching the program.

R3SETUP.EXE -f NTCMIGNB.R3S

11.You are asked to provide many values you must take from Table 32. Do notchange any value in the table. We particularly recommend you be careful withthe DBHOSTNAME. You must use servera again, not serverb.

5.7 Oracle cluster conversion

Step 24.16: Converting the R/3 database to Fail Safe on A

Do the following to convert the database installed on Node A to a Fail Safedatabase:


1. Shut down Node A.

2. Shut down Node B.

3. Restart Node A.

4. Restart Node B.


6. Restart the Oracle Instance by clicking Start > Oracle Enterprise Manager >Instance Manager and connect using these values:

User name: internalPassword: oracleService: ITS.WORLDConnect as: Normal

7. Select Database Open as shown in Figure 64:

Figure 64. Starting the instance on Node A

8. Your are asked to provide the local parameter file as shown in Figure 65.Select initITS.ora


Figure 65. Specify the local parameter file

Step 24.17: Converting the R/3 database to Fail Safe on B

Do the following to convert the database on Node B to a Fail Safe database:

1. Copy the INIT<SID>.ORA file (in our lab, INITITS.ORA) from\<Oracle_Home>\Database (in our lab c:\orant\database) in Node A to thesame directory on Node B.

2. Click on Start > Programs > Oracle Enterprise Manager > Oracle Fail SafeManager and connect using the values in Table 33.

Table 33. Connecting to Fail Safe Manager

3. Create the Fail Safe group ORACLE<SID> (in our lab ORACLEITS). In OracleFail Safe Manager, click Groups > Create.

Provide information from Table 34.

Table 34. Conversion of Oracle DB to a Fail Safe DB: Table 1

Parameters Meanings Lab values Your values

User Name User ID of the account used for theCluster Server service

ClusSvc

Password Password of the account used for theCluster Server service

ibm

Cluster Alias Cluster Group network name SAPCLUS

Domain Windows NT domain SAPDOM


Group Name Name of the Oracle ClusterGroup

ORACLEITS

Network Network used to connect to theOracle Cluster Group

Public

Network name Oracle Cluster Group: NetworkName

ITSORA

IP address Oracle Cluster Group: IPAddress

192.168.0.52


4. In Oracle Fail Safe Manager click SAPCLUS > Databases > StandaloneDatabases, right-click ITS.WORLD and select Add to Group.

5. Provide the following values:

Table 35. Conversion of a Oracle DB to a Fail Safe DB: Table 2

6. Confirm the operation Add Database to Fail Safe Group and click Yes.

7. When the Add Agent to Group “OracleITS” window appears select M: (this isthe SAP Archive drive in our lab) as the disk for the Oracle Fail Saferepository.

8. Move the SAP cluster group to Node B using Microsoft Cluster Administrator.

9. Using either SAPPAD.EXE or Notepad, open the fileJ:\usr\sap\Its\sys\profile\Default.pfl and modify the SAPDBHOST line asshown in Figure 66. Change servera to itsora as shown.

Subnet Mask Oracle Cluster Group: SubnetMask

255.255.255.0

Failover Period Leave default 6 Hours

Failover Threshold Leave default 10 times

Failback mode Prevent failback Prevent failback

Values Meanings Lab values Your values

Group Name ORACLEITS

Service Name ITS.WORLD

Instance Name ITS

Database Name <SID> ITS

Parameter file \<Oracle_Home>\database\init<SID>.ora where<Oracle_Home> isenvironment variable

c:\orant\database\initITS.ora

Account to access thedatabase

Internal Internal

Password oracle oracle

Oracle Fail Safe Policy:Pending Timeout

Leave default 180 sec

Oracle Fail Safe Policy:“Is Alive” interval

Leave default 60000 millisecs

Oracle Fail Safe Policy:The cluster softwarehas to restart the DB incase of failure

Yes



Figure 66. Changing DEFAULT.PFL

For troubleshooting, detailed information can be found in Oracle Fail SafeConcepts and Administration Release 2.0.5 (doc. A57521-01). Particularlyrelevant are:

• The description of the modifications in the TNSNAMES.ORA andLISTENER.ORA files in Section 2.5.

• The Troubleshooting chapter, Chapter 6.

• Appendix B, containing the list of Oracle Fail Safe Messages.

5.8 Completing the migration to an MSCS

Step 24.18: Completing the migration to MSCS on Node A

Do the following:


2. Using Microsoft Cluster Administrator:

– Move the Quorum Disk resource to the Cluster Group if have not alreadydone that.

– Delete the empty Cluster Groups.– Move all the Cluster Groups to Node A.– Take the SAP-R/3 ITS resource online on Node A.

3. Click Start > Programs > SAP R/3 Setup for ITS > SAP R/3 Setup -Completing the Migration to an MSCS (Instvers), which corresponds to thecommand:

R3Setup.exe -f UPDINSTV.R3S


4. You are asked to provide some values we have collected in Table 36.

Table 36. Completion of the MSCS migration on Node A

5. Reboot Node A.

Step 24.19: Completing the migration to MSCS on Node B

Do the following:

6. Log on to Node B as the installation user (in our lab, sapinst).

7. Move all cluster groups from Node A to Node B using Microsoft ClusterAdministrator.

8. Click on Start > Programs > SAP R/3 Setup for ITS > SAP R/3 Setup -Completing the Migration to an MSCS (Instvers), which corresponds tousing the command:

R3Setup.exe -f UPDINSTV.R3S

9. You are asked to provide some values we have collected in Table 37.

Table 37. Completion of the MSCS migration on Node B

10.Reboot Node B.

Return to step 25 on page 110 to continue the installation.


Network Name for ClusterResource Central Instance(CIHOSTNAME)

ITSSAP


See R/3 Installation onWindows NT: OracleDatabase Release 4.5 B.page 4-5. Use uppercasecharacters.

ITS

Number of the centralsystem (SAPSYSNR)

Any two digit numberbetween 00 and 97

00


Network Name for ClusterResource Central Instance(CIHOSTNAME)

ITSSAP


See R/3 Installation onWindows NT: OracleDatabase Release 4.5 B.page 4-5. Use uppercasecharacters.

ITS


Any two-digit numberbetween 00 and 97.

00


Chapter 6. Installation using DB2

In this chapter we describe the installation of SAP R/3 Release 4.5B on DB2RDBMS 5.2 patch level 7. This material forms part of the overall installation asdescribed in Chapter 4, “Installation and verification” on page 85. Specifically, thischapter is referenced in step 24 on page 108.


An overall description of the installation process is shown in Figure 67. You willnote that some steps are to be done on Node A, some on Node B, and some areto be carried out on both nodes.

The main references for the installation are:

• R/3 Installation on Windows NT DB2 Common Server Release 4.5B (doc.5100 5502).

• Conversion to Microsoft Cluster Server: DB2 Universal Database Server 4.5B(doc. 5100 6418).

• The redpaper, SAP R/3 and DB2 UDB in a Microsoft Cluster Environment,available from http://www.redbooks.ibm.com.



Step 24.2: Install DB2 on Node A, page 146Step 24.3: Install DB2 FixPak on Node A, page 147Step 24.4: Dropping the sample DB and rebooting Node A, page 148Step 24.5: Install DB2 on Node B, page 148Step 24.6: Install DB2 FixPak on Node B, page 149Step 24.7: Dropping the sample DB and rebooting Node B, page 149

Step 24.8: Create a new source for the SAP Kernel CD-ROM, page 149Step 24.9: Check/Correct Conversion R3S Files, page 150Step 24.10: Install the R3SETUP tool on Node A, page 150

Step 24.17: Migrating MSCS on Node A, page 156Step 24.18: Check the Services file on Node A, page 157Step 24.19: Check the DB2<SID> Service settings on Node A, page 157Step 24.20: Check the Services file on Node B, page 158Step 24.21: Check the DB2<SID> Service settings on Node B, page 158

Step 24.15: Converting Node A for operation in a cluster, page 154Step 24.16: Converting Node B for operation in a cluster, page 155

Step 24.13: Install R3Setup files for cluster conversion on A, page 153Step 24.14: Install R/3 files for cluster conversion on Node B, page 154

Step 24.11: Install the CI and DI on Node A, page 151

Step 24.12: Modify DB2MSCS.CFG & run DB2MSCS on Node A, page 152

A

B

A

A

B


Preliminary work

DB2 install






DB2 cluster conversion

Steps for a DB2installation

A

A

B

B

A

A

A

A

B

A

B

A

A

A

B

B

A



• DB2 Universal Database and SAP R/3 Version 4, SC09-2801.

• The continually updated OSS note 0134135 4.5 B R/3 Inst. on Windows NT(general).

• Checklist - Installation Requirements: Windows NT. The main points of thechecklist are discussed in 3.1, “Checklist for SAP MSCS installation” on page40.

• The continually updated OSS note 0134159 4.5 B Installation on Windows NT:DB2/CS.



Before you begin, you should fill in all the SAP installation worksheets in thischapter.

6.2 DB2 installation

Step 24.2: Install DB2 on Node A

1. Check SAPNET for more DB2 information by viewing OSS Notes 0134135 and0134159.

2. Log on as the installation user (in our lab, sapinst).

3. Insert the DB2 RDBMS CD-ROM and start the DB2 Installer by runningSETUP.EXE in the \NT_i386\DBSW directory. Figure 68 appears.

Throughout the entire installation process, make sure you are always loggedon as the installation user (in our lab, sapinst). This user must be a domainadministrator as described in step 23.


Do not restart the system until after dropping the DB2 database. If you dorestart the system, results are unpredictable and you may have to restart theinstallation process.

Attention


Chapter 6. Installation using DB2 147

Figure 68. Selection of products

4. Follow the installation process to install DB2 on Node A. Use Table 38 to helpyou enter all the installation parameters.

Note: Do not reboot Node A at the end of the installation.

Table 38. DB2 installation values for Node A

Step 24.3: Install DB2 FixPak on Node A

1. Insert the DB2 RDBMS CD and start the DB2 Installer by running SETUP.EXEin the \NT_i386\FIXPAK directory. Figure 69 appears.


Select products DB2 Universal DatabaseEnterprise Edition

Installation type Typical

Destination directory Leave default. DestinationDrive needs to be a localnon-shared drive.

db2admin and password Leave default name. Enterpassword

Restart system No


Figure 69. Start options

2. Deselect both values.

3. When prompted, do not reboot the server.

Step 24.4: Dropping the sample DB and rebooting Node A

1. Log off and log on as the installation user (in our lab, sapinst) to enable thechanges to the environment variables such as the search path.

2. Open a command prompt.

3. Remove the default database instance of DB2 using the command below.

C:\DB2IDROP DB2

4. Reboot Node A.

Step 24.5: Install DB2 on Node B

1. Insert the DB2 RDBMS CD and start the DB2 Installer by running SETUP.EXEin the \NT_i386\DBSW directory.

2. Follow the installation process to install DB2 on Node B. Use Table 39 to helpyou enter all the installation parameters.

Note: Do not reboot Node B at the end of the installation.

Table 39. DB2 Installation values for Node B

Parameter Lab values Your values

Select products DB2 Universal DatabaseEnterprise Edition

Installation type Typical

Do not restart the system until after dropping the DB2 database. If you dorestart the system, results are unpredictable and you may have to restart theinstallation process.

Attention


Step 24.6: Install DB2 FixPak on Node B

1. Insert the DB2 RDBMS CD and start the DB2 Installer program by doubleclicking the program SETUP.EXE in the \NT_i386\FIXPAK directory. Figure 69on page 148 appears.

2. Deselect both values.

3. When prompted, do not reboot the server.

Step 24.7: Dropping the sample DB and rebooting Node B

1. Log off and log on as the installation user (in our lab, sapinst) to enable thechanges to the environment variables such as the search path.

2. Open a command prompt.

3. Remove the default database instance of DB2 using the command below.

C:\DB2IDROP DB2

4. Reboot Node B.

6.3 Install the R3Setup tool on Node A

Step 24.8: Create a new source for the SAP Kernel CD-ROM

If it exists (there may not be any updates), an updated version ofR3SETUPCOMMON_1.CAR may be found at:

ftp://sapserv3/general/R3server/patches/rel45B/NT/<PROC>/DB6

Note: If the update exists, you must apply it.

See B.5, “SAPSERV FTP site” on page 186 for details on how to access this FTPsite. Always obtain the most current version of the package files in case there arebug fixes. Download the file with the highest version number. If no file is availablefor your platform and database, you do not need one.

Note: The R3Setup profiles delivered on the 4.5B CD-ROMs from SAP areincorrect and you will need to download the latest version of them as well.

1. Create a directory C:\SAPKERNEL.

2. Copy the entire contents of the SAP Kernel CD-ROM into this directory.

3. Create a SAPPATCHES subdirectory.

4. Download the above CAR file from sapserv into the SAPPATCHES directory.

5. Uncompress the R3SETUPCOMMON_1.CAR file into the SAPPATCHESdirectory using the command:

car -xvf R3SETUPCOMMON_1.car

Destination directory Leave default

db2admin and password Leave default name

Restart system No

Parameter Lab values Your values


6. Copy the content of the SAPPATCHES directory intoC:\SAPKERNEL\NT\COMMON.

7. This is now the location of the new Kernel CD when the R3Setup program askfor the Kernel CD.

Step 24.9: Check/Correct Conversion R3S Files

The files that convert the stand-alone server to a cluster is administered by twofiles called NTCMIGNA.R3S for Node A and NTCMIGNB.R3S for Node B. Thesefiles are located in the C:\SAPKERNEL\NT\COMMON directory.

Open each file with C:\SAPKERNEL\NT\I386\SAPPAD.EXE. Verify the l(lowercase L) is present in the DB2INSTANCE variable as shown in Figure 70:

Figure 70. NTCMIGNA.R3S and NTCMIGNB.R3S — changes required

Step 24.10: Install the R3SETUP tool on Node A


2. Move all cluster resources to Node A.

3. Explore the SAPKERNEL CD directory and start the R3SETUP program bydouble clicking the program R3SETUP.BAT in theC:\SAPKERNEL\NT\COMMON directory.

4. Proceed with the installation using Table 40 for the installation parameters.

Table 40. Values for the installation of R3SETUP on Node A

Value Restrictions andrecommendations

Lab value Your values

SAP system nameSAPSYSTEMNAME

See R/3 Installation onWindows NT: DB2 Database.Release 4.5 B. page 4-5

ITS

Path to installation directoryINSTALL PATH

Leave default Default

Do you want to log off?CDINSTLOGOFF_NT_IND

Yes

[DB_ENV]DB2DBDFT=@SAPSYSTEMNAME@DB2DB6EKEY=@DB2DB6EKEY@DB6EKEY=@DB2DB6EKEY@DSCDB6HOME=@CIHOSTNAME@DB2INSTANCE=db2l@LOWER_SAPSYSTEMNAME@DB2CODEPAGE=819

[INST_ENV]DB2DBDFT=@SAPSYSTEMNAME@DB2DB6EKEY=@DB2DB6EKEY@DB6EKEY=@DB2DB6EKEY@DSCDB6HOME=@CIHOSTNAME@DB2INSTANCE=db2l@LOWER_SAPSYSTEMNAME@DB2CODEPAGE=819

Ensure there is a lowercase Lin each of these two locationsin both files. If the lowercase Lis not there, enter one.

Note: Later versions of thefiles may have this corrected.


6.4 Install the Central Instance and Database Instance

Step 24.11: Install the CI and DI on Node A

1. Click on Start > Programs > SAP R3 Setup > SAP R3 setup - Central &Database Instance.

2. Proceed with the installation using Table 41 on page 151 for the installationparameters.

Table 41. Values for the installation of Central Instance and Database Instance




See R/3 Installation on WindowsNT: DB2 Database Release 4.5 B.page 4-5

ITS


Any two digit number between 00and 97

00

Drive of the \usr\sap directorySAPLOC

On the shared disksNot on the Quorum disksNot on DB2 disks

J: (SAP softwareshared disk)


SAPDOM

Central transport hostSAPTRANSHOST

SAPTRANSHOST

Encryption KeyDB2DB6EKEY

<SID><servername> ITSservera

DB2<sid> password ibm


SAPServer<SID> password ibm

DB2 instance directoryDFTDBPATH

LOG_ARCHIVE directory onshared disks

L:

DIAGPATH L:

LOGDIR_DRIVE LOG_DIR directory on shareddisks

K:

LOGARCHIVEDIR_DRIVE LOG_ARCHIVE directory onshared disks

L:

SAPREORGDIR_DRIVE SAPREORG subdirectory on thesame drive as the LOG_ARCHIVEdirectory (shared disks)

L:

SAPDATA1_DRIVE On the shared disksNot on the Quorum disksNot on SAPLOC diskNot on the LOG_DIR disksNot on the LOG_ARCHIVE disks

N:

SAPDATA2_DRIVE Same as above O:

SAPDATA3_DRIVE Same as above P:


6.5 DB2 cluster conversion

Step 24.12: Modify DB2MSCS.CFG & run DB2MSCS on Node A

On Node A, do the following:


2. Copy the file DB2MSCS.CFG from the C:\SAPKERNEL\NT\Common directoryto the install directory, C:\USERS\<sid>adm\INSTALL.

3. Edit DB2MSCS.CFG using the SAPPAD utility. Figure 71 shows themodifications we did for our lab environment.

SAPDATA4_DRIVE Same as above P:

SAPDATA5_DRIVE Same as above N:

SAPDATA6_DRIVE Same as above O:

sapr3 user password ibm


default

Kernel CD location This is the copy of the CD on thelocal drive.

C:\SAPKERNEL\

Port number of the message server(PORT)

Leave the default value 3600(default)

Database Services(PORT)

Leave the default value 5912(default)

R2_Connection No

SMSTEMP No

Number of R3load processes(PROCESSES)

• When RAM is ≤ 512 MB, use 2.• When RAM is > 512 MB, use a

value equal to the number ofCPUs.

4

SAP gateway No

REPORT_NAMES Windows NT




Figure 71. DB2MSCS.CFG modifications

4. Stop the service DB2<SID> by issuing DB2STOP at a command prompt.

Note: If executing the command DB2STOP you may get the error: SQL1390CThe environment variable DB2INSTANCE is not defined or is invalid. If thishappens, check the value of the environment variable DB2INSTANCE. If thisvalue is still DB2 then you must do the following:

a. Delete the environment variable DB2INSTANCE from the systemenvironment using the System applet in Control Panel.

b. Set the value for DB2INSTANCE in your current command prompt sessionto DB2<SID> using the command:set DB2INSTANCE=DB2<SID>

c. Re-execute the DB2STOP command.

5. Change to the C:\USERS\<SID>ADM\INSTALL directory.

6. Run the utility DB2MSCS.

Note: The values for the DISK_NAME and INSTPROF_DISK are not onlydependent on the drive letter, but also the type of disk subsystem used. For ourexample we used a Fibre Channel controller, thus there was no need for theprefix IPSHA.

6.6 Install R3SETUP files for cluster conversion

Step 24.13: Install R3Setup files for cluster conversion on A

• Open the explorer to C:\SAPKERNEL\NT\Commom, start the conversionprogram by double-clicking the program NTCLUST.BAT.

Table 42. Values for the cluster conversions on Node A


System Name ITS

Install Directory c:\users\itsadm\install

Logoff Yes

DB2MSCS.CFG before modifications

CLUSTER_NAME=<cluster name>GROUP_NAME=DB2 <SAPSID> GroupDB2_INSTANCE=DB2<SAPSID>IP_NAME=DB2 IP <SAPSID>IP_ADDRESS=<virtual IP address of DB2 group>IP_SUBNET=<subnet mask of DB2 group>IP_NETWORK=<network named used to communicate>NETNAME_NAME=DB2 NetName <SAPSID>NETNAME_VALUE=<hostname of DB2 group>DISK_NAME=<resource name of shared disk containing

database files>DISK_NAME=<...more shared disks>INSTPROF_DISK=<resource name of the shared disk

where DBDFTPath is pointing to>

DB2MSCS.CFG with lab modifications

CLUSTER_NAME=SAPCLUSGROUP_NAME=DB2 ITS GroupDB2_INSTANCE=DB2ITSIP_NAME=DB2 IP ITSIP_ADDRESS=192.168.0.52IP_SUBNET=255.255.255.0IP_NETWORK=PUBLICNETNAME_NAME=DB2 NetName ITSNETNAME_VALUE=ITSDB2DISK_NAME=DISK K:DISK_NAME=DISK L:DISK_NAME=DISK M:DISK_NAME=DISK N:DISK_NAME=DISK O:DISK_NAME=DISK P:INSTPROF_DISK=DISK L:


Step 24.14: Install R/3 files for cluster conversion on Node B

• Copy the new SAPKERNEL directory from Node A to Node B.

• Open the explorer to C:\SAPKERNEL\NT\Commom, start the conversionprogram by double-clicking the program NTCLUST.BAT.

Table 43. Values for the cluster convention on Node B


Step 24.15: Converting Node A for operation in a cluster

1. Click Start > Program > SAP R/3 Setup > Configuring Node A for a MSCS.

2. Proceed with the conversion using Table 44 for the parameters.

Table 44. Values for converting Node A to MSCS


System Name ITS

Install Directory c:\users\itsadm\install

Logoff Yes



Virtual Name for the R/3Setup Group

ITSSAP

SAP R/3 Group IPAddress

192.168.0.51

Subnet Mask 255.255.255.0

Network to Use Default; Public


See R/3 Installation onWindows NT: DB2Database Release 4.5 B.page 4-5

ITS


Any two-digit numberbetween 00 and 97

00

Drive of the \usr\sapdirectory SAPLOC

• On the shared disks• Not on the Quorum disks• Not on DB2 disks

J: (SAPsoftwareshared disk)

When you initially launch the Configuring Node A for an MSCS program,youwill get the following error message:

Error: OSUSERSIDADMRIGHTS_NT_DB6 installationDoPhase failed

You get this because the R/3 setup program does not wait for the resources togo online. You will need to launch the program a second time.

Error message


Step 24.16: Converting Node B for operation in a cluster


2. Take SAP_R/3 <SID> offline. (Note: only take this resource offline.)

3. Move all resources to Node B and bring online.

4. Click on Start > Program > SAP R/3 Setup > Configuring Node B for anMSCS.

5. Enter the values as per Table 45.

Table 45. Values for converting Node B to MSCS

Windows NT domainname SAPNTDOMAIN

SAPDOM

Hostname of R/3Database Server(DBHOSTNAME)

See Figure 71 on page 153 ITSDB2

Encryption KeyDB2DB6EKEY

<SID><servername> ITSservera

DB2<sid> password ibm


RAM that is reserved tothe R/3 system(RAM_INSTANCE)

Default

SAPService<SID>password

ibm

Gateway R2_Connection No



Virtual Name for the R/3Setup GroupNETWORKNAME

ITSSAP

SAP R/3 Group IPAddress IPADDRESS

192.168.0.51

Subnet Mask 255.255.255.0

Network to UseNETWORKTOUSE

DefaultPublic



ITS



00




6.8 Complete the Migration to MSCS

Step 24.17: Migrating MSCS on Node A

Note: This section replaces the section “Completing the Cluster Conversion” inthe SAP manual Conversion to Microsoft Cluster Server: DB2 Universal DatabaseServer.

The SAP R/3 system contains a table INSTVERS where the R/3 server name andthe latest installation status are stored. During each startup SAP R/3 checks if thecurrent R/3 server exists in this table and if the installation status is correct. If theserver name is not found or the installation status is invalid, the system shows anerror message saying that the installation was not completed successfully.

During the initial installation of the non-clustered R/3 system an entry for serverNode A was made with the correct status. After the conversion to a clustered R/3system, the name of the R/3 server is changed to the virtual name, for example,RSDR3. To avoid the error message, an entry for this virtual server name has tobe created in the table INSTVERS. This can be done with the help of R3SETUPin the following way:


2. Move all resources to Node A and bring online.

3. The R3SETUP template file UPDINSTV.R3S is in your installation directory,for example, C:\users\<sid>adm\Install. Verify the following lines:

[EXE]10=RFCUPDATEINSTVERS_IND_IND

Drive of the \usr\sapdirectory (SAPLOC)

• On the shared disks• Not on the Quorum disks• Not on DB2 disks

J: (SAPsoftwareshared disk)

Windows NT domainname (SAPNTDOMAIN)

SAPDOM

Hostname of R/3Database Server(DBHOSTNAME)

See Figure 71 on page 153 ITSDB2

Encryption Key(DB2DB6EKEY)

<SID><servername> ITSserverb

DB2<SID> password ibm


Port number of themessage server (PORT)

Leave the default value 3600

Database Services port Leave the default value 5912

SAPService<SID>password

ibm

SAPR3 Password ibm




[RFCUPDATEINSTVERS_IND_IND]CLASS=CRfcJobRFCREPNAME=NORFCSTEP=3CIHOSTNAME=SAPSYSTEMNAME=SAPSYSNR=CONFIRMATION=CIHOSTNAME SAPSYSTEMNAME SAPSYSNRStart / verify the R/3 system is running.

4. Click Start > Program > SAP R/3 Setup > Completing the Migration toMSCS.

Table 46. Values for completing the migration to MSCS

Step 24.18: Check the Services file on Node A

The file C:\WINNT\System32\drivers\etc\Services on both nodes need to containthe lines below if the defaults were taken for the port numbers:

Step 24.19: Check the DB2<SID> Service settings on Node A

The service DB2<SID> needs to be set to manual and is to be administered bydb2<sid>. This is because this service is at the cluster level and cannot berunning on both nodes at the same time in the same cluster.

Parameter Restrictions andrecommendations


Virtual Name for the R/3Setup Group(CIHOSTNAME)

ITSSAP



ITS



00

sapdp00 3200/tcpsapdp00s 4700/tcpsapgw00 3300/tcpsapgw00s 4800/tcpsapmsITS 3600/tcpsapdb2ITS 5912/tcpsapdb2ITSi 5913/tcp


Figure 72. DB2ITS Service

Step 24.20: Check the Services file on Node B

Repeat step 18 on page 157 for Node B.

Step 24.21: Check the DB2<SID> Service settings on Node B

Repeat step 19 on page 157 for Node B.



Chapter 7. Installation using SQL Server

In this chapter we describe the installation of SAP R/3 Release 4.5B usingMicrosoft SQL Server 7.0 as the database. This material forms part of the overallinstallation as described in Chapter 4, “Installation and verification” on page 85.Specifically, this chapter is referenced in step 24 on page 108.


An overall description of the installation process is shown in Figure 73. You willnote that some steps are to be done on Node A, some on Node B, and some areto be carried out on both Nodes.

The main references for the SQL Server installation are:

• R/3 Installation on Windows NT: Microsoft SQL Server, Release 4.5B(document number 51005503)

• Conversion to Microsoft Cluster Server: MS SQL Server, 4.0B 4.5A 4.5B(document number 51005504)

• How to Install SQL Server 7.0, Enterprise Edition on Microsoft Cluster Server:Step by Step Instructions. (white paper from http://www.microsoft.com/sql)



Step 24.2: Install SQL Server 7.0 EE on Node A, page 160Step 24.3: Install SQL Server 7 SP1 on Node A, page 161

Step 24.4: Install the R/3 Setup tool on Node A, page 161

Step 24.11: Completing the migration to MSCS, page 166Step 24.12: Removal of unused resources, page 167

Step 24.9: Converting Node A for operation in a cluster, page 165Step 24.10: Converting Node B for operation in a cluster, page 166

Step 24.7: Install the R/3 cluster conversion tool on Node A, page 164Step 24.8: Install the R/3 cluster conversion tool on Node B, page 165

Step 24.5: Install the R/3 CI and DBMS on Node A, page 162

Step 24.6: Run the Failover Cluster Wizard on Node A, page 164

A

A

A

A

A

B

A

B

A

A

A

B


Preliminary work

SQL Server install






SQL Server cluster conversion

Steps for an SQL Serverinstallation

A

If you do not install the programs in the following order, the software productscan fail and require that the disks be reformatted and installation restarted.

Attention

http://www.microsoft.com/sql


From this point, we assume that Windows NT 4.0 Enterprise Edition is installedon both nodes with Service Pack 3, and MSCS is running.



Before beginning, you should fill in all the SAP installation worksheets in thischapter.

7.2 Install SQL Server and SAP R/3

Step 24.2: Install SQL Server 7.0 EE on Node A

An SAP cluster with MSCS requires you to install SQL Server in theactive/passive mode. This means that only one copy of SQL Server has to beinstalled. This installation procedure is performed only on one node, no matterwhich one. We will install it on Node A.

The target directory must be on a shared disk. Refer to section 3.4.2.5, “SQLServer 7.0 files” on page 61 to know which disk to use for the different SQLServer components (software files, temporary files, log files, and data files).

Do the following:

1. Check that all the cluster resources are owned by Node A.

2. Insert the SAP RDBMS CD-ROM for SQL Server Enterprise Edition (not theStandard Edition). Start the program SETUP.BAT from the root directory.

3. The installation program guides you through a series of windows. Use Table47 as a reference for the most important parameters:

Table 47. Installation of SQL Server 7


Setup type Select Custom Custom

Destinationfolders

• See 3.4.2.5, “SQL Server 7.0 files” on page 61to know to which target disk drive to install.

• Program files: The default location is \MSSQL7• Database files for the Master, Msdb, and Pubs

databases: We recommend you install them ina different directory: \MSSQL7DB

K:\MSSQL7K:\MSSQL7DB

Selectcomponents

Confirm the default settings Leave default

These steps are part of installation step 24 as described on page 108 inChapter 4.

Step 24

Throughout the entire installation process, make sure you are always loggedon as the installation user (in our lab, sapinst).


Chapter 7. Installation using SQL Server 161

4. After SQL Server is installed, it is normal for the cluster group to be offline. Bydefault, neither SQL Server nor SQL Server Agent are automatically startedwhen the installation is complete.

5. Reboot the sever and test the SQL Server installation as follows:

– Start SQL Server on Node A– Register Server A– Perform simple queries– Set up SQLMail if you intend to use it– Stop SQL Server

Step 24.3: Install SQL Server 7 SP1 on Node A

1. Download Service Pack 1 from the Microsoft FTP server at the followingaddress or get it from the Microsoft TechNet CDs:

ftp://ftp.microsoft.com/bussys/sql/public/fixes/usa/sql70

2. All the cluster resources must be online on Node A.

3. Start the Service Pack installation program, and use the information in Table48:

Table 48. Service Pack 1 for SQL Server 7

4. Reboot both nodes after completion.

7.3 Install the R/3 Setup tool

Step 24.4: Install the R/3 Setup tool on Node A

To install R3SETUP on Node A, do the following:

1. Log on to the Windows NT system on Node A as the installation user (in ourlab, sapinst).

Character set/sort order

Required values Char Set: 850 MultilingualSort Order: Binary OrderUnicode Collation: BinaryOrder

Networklibraries

The components Named Pipes, TCP/IP sockets,and Muliprotocol must be installed.

Leave default

Servicesaccounts

Specify the user name of the two serviceaccounts created prior to SQL Serverinstallation. Repeat this step for both SQL Serverand SQL Agent.Do not select AutoStart service.

• Customize the settingsfor each service

• SQL Server: sqlsvc /password / SAPDOM

• SQL Agent: sqlagent /password / SAPDOM

Remoteinformation

(For the cluster administrator account)Specify the cluster service account information

clussvc/ password/SAPDOM

Values Restrictions and recommendations Lab values

Connect to Server Windows NT authentication

Remote information Specify the SAP clusteradministrator account

itsadm/password/SAPDOM


ftp://ftp.microsoft.com/bussys/sql/public/fixes/usa/sql70


2. Check that the TEMP environment variable has been set, using the Systemapplet in the Windows NT Control Panel. TEMP is normally set to C:\TEMP.Make sure that the specified directory really exists in the file system.

3. From the SAP Kernel CD-ROM, start the program \NT\COMMON\R3SETUP.BAT.

4. You are asked to provide values you can take from the following table:

Table 49. Installation of R/3 Setup tool

5. At the end of this step, you will be logged off from your Windows NT session.For the next step, you must log on with the same Windows NT user account,because the installation of the R/3 setup tool assigns the rights necessary forperforming an installation to the user is installing the R/3 setup tool.

7.4 Install the SAP R/3 Central Instance

Step 24.5: Install the R/3 CI and DBMS on Node A

1. On Node A, log on as the installation user (in our lab, sapinst).

2. Check that all the cluster resources are owned by Node A, and move them toNode A if necessary.

3. Click Start > Programs > SAP R/3 Setup > Central and Database Instance.This will install a central instance, and build and load the database.

4. You are prompted to enter values for a number of parameters. The followingtable summarizes these:

Table 50. Installation of the R/3 Central and Database instance



SAP system nameSAPSYSTEMNAME

<SID> ITS

Path to installation directoryINSTALL PATH



Yes




<SID> ITS

Number of the centralsystem (SAPCISYSNR)

Any two-digit number between 00 and97

00

SAPLOC - Drive of the\usr\sap directory

• On the shared disks• Not on the Quorum disks• Not on SQL Server disks

J: (SAP softwareshared disk)


SAPDOM

Central transport host(SAPTRANSHOST)

SAPTRANSHOST


5. When R3Setup has obtained all the information it needs, it automaticallybegins with installation processing. This phase can take up to several hoursdepending on the global configuration and the server performance.

6. Reboot Node A.

RAM that is reserved to theR/3 system(RAM_INSTANCE)

Leave the default (2176MB in ourconfiguration)


Leave the default value 3600 (default)

<SID>adm password password

SAPService<SID> password password

SAP gateway No

TEMPDATAFILESIZE Leave the default value 300 (default)

TEMPDATAFILEDRIVE See 3.4.2.5, “SQL Server 7.0 files” onpage 61

K:

Type of installation Choose custom if you intend tooptimize the parameters for data andlog files. Otherwise, select No.

Automatic installation

DATAFILEDRIVE1 • On the shared disks• Not on the Quorum disks• Not on the SAPLOC disk• Not on the Log disks• Not on the Archive disks

M:


N:


O:

LOGFILEDRIVE • On the shared disks• Not on the Quorum disks• Not on the SAPLOC disk• Not on SAPDATA disks• Not on the Archive disks

L:

Number of R3loadprocesses (PROCESSES)

Number of CPUs 4

REPORT_NAMES NT




7.5 Convert the database to cluster operation

Step 24.6: Run the Failover Cluster Wizard on Node A

The Failover Cluster Wizard is a tool that is part of SQL Server making thedatabase aware of the cluster. To run the wizard do the following:


2. Click Start > Programs > SQL Server > Failover Cluster Wizard.

3. Enter the information you are prompted using Table 51 as a guide:

Table 51. SQL Failover Cluster Wizard

4. After the installation process, you have to reboot Node B and Node A.

5. When the system is restarted, log on to Node A as the installation user (in ourlab, sapinst).

6. Move all resources back to Node A.

7. Redefine the shares on the \USR\SAP directory. You have to manuallyconfigure two shares: SAPLOC and SAPMNT (you must respect thesenames), which points to the same directory: \USR\SAP.

8. Restart the Windows NT services: SAPOSCOL and SAP<SID>.

9. Restart the database service in the Cluster Administrator.

10.All the resources for SQL Server are now displayed in the ClusterAdministrator and the database can be moved between nodes.

7.6 Install the cluster conversion tool

Step 24.7: Install the R/3 cluster conversion tool on Node A

Complete the following steps to install the SAP cluster conversion tool:

1. Make sure that you are logged on to Node A as the installation user (in our lab,sapinst).

2. Insert the SAP Kernel CD-ROM and start the installation of the conversionprogram by running NTCLUST.BAT from the \NT\COMMON directory.



SA password By default there is nopassword.

(empty)

SQL Serverservice account

Type the SQL Server serviceaccount.

password

IP Address This is the virtual IP addressfor accessing the database onthe public network, with thecorrect subnet mask.

192.168.0.52

255.255.255.0

Server Name Name of the virtual server. ITSSQL


3. You are asked to provide the installation directory: enter the same directoryused during the installation of the R/3 instance.

Table 52. Installation of the cluster conversion tool

4. You are automatically logged off from your Windows NT session.

Step 24.8: Install the R/3 cluster conversion tool on Node B

Log on to Node B as the installation user (in our lab, sapinst), and runNTCLUST.BAT per step 24.7.


Step 24.9: Converting Node A for operation in a cluster


2. Make sure that all the cluster resources within MSCS are owned by Node A.

3. Click Start > Programs > SAP R/3 Setup > Configuring Node A for MSCS.

4. You are prompted to enter values for a number of parameters described inTable 53:

Table 53. Converting Node A for operation in a cluster



SAPSYSTEMNAME <SID> ITS

\users\<SID>adm\install C:\users\ITSadm\install


Yes



Network name(NETWORKNAME)

Virtual name for the R/3 system. Thename of the SAP cluster group. Donot enter the name of the cluster.

ITSSAP

IP Address(IPADDRESS)

Virtual IP address for the R/3 systemon the public network. The IP addressfor the SAP cluster group. Do notenter the IP address of the cluster.

192.168.0.51

Subnet mask 255.255.255.0

Network to use(NETWORKTOUSE)

Specify the name of the publicnetwork.

public


<SID> ITS


Any two-digit number between 00 and97

00

SAPLOC - drive of the\USR\SAP directory

SAP software shared disk J:


When all entries have been made, the R3SETUP converts the R/3 instance onNode A for operation in a cluster.

5. When the processing is finished, take the SAP R/3 cluster group offline in theClsuter Administrator, and move it to Node B. On Node B, bring all theresources in that group online, except the R/3 resource.

Step 24.10: Converting Node B for operation in a cluster


2. Make sure that the cluster resources for the R/3 system (SAP_R/3<SID>)within MSCS are owned by Node A.

3. Click Start > Programs > SAP R/3 Setup > Configuring Node B for MSCS.

4. You are prompted to enter values for a number of parameters. Use Table 53 forassistance.

When all entries have been made, the R3SETUP converts the R/3 instance onNode B for operation in a cluster.

5. When R3SETUP has finished, start the R/3 cluster resource SAP_R/3<SID>.You should now have the SQL Server group online on Node A and the SAPR/3 group online on Node B.

At this stage, you can swap the two cluster groups between the two nodesmanually to make them run where you want them to run.

7.8 Complete the MSCS migration

Step 24.11: Completing the migration to MSCS

To complete the cluster conversion, go to the node where the R/3 system isrunning. Make sure you are logged on as the installation user (in our lab, sapinst).

1. Click Start > Programs > SAP R/3 Setup > Completing the Migration to anMSCS.

Windows NT domainname (SAPNTDOMAIN)

SAPDOM

Database virtual nameDBHOSTNAME

ITSSQL

RAM that is reserved toR/3 RAM_INSTANCE

Leave the default (2176MB in our configuration)

<sid>adm password password

SAP gateway(R2_CONNECTION)

NO




2. You are prompted to enter values for a number of parameters described inTable 54:

Table 54. Completing the migration to MSCS

3. Restart the server.

The R/3 system has now been fully converted and is able to operate in the clusterand make use of the cluster features. Test whether the clsuter failover mechanismis working properly by simulating a failover, first on the R/3 group and then on thedatabase group.

7.9 Removal of unused resources

Step 24.12: Removal of unused resources

At the end of the installation the following resources/dependencies are created inthe cluster group:

• MSDTC < Cluster Name < Cluster IP Address• MSDTC < Quorum Disk

The MSDTC resource must be deleted. Before doing that remember to delete thedependencies.



Labvalues

Your values

Central Instance hostname CIHOSTNAME

Specify the same SAP servervirtual name.

ITSSAP


<SID> ITS


00


Chapter 8. Verifying the installation

This chapter provides detailed information about the SAP configuration, whichcan be useful while troubleshooting the installation:

• 8.1, “How to troubleshoot the system at the end of the installation” on page169 provides some hints on how to approach the problem to troubleshoot thesystem.

• 8.2, “Log files” on page 170 contains a list of main logs useful to betterunderstand what is happening. High skills are generally required tounderstand these logs.

• 8.3, “Services” on page 173 contain a list of services you should expect to seeafter the installation. This section has to be used as an easy reference in orderto see if everything is correctly installed.

• 8.4, “Accounts and users” on page 175 contain a list of accounts and rights atthe end of the installation. This section can be a useful reference if you meetpermission problems during or after the installation.

• 8.5, “R3Setup” on page 179 contain a description of how to circumvent atypical unpredictable error one can sometime seen during the R3Setup usage.

8.1 How to troubleshoot the system at the end of the installation

If you get any problems at the end of a SAP installation, it can be very difficult tounderstand what is the cause of these errors. This section aims to provide somehints to face these situations. The main components contributing to the finalconfiguration are

• Hardware• Windows NT software• Microsoft Cluster• SAP software• DBMS software• Network• Clients

Your first step should be to try to isolate which of these components needs athorough analysis. The following hints can be useful in this phase:

• If you have connectivity problems from a SAPGUI or you are not able to failover from one node to the other node, this does not necessarily mean youhave a problem with the SAP system. Quite often, network switches or routersare the causes of a non-working SAP system. Also, routing tables on the SAPservers or on the clients must be examined. A network expert able to analysethe network traffic should be called.

• If you need to understand whether you have a cluster or SAP problem,proceed as described in this point. In both Node A and Node B create ahardware profile (NoSAPNoDB) in which all the SAP and DBMS Services aredisabled. You can get a list of services you need to disable from 8.3,“Services” on page 173.

Reboot both nodes and test the cluster using Microsoft CASTEST utility (see4.9.2, “Test the failover process” on page 107 for more details).


– If the test is successful you know that your cluster is properly working andso the problem can be looked for in the SAP or DBMS installation.

– If the test is not successful you can use the CASTEST log and MicrosoftCluster log in order to understand what is not working.

8.2 Log files

The first place to check when troubleshooting is the various log files.

8.2.1 MSCS log fileThe Microsoft Cluster Server log is the most powerful tool to troubleshoot thecluster. To enable logging it is necessary to create a new system variable,ClusterLog, shown in Figure 74.

Figure 74. Enabling the MSCS log

To provide you with some information about the interpretation of this log, we willshow some small excerpts of real logs.

This first excerpt, Figure 75, describes the start of the Cluster Service on Nodeservera.

Chapter 8. Verifying the installation 171

Figure 75. MSCS log excerpt — Cluster Service startup

The circled line shows the Cluster Service starting:

11f is the ID of the thread issuing the log19-17:59:33.967 is the GMT time stampCluster Service started - Cluster Version 2.224 is the event description

The central lines are characterized by the element [DM]. This is an acronym forDatabase Manager, the cluster component through which the changes in thecluster configuration are done. Table 55 is a list of typical components found in acluster log:

Table 55. Components listed in the log

To understand what the cluster does, it is necessary to understand the internalarchitecture of MSCS. Good references are:

• Clustering Architecture (Microsoft white paper)

• Windows NT Microsoft Cluster Server by Richard R. Lee

Acronym Component

DM Database Manager

NM Node Manager

FM Failover Manager

API API support

LM Log Manager

CS Cluster Service

INIT State of a node before joining the cluster

JOIN State of the node when the node tries to join the cluster

EP Event Processor

RM Resource Monitor

GUM Global Update Manager

189::19-17:59:33.935 [CPROXY] Service Starting...189::19-17:59:33.935 [CPROXY] Service Registered189::19-17:59:33.951 [CPROXY] Process created.189::19-17:59:33.951 [CPROXY] Service Started.11f::19-17:59:33.967

11f::19-17:59:33.967 Cluster Service started - Cluster Version 2.224.11f::19-17:59:33.967 OS Version 4.0.1381 - Service Pack 4.11f::19-17:59:33.967 We're initing Ep...11f::19-17:59:33.967 [DM]: Initialization11f::19-17:59:33.967 [DM] DmpRestartFlusher: Entry11f::19-17:59:33.967 [DM] DmpStartFlusher: Entry11f::19-17:59:33.967 [DM] DmpStartFlusher: thread created11f::19-17:59:33.967 [NM] Initializing...11f::19-17:59:33.967 [NM] Local node name = servera.


A fundamental document containing a complete list of Event Viewer errors due tocluster problems with description enclosed is:

• MS Cluster Server Troubleshooting and Maintenance by Martin Lucas

The next excerpt from the MSCS log, Figure 76, shows the completion of the startprocess:

Figure 76. MSCS log excerpt — cluster started

The next lines in the log, shown in Figure 77, describe the arbitration process toget access to the quorum:

Figure 77. MSCS log excerpt — quorum arbitration

8.2.2 SAP log filesThe most important logs useful to troubleshoot the SAP R/3 installation are:

• \Users\itsadm\Install\CENTRDB.LOG• \Users\itsadm\Install\NTCMIGNA.LOG• \Users\itsadm\Install\NTCMIGNB.LOG• \Users\itsadm\Install\R3CLUS.LOG• \Users\itsadm\Install\UPDISTINTV.LOG• \Temp\R3Setup\NTCLUSCD.LOG• \usr\sap\ITS\sys\DVEBMGS00\work\SAPSTART.LOG

The SAP installation procedure was changed with the release of R/3 4.5A. Asdescribed in OSS note 0138765 Cluster Migration: Terminology and Procedure,the installation is composed of two main phases:

• The ordinary (non-cluster) installation• Cluster migration

In the first phase, the CENTRDB.TPL template is used and relevant information islogged in the CENTRDB.LOG file. In the second phase, the NTCMIGNA.TPL (forNode A) and NTCMIGNB.TPL (for Node B) templates are used. In this phase thecorresponding logs NTCMIGNA.LOG, and NTCMIGNB.LOG should be analyzed.

During the creation of the cluster group, three main programs are used:

11f::19-17:59:35.857 [FM] FmJoinPhase2 complete, now online!11f::19-17:59:35.860 [INIT] Cluster Started! Original Min WS is 204800, Max WS is

1413120.189::19-17:59:35.860 [CPROXY] clussvc initialized140::19-17:59:42.656 Time Service <Time Service>: Status of Time Service request

to sync from node serverb is 0.

16a::19-18:00:07.672 [NM] Checking if we own the quorum resource.180::19-18:00:07.672 Physical Disk <Disk I:>: SCSI, error reserving disk, error 170.180::19-18:00:36.860 Physical Disk <Disk I:>: Arbitrate returned status 0.16a::19-18:00:36.860 [FM] Successfully arbitrated quorum resource

9abfb375 -540e- 11d3- bd6a- 00203522d044.16a::19-18:00:36.860 [FM] FMArbitrateQuoRes: Current State 2 State=2 Owner 216a::19-18:00:36.860 [FM] FMArbitrateQuoRes: Group state :Current State 0 State=0

Owner 216a::19-18:00:36.860 [NM] We own the quorum resource.


• INSAPRCT — responsible for registering the SAP Resource Type• CRCLGRP — responsible for creating the SAP cluster group• COCLGRP — responsible for creating the R/3 resource

These three programs write errors in the R3CLUS.LOG file. See OSS note0112266 R/3 + MSCS Cluster Server: Frequent Questions + Tips for furtherinformation.

As of Release 4.5A, there is a specific R3Setup step dedicated to the correctionof the table INSTVERS whose name is “Completing cluster installation (Instvers)”(see OSS note 0112266 R/3+MSCS Cluster Server: Frequent Questions + Tips).These steps are logged in the UPDISTINTV.LOG file.

If the installation is complete, but the instance does not start, a good source ofinformation can be STARTUP.LOG where the start of the instance is logged.Further information can be found in OSS note 0002033 Startup fails,sapstart.sem, startsap, sapstart.

8.3 Services

It is possible to take a complete description of the services and of their runningstatus using the SCLIST utility from the Windows NT Resource Kit. The followingsubsections contain the list of services at the end of the installation with a fewcomments about their meaning.

8.3.1 OracleTable 56 contains a complete list of relevant services running on a cluster nodejust after the completion of the installation.

Table 56. Oracle Services

Service User account Startup Meaning

Cluster Server ClusSvc Automatic Windows NT4.0 EEService implementing thecluster features likeresource monitoring,failover, and so on.

OracleAgent80 System Manual Listens for and respondsto job and eventrequestes sent from theOEM console2.

OracleAgent80ITSORA System Manual Listens for and respondsto job and eventrequestes sent from theOEM console

OracleClientCache80 System Manual Oracle version 8 providesa client cache service thatallows a client on mostplatforms to storeinformation retrieved froma Names Server in itslocal cache


8.3.2 DB2Table 57 contains a complete list of relevant DB2 services running on a clusternode just after the completion of the installation:

Table 57. DB2 Services

OracleDataGatherer System Manual Gathers performancestatistics for the OraclePerformance Manager.

OracleExtprocAgent System Manual Enables information fromdatabase queries to bepublished to a Web pageat specified timeintervals2.

OracleFailSafe System Manual Oracle cluster service.

OracleServiceITS1 System Manual Oracle instance ITSservice.

OracleTNSListener80Fslitsora System Manual Listens for and acceptsincoming connectionrequests from clientapplications2.

SAPITS_00 SAPServiceITS Manual SAP instance service

SAPOSCOL SAPServiceITS Automatic SAP Operating SystemCollector service.

Notes:1 For this service the setting Allow Service to Interact with Desktop must be checked.2 From Oracle book: Oracle 8. Getting Started


Cluster Server ClusSvc Automatic Windows NT4.0 EE Serviceimplementing the clusterfeatures like resourcemonitoring, failover, ...

DB2 - DB2DAS00 db2admin Automatic DB2 Administration Server(DAS) instance1

DB2 - DB2LITS System Automatic Local database instance usedby work processes to accessthe R/3 database2

DB2 Governor db2admin Manual This service controlsapplication behavior by settinglimits and defining actions whenthe limits are exceeded.

DB2 JDBC Applet Server SYSTEM Manual DB2 Java Applet server - tosupport Java Applets

DB2-DB2ITS(or DB2ITS)

db2its Manual D B2 ITS instance

DB2 Security Server SYSTEM Automatic DB2 Security Service3



8.3.3 SQL ServerTable 58 contains a complete list of relevant SQL Server services running on acluster node just after the completion of the installation.

Table 58. SQL Server services

8.4 Accounts and users

The next step is to look at the accounts and users on your system.

SAPITS_00 sapseITS Manual SAP instance ITS

SAPOSCOL sapseITS Automatic SAP Operating SystemCollector

Notes:1 See Chapter 4 in The Universal Guide to DB2 for Windows NT, SC09-2800, for details on

the meaning of this instance.2 See SAP R/3 and DB2 UDB in a Microsoft Cluster environment, , section 3.3.2 for more

details.3)See Chapter 2 in The Universal Guide to DB2 for Windows NT, where the limited usage of

this service in the most recent releases of DB2 is explained.


Cluster Server ClusSvc Automatic Windows NT4.0EE Serviceimplementing thecluster featureslike resourcemonitoring,failover, ...

MSSQLServer$ITSSQL sqlsvc MS SQL Server(instance ITS)1

SQLServerAgent$ITSSQL sqlsvc MS SQL Serveragent (instanceITS) allowing thescheduling ofperiodic activities1

VSrvSvc$ITSSQL SYSTEM Virtual ServerService forintance ITS

SAPITS_00 SAPServiceITS SAP instance ITS

SAPOSCOL SAPServiceITS SAP OperatingSystem Collector

Notes1 See MS SQL Server Introduction (Microsoft Technet), Chapter 7 for details



8.4.1 OracleTable 59 shows the accounts stored in the Windows NT account databases onthe primary domain controller (PDC) of the SAP domain and in the accountdatabase of the cluster nodes:

Table 59. Accounts on the PDC

Table 60 shows the accounts configured in the Windows NT account database onboth nodes of the cluster:

Table 60. Accounts on the cluster nodes

Table 61 shows the users configured in Oracle tables:

Table 61. Users in the Oracle tables

Account User rights onthe PDC

User rights on the clusternodes

Cluster Service AccountLab value: ClusSvcBelongs to: Domain users

Back up files and directories,Increase quotas, Increasescheduling priority, Loadand unload device drivers,Lock pages in memory, Logon as a service, Restorefiles and directories

<sapsid>admLab value: itsadmBelongs to: Domain users, DomainAdmins, SAP_ITS_GlobalAdmin

Act as a part of theOS, Log on as aservice, Replace aprocess level token

Act as part of the operatingsystem, Increase quotas,Replace a process leveltoken

SAPService<SAPSID>Lab value: SAPServiceITSBelongs to: Domain users,SAP_ITS_GlobalAdmin

Access this computer fromnetwork, Log on as a service

Global GroupSAP_<SAPSID>_GlobalAdminLab value: SAP_ITS_GlobalAdminContains: itsadm and SAPServiceITS

Group Contains

Local Group ORA_<SAPSID>_DBA(lab value ORA_ITS_DBA)

Contains SAPDOM\ClusSvc andSAPDOM\itsadm

Local Group ORA_<SAPSID>_OPER (labvalue ORA_ITS_OPER)

Contains SAPDOM\itsadm andSAPDOM\SAPServiceITS

Local Group SAP_<SAPSID>_Local Admin(lab value SAP_ITS_Local Admin)

Contains SAPDOM\SAP_ITS_GlobalAdmin

User Granted roles System privileges

DBSNMP Connect, resource,SNMPagent

Create publicsynonym, unlimitedtablespace

OPS$<SAPSID>ADMLab value: OPS$ITSADM

Connect, resource, SAPDBA Unlimited tablespace

OPS$SAPSERVICE<SAPSID>Lab value: OPS$SAPSERVICEITS

Connect, resource Unlimited tablespace


8.4.2 DB2 usersTable 62 shows the accounts stored in Windows NT database on the PDC.

Table 62. Accounts stored in Windows NT database on the PDC

Table 63 shows the accounts stored in Windows NT databases in Server A andServer B.

SAPR3 Connect, DBA, resource Unlimited tablespace

SYS All roles All system privileges

SYSTEM DBA Unlimited tablespace

Account/Group User rights on PDC User cluster nodes

Cluster Service Account(lab value: ClusSvc)Belongs to: Administrators,Domain users

Back up files and directories,Increase quotas, Increasescheduling priority, Load andunload device drivers, Lockpages in memory, Log on asa service, Restore files anddirectories

Back up files and directories,Increase quotas, Increasescheduling priority, Load andunload device drivers, Lockpages in memory, Log on asa service, Restore files anddirectories

db2<sapsid> (lab value:db2its)Belongs to: Domain users,SYSADM

Not applicable Access this computer fromthe network, Act as part ofthe operating system, Log onas a service, Replace aprocess-level token

<sapsid>adm (lab value:itsadm)Belongs to: DomainAdmins, Domain users,SAP_ITS_GlobalAdmin,SYSCTRL

Act as a part of the operatingsystem, Increase quotas,Log on as a service, Replacea process level token

Access this computer fromthe network, Act as part ofthe operating system,Increase quotas, Log on as aservice, Replace a process-level token

sapse<sapsid> (lab value:sapseits)Belongs to: Domain users,SAP_ITS_GlobalAdmin,SYSCTRL

Not applicable Access this computer fromthe network, Log on as aservice

Global GroupSAP_<SAPSID>_GlobalAdmin (lab valueSAP_ITS_GlobalAdmin)Contains: itsadm

Not applicable Not applicable

Local Group SYSADMContains: db2its


Local Group SYSCTRLContains: itsadm, sapseits


User Granted roles System privileges


Table 63. Accounts stored in Windows NT databases in servera and serverb

Table 64 shows the database access rights granted to users.

Table 64. Database access rights granted to users

8.4.3 SQL ServerTable 65 shows the accounts specific for an SQL Server installation:

Table 65. SQL Server-specific accounts

Accounts User rights

db2adminBelongs to: Administrators

Act as part of the operating system, Create a tokenobject, debug programs, increase quotas, Log on as aservice, replace a process level token,

sapr3Belongs to: Users

Local group:SAP_<SAPSID>LocalAdminContains the global groupSAPDOM\SAP_ITS_GlobalAdmin

Users/Groups Authorities

User: DB2ITS All

User: ITSADM All

User: SAPR3 Connect database, create tables, create packages, create schemasimplicitly

User: SAPSEITS All

Group: PUBLIC Create schemas implicitly

Account Where Details Lab value

MS SQL ServerService account

Domaincontroller of theSAPDOMdomain

Account name rules sqlsvc

Standard MS SQLlogin withadministratorprivileges

sa (systemsadministrator)

sa

Account previouslyused to connect thework processes tothe DBMS

Login stored inthe MS SQLServer DB

sapr31 sapr3

Local account onthe MS SQLServers

SQLAgentCMDExec2 SQLAgentCMDExec

Local account onthe MS SQLServers

MTSImpersonators MTSImpersonators


8.5 R3Setup

Sometimes the SAP installation program R3Setup shows an unpredictablebehavior. Errors like the one in Figure 78 can appear:

Figure 78. Installation error

The general strategy to face any R3Setup error is: exit from the R3Setup and tryagain. If the problem persists an error analysis will be necessary.

Notes:1 The account sapr3 still exists in SAP R/3 Release 4.5x but is no longer used as described

in OSS note 01573722 See MS SQL Server Transact-SQL and Utilities Reference. Volume 2 in Microsoft

Technet for a description of meaning of this account

Account Where Details Lab value


Appendix A. Special notices

This publication is intended to help the Windows NT and SAP R/3 consultantinstall SAP R/3 4.5B in a Microsoft Cluster Server configuration using eitherOracle, DB2 or SQL Server as the database. The information in this publication isnot intended as the specification of any programming interfaces that are providedby SAP and Microsoft for SAP R/3 and Windows NT. See the PUBLICATIONSsection of the IBM Programming Announcement for for more information aboutwhat publications are considered to be Netfinity product documentation.

References in this publication to IBM products, programs or services do not implythat IBM intends to make these available in all countries in which IBM operates.Any reference to an IBM product, program, or service is not intended to state orimply that only IBM's product, program, or service may be used. Any functionallyequivalent program that does not infringe any of IBM's intellectual property rightsmay be used instead of the IBM product, program or service.

Information in this book was developed in conjunction with use of the equipmentspecified, and is limited in application to those specific hardware and softwareproducts and levels.

IBM may have patents or pending patent applications covering subject matter inthis document. The furnishing of this document does not give you any license tothese patents. You can send license inquiries, in writing, to the IBM Director ofLicensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 USA.

Licensees of this program who wish to have information about it for the purposeof enabling: (i) the exchange of information between independently createdprograms and other programs (including this one) and (ii) the mutual use of theinformation which has been exchanged, should contact IBM Corporation, Dept.600A, Mail Drop 1329, Somers, NY 10589 USA.

Such information may be available, subject to appropriate terms and conditions,including in some cases, payment of a fee.

The information contained in this document has not been submitted to any formalIBM test and is distributed AS IS. The use of this information or theimplementation of any of these techniques is a customer responsibility anddepends on the customer's ability to evaluate and integrate them into thecustomer's operational environment. While each item may have been reviewed byIBM for accuracy in a specific situation, there is no guarantee that the same orsimilar results will be obtained elsewhere. Customers attempting to adapt thesetechniques to their own environments do so at their own risk.

Any pointers in this publication to external Web sites are provided forconvenience only and do not in any manner serve as an endorsement of theseWeb sites.

The following terms are trademarks of the International Business MachinesCorporation in the United States and/or other countries:

AIX AS/400DB2 ESCONIBM MQMQSeries Netfinity


The following terms are trademarks of other companies:

C-bus is a trademark of Corollary, Inc.

Java and HotJava are trademarks of Sun Microsystems, Incorporated.

Microsoft, Windows, Windows NT, and the Windows 95 logo are trademarksor registered trademarks of Microsoft Corporation.

PC Direct is a trademark of Ziff Communications Company and is usedby IBM Corporation under license.

Pentium, MMX, ProShare, LANDesk, and ActionMedia are trademarks orregistered trademarks of Intel Corporation in the U.S. and othercountries.

SET and the SET logo are trademarks owned by SET Secure ElectronicTransaction LLC.

UNIX is a registered trademark in the United States and othercountries licensed exclusively through X/Open Company Limited.

Other company, product, and service names may be trademarks orservice marks of others.

Netfinity Manager OS/390Parallel Sysplex RS/6000S/390 ServeRAIDServerProven TrackPoint


Appendix B. Related publications

The publications listed in this section are considered particularly suitable for amore detailed discussion of the topics covered in this redbook.

B.1 International Technical Support Organization publications

For information on ordering these ITSO publications see “How to get ITSOredbooks” on page 193.

• Optimizing IBM Netfinity Servers for SAP R/3 and Windows NT, SG24-5219

• Netfinity Performance Tuning with Windows NT 4.0, SG24-5287

• Implementing Netfinity Disk Subsystems: ServeRAID SCSI, Fibre Channeland SSA, SG24-2098

• Windows NT Backup and Recovery with ADSM, SG24-2231

• Disaster Recovery with HAGEO: An Installer’s Companion, SG24-2018

• Bullet-Proofing Your Oracle Database with HACMP: A Guide to ImplementingAIX Databases with HACMP, SG24-4788

• Oracle Cluster POWERsolution Guide, SG24-2019

• High Availability Considerations: SAP R/3 on DB2 for OS/390, SG24-2003

• Using Tivoli Storage Management in a Clustered NT Environment, SG24-5742

B.2 Redbooks on CD-ROMs

Redbooks are also available on the following CD-ROMs. Click the CD-ROMsbutton at http://www.redbooks.ibm.com/ for information about all the CD-ROMsoffered, updates and formats.

B.3 Related Web sites

This appendix lists the Web sites that are relevant to the topics discussed in thisredbook.

CD-ROM Title Collection KitNumber

System/390 Redbooks Collection SK2T-2177Networking and Systems Management Redbooks Collection SK2T-6022Transaction Processing and Data Management Redbooks Collection SK2T-8038Lotus Redbooks Collection SK2T-8039Tivoli Redbooks Collection SK2T-8044AS/400 Redbooks Collection SK2T-2849Netfinity Hardware and Software Redbooks Collection SK2T-8046RS/6000 Redbooks Collection (BkMgr Format) SK2T-8040RS/6000 Redbooks Collection (PDF Format) SK2T-8043Application Development Redbooks Collection SK2T-8037IBM Enterprise Storage and Systems Management Solutions SK3T-3694

http://www.redbooks.ibm.com/


B.3.1 Netfinity technology

Internet sites:

• http://www.pc.ibm.com/netfinity

• http://www.pc.ibm.com/support

IBM intranet sites (available within IBM only):

• http://performance.raleigh.ibm.com/ — Netfinity performance Web site

• http://netfinity.sl.dfw.ibm.com/ — IBM ATS Web site

• http://argus.raleigh.ibm.com/ — Netfinity hardware development

• http://devtlab.greenock.uk.ibm.com/ — Greenock development

B.3.2 Windows NT

• http://www.microsoft.com/security

• http://NTSecurity.ntadvice.com

• http://www.trustedsystems.com

• http://www.microsoft.com/hcl — Microsoft Hardware Compatibility List(select Cluster)

B.3.3 Microsoft Cluster Server

• http://www.microsoft.com/ntserver/ntserverenterprise/

B.3.4 SAP

Internet sites:

• http://www.sap.com — Main SAP AG Web site

• http://www.sapnet.sap.com — Technical Web site; to access this site it isnecessary to have an account

– http://www.sapnet.sap.com/r3docu — Documentation

– http://www.sapnet.sap.com/technet — TechNet

– http://www.sapnet.sap.com/securityguide — Security

– http://www.sapnet.sap.com/notes — OSS notes

• http://www.r3onnt.com/ — IXOS Web site containing certified platforms

• http://www.ibm.com/erp/sap — IBM-SAP alliance page

• http://www.microsoft.com/industry/erp/sap/ — Microsoft-SAP alliance

• http://www.microsoft.com/germany/sap/ — Microsoft-SAP Germany

• http://www.sapfaq.com/ — Frequently asked questions Web site

• http://www.sap-professional.org/ — SAP professional organization

• http://www.saptechjournal.com/ — SAP Technical Journal online

IBM intranet sites (accessible to IBM employees only):

• http://w3.isicc.de.ibm.com/ — ISICC Web page (Germany)

• http://w3.isicc.ibm.com/ — ISICC Web page

http://www.pc.ibm.com/netfinity


http://performance.raleigh.ibm.com/

http://netfinity.sl.dfw.ibm.com/

http://argus.raleigh.ibm.com/

http://devtlab.greenock.uk.ibm.com/

http://www.microsoft.com/security

http://NTSecurity.ntadvice.com

http://www.trustedsystems.com

http://www.microsoft.com/hcl

http://www.microsoft.com/ntserver/ntserverenterprise/

http://www.sap.com

http://www.sapnet.sap.com

http://www.sapnet.sap.com/r3docu

http://www.sapnet.sap.com/technet

http://www.sapnet.sap.com/securityguide

http://www.sapnet.sap.com/notes

http://www.r3onnt.com/

http://www.ibm.com/erp/sap

http://www.microsoft.com/industry/erp/sap/

http://www.microsoft.com/germany/sap/

http://www.sapfaq.com/

http://www.sap-professional.org/

http://www.saptechjournal.com/

http://w3.isicc.de.ibm.com/

http://w3.isicc.ibm.com/

Appendix B. Related publications 185

B.3.5 Oracle

• http://www.oracle.com/ — Oracle Web site

• http://technet.oracle.com/ — Oracle TechNet

B.3.6 DB2

• http://www.software.ibm.com/data/db2/udb/udb-nt/ — IBM DB2 on WindowsNT

• http://www.software.ibm.com/data/partners/ae1partners/ — IBM DB2partners

B.3.7 SQL Server

• http://www.microsoft.com/sql/ — Microsoft SQL Server Web site

B.4 Downloadable documents

These documents are referenced in this redbook.

B.4.1 Microsoft Windows NT

• Windows NT Security Guidelines by Steve Sutton, available fromhttp://www.trustedsystems.com/NSAGuide.htm

B.4.2 Microsoft Cluster

• IBM Netfinity High-Availability Cluster Solutions Using the IBM ServeRAID -3Hand IBM ServeRAID-3HB Ultra2 SCSI Controllers Installation and User’sGuide, available from http://www.pc.ibm.com/netfinity/clustering

• IBM Shared Disk Clustering Hardware Reference, available fromhttp://www.pc.ibm.com/netfinity/clustering

• FAQ: All You Ever Wanted to Know about Windows NT Server 4.0 EnterpriseEdition

• Windows NT Server, Enterprise Edition Administrator's Guide and ReleaseNotes from http://www.microsoft.com/technet/

• MS Cluster Server Troubleshooting and Maintenance by Martin Lucas fromhttp://www.microsoft.com/technet/

• Deploying Microsoft Cluster Server from http://www.microsoft.com/technet/

• MSCS Administrator’s Guide, from http://www.microsoft.com/technet/

• Microsoft Cluster Server Release Notes, from the Windows NT 4.0 EnterpriseEdition CD

• IBM Cluster Checklist, available fromhttp://www.pc.ibm.com/us/searchfiles.html

B.4.3 SAP R/3

Installation

• R/3 Installation on Windows NT Oracle Database, Release 4.5B, 51004599(May 1999)

http://www.microsoft.com/technet/

http://www.pc.ibm.com/us/searchfiles.html

http://www.trustedsystems.com/NSAGuide.htm






http://www.oracle.com/

http://technet.oracle.com/

http://www.software.ibm.com/data/db2/udb/udb-nt/

http://www.software.ibm.com/data/partners/ae1partners/

http://www.microsoft.com/sql/


• R/3 Installation on Windows NT DB2 Common Server, Release 3.5B,51005502

• R/3 Installation on Windows NT MS SQL Server, Release 3.5B, 51005503

• Conversion to Microsoft Cluster Server: IBM DB2 for NT, Release 3.5B,51006418

• Conversion to Microsoft Cluster Server: Oracle 4.0B 4.5A 4.5B, 51005504

• Conversion to Microsoft Cluster Server: MS SQL Server, Release 4.0B, 4.5A,4.5B, 51005948

Security

• R/3 Security Guide: Volume I. An Overview of R/3 Security Services

• R/3 Security Guide: Volume II. R/3 Security Services in Detail

• R/3 Security Guide: Volume III. Checklist

Networks

• Network Integration of R/3 Frontends, 51006473

• Network Integration of R/3 Servers, 51006371

Tuning

• Tuning SAP R/3 for Intel Pentium Pro Processor-based Servers RunningWindows NT Server 3.51 and Oracle 7 Version 7.2 fromhttp://www.intel.com/procs/servers/technical/SAP/281860.pdf

• SAP/Oracle/AIX. Performance Tuning Tips by John Oustalet and Alter Orb

• SAP R/3 Performance Tuning Guide for Microsoft SQL Server 7.0 fromhttp://www.microsoft.com/SQL/productinfo/sapp.htm

B.5 SAPSERV FTP site

Here we describe how to connect to SAP’s SAPSERVx system by way of anMPN/SOCKSified browser to download patches.

Note: These steps are related to access through the IBM intranet only. Moreinformation regarding access to SAPSERVx can be found in the following OSSnotes:

• Note Number: 0063786 — FAQ on SAPSERVx.

• Note Number: 0019466 — Downloading a Patch from SAPSERVx.

For IBM intranet access by way of SOCKSified FTP, do the following:

1. Update your SOCKS.CNF file with the following detail:

http://www.intel.com/procs/servers/technical/SAP/281860.pdf

http://www.microsoft.com/SQL/70/whpprs/SAPSQL.htm


Updates to the SOCKS.CNF file are maintained by the ISICC team in Walldorf,Germany. The information above is current as of August 19, 1999. Send ane-mail to [email protected] if you have questions. The current ISICCSOCKS.CNF information can be found at either of the following URLs:

– ftp://sicc980.isicc.de.ibm.com/perm/socks/os2/socks.conf– ftp://9.165.228.33/perm/socks/os2/socks.conf

2. Configure your browser for a manual proxy configuration with SOCKS server9.165.214.110 and port:1080.

3. Make sure you do not have an FTP proxy server identified.

4. Use ftp://147.204.2.5 as the address to access SAPSERVx.

Note: You should reset your browser’s proxy configuration to its original settingswhen you are finished with access to SAPSERVx.

B.6 OSS notes

Refer to B.5, “SAPSERV FTP site” on page 186 for details on how to access theOSS notes.

0008523 DB Backups Using Ccms Do Not Work0030478 Service Packs on Windows NT (current support status)0068544 Memory Management under Windows NT0088416 Zero Administration Memory Management from 4.0A/NT0098385 R/3 MSCS Installation With Multiple NICs0098717 4.0B R/3 Installation on Windows NT: DB2/CS0100163 External Command for Windows Nt Not Found0101412 DBCC Checks with sap_mon_dbcheck0101896 General problems for Microsoft cluster installation0103747 Performance 4.0/4.5: Parameter Recommendations0106275 Availability of R/3 on Microsoft Cluster Server (support status)0107534 Migration on Microsoft Cluster Server0107591 INFORMIX: migration stand-alone R/3 system to MSCS (3.1I)0110172 NT: Transactions with Large Storage Requirements0110507 Problems with PREPARE/STARTUP in Cluster Environment0112266 R/3 + MS cluster server: frequent questions + tips0114287 SAPDBA in a Microsoft Cluster Server environment0117294 4.5A R/3 Installation on Windows NT: MS SQL Server0117295 4.5A R/3 Installation on Windows NT: Oracle0117296 Inst.: Rel. 4.5A R/3 Installation on Windows NT: DB2/CS0117305 ORACLE: Migration to a Microsoft cluster server 4.5A0120211 INFORMIX: Migrating to a Microsoft cluster server 4.5A

This is screen.# IBM internal Network - without socks-server#-----------------------------------------------direct 9.0.0.0 255.0.0.0# SAP's DMZ - socks-server siccfw1.isicc.ibm.com#-----------------------------------------------sockd @=9.165.214.110 147.204.0.0 255.255.0.0# Internet - socks-server socks.de.ibm.com#-----------------------------------------------sockd @=9.165.255.62 0.0.0.0 0.0.0.0

ftp://sicc980.isicc.de.ibm.com/perm/socks/os2/socks.conf

ftp://9.165.228.33/perm/socks/os2/socks.conf

ftp://147.204.2.5


0124141 Hot Package 40B08 (IPA-TEM)0126985 Configuration of Ataman Remote Shell for DB2CS/NT0128167 Service Pack 4 on NT MSCS with Oracle products0132738 INFORMIX: Using SAPDBA in MSCS or distributed environment0134073 4.5B R/3 Installation on Windows NT: MS SQL Server0134135 4.5B R/3 Installation on Windows NT (General)0134141 Conversion to a Microsoft Cluster Server 4.5B0134159 4.5B R/3 Installation on Windows NT: DB2/CS0138765 Migration to a Microsoft Cluster Server 4.5A0140960 MSCS Installation R/3 3.x on MS SQL Server 7.00140990 NT MSCS: How to backup/recover the CLUSDB0142731 DBCC Checks for SQL Server 7.00144310 Installing the NT SP4 on R/3 MSCS clusters0146751 Converting MS SQL Server 6.5 to 7.0 in cluster0151508 Resource Requirements for Release 4.6A0154700 MSCS Cluster Verification Utility0156363 MSCS: NET8 Configuration for Oracle0166966 Printing in Microsoft Cluster Environment

B.7 Knowledge Base articles

The following Microsoft Knowledge Base articles are relevant to this redbook.They can be found at:

http://support.microsoft.com/search/

Hardware installation

Q169414 Cluster Service may stop after failoverQ171793 Information on application use of 4GT RAM tuning

MSCS software installation

Q185752 MSCS doesn't run with NT WorkstationQ175779 MSCS requires SP3 or laterQ174617 Chkdsk runs while running MSCS setupQ171883 Visual C++ runtime error when installing MSCSQ174332 How to install additional cluster administratorsQ171265 Unable to uninstall ClusterAdminQ190354 Unattended MSCS setup with -JOIN requires user input (fixed with

SP4)Q214680 Cluster setup does not recognize disk with more than three logical

drives (fixed with SP5)Q232910 Error 1044 when attempting to create a new cluster

MSCS upgrades and hotfixes

Q178924 How to upgrade an evaluation version of MSCSQ179776 Availability of hotfixes for MSCSQ174799 How to install service packs in a cluster

Groups, resources, and dependencies

Q169017 Groups and resources in MSCSQ197047 Failover/failback policies on MSCSQ171791 Creating dependencies in MSCS

http://support.microsoft.com/search/


Q178276 Dependencies unavailable in Properties tabQ174928 Dependencies page empty when running Resource WizardQ182193 Error: Cluster resource dependency can not be foundQ174641 Resource Parameters tab is missingQ172507 Resources go offline and online repeatedlyQ171277 Resource failover timeQ168948 Information about the cluster groupQ225329 Access violation in resource monitor (fixed with SP5)

Time service resource

Q174331 Error when adding second time serviceQ174398 How to force time synchronization between MSCS nodes

Quorum resource

Q175664 Error creating dependency for quorum resourceQ172944 How to change quorum disk designationQ172951 How to recover from a corrupted quorum logQ225081 Cluster resources quorum size defaults to 64 KBQ238173 Quorum checkpoint file may be corrupted at shutdown

Cluster disks

Q171052 Software FT sets are not supported in MSCSQ175278 How to install additional drives on shared SCSI busQ175275 How to replace shared SCSI controller with MSCSQ176970 Chkdsk /f does not run on the shared cluster diskQ174797 How to run Chkdsk on a shared driveQ196655 How to set up file auditing on cluster diskQ189149 Disk counters on clustered disk record zero valuesQ172968 Disk subsystem recovery documentation errorQ195636 Fibre Channel system loses SCSI reservation after multiple restarts

(fixed with SP4)Q193779 MSCS drive letters do not update using DiskAdmin (fixed with SP4)Q215347 Cluster disk with more than 15 logical drives fails to go online (fixed

with SP5)

Cluster networks — general, IP protocols

Q101746 TCP/IP Hosts file is case sensitiveQ158487 Browsing across subnets w/ a multihomed PDC in Windows NT 4.0Q171390 Cluster service doesn't start when no domain controller availableQ171450 Possible RPC errors on cluster startupQ168567 Clustering information on IP address failoverQ170771 Cluster may fail if IP address used from DHCP serverQ178273 MSCS documentation error: no DHCP server failover supportQ174956 WINS, DHCP, and DNS not supported for failover

Cluster networks — name resolution

Q195462 WINS registration and IP address behavior for MSCSQ193890 Recommend WINS configuration for MSCSQ217199 Static WINS entries cause the network name to go offlineQ183832 GetHostName() must support alternate computer names (fixed with

SP4)Q171320 How to change the IP address list order returned


Q164023 Applications calling GetHostByName() for the local host name maysee the list of IP addresses in an order that does not match the bindingorder (fixed with SP4)

Cluster networks — network interfaces

Q174812 Effects of using autodetect setting on cluster NICQ201616 Network card detection in MSCSQ175767 Behavior of multiple adapters on same networkQ176320 Impact of network adapter failure in a clusterQ175141 Cluster service ignores network cardsQ174945 How to prevent MSCS from using specific networksQ174794 How to change network priority in a cluster

Applications and services — general

Q171452 Using MSCS to create a virtual serverQ175276 Licensing policy implementation with MSCSQ174837 Microsoft BackOffice applications supported by MSCSQ188984 Office 97 not supported in a clustered environmentQ198893 Generic application: Effects of checking “Use Network Name for

Computer Name” in MSCSQ174070 Registry replication in MSCSQ181491 MS Foundation Class GenericApp resources failQ224595 DCOM client cannot establish CIS session using TCP/IP address

(fixed with SP5)Q188652 Error replicating registry keys (fixed with SP4)Q184008 SQL Server cluster setup may fail on third-party disk drivesQ176522 IIS Server instance error message with MSCS

Applications and services — Microsoft SQL Server

Q192708 Installation order for MSCS support for SQL Server V6.5 or MSMessage Queue Server

Q187708 Cannot connect to SQL Virtual Server via sockets (fixed with SP4)Q185806 SQL Server service stopped when IsAlive fails to connect (fixed with

SQL Server SP5a (U.S.) for V6.5)Q216674 Automatic SQL cluster failover does not work with WNT 4.0 SP4Q195761 SQL Server 7.0 frequently asked questions: failoverQ219264 Order of installation for SQL Server 7.0 clustering setupQ223258 How to install the WinNT Option Pack on MSCS with SQL Server 6.5

or 7.0Q183672 How to upgrade a clustered MS MessageQueue SQL to SQL

Enterprise Edition

Applications and services — Oracle Fail Safe

Q219303 Oracle Fail Safe does not function after SP4 installed (fixed with SP5)

Troubleshooting and debugging

Q168801 How to enable cluster logging in MSCSQ216237 Cluster server will not start if cluster log directory is not created (fixed

with SP5)Q216240 Cluster log is overwritten when cluster server startsQ216329 Cluster log filling with erroneous security descriptor information (fixed

with SP5)


Q174944 How to use the -debug option for cluster serviceQ189469 ClusterAdmin can connect to all NetBIOS namesQ197382 How to keep ClusterAdmin from reconnecting to a clusterQ171451 Cluster node may fail to join clusterQ185051 Restarting cluster service crashes services.exe (fixed with SP4)Q193654 Services continue to run after shutdown initiated (fixed with SP4)Q216064 Cluster server has Clusdb corruption after power outage (fixed with

SP5)Q219309 Disk error pop-up causes cluster service to stop (fixed with SP5)Q233349 Cluster service issues event 1015 every four hours after applying SP5

B.8 Other publications

These publications are also relevant as further information sources:

Clustering

• Gregory F. Pfister, In Search of Clusters, Second Edition, Prentice Hall PTR,ISBN 0138997098

• Mark Sportack, Windows NT Clustering Blueprints, SAMS Publishing, ISBN0672311356

Windows NT

• David A. Salomon, Inside Windows NT, Second Edition, Microsoft Press, ISBN1572316772

• Richard R. Lee, Windows NT Microsoft Cluster Server, Osborne McGraw-Hill,ISBN 0078825008

SAP R/3

• Liane Will, SAP R/3 System Administration, Sybex, ISBN 0782124267l

• Hartwig Brand, SAP R/3 Implementation with ASAP, Sybex, ISBN 0782124725

• Diane Bullock et al, DB2 Universal Database and SAP R/3 Version 4, IBM,SC09-2801

• Robert E. Parkinson et al, Basis Administration for SAP, Prima Tech, ISBN0761518878

• The redpaper, SAP R/3 and DB2 UDB in a Microsoft Cluster Environment,available from http://www.redbooks.ibm.com

Oracle

• Liane Hobbs, Oracle 8 on Windows NT, Digital Press, ISBN 1555581900

• Steve Bobrowski, Oracle Architecture, Osborne McGraw-Hill, ISBN0078822742

• Michael J. Corey et al, Oracle 8 Tuning, Osborne McGraw-Hill, ISBN0078823900

DB2

• Jonathan Cook et al, The Universal Guide to DB2 for Windows NT, IBM,SC09-2800


How to get ITSO redbooks

This section explains how both customers and IBM employees can find out about ITSO redbooks, redpieces, andCD-ROMs. A form for ordering books and CD-ROMs by fax or e-mail is also provided.

• Redbooks Web Site http://www.redbooks.ibm.com/

Search for, view, download, or order hardcopy/CD-ROM redbooks from the redbooks Web site. Also readredpieces and download additional materials (code samples or diskette/CD-ROM images) from this redbooks site.

Redpieces are redbooks in progress; not all redbooks become redpieces and sometimes just a few chapters willbe published this way. The intent is to get the information out much quicker than the formal publishing processallows.

• E-mail Orders

Send orders by e-mail including information from the redbooks fax order form to:

• Telephone Orders

• Fax Orders

This information was current at the time of publication, but is continually subject to change. The latest informationmay be found at the redbooks Web site.

In United StatesOutside North America

e-mail [email protected] information is in the “How to Order” section at this site:http://www.elink.ibmlink.ibm.com/pbl/pbl

United States (toll free)Canada (toll free)Outside North America

1-800-879-27551-800-IBM-4YOUCountry coordinator phone number is in the “How to Order” section atthis site:http://www.elink.ibmlink.ibm.com/pbl/pbl

United States (toll free)CanadaOutside North America

1-800-445-92691-403-267-4455Fax phone number is in the “How to Order” section at this site:http://www.elink.ibmlink.ibm.com/pbl/pbl

IBM employees may register for information on workshops, residencies, and redbooks by accessing the IBMIntranet Web site at http://w3.itso.ibm.com/ and clicking the ITSO Mailing List button. Look in the Materialsrepository for workshops, presentations, papers, and Web pages developed and written by the ITSO technicalprofessionals; click the Additional Materials button. Employees may access MyNews at http://w3.ibm.com/ forredbook, residency, and workshop announcements.

IBM Intranet for Employees

http://www.redbooks.ibm.com/

http://www.elink.ibmlink.ibm.com/pbl/pbl



mailto:[email protected]

http://w3.itso.ibm.com/

http://w3.ibm.com/





IBM Redbook fax order formPlease send me the following:

We accept American Express, Diners, Eurocard, Master Card, and Visa. Payment by credit card notavailable in all countries. Signature mandatory for credit card payment.

Title Order Number Quantity

First name Last name

Company

Address

City Postal code

Telephone number Telefax number VAT number

Invoice to customer number

Country

Credit card number

Credit card expiration date SignatureCard issued to


List of abbreviations

ABAP Advanced Business ApplicationProgramming

ADSI Active Directory Service Interfaces

ADSM ADSTAR Distributed Storage Manager

API application programming interface

ARCH archiver

ASCII American National Standard Code forInformation Interchange

ATS Advanced Technical Support

BIOS basic input/output system

BLOB binary large objects

BTC batch

CCMS SAP Computer Center ManagementSystem

CD-ROM compact disk-read only memory

CI central instance

CMT IBM Center for Microsoft Technologies

CPU central processing unit

DAT digital audio tape

DB database

DBA database administrator

DBMS database management system

DBWR database writer

DHCP Dynamic Host Configuration Protocol

DI database instance

DIA dialog

DLL dynamic linked library

DLT digital linear tape

DNS domain name server

DTC Distributed Transaction Coordinator

EDI electronic data interchange

EE Enterprise Edition

ENQ enqueue

ERP Enterprise Resource Planning

ESCON enterprise systems connection

ESM Environmental Services Monitor

FAQ frequently asked questions

FC Fibre Channel

FCAL Fibre Channel Arbitrated Loop

FDDI fiber distributed data interface

FI financial accounting

FLA Fabric Loop Attach

FTP file transfer protocol

GB gigabytes

GBIC Gigabit Interface Converter

GL general ledger

GMT Greenwich mean time

GUI graphical user interface

HACMP high availability cluster multi-processing

HAGEO High Availability Geographic Cluster

HCL hardware compatibility list

HCT hardware compatibility test

HDR High-availability Data Replication

IBM International Business Machines

ICMP Internet control message protocol

ICSM IBM Cluster Systems Management

IE Internet Explorer

IIS Internet Information Server

ISICC IBM/SAP International CompetencyCenter

LAN local area network

LED light emitting diode

LGWR log writer

LUN logical unit number

LVDS low voltage differential signalling

MMC Microsoft Management Console

MS Microsoft

MSCS Microsoft Cluster Server

MSDTC Microsoft Distributed TransactionCoordinator

MSMQ Microsoft Message Queue Server

MSSQL Microsoft SQL Server

NNTP NetNews transfer protocol

NTC NT Competency Center

ODBC open database connectivity

OEM other equipment manufacture

OFS Oracle Fail Safe

OLTP online transaction processing

OPS Oracle Parallel Server

OS operating system


OSS online service system

PA Personnel Administration

PAE Physical Address Extension

PDC primary domain controller

PLDA Private Loop Direct Attach

PSE Page Size Extension

RAID redundant array of independent disks

RAM random access memory

RDAC redundant disk array controller

RDBMS relational database management system

RISC reduced instruction set computer

RPM revolutions per minute

SAPS SAP Application Performance Standard

SCSI small computer system interface

SD sales and distribution

SGA system global area

SID system identification

SMS Systems Management Server

SMTP simple mail transfer protocol

SNA systems network architecture

SNMP simple network management protocol

SP service pack

SPO spooler

SQL structured query language

SSA serial storage architecture

TCP/IP transmission control protocol/internetprotocol

UDB Universal Database

UPD update

UPS uninterruptable power supply

VHDCI very high density connector interface

WHQL Windows Hardware Quality Labs

WINS Windows Internet Name Service


Index

Numerics01K7296 70, 7201K7297 69, 7201K8017 6601K8028 6601K8029 6603K9305 7203K9306 7203K9307 7203K9308 7203K9310 6603K9311 6635231RU 70, 7235261RU 69, 724 GB tuning 9976H5400 66

Aaccounts 88

<sid>adm 108DB2 177Oracle 176SAPService<SID> 108SQL Server 178

Active Directory Services Interface 105ADSM 35AIX and NT solutions 16asynchronous replication 11Ataman 127auto-sensing network adapters 81

Bbackbone configuration 114backbone network 77, 79backup 33

alias definitions 35MSCS open files 37offline backups 37scheduling 36virtual names 36

backup copy of Windows NT 90batch work processes 120binding order 80, 100BLOBs 12block size 67BOOT.INI 94, 99browser service error 95buffer analysis 125

CCASTEST 107CDINSTLOGOFF_NT_IND

DB2 150SQL Server 162, 165

CENTRDB.R3S 133

certification 42categories 43hardware components 43HCL 39IBM 42iXOS 39, 42Microsoft 8, 39, 44SAP 8

CIHOSTNAMEDB2 157Oracle 144SQL Server 167

CLUSDB files, backing up 37CLUSRES.DLL 105, 117cluster

cluster conversion filesOracle 135

logging tool 102Cluster Diagnostic Utility 96Cluster Verification Utility 96clustering 1

backups 33cold standby 10configurations 3EXP15, use with 64Fibre Channel 72replicated database 11replicated DBMS server 14ServeRAID 66shared disks 3shared nothing 50swing-disk 3

ClusterProven 42cold standby 10components 171computer name 90configuration 39

Ddatabase reorganization 126DATAFILEDRIVE

SQL Server 163DB13 126DB2

accounts 177active database logs 59CAR files 149CDINSTLOGOFF_NT_IND 150CD-ROM copy 149central instance installation 151CIHOSTNAME 157cluster conversion files 153cluster, convert Node A to 154database instance 151DB2<SID> Service 157DB2DB6EKEY 151, 155DB2INSTANCE 150


DB2MSCS.CFG 152DB2UE (user exit) 59DBHOSTNAME 155DFTDBPATH 151disk layout 59dropping the database 148fixpack 147, 149installation 145INSTVERS 156IPADDRESS 155MSCS, migration to 156NETWORKNAME 155NETWORKTOUSE 155Node A installation 146Node B installation 148NTCLUST.BAT 153NTCMIGNA.R3S 150package files 149PORT 152PROCESSES 152R2_CONNECTION 152R3Setup install 150RAID levels 60RAM_INSTANCE 152, 155REPORT_NAMES 152SAPDATA 151SAPLOC 151, 156SAPNTDOMAIN 151, 155SAPSYSNR 151, 154, 157SAPSYSTEMNAME 150, 154, 157SAPTRANSHOST 151security planning 88services 174SMSTEMP 152user exit (DB2UE) 59verify installation 114worksheets

central instance 151cluster conversion files 153convert to cluster 154MSCS, migrating to 157Node A installation 147Node B installation 148R3Setup on Node A 150

DBHOSTNAMEDB2 155SQL Server 166

DEFAULT.PFL 111dependencies, implications of (in MSCS) 23DFTDBPATH

DB2 151DHCP 81, 91DHCP, use of (in MSCS) 24dialog traffic 121dialog work processes 119disk layout 50, 52

DB2 59hot spares 65log files 50, 65merge groups (ServeRAID) 51

Netfinity servers 54operating system 50Oracle 56, 58Oracle Fail Safe Respository 57page file 50, 51, 53, 54performance 65quorum 50RAID levels 65recommendation

DB2 60Oracle 58SCSI 67

redo logs 59SAP R/3 files 55shared disks 55size of disks 65SQL Server 61

disk space required 41disk subsystem

quorum resource in MSCS 24terminology, common or shared disk? 18

DNS 7, 79, 91, 103domain controller 7domain name 91domain requirements for MSCS 27

Eenqueue work processes 119EXP15 62, 69

sample configurations 66, 72

Ffailover

See also MSCSFailover Cluster Wizard (SQL Server) 164fault tolerance 1Fibre Channel 3, 39, 68

active/active mode 70, 76cache 70cache policy 59clustering configurations 72components 68configurations 41Ethernet connectors 70EXP15 62, 69failover 70FailSafe RAID Controller 73GBIC 71hardware 68host adapter 69hub 70logical units (LUNs) 70long-wave 71LUNs 75LUNs, number of 75maximum disks per LUN 75nodes 71performance 75quorum 55

Index 199

RAID controller 69RAID levels 75RDAC driver 74redundant configuration 74redundant pair 70RS-232 70segment size 76shared disks 73short-wave 71single points of failure 73SYMplicity Storage Manager 74, 76topology 68write cache 76

GGBIC 71group

See resource

HHACMP 16HAGEO 16hardware configuration 39HCL (hardware compatibility list) 39heartbeat

redundant network adapters, use of 82high availability 1HOSTS file 78, 80, 94, 103

IIBM Cluster System Manager

See ICSMICSM

IBM Cluster System Managerfailover, scheduled 27

Informix High-availability Data Replication 12INIT.ORA 14installation 85

4 GB tuning 99ADSI 105backbone configuration 114BOOT.INI 94, 99browser service error 95cluster logging tool 102cluster verification 117Cluster Verification Utility 96computer name 90database installation 108database verification 114DB2

See DB2, installationDHCP 91disk configuration 94DNS 91, 103domain controller 90domain name 91drivers, unnecessary 100failover test 107

hardware verification utility 96HOSTS file 94installation check 110internal disks 89Internet Explorer 105IP addresses 91LMHOSTS 91MMC 105MSCS verification 105Network Monitor 102Oracle

See Oracle, installationoverview 85page file size 98PDC 90PING tests 106private network 91protocols, unnecessary 100R3DLLINS 105sapinst account 107SAPNTCHK 113SAPService<SID> 108SAPWNTCHK 113security 86Server service 97Service Packs 94, 102

fixes 105services, unnecessary 100<sid>adm 108SQL Server

See SQL ServerSYMplicity Storage Manager 95tests 110user account 107verify the SAP install 110verifying the installation 169WINS 91, 103

installation HOSTS file 103INSTVERS 156interconnect

MSCS 17redundant network adapters, use of 82

Internet Explorer 105IP address

MSCS, potential problem 25IP addresses 91IPADDRESS

DB2 155Oracle 136SQL Server 165

IPSHA.DLL 117ISICC 49iXOS 39, 42

KKILL.EXE 118Knowledge Base articles

Q101746 94, 103Q114841 52Q158487 95


Q164023 80, 106Q168801 102Q170771 24Q171793 100Q172944 37Q172951 37Q174812 82Q175767 82Q176320 82Q193890 94, 103Q195462 94, 103Q217199 94, 103

LLMHOSTS 91LMHOSTS file 79load balancing 2, 83log files 50, 65

MSCS 170SAP R/3 172

log-based replication 11LOGFILEDRIVE

SQL Server 163logical drives 65LUNs

adding 76maximum disks 75number of 75

LVDS 64, 72

Mmemory management 121message work processes 119Microsoft Cluster Server 3

backup of open files 37certified hardware 8, 39checklist for installation 40CLUSRES.DLL 105groups

SAP-R/3 <SAPSID> 32SQL <SAPSID> 32

HCL 39log files 170Oracle Parallel Server, not compatible with 15quorum 55Service Packs 41verification 105worksheet 101

Microsoft HCL 39Microsoft Management Console 105Microsoft SQL Server Replication 12MSCS

Microsoft Cluster Serverapplication failure, support for 17dependencies between resources 19DHCP, use of 24domain requirements 27failback 17, 29failback policy 29

failover 27failover example 28failover properties for resources and groups 27failover, phases of 27failover, smallest unit of 22hardware configuration 17importance of 17IP address, potential problem 25IPX, use of 25IsAlive 29load-balancing 29LooksAlive 29managing with ICSM 27NetBEUI use of 25nodes, number supported in a cluster 17Oracle FailSafe 25preferred owner 29quorum resource 24resource group states 23resource groups 22resource hierarchy 20Resource Monitor 19, 29resource states 22resource types 20resources 18SAP R/3 25TCP/IP, role of 24virtual servers 23

multiplatform solutions 16

NNetfinity Cluster Enabler 68network configuration 76

auto-sensing adapters 81backbone 77, 79binding order 80, 100DHCP 81DNS 79heartbeat 78HOSTS file 78, 80interconnect 78load balancing 83multiple adapters 80name resolution 79PING errors with multiple adapters 80private network 78public network 77redundancy 81server names 78TCP/IP addresses 78VLAN 77WINS 79

Network Monitor 102NETWORKNAME


NETWORKTOUSEDB2 155Oracle 136

Index 201

SQL Server 165NLS_CHARACTERSET

Oracle 134, 137NTCLUSCD.R3S 135NTCMIGNA.R3S 136NTCMIGNB.R3S 139

Ooffline backups 37Oracle

accounts 176ARCH 56central instance 133CIHOSTNAME 144cluster conversion files 135convert to cluster 136DBHOSTNAME 137DBWR 57installation 129Instance Manager 138, 140IPADDRESS 136LGWR 56LISTENER.ORA 143migration to MSCS, completing 143NETWORKNAME 136NETWORKTOUSE 136NLS_CHARACTERSET 134, 137Node A installation 130Node B installation 131Oracle Fail Safe 131

converting to 139group ORACLE<SID> 141installing 131Is Alive interval 142OFS Repository 57patch 2.1.3.1 132pending timeout 142

OracleTNSListener80 service 132OSS notes 129patch 8.0.5.1.1 131PORT 135, 137PROCESSES 135R2_CONNECTION 135, 138R3Setup installation 133RAM_INSTANCE 134, 137redo logs 56, 57SAPDATA 134, 137SAPDATA_HOME 134SAPDBHOST 142SAPLOC 134, 137SAPNTDOMAIN 134, 137SAPSYSNR 134, 137, 144SAPSYSTEMNAME 133, 144SAPTRANSHOST 134security planning 88services 173tablespaces 57TNSNAMES.ORA 143verify installation 114worksheets

central instance 133cluster conversion 136cluster conversion files 135, 136MSCS, migration to 144OFS group 141Oracle Fail Safe 132Oracle install 130Oracle patch 131R3Setup 133

Oracle Fail Safeconverting to 139Is Alive interval 142OFS Repository 75pending timeout 142verify installation 114worksheet 132

Oracle Parallel Server 14, 68MSCS, not compatible with 15

Oracle Standby Database 12Oracle Symmetric Replication 12OS/390 and NT solutions 16

PPAE 53page file 50, 51, 53, 54, 98, 121PCI scan order 67PDC 90PING errors with multiple adapters 80PING tests 106PORT

DB2 152Oracle 135, 137SQL Server 163

private network 78, 91PROCESSES


PSE 53public network 77

QQuickSizer 50quorum 50, 55, 65

SCSI heartbeat cable 66quorum resource

MSCS 24

RR2_CONNECTION


R3DLLINS 105R3Setup

DB2central instance 151cluster conversion files 153


convert to cluster 154installation 150

migration to MSCS 166Oracle

central instance 133cluster conversion files 135convert to cluster 136install 133migration to MSCS, completing 143worksheet 133

SQL Server 161cluster conversion tool 164SAP cluster conversion 165

RAID-1, use of 51RAM_INSTANCE

DB2 152, 155Oracle 134, 137SQL Server 163, 166

ratio of work processes 120redo logs 13, 56redundancy

server components 17redundant network adapters 82

use on interconnect 82redundant network path 81remote shell 127replicated database 11

asynchronous 11BLOBs 12failover 14issues 11, 13levels 12log-based 11products 12redo logs 13replicated DBMS server, compared with 15SAP R/3, use with 12standby databases 12statement-based 11synchronous 11

replicated DBMS server 14failover 14issues 15Oracle Parallel Server 14redo logs 14replicated database, compared with 15

Replicated Standby Database for DB2/CS 12REPORT_NAMES

DB2 152SQL Server 163

resourceSee MSCS

RMPARAMS file 76RZ10 111

SS/390 and NT solutions 16SAP QuickSizer 50SAP R/3

availability features 5

buffer analysis 125certified hardware 8, 39checklist for installation 40cluster verification 117configurations supported 8connection test 110database reorganization 126dialog traffic 121disk space required 41features, availability 5hardware certification 39hardware minimums 40installation 85installation check 110log files 172MSCS, use with 7multiple instances 5network configuration 76process restart 5profiles 111quorum 65reconnect 5remote shell 127restart of processes 5security 86single points of failure 6sizing 46system log 110tests 110tuning 119, 122

publications 126ServeRAID 67

update traffic 121verify cluster operation 117verify the installation 110work processes 119

SAPCISYSNRSQL Server 162, 165

SAPDATADB2 151Oracle 134, 137

SAPDATA_HOMEOracle 134

SAPLOCDB2 151, 156Oracle 134SQL Server 165

SAPLOGON 120SAPNTCHK 113SAPNTDOMAIN

DB2 151, 155Oracle 134, 137SQL Server 162, 166

SAP-R/3 <SAPSID> cluster group 32SAPS 46SAPService<SID> 108SAPSYSNR

DB2 151, 154, 157Oracle 134, 144

SAPSYSTEMNAME

Index 203

DB2 150, 151, 154, 157Oracle 133, 144SQL Server 162, 165

SAPTRANSHOST 104DB2 151Oracle 134SQL Server 162

SAPWNTCHK 113scan order 67scheduling backups 36SCSI

recommendations 67security 86segment size 76Server service 97ServeRAID 3, 62

battery backup cache 67cache policy 59, 67clustering 66configurations 39, 62EXP15 62failover 65hot spares 65logical drives 65LVDS 64merge groups 51quorum 55RAID levels 59read ahead 67recommendations 67scan order 67stripe size 67tuning recommendations 67Ultra2 SCSI 64

ServerProven 42Service Packs 41services


shadow databaseSee replicated database

shared disks 55shared nothing 50<SID> 50<sid>adm 108single points of failure 6sizing 46

methodology 47, 48Netfinity servers 54SAPS 46tools 50transaction-based sizing 48user-based sizing 48

SM21 110SM28 110SM51 112SMSTEMP

DB2 152SQL <SAPSID> cluster group 32

SQL Server 159accounts 178CDINSTLOGOFF_NT_IND 162, 165central instance 162CIHOSTNAME 167client component 61cluster conversion tool 164DATAFILEDRIVE 163DBHOSTNAME 166devices 61disk layout 61Failover Cluster Wizard 164installation 159, 160IPADDRESS 165LOGFILEDRIVE 163MSCS, migration to 166NETWORKNAME 165NETWORKTOUSE 165Node A 160PORT 163PROCESSES 163R2_CONNECTION 166R3Setup install 161RAID, use of 61RAM_INSTANCE 163, 166REPORT_NAMES 163SAP cluster conversion 165SAP gateway 163SAP service 164SAPCISYSNR 162, 165SAPLOC 165SAPNTDOMAIN 162, 166SAPOSCOL service 164SAPSYSTEMNAME 162, 165SAPTRANSHOST 162Service Pack 161services 164, 175TEMPDATAFILE 163verify installation 114verify the install 161worksheets

central instance 162cluster conversion tool 165Failover Cluster Wizard 164migration to MSCS 167R3Setup install 162SAP cluster conversion 165SQL Server 7 160

SQL1390C (error message) 153ST02 125ST04 123standby database 12standby server 16statement-based replication 11stripe size

Fibre Channel 76ServeRAID 67

swing-disk 3SYMplicity Storage Manager 74, 76, 95

RMPARAMS file 76


synchronous replication 11

Ttape drives 34TCP/IP

DHCP in MSCS configurations 24MSCS 24

TCP/IP addresses 78TEMPDATAFILE

SQL Server 163transaction codes

DB13 37, 126RZ10 111SM21 110SM28 110SM51 112ST02 125ST04 123

transport host 104trouble shooting 169tuning 67, 119

block size 67database tuning 127Fibre Channel 76page file size 98recommendations 67stripe size 67Windows NT 96

UUltra2 SCSI 64update traffic 121update work processes 119UPDINSTV.R3S 143, 144, 156UPS 39

Vverification 117verifying the installation 169VHDCI cables 66Vinca StandbyServer 9virtual names for backup 36virtual servers

MSCS 23VLAN 77

Wwhat the user sees 4Windows 2000

certification 45PAE memory addressing 53

Windows NT4 GB tuning 99accounts 88backup copy 53, 90drive letters 55Event Viewer errors 98Network Monitor 102

NT Resource Kit 96page file 50, 51, 53, 54, 98, 121security 87Server service 97Service Packs 41, 94, 102

fixes 105tuning 96, 119Windows Load Balancing Service 2

Windows NT ServerUNX, as an alternative to 17

WINS 7, 79, 91, 103work processes 119worksheets

DB2central instance 151cluster conversion files 153convert to cluster 154MSCS, migrating to 157Node A installation 147Node B installation 148R3Setup on Node A 150

general 90Microsoft Cluster Server 101Node A 90Node B 92Oracle

central instance 133cluster conversion 136cluster conversion files 135, 136installation 130MSCS, migration to 144Oracle Fail Safe 132Oracle Fail Safe group 141patch 131R3Setup 133

SQL Servercentral instance 162cluster conversion tool 165Failover Cluster Wizard 164migration to MSCS 167R3Setup install 162SAP cluster conversion 165SQL Server 7 install 160


ITSO redbook evaluation

Implementing SAP R/3 4.5B Using Microsoft Cluster Server on IBM Netfinity ServersSG24-5170-01

Your feedback is very important to help us maintain the quality of ITSO redbooks. Please complete thisquestionnaire and return it using one of the following methods:

• Use the online evaluation form found at http://www.redbooks.ibm.com• Fax this form to: USA International Access Code + 1 914 432 8264• Send your comments in an Internet note to [email protected]

Which of the following best describes you?_ Customer _ Business Partner _ Solution Developer _ IBM employee_ None of the above

Please rate your overall satisfaction with this book using the scale:(1 = very good, 2 = good, 3 = average, 4 = poor, 5 = very poor)

Overall Satisfaction __________

Please answer the following questions:

Was this redbook published in time for your needs? Yes___ No___

If no, please explain:

What other redbooks would you like to see published?

Comments/Suggestions: (THANK YOU FOR YOUR FEEDBACK!)

SG24-5170-01

Printed in the U.S.A.

Implem

entingSA

PR

/34.5B

Using

Microsoft

Cluster

Serveron

IBM

Netfinity

ServersSG

24-5170-01

Implementing SAP R3 4.5B Using Microsoft

Documents

Transcript of Implementing SAP R3 4.5B Using Microsoft