Lotus Domino for S/390 Release 5: Performance Tuning … Tuning and Capacity Planning Mike Ebbers...

ibm.com/redbooks

Lotus Domino for S/390 Release 5:Performance Tuning andCapacity Planning

Mike EbbersMisao Ishikawa

Mike LangMilan Otajovic

Max Santos

How to plan for Domino capacity on an OS/390 system

How to tune an OS/390 system for Domino performance

How to tune and monitor Domino for OS/390

http://www.redbooks.ibm.com/


International Technical Support Organization SG24-5149-02

Lotus Domino for S/390 Release 5:Performance Tuning andCapacity Planning

January 2001

© Copyright International Business Machines Corporation 1998, 2000, 2001. All rights reserved.Note to U.S Government Users - Documentation related to restricted rights - Use, duplication or disclosure is subject to restrictionsset forth in GSA ADP Schedule Contract with IBM Corp.

Third Edition (January 2001)

This edition applies to Domino for S/390 V5R5.0.4, Program Number 5655-B86 for use with the OS/390 Version 2 operating system, Releases 6 through 10.

Comments may be addressed to:IBM Corporation, International Technical Support OrganizationDept. HYJ Mail Station P0992455 South RoadPoughkeepsie, NY 12601-5400

When you send information to IBM, you grant IBM a non-exclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you.

Before using this information and the product it supports, be sure to read the general information in Appendix E, “Special notices” on page 141.

Take Note!

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiThe team that wrote this redbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiComments welcome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

Chapter 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11.1 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11.2 The unique benefits of S/390 as a Domino platform . . . . . . . . . . . . . . . . . .21.3 More to come . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2

Chapter 2. OS/390 tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52.1 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .52.2 Planning the server architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

2.2.1 OS/390 address spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .62.2.2 Multiple Domino servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .92.2.3 Domino on symmetric multiprocessors . . . . . . . . . . . . . . . . . . . . . . . .92.2.4 Setting OS/390 parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

2.3 Managing the Domino workload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .102.3.1 Workload Manager goal mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112.3.2 Compatibility mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12

2.4 Domino's use of storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .132.4.1 Processor storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .142.4.2 Page data sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .152.4.3 Place program modules in LPA . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16

2.5 Monitoring OS/390 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182.5.1 Monitoring OS/390 with RMF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .182.5.2 Monitoring OS/390 with SMF. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .25

Chapter 3. DASD subsystem tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .293.1 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .293.2 DASD subsystem considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29

3.2.1 IBM RAMAC Virtual Array (RVA) . . . . . . . . . . . . . . . . . . . . . . . . . . . .303.2.2 The Enterprise Storage Server (ESS) . . . . . . . . . . . . . . . . . . . . . . . .30

3.3 Monitoring DASD I/O activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .323.3.1 RMF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .323.3.2 SMF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .343.3.3 UNIX statistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34

3.4 DFSMS/MVS 1.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .343.4.1 HFS performance enhancement . . . . . . . . . . . . . . . . . . . . . . . . . . . .343.4.2 Setting DFSMS parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .363.4.3 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .373.4.4 Recommendation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .42

3.5 Hierarchical file system (HFS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .423.5.1 HFS dataset allocation and DFSMS/MVS . . . . . . . . . . . . . . . . . . . . .423.5.2 HFS dataset management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .433.5.3 Domino I/O statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .45

3.6 Multivolume HFS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .463.6.1 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .463.6.2 Planning the HFS structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .473.6.3 Multiple extents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .473.6.4 Recommended HFS structure for UNIX System Services . . . . . . . . .483.6.5 Recommended HFS structure for Domino . . . . . . . . . . . . . . . . . . . . .51

© Copyright IBM Corp. 2000 iii

3.6.6 Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543.6.7 SMS-managed or not . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543.6.8 Monitoring HFS space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Chapter 4. Network tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574.1 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574.2 TCP/IP. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

4.2.1 TCP/IP release level . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574.2.2 Monitoring TCP/IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 574.2.3 Tuning TCP/IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

4.3 Network connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634.3.1 Monitoring connection activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

4.4 Network performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 644.4.1 Splitting the network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

4.5 Domino network statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

Chapter 5. Domino tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675.1 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675.2 Domino initialization parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

5.2.1 Changing parameters in notes.ini. . . . . . . . . . . . . . . . . . . . . . . . . . . 685.2.2 The Notes_SHARED_DPOOLSIZE parameter . . . . . . . . . . . . . . . . . 685.2.3 Thread parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 695.2.4 Make Domino server tasks non-swappable . . . . . . . . . . . . . . . . . . . 69

5.3 NSF bufferpool size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715.3.1 Setting the parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715.3.2 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715.3.3 Tuning NSF_Buffer_Pool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 725.3.4 Recommendation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

5.4 Transaction logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 745.4.1 Setting up transaction logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 755.4.2 Test result of transaction logging . . . . . . . . . . . . . . . . . . . . . . . . . . . 755.4.3 Recommendation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

5.5 Domino server tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 765.5.1 Domino tasks started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

5.6 Replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 785.6.1 Replication between servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 795.6.2 Replication between a client and a server . . . . . . . . . . . . . . . . . . . . 79

5.7 Indexing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 805.7.1 Update and Updall tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 815.7.2 Full text index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 815.7.3 Number of indexing tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825.7.4 Indexing parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

5.8 Compact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825.9 Unread marks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 845.10 Multiple MAIL.BOX database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 845.11 Web server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

5.11.1 Domino Web server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 855.11.2 Domino Web server parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . 865.11.3 Domino for IBM HTTP server connector. . . . . . . . . . . . . . . . . . . . . 86

Chapter 6. Domino tuning (advanced functions) . . . . . . . . . . . . . . . . . . . . 896.1 Partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

6.1.1 Costs of partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 896.1.2 Efficiency of partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92

iv Lotus Domino for S/390 Release 5: Performance Tuning & Capacity Planning

6.2 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .936.2.1 Cluster components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .936.2.2 Parameters that affect clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . .946.2.3 Costs of clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .946.2.4 Monitoring clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .94

Chapter 7. Domino monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .977.1 SMF type 108 record. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .97

7.1.1 Record mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .987.1.2 RMF Lotus Domino server report . . . . . . . . . . . . . . . . . . . . . . . . . .106

7.2 Notes log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1117.3 Statistics and events log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1117.4 Statistics and reporting database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112

7.4.1 Reporting period . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1127.4.2 Domino statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1127.4.3 Coordinating SMF, RMF and Domino data collection. . . . . . . . . . . .1127.4.4 Statistics commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .112

7.5 Processing Domino statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1137.5.1 Exporting the contents of a database . . . . . . . . . . . . . . . . . . . . . . .113

Chapter 8. Capacity planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1158.1 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1158.2 Estimating processor capacity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .116

8.2.1 Calculating the processor capacity needed . . . . . . . . . . . . . . . . . . .1168.2.2 Number of users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1168.2.3 Transaction rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1168.2.4 CPU time used per transaction . . . . . . . . . . . . . . . . . . . . . . . . . . . .1178.2.5 Consider the peak times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1178.2.6 Making an initial estimate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1188.2.7 Capacity planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .118

8.3 Estimating processor storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1188.3.1 Estimating processor storage for Notes clients . . . . . . . . . . . . . . . .1198.3.2 Estimating processor storage for POP3 clients . . . . . . . . . . . . . . . .1208.3.3 Estimating processor storage for IMAP clients . . . . . . . . . . . . . . . .1218.3.4 Estimating processor storage for WebMail clients . . . . . . . . . . . . . .121

8.4 The use of Domino partitioned servers . . . . . . . . . . . . . . . . . . . . . . . . . .1228.5 The use of S/390 LPARs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1228.6 Estimating DASD required . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .122

8.6.1 Estimating DASD space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1228.6.2 Estimating DASD I/O rates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .123

8.7 Network considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1238.8 The production Domino for S/390 servers running in Poughkeepsie . . . .123

8.8.1 Hardware and software environment . . . . . . . . . . . . . . . . . . . . . . . .1248.8.2 Workload. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .125

8.9 Other production Domino for S/390 servers . . . . . . . . . . . . . . . . . . . . . .1258.9.1 A large production DPAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1258.9.2 A large POP3 installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1268.9.3 Leading edge installations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .126

8.10 Capacity planning assistance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1268.10.1 TechXpress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1268.10.2 SNAP/SHOT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1278.10.3 IBM Testing Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .127

v

Appendix A. Workload Manager (WLM) example . . . . . . . . . . . . . . . . . . . . .129A.1 Workload and service class descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . .129A.2 Report classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .130A.3 Classification rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .130A.4 Service definition goals for service classes . . . . . . . . . . . . . . . . . . . . . . . . . .131

Appendix B. Domino parameters that affect performance . . . . . . . . . . . . .133B.1 System parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133B.2 Monitoring parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133B.3 Administration automation parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . .133B.4 Indexing parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .134B.5 Mailing parameters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .134

Appendix C. Sample notes.ini file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .135

Appendix D. Sample output of the show statistics command . . . . . . . . . .137

Appendix E. Special notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .141

Appendix F. Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .143F.1 IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .143F.2 IBM Redbooks collections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .143F.3 Other resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .143F.4 Referenced Web sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .144F.5 Lotus Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .144

How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145IBM Redbooks fax order form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .146

Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

IBM Redbooks review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161

vi Lotus Domino for S/390 Release 5: Performance Tuning & Capacity Planning

Preface

This IBM Redbook helps OS/390 system programmers and Domino administrators to monitor and tune Domino for S/390. It identifies factors that affect Domino R5 performance on S/390, and supplies recommendations to tune the configuration and parameter settings for optimal performance.

In addition, it provides guidelines for capacity planning. A background in OS/390 and experience with Domino is assumed.

This updated version describes the new SMF type 108 records and covers clustering, shared CPs, and RPCs.

The team that wrote this redbook

This redbook was produced by a team of specialists from around the world working at the International Technical Support Organization Poughkeepsie Center.

Mike Ebbers is a Senior I/T Specialist at the International Technical Support Organization, Poughkeepsie Center. He writes about and teaches IBM classes on all areas of Domino for S/390. Mike has been with IBM for 27 years; 10 in the field, 11 in education and six with the ITSO.

Misao Ishikawa is an Advisory I/T Specialist, supporting Lotus Domino for S/390 and Intelligent Miner for OS/390 in IBM Japan Ltd. since 1998. He joined IBM in 1987 and provided various kind of technical supports, including database areas such as IMS and DB2 for six years. He worked on the NTT CUSTOM project and the 1998 Nagano Olympics project as a DB2 specialist, supporting DB2 for OS/390.

Mike Lang is a Consulting IT Specialist in the IBM Lotus Integration Center in the Dallas Systems Center located in Dallas, Texas. He has provided support for OS/390 office systems for 20 years and is currently supporting Domino for S/390. He specializes in capacity planning and performance tuning. Prior to joining the Dallas Systems Center, Mike was a Systems Engineer for 13 years in Cincinnati, Ohio supporting CICS and IMS application development.

Milan Otajovic is a Sales Specialist in IBM Slovaka. He has 12 years of experience in the S/390, MVS and OS/390 fields. He has worked at IBM for six years. His areas of expertise include MVS system programming, DB2 administration and application programming, CICS applications programming, and Lotus Domino.

Max Santos is an IT Specialist at a large bank in Brazil. He has eight years of experience on UNIX systems and other platforms. His areas of expertise also include TCP/IP network applications, network design, Web services, and Domino servers. He can be reached by e-mail at [email protected].

Thanks to the following people for their invaluable contributions to this project:

Richard ConwayRobert HaimowitzInternational Technical Support Organization, Poughkeepsie Center

© Copyright IBM Corp. 2000 vii

Robert BeattieJoseph BostianMark HoffstatterSonomi MukaidaIBM Server Group, Poughkeepsie

Jim GoldingSreehari HaridevaraJohn JustinElsie RamosIBM Global Services, Poughkeepsie

Robert Morse IIIIBM Washington Systems Center, Gaithersburg

Comments welcome

Your comments are important to us!

We want our Redbooks to be as helpful as possible. Please send us your comments about this or other Redbooks in one of the following ways:

• Fax the evaluation form found in “IBM Redbooks review” on page 161 to the fax number shown on the form.

• Use the online evaluation form found at ibm.com/redbooks

• Send your comments in an Internet note to [email protected]

viii Lotus Domino for S/390 Release 5: Performance Tuning & Capacity Planning



http://www.redbooks.ibm.com/contacts.html

Chapter 1. Introduction

This book provides information to help you tune the performance of Lotus Domino for S/390 Release 5, and to assist you in planning the capacity for a Domino workload on a S/390 platform.

As you approach this topic, it is useful to remember the following:

• Domino was originally written for the PC platform. Therefore, it was developed with different requirements from those for traditional S/390 applications, such as CICS, IMS and DB2. One of our objectives is to explain the characteristics of Domino so that you are better able to monitor and tune the system.

• Domino was written to be platform-independent. This means that it does not exploit all OS/390 functions directly. Contrast that with CICS, IMS, and DB2, which were all written for OS/390 and designed specifically to exploit S/390 capabilities.

• Domino was ported to the OS/390 UNIX System Services (formerly called OpenEdition). This book will guide you in performance considerations for OS/390 UNIX.

This redbook assumes knowledge of Lotus Domino for S/390, which can be obtained by reading Lotus Domino for S/390 Install Guide for Servers, which comes with the product. In addition, Lotus Domino for S/390 Release 5: Installation, Customization and Administration (in particular, Chapter 15) describes how Domino interfaces to OS/390 and recommends values for many OS/390 parameters. We do not repeat that information here, except where necessary for the topic being discussed.

1.1 Recommendations

Our key recommendations for performance tuning, performance monitoring, and capacity planning of Lotus Domino for S/390 are:

• Monitor OS/390 closely as you introduce Domino.

Domino does not run the same way as other S/390 workloads that you may be used to. Make sure that you are not artificially constraining Domino with parameter settings designed for a different type of application.

• Use a team approach. Domino tuning on S/390 is a combination of:

• OS/390 tuning, including UNIX System Services

• Domino server tuning

• DASD tuning

• Network tuning

Therefore, the Notes administrator, the OS/390 performance specialist, the DASD administrator, and the network administrator must work together to optimize Domino performance.

• Consult our list of tuning recommendations at the start of each chapter in this book.

• Use valid measurements for capacity planning.

© Copyright IBM Corp. 2000 1

We discuss some valid capacity planning approaches in Chapter 8, “Capacity planning” on page 115. Avoid using simple single-function tests as an indicator of the capacity of S/390 as a Domino server. These tests are easy to run, but they can give very misleading results.

1.2 The unique benefits of S/390 as a Domino platform

While S/390 appears to Notes users as just another server, the S/390 server platform provides a lot more function than other servers. The S/390 platform, along with its OS/390 operating system (previously called MVS), has a long history in the commercial marketplace. It has matured into a powerful server platform that meets the needs of users of commercial systems.

Compare that to the history of UNIX in the commercial marketplace, as well as the PC operating systems, which are relatively young. Key functions, such as clustering (for scalability and availability) and systems management tools, that are important in commercial systems, have been available on S/390 for many years and are more mature than on many other servers. Users of S/390 recognize and value that maturity.

A S/390 server is able to support large workloads because IBM has invested in many capabilities in the processor and software to provide scalability. Here is a list of the key differences that you need to be aware of in your planning:

• Very efficient use of symmetric multiprocessors (SMPs), even up to 12-engine machines

• Addressing of 4 terabytes of expanded processor storage, along with the 31-bit (2 GB) storage addressing that it has had for many years

• Efficient techniques for keeping data in processor memory where it can be accessed quickly

• A sophisticated job scheduler

• The removal of bottlenecks

• Shared access to all disks by all processors

• Separate, dedicated I/O processors

Many servers may claim large capacity numbers. Not many are achieving it in practice. Domino for S/390 has earned a certified Notesbench run of 32,000 mail users with response times of less than 1 second (see http://www.notesbench.org). Each new release of code brings additional performance and capacity.

1.3 More to come

We expect to update this book with new tips and techniques from time to time. There are performance and capacity improvements in plan for:

• Lotus Domino

• OS/390 and its OS/390 UNIX interface

• OS/390 Hierarchical File System

• S/390 hardware

2 Lotus Domino for S/390 Release 5: Performance Tuning & Capacity Planning

However, we hope that this current work will assist you to better tune and manage your Domino servers on S/390.

Note: We also recommend two other redbooks that contain information on tuning:

1. Lotus Domino for S/390 Release 5: Installation, Customization and Administration. This redbook tells how to install Domino using the recommended parameter values and environment setup conditions.

2. Lotus Domino for S/390 Release 5: Problem Determination Guide. This redbook tells how to diagnose and solve problems, including those related to performance.

Chapter 1. Introduction 3

Chapter 2. OS/390 tuning

This chapter describes how Domino interfaces to OS/390, using the Domino address space and task structure. We discuss the tuning considerations for OS/390 when you are running one or more Domino servers. We discuss this topic in detail; but first, here is a summary of our recommendations.

2.1 Recommendations

These are the key recommendations for tuning OS/390 to run Domino:

• Make sure you have enough storage for Domino.

Domino creates multiple address spaces, some of which have large working sets. OS/390 UNIX System Services also makes extensive use of storage for performance in the kernel address space and DFSMS dataspaces. You need enough processor storage for these or you will page to DASD, which will impact Domino response times.

• In the BPXPRMxx member in SYS1.PARMLIB, check your OS/390 UNIX System Services settings and make them large enough.

There are several OS/390 UNIX System Services parameters in BPXPRMxx which, if too low, will cause problems in the Domino server. The server may not be able to create new tasks or get the virtual storage it requires. See Domino S/390 5.0 Install Guide for the recommended values and follow them. It ships with the product and is available on the Web at:

http://notes.net/doc

On the Web, from View the documentation by, select Product name, select S/390, and select the release number of the documentation that you would like to access (for example, 5.0.5).

Also see Lotus Domino for S/390 Release 5: Installation, Customization and Administration, SG24-2083 (edition 02) for further information.

• Make the Domino server a high priority OS/390 workload.

The Domino server is an online system. It needs the same level of service that you would give to your CICS or IMS production systems. If the priority within OS/390 is not set high enough, then response times to the Notes client will suffer and client requests will time out.

• If you are running on a G3 or a G4 processor, we recommend that you upgrade your microcode level to level 98, which was available in November 1997 for the Generation 4 processors and in January 1998 for the Generation 3 processors.

• Tune and monitor the entire OS/390 system.

If you add a Domino server to a badly tuned OS/390 system, it will suffer. Do all the things you would normally do for performance. Get IBM to help set up the system before you install the Domino server, if you need the expertise. Then as you start to use Domino, monitor the system closely to see how Domino performs.

• Check out the OS/390 UNIX System Services performance tuning tips on the World Wide Web at:


http://www.s390.ibm.com/products/oe/bpxpftgt.html

You can also read Debugging UNIX System Services, Lotus Domino, Novell Network Services, and other Applications on OS/390.

2.2 Planning the server architecture

Now we get into more details about the OS/390 environment and some tuning techniques. This section deals with how Domino uses address spaces in OS/390, how to run multiple Domino servers in one system or LPAR, and what OS/390 parameters to set for optimum performance.

2.2.1 OS/390 address spacesLotus Domino for S/390 Release 5 uses multiple OS/390 address spaces, as shown in Figure 1.

Figure 1. Domino tasks in OS/390 address spaces

At initialization, the main server address space is always started. Additional server address spaces are created at startup, depending on the functions requested in the initialization control file, called notes.ini. These additional server address spaces perform background tasks or functions requested by threads in the main server address space in order to service client requests. Server address spaces can also be started at the request of the Domino administrator while the Domino server is running. Domino also dynamically starts and stops server address spaces to perform some long running background tasks.

The Domino address spaces are created and managed by the Workload Manager. After a Domino process has completed, the Workload Manager keeps the address space available for 30 minutes to allow reuse of the address space by another Domino process. Address spaces are retained in both Workload Manager goal mode and compatibility mode. Domino address spaces retained by

Main Server Task

Server Tasks as defined in notes.ini


the Workload Manager keep their Domino job name but have a step name of BPXAS.

2.2.1.1 Address spaces for user connectionsIn R4 (the previous release of Domino), as each client connected to the Domino server, it was assigned a thread. This thread had its own subtask within an address space and thus its own task control block (TCB). A user address space could contain 100 threads. Thus the first 100 users executed in one address space. The 101st user caused a new address space to be created, which contained the next 100 users. As a user connection was terminated, the thread became available for reuse.

This model was dropped in Domino R5 in favor of a thread pool model. In this design, each server thread supports many connections. Figure 2 shows the thread pool model and how the server process uses it.

Figure 2. Thread pool and server process

When a user connects to the server, a client session is created with a unique socket descriptor and a virtual thread ID is assigned. When a client performs I/O, a physical thread is associated with the virtual thread and does the required work on behalf of the client’s session. After the work is completed, the physical thread is returned to the pool of threads. It is no longer associated with the virtual thread and is available to service another request. With this reusable thread model, the server needs far fewer threads than the maximum number of concurrent user connections.

By utilizing a single main server address space and a small fixed number of threads to service client requests, Domino's footprint on the system is much smaller. This means less contention for common system resources, and less memory consumed per concurrent user request. The end result is a much more linear scale curve with none of the bottleneck problems of the Release 4 server.

2.2.1.2 Address space namingThe Domino server creates all its address spaces with a job name of the user ID under which the server is started, followed by a number. This is shown in Figure 3

Chapter 2. OS/390 tuning 7

on page 8. This system is running a Domino server. The display in Figure 3 is produced using the MVS command D A,L.

Figure 3. Domino address space names - D A,L command

The Domino server was started from the user ID BETA and therefore the server address space names are BETAx. As you see, you can have duplicate job names. You cannot predict what the job name will be for a specific Domino function. Therefore, you cannot differentiate between the various address spaces created by a Domino server.

Note: Beginning with Release 5.03 of Domino, you can optionally name the Domino address spaces with your own naming convention, by overriding the default jobname prefix.

2.2.1.3 Address space and program nameIf you wish to obtain the function names of address spaces on a running Domino server you should issue the OS/390 command D OMVS,U=userid. This user ID is the one which is used when you start the server. Figure 4 is a sample output of this command.

Figure 4. Output from D OMVS,U=userid command

The function being performed by this Domino address space can be determined from the program name, which is the last qualifier in the command display shown

LOTUS4 d a,lLOTUS4 IEE114I 19.01.46 1999.335 ACTIVITY 667JOBS M/S TS USERS SYSAS INITS ACTIVE/MAX VTAM OAS00012 00007 00000 00025 00022 00000/00300 00018VLF VLF VLF NSW S LLA LLA LLA NSW SVTAM VTAM VTAM NSW S JES2 JES2 IEFPROC NSW SRMF RMF IEFPROC NSW S TCAS TCAS TSO OWT SSYSLOGD5 STEP1 IBMUSER IN AO TCPIPA TCPIPA TCPIP NSW SOINETD1 STEP1 IBMUSER OWT AO FTPOE1 STEP1 IBMUSER OWT AOBETA STEP1 BETA IN AO BETA8 STEP1 BETA OWT AOBETA5 STEP1 BETA NSW AO BETA6 STEP1 BETA NSW AOBETA7 STEP1 BETA OWT AO BETA8 STEP1 BETA IN AOBETA9 STEP1 BETA IN AO BETA9 STEP1 BETA IN AOBETA1 STEP1 BETA IN AO

LOTUS4 d omvs,u=betaLOTUS4 BPXO040I 19.05.24 DISPLAY OMVS 676OMVS 000E ACTIVE OMVS=(X4)USER JOBNAME ASID PID PPID STATE START CT_SECSBETA BETA9 0040 402653191 905969679 1D 18.59.22 .397LATCHWAITPID= 0 CMD=/usr/lpp/lotus/notes/latest/os390/amgr

BETA BETA9 002A 838860813 536870939 1D 18.59.53 .319LATCHWAITPID= 0 CMD=/usr/lpp/lotus/notes/latest/os390/calcon

BETA BETA1 003B 536870926 536870939 1S 19.00.04 4.188LATCHWAITPID= 0 CMD=/usr/lpp/lotus/notes/latest/os390/sched

BETA BETA8 0034 905969679 536870939 1R 18.59.18 58.332LATCHWAITPID= 0 CMD=/usr/lpp/lotus/notes/latest/os390/amgr

BETA BETA 002E 939524115 318767126 1W 11.38.41 6.576LATCHWAITPID= 0 CMD=sh -L

BETA BETA 002E 318767126 9 1F 11.38.05 6.576LATCHWAITPID= 0 CMD=rlogind2 -a -m dtterm/9600 beta lotush.p

BETA BETA5 0031 536870939 402653212 HS 18.58.15 16.768LATCHWAITPID= 0 CMD=/usr/lpp/lotus/notes/latest/os390/server

BETA BETA8 003E 402653212 939524115 1WI 18.58.13 .827LATCHWAITPID= 0 CMD=sh -L

BETA BETA7 0039 251658269 536870939 1S 18.59.10 .336LATCHWAITPID= 0 CMD=/usr/lpp/lotus/notes/latest/os390/stats

BETA BETA6 003A 1442840707 536870939 1S 18.58.44 .532LATCHWAITPID= 0 CMD=/usr/lpp/lotus/notes/latest/os390/router


in Figure 4 on page 8 (the CMD field). For example, the last address space is executing the router program.

Note: If the path name is too long, the CMD field may not be large enough to display the full program name.

A list of the program names is provided in 5.5, “Domino server tasks” on page 76. For a full list, refer to the Domino 5 Administration Help database. Note that the program names may appear in upper case or lower case, but they are the same program.

2.2.2 Multiple Domino serversMultiple Domino servers can run on a single OS/390 image. An instance of a Domino server is called a Domino partition (DPAR). Each Domino partition has the set of address spaces just described. Each Domino partition is independent of other Domino partitions, with its own address spaces and HFS files.

Two Domino partitions use TCP/IP to communicate and transfer data. This means they are not dependent on being in the same image or processor.

This gives you a high degree of flexibility on where to run a Domino server. It can easily be moved from one OS/390 image to another. The system and user files for the Domino partition can be unmounted from the HFS directory on one OS/390 image and mounted in the directory of another OS/390 image.

As the number of users increases, it may be necessary to create more Domino partitions. While in Domino Release 4 it was often necessary to use several DPARs to support a large number of users, this is not necessarily the case with Release 5. The number of production users that can be supported by a single Release 5 DPAR is significantly higher than for Release 4 because many of the bottlenecks that affected the scale characteristics of the server have been removed. The actual recommendations for the number of users in each Domino partition are in 6.1.2, “Efficiency of partitioning” on page 92.

You can run multiple Domino partitions in the same OS/390 image, or in different PR/SM Logical Partitions (LPARs) on a single processor, or spread across more than one processor. You can configure the Domino servers to meet the capacity required to support your Notes clients.

2.2.3 Domino on symmetric multiprocessorsDomino makes effective use of multiple processors by using multiple threads and processes to execute work concurrently.

You can specify that more than one copy of a server function is started in the notes.ini initialization parameters, if required. The following is an example.

ServerTasks = Replica,Replica,Router,Update,Stats........

2.2.4 Setting OS/390 parametersIt is important to set up OS/390 and OS/390 UNIX System Services to allow for the number of address spaces created by a Domino server. Since Domino also makes extensive use of storage, it is important that the OS/390 UNIX System Services storage parameters in SYS1.PARMLIB(BPXPRMxx) are set to the recommended values.


These values are listed in Domino S/390 5.0 Install Guide. They are discussed in detail in Lotus Domino for S/390 Release 5: Installation, Customization and Administration, SG24-2083 (edition 02). Table 1 lists the parameters that you should check.

Table 1. OS/390 parameters

2.3 Managing the Domino workload

Your Domino servers should get the same consideration that you would give a production CICS or IMS system. It should have the resources required to give good response times to the user.

Parameter type

SYS1.PARMLIB member

Parameters Recommendation

OS/390 IEASYSxx MAXUSER Add at least 10 to your current value

OS/390 UNIX BPXPRMxx FORKCOPY COPY

IPCSEMNIDS 20,000 (maximum)

IPCSEMNOPS 32,767 (maximum)

IPCSHMMPAGES 25,600 (maximum)

IPCSHMNIDS 500 (default)

IPCSHMNSEGS 1,000 (maximum)

IPCSHMSPAGES 2,621,440 (maximum)

MAXASSIZE 2,147,483,647 (maximum)

MAXCPUTIME 2,147,483,647 (max)

MAXFILEPROC 65,535 (maximum)

MAXQUEUEDSIGS 100,000

MAXSHAREPAGES 32,768,000(max)

MAXTHREADS 1,000

MAXTHREADTASKS 1,000

NETWORK DOMAINNAME (AF_UNIX) MAXSOCKETS

10,000

NETWORK DOMAINNAME (AF_INET) MAXSOCKETS

35,000

OS/390 CSM IVTPRMxx FIXED MAX 120M

ECSA MAX 30M


We recommend that you use the Workload Manager in goal mode to manage your OS/390 system when running a Domino server. The Workload Manager can make sure that your Domino server receives the needed resources.

It is not possible, using Workload Manager goal mode or compatibility mode, to single out individual Domino functions for specific treatment. In fact, it is recommended that you do not attempt to single out individual Domino address spaces for higher or lower priority. Domino uses extensive locking between tasks, so all tasks in a Domino server should run at the same priority.

You can manage and report Domino server partitions separately if each Domino server is started under a different user ID.

If you plan to run the Domino server and other high priority workloads in the same OS/390 image, ensure that you have enough processor capacity to meet the needs of all workloads without impacting the responsiveness of each.

It is important to maintain the response time of the Domino server to the Notes client. If the Domino server does not respond within a specified period, as set on the Notes client, then the client treats this as though the server is not available. This TCP/IP session timeout is specified in the Notes Ports dialog box on the client (see Domino 5 Administration Help database). The default value is 5 seconds. The user will then have to decide whether to retry the request, switch to another server or work locally on the client. This situation should be avoided.

2.3.1 Workload Manager goal modeOS/390 Workload Manager goal mode is the recommended way to manage Domino servers. Even if you are only running a single Domino server on OS/390, goal mode simplifies the setup of your system to meet its performance objectives. It avoids potential errors that can result from setting parameters in compatibility mode. Most people find that goal mode can be set up fairly quickly on a system, even if they have not used it before.

In IBM, we use Workload Manager to control our Domino servers for a large user population. An example of the Workload Manager definitions for a set of production Domino servers can be found in Appendix A, “Workload Manager (WLM) example” on page 129.

The Domino server should be placed in a service class with a high velocity. We recommend a value of at least 60 or 70 for the velocity. The Domino server should also have a high importance relative to other workloads.

When setting up Workload Manager for a Domino server, you need to:

• Define a workload for OS/390 UNIX System Services work.

• Define service classes for OS/390 UNIX System Services work:

• OS/390 UNIX System Services startup processes which run under the USERID OMVSKERN, in the OMVS subsystem.

• Forked address spaces, such as Domino server address spaces, which run under the user ID used for the Domino server, in the OMVS subsystem.

• Define classification rules for the OMVS subsystem:

• Start up processes under USERID OMVSKERN.


• Forked address spaces, such as the Domino servers.

If you are running OS/390 V2R8 or later, consider turning the I/O priority management on. You can do this from ISPF WLM administrative application, option Service Coefficients/Options. If you have ESS installed, turn on the Dynamic Alias Management option as well. Dynamic alias management requires I/O priority management to be active. Dynamic alias management, together with the PAV feature of ESS, can greatly improve performance of your Domino installation.

2.3.2 Compatibility modeIf you are not running in Workload Manager goal mode, you have to take extra care with your compatibility mode parameters. The Domino server can create a number of address spaces, both at initialization and dynamically during the running of the server. You need to make sure that your IEAIPSxx specifications in SYS1.PARMLIB take these additional address spaces into account. IEAIPSxx contains parameters for an installation performance specification to control the Workload Manager (WLM) function of the System Resources Manager (SRM).

2.3.2.1 Multiprogramming level (MPL)In compatibility mode, the multiprogramming levels are controlled by the MPL parameter in IEAIPSxx in SYS1.PARMLIB. You must make sure that the maximum MPL for the domain is larger than the number of address spaces the server will create. If you have more address spaces than the maximum MPL, the SRM will have to keep swapping Domino address spaces to keep the number below this maximum. If a Domino address space is swapped, response time to the user will be severely affected and the Notes clients may time out. Make sure that you avoid the situation where the number of Domino address spaces exceeds the maximum MPL for the domain. In those circumstances the Domino server will not perform well, and it may not even start successfully.

2.3.2.2 PriorityMake sure that the Domino server is not waiting for CPU because of higher priority tasks. The Domino server address spaces should be above any batch processing, TSO (other than maybe a short period one) and other online systems that consume large amounts of CPU.

Only workloads that act as servers to Domino should be at a higher priority. These include:

• TCP/IP

• OMVS (OS/390 UNIX System Services Kernel address space)

• OS/390 system address spaces

• SYSBMAS address space

• Monitors such as RMF

If you are running an OS/390 system with DFSMS prior to version 1.5, you should define SYSBMAS (DFSMS buffer manager) address space with a priority higher than that of any users accessing HFS files, including Domino address spaces.

If you have two online production workloads, such as CICS and a Domino server, you will have to make a business decision on their relative importance.


The priority of address spaces in compatibility mode is controlled by the DP parameter in IEAIPSxx in SYS1.PARMLIB. The absolute priority of an address space is not important. What is important is where the address space sits in the order of priority relative to all the other address spaces in the OS/390 image.

2.3.2.3 Storage isolationIn goal mode, the Workload Manager handles the working set of central and expanded storage for you. In compatibility mode, if there is not enough free storage available, OS/390 can take storage from Domino servers that are not protected. The way to protect the Domino server address spaces is to use storage isolation. This attempts to protect the processor storage working set of selected workloads. If you use storage isolation, then OS/390 prefers to take processor storage from address spaces that do not have storage isolation.

You can specify that OS/390 controls the working set size according to the paging rate of the address space; this is recommended. You should try to achieve zero paging for the Domino address spaces. The working set controls for an address space are PWSS and PPGRTR in IEAIPSxx in SYS1.PARMLIB.

Note: Storage isolation for Domino is important if you are short of central or expanded storage. It is required if you are using storage isolation for other workloads. Otherwise, OS/390 steals pages from the Domino server and gives them to the other workloads.

2.3.2.4 Setting the definitionsTo install the definitions for running a Domino server in compatibility mode, you need to update the IEAICSxx and IEAIPSxx members of SYS1.PARMLIB as follows:

• Add a subsystem definition, SUBSYS=OMVS, for OS/390 UNIX System Services address spaces.

• In the OMVS subsystem, define a USERID=OMVSKERN for OS/390 UNIX System Services initialization processes and give it a high priority.

• Also define the USERIDs you will run the Domino servers under and give them sufficiently high priority, as discussed.

2.4 Domino's use of storage

As described in 2.2.1, “OS/390 address spaces” on page 6, a Domino server creates a large number of address spaces. Some of these address spaces, such as the main server address space, use a large amount of virtual storage, particularly in extended private (above 16 MB addressing) storage.

The virtual storage requirements for a Release 4 server were never a problem because, for every 100 users, the server creates a new address space and continues processing. So the total amount of virtual storage per address space required to service all client connections never got very large.

For Release 5, however, each connection is represented by a virtual thread which exists in the main server address space. So the storage to support the virtual threads which service the user connections is allocated from a single address space, instead of many different address spaces. While virtual storage will not be


a constraint for server loads of less than several thousand users in Release 5, it may become a consideration for very large installations.

While Domino R4 used large amounts of common storage, Domino R5 has been redesigned and now does not require much additional space in ESQA.

2.4.1 Processor storageSince the Release 5 server can support much larger workloads than the Release 4 server, it is not surprising that the server will also require a larger amount of storage to handle such loads. This processor storage can be a mixture of central and expanded storage. Make sure you have sufficient central storage to avoid excessive page movement between central and expanded storage.

You should not add Domino servers to an OS/390 image without sufficient processor storage to support the combined workloads. This would cause paging, affecting the Domino servers and the other workloads on the OS/390 image.

When you have several large Domino partitions in a single OS/390 image, or where a Domino server is added to an OS/390 image that is already using large amounts of central storage, you may reach the limit of 2 GB for an OS/390 image. You should then use expanded storage, or split the workloads over more than one LPAR. A processor can be configured with more than 2 GB of central storage, but a single LPAR can only have up to 2 GB of central storage.

The processor storage required to support a number of Domino users is given in 8.3, “Estimating processor storage” on page 118. The processor storage required is influenced by the number of address spaces and the working set of these address spaces. The working set size is the number of 4 KB pages that the program frequently references. These pages include program instructions and data. This is smaller than the virtual storage size, because there are parts of programs and data that are not used during normal running of the server. These include parts used only at Domino startup, error routines, and infrequently-used functions.

Unreferenced pages in the working set will be stolen (paged out) by OS/390 so their storage can be used by other address spaces. If the OS/390 image has far more storage than is required to run its workloads, then these pages are not stolen and the working set remains larger than is actually required.

Table 2 shows an example of processor storage use for a system that has two Domino server partitions, using the Mail with Calendar and Scheduling workload. There is some additional support-type work running. We collected this data from an RMF report.

Table 2. Processor storage use

Component Processor storage used (MB)

Total processor storage used- Central storage: - Expanded storage:

2624- 2046- 578

Shared storage 1035

OMVS 337


Table 3 shows how much storage is used for each Domino server task. We collected this data from SMF 30 records; therefore it doesn’t include the shared storage.

Table 3. Storage used for Domino tasks

2.4.2 Page data setsWhen supporting large Domino server workloads, there may be a significant increase in page space needed. Therefore, you should evaluate the size and number of page data sets on the system. The page data sets must be able to hold the virtual storage requirements of the address spaces in the OS/390 image. The size of the page data sets and the number used can be found in the PAGESP report in RMF.

Always allocate far more page dataset space than you require for the normal running of your OS/390 system. Then, if you experience an increased demand for processor storage, such as when OS/390 takes a system dump of a large address space or part of the processor storage goes offline, you will have enough page dataset space to handle these situations.

TCP/IP 31

ESQA 32

Domino server 1- 1849 registered users- 681 connected users

316

Domino server 2- 489 registered users- 144 connected users

177

Domino task Server 1 (MB) Server 2 (MB)

server 21 16

sched 1 0

amgr 90 11.9

event 1 1

adminp 3 1

collect 0.8 0.8

update 77 27

callconn 1 0.3

router 25 17

replica 4 4

stats 5 7

Component Processor storage used (MB)


2.4.3 Place program modules in LPAWe recommend that you place certain Domino and C/C++ Runtime Library programs in the Link Pack Area (LPA). (Nearly all the modules go above 16 MB in the Extended Link Pack Area (ELPA).) This reduces the virtual and processor storage used. When multiple address spaces use the same program, it is more efficient to place a single copy in common storage than a copy in each address space.

2.4.3.1 Domino program modules in LPAThe Domino server installation process provides you with sample JCL members (ALOCPROD, PUTINLPA, MDFYPROD) that place certain modules into dynamic LPA. It is important that you run these jobs. If the Domino server modules are not in LPA, approximately 35 MB of program modules will be loaded into each Domino address space.

The procedure to place these modules in LPA is described in Domino S/390 5.0 Install Guide and also in Lotus Domino for S/390 Release 5: Installation, Customization and Administration, Chapter 8, “Preparing Dynamic LPA”. We provide sample SYSLIN cards for a PUTINLPA job in Figure 5 on page 18.

You should also check that the sticky bit is set on for those programs in the Hierarchical File System (HFS), as described in these books. With the sticky bit on, the module is accessed from LPA rather than loaded from the HFS.

To make absolutely sure that you do not load the modules from the HFS into the Domino address spaces, you can replace the modules in the HFS with dummy modules, such as an empty file. Just take a backup copy of the real modules, then replace them with empty files with the same permission bit settings. Make sure that these dummy files have the sticky bit on. If everything is working correctly, the original programs are loaded from LPA. However, if there is an error, such as if the modules are removed from the LPA data sets or if the HFS sticky bits are set off, then the empty modules will be loaded. Since these are not valid modules, this will cause an error in the Domino server. This may be preferable to having the modules loaded into many address spaces, which will degrade system performance but give no indication of the problem.

The way to create such a dummy module is to change to the executable directory (usually /usr/lpp/lotus/notes/latest/os390), and execute the following commands for each module that you want to exist:

a. mv module_name backup_dir/module_name

b. touch module_name

c. chmod 754 module_name

d. chmod +t module_name

Table 4 provides a recommendation about the modules that should reside in LPA.

Table 4. Recommended module names for LPA

Module name Purpose

libnotes The main Domino Dynamic Link Library (dll). Used by all Domino executables.


Of the modules listed in Table 4, it is most important to put the first four (libnotes, ftgtr, domliba, and decsext) into LPA. If these are not put into LPA, an additional 35 MB of storage will be consumed in each Domino address space. Failure to include all four will also cause an authorization failure when the server tries to load libnotes, and the server will not start. If the server is correctly set up according to the Domino S/390 5.0 Install Guide, these modules (libnotes, ftgtr, domliba, and decsext) should be in LPA.

In addition, it is always a good idea to include the server, update, and replica executables in LPA as well. These are executables that are nearly always used, and will provide some storage savings when multiple DPARs are used within one LPAR.

The router executable should also be included in LPA for mail servers which support traditional Notes mail. If you are using your Domino server to support one of the Internet mail protocols (POP3, IMAP, or HTTP in conjunction with SMTP), then it is not necessary to put the router in LPA.

We recommend that you do not include any of the Internet protocol executables or the agent manager in LPA, because of potential authorization problems at load time. It may still be advantageous to load other modules into LPA if they meet the following criteria:

• Multiple instances of the module are in use at the same time.

• All of the subroutines in the module call only subroutines which reside in that same module, or another module that is loaded in LPA.

To implement the recommendations in Table 4, you may execute the PUTINLPA job, including the SYSLIN cards shown in Figure 5 on page 18.

Note: The letters of each module name in OS/390 UNIX System Services are case-sensitive. Each member name of OS/390 must be upper case letters.

ftgtr or ftgtr34 The full text index dll. This contains subroutines that are called by some of the functions in libnotes, and must be included in LPA if libnotes is included.

domliba A dll is used by LS:DO.

decsext The decs dll. This contains subroutines that are called by some of the functions in libnotes, and must be included in LPA if libnotes is included.

server The main server executable.

update Scans all of the databases in the Notes data directory for views that need to be rebuilt.

replica Scans all of the databases in the Notes data directory for scheduled replication events.

router Routes standard Lotus Notes Mail.

Module name Purpose


Figure 5. SYSLIN cards for PUTINLPA

2.4.3.2 C/C++ Runtime Library program modulesFor a list of the C/C++ Runtime Library program modules to place in LPA and how to do it, see OS/390 UNIX System Services Planning, SC28-1890 (edition 07), Chapter 16, “Tuning Performance”.

The C/C++ Runtime Library programs that you do not put into LPA should be cached in the Virtual Lookaside Facility (VLF).

2.5 Monitoring OS/390

An essential part of running any application is to be able to monitor the resources consumed. This provides input into both performance tuning and capacity planning. In this section we discuss the major sources of information about a Domino server from the OS/390 system on which it runs. We discuss Domino monitoring techniques in Chapter 7, “Domino monitoring” on page 97.

Traditional OS/390 information sources provide a lot of information for capacity planning, resource accounting, and performance tuning. These have been enhanced for applications such as Domino that use the OS/390 UNIX System Services interfaces. First we discuss the Resource Measurement Facility (RMF), which provides system-level information that is useful for capacity planning and tuning. Then we look at the data available in System Management Facilities (SMF), which is useful for monitoring Domino resource usage, both CPU and I/O.

Additionally, Domino R5 provides information to a new SMF record that includes Domino transaction information. See 7.1, “SMF type 108 record” on page 97 for more information.

2.5.1 Monitoring OS/390 with RMFThe Resource Measurement Facility (RMF) is used to monitor the activity of the OS/390 system. It is useful for capacity planning, performance tuning, and

//SYSLIN DD *INCLUDE INLMOD(server)ENTRY CEESTARTSETCODE AC(1)NAME SERVER(R)INCLUDE INLMOD(update)ENTRY CEESTARTSETCODE AC(1)NAME UPDATE(R)INCLUDE INLMOD(replica)ENTRY CEESTARTSETCODE AC(1)NAME REPLICA(R)INCLUDE INLMOD(router)ENTRY CEESTARTSETCODE AC(1)NAME ROUTER(R) :

//SYSTSIN DD *bpxbatch SH chmod o=t /usr/lpp/lotus/notes/latest/os390/serverbpxbatch SH chmod o=t /usr/lpp/lotus/notes/latest/os390/updatebpxbatch SH chmod o=t /usr/lpp/lotus/notes/latest/os390/replicabpxbatch SH chmod o=t /usr/lpp/lotus/notes/latest/os390/router


problem diagnosis. RMF gives information about system resources, workload groups, DASD controllers, and DASD volumes.

There is a lot of detailed information available with RMF. Here we include a discussion of the OS/390 UNIX System Services OMVS report, some key system indicators, and useful data about a Domino server workload. Details on producing RMF reports and interpreting them are found in:

• OS/390 RMF User’s Guide, SC28-1949 (edition 04)

• OS/390 RMF Report Analysis, SC28-1950 (edition 04)

RMF has two online monitors, which you can access from TSO:

• The RMFMON (Monitor II) command shows data about processor CPU, storage, address spaces, and DASD volumes.

• The RMFWDM (Monitor III) shows how an address space is being delayed and what is causing the delay. RMFWDM keeps information in storage or on DASD so you can look at delays after the event.

Table 5 shows you which report option to specify and which SMF records to collect when generating the following RMF reports:

Table 5. Interval and duration reports

Figure 6 is a sample of a SYSIN statement for an RMF report.

Figure 6. SYSIN statement for RMF report

2.5.1.1 RMF summary report -- system performanceYou should monitor the same things for Domino servers as for other workloads that require good response times. Areas to consider are CPU, DASD response times, paging, and storage.

Report name Report option Gathered by SMF Record

Summary SUMMARY(INT | TOT) Monitor I

OMVS Kernel Activity REPORTS(OMVS) Monitor III 74.3

CPU Activity REPORTS(CPU) Monitor I 70

Paging Activity REPORTS(PAGING) Monitor I 71

Workload Activity(Goal mode)

SYSRPTS(WLMGL) Monitor I 72.3

Workload Activity(Compatibility mode)

REPORTS(WKLD) Monitor I 72.1

Device Activity REPORTS(DEVICE) Monitor I 74.1

//SYSIN DD * SYSOUT(H)

DINTV(0030)REPORTS(CPU,PAGING,OMVS,DEVICE)

SYSRPTS(WLMGL(POLICY,SCLASS,RCLASS)) /*


The RMF Summary Report provides a useful overview of the system. An example is shown in Figure 7.

Figure 7. RMF summary report

You should produce a summary report for the whole day and check the following:

• CPU BUSY is not too high.

If the utilization is high (over 80% during peak periods), you need to make sure that the Domino server is set to a high enough priority so that it is not impacted. If the utilization reaches 100%, you are running more work than the OS/390 image can support and workloads are suffering. To ensure that this does not affect the Domino server, make it a high priority workload.

• DASD average response time is not high, preferably less than ten milliseconds.

• DEMAND PAGING is close to zero.

If there is paging, make sure that the Domino address spaces and the common storage areas are not impacted. Use storage isolation in compatibility mode.

2.5.1.2 RMF OMVS kernel activity reportA new OMVS kernel activity report is included for OS/390 UNIX System Services. RMF Monitor III must be active to collect the information for this OMVS report.

This report is useful to see if your values in the OS/390 UNIX System Services SYS1.PARMLIB(BPXPRMxx) member are too low. It shows the maximum values you have set, and also minimum, average, and maximum values observed during the RMF reporting interval. It also shows the number of overruns, that is, the number of times processes could not be created or storage was not available.

Figure 8 on page 21 is an example of an OMVS kernel activity report. Note that this report was run on our test system and therefore shows very little activity.

R M F S U M M A R Y R E P O R TPAGE 001

OS/390 SYSTEM ID MVS1 START 03/09/1999-08.02.26 INTERVAL 00.29.59REL. 02.07.00 RPT VERSION 2.7.0 END 03/09/1999-12.02.26 CYCLE 1.000 SECONDS

NUMBER OF INTERVALS 8 TOTAL LENGTH OF INTERVALS 03.59.56DATE TIME INT CPU DASD DASD JOB JOB TSO TSO STC STC ASCH ASCH OMVS OMVS SWAP DEMAND SERVICEMM/DD HH.MM.SS MM.SS BUSY RESP RATE MAX AVE MAX AVE MAX AVE MAX AVE MAX AVE RATE PAGING RATE

03/09 08.02.26 29.59 22.6 12 465.9 4 2 64 51 76 73 0 0 0 0 0.00 0.00 17379703/09 08.32.26 30.00 28.9 17 642.2 7 4 89 77 74 71 0 0 0 0 0.00 0.00 15467203/09 09.02.26 29.59 26.0 19 566.7 8 5 103 97 73 71 0 0 0 0 0.00 0.00 11808503/09 09.32.26 30.00 27.8 12 583.5 7 4 103 100 75 70 0 0 0 0 0.00 0.00 131995

03/09 10.02.26 29.59 25.5 20 542.6 8 6 107 102 70 68 0 0 0 0 0.00 0.00 10169303/09 10.32.26 29.59 43.7 11 545.3 7 4 106 99 73 70 0 0 0 0 0.00 0.00 30022703/09 11.02.26 30.00 48.6 13 470.1 10 6 98 94 72 69 0 0 0 0 0.00 0.00 713475


Figure 8. RMF kernel activity report

The RMF reporting interval is typically set at 15 minutes. You are only interested in whether you are exceeding the values in BPXPRMxx and not in how the values change during the day. We suggest that you run an RMF report just for the OMVS kernel activity specifying a complete day, or more, in the DINTV parameter. The DINTV has a maximum value of 100 hours.

You should check periodically that the maximum used values (CURRENT (TOT) in the MAXIMUM column) are well below (MAXIMUM (TOT)) in all the process and interprocess communication and memory map sections. The MAXIMUM (TOT) fields are the values from the active BPXPRMxx in SYS1.PARMLIB. You should not allow any overruns (OVERRUNS (N/S) in the MAXIMUM column) when running a Domino server, because Domino would be unable to start processes or obtain storage.

2.5.1.3 RMF CPU activity report When running in logical partition (LPAR) mode, make sure the processor is not at high utilization. Also make sure that the LPAR has a sufficient share of the processor so that it is not being constrained by other LPARs. You can see this on the partition data report, shown in Figure 9 on page 22.

O M V S K E R N E L A C T I V I T YPAGE 1

OS/390 SYSTEM ID AQTS DATE 03/08/1999 INTERVAL 30.00.000REL. 02.07.00 RPT VERSION 2.7.0 TIME 13:00:00 CYCLE 1.000 SECONDS

TOTAL SAMPLES = 1,800

OMVS SYSTEM CALL ACTIVITY---------------------------------------------------------------------------------------------------------------------------------

MINIMUM AVERAGE MAXIMUM---------------------------------------------------------------------------------------------------------------------------------SYSCALLS (N/S) 23.5 2300* 5699CPU TIME (H/S) 16 47* 88

OMVS PROCESS ACTIVITY---------------------------------------------------------------------------------------------------------------------------------

PROCESSES USERS PROCESSES PER USERMAXIMUM (TOT) 1200 50 12

---------------------------------------------------------------------------------------------------------------------------------MINIMUM AVERAGE MAXIMUM MINIMUM AVERAGE MAXIMUM MINIMUM AVERAGE MAXIMUM

---------------------------------------------------------------------------------------------------------------------------------CURRENT (TOT) 99 854 1200 12 13 23OVERRUNS (N/S) 0 5.5* 333 0 1.5* 4.8 0 1.0* 5.3

OMVS INTER-PROCESS COMMUNICATION

---------------------------------------------------------------------------------------------------------------------------------

MESSAGE QUEUE IDS SEMAPHORE IDS SHARED MEMORY IDS SHAREDMEMORY PAGES

MAXIMUM (TOT) 500 500 500 262144

---------------------------------------------------------------------------------------------------------------------------------

MINIMUM AVERAGE MAXIMUM MINIMUM AVERAGE MAXIMUM MINIMUM AVERAGE MAXIMUM MINIMUMAVERAGE MAXIMUM


Figure 9. RMF partition data report

Create the report at times of peak loading for the processor and check the following:

• PHYSICAL PROCESSORS TOTAL (TOTAL line)

This shows how busy the processor is. If it is at or near 100 percent, the LPARs will receive resources according to their weights. If the OS/390 image is in an LPAR with dedicated engines, you should consider only the LPAR utilization.

• WEIGHTS

The weights are used to allocate CPU resources between LPARs that are sharing engines. These weights come into effect when the engines assigned to shared LPARs are 100 percent busy. An LPAR receives its weight divided by the sum of the weights of the active LPARs. In Figure 9, the SYS1 LPAR would receive 75/(2+75+3+20), which is 75 percent of the shared engines.

• NUMBER OF LOG PRCRS

Make sure that you are not constraining an LPAR by giving it too few logical processors (CPU engines). IFigure 9 shows we have six engines on the processor, but the AQXS LPAR has only two engines specified. So it can never use more than 33% of the processor, even if the other LPARs are idle.

• CAPPING

We do not recommend capping an LPAR running a Domino server. You could affect the server response time when CPU resource is available by restricting the LPAR to its specified weight.

You can find more information about LPARs in S/390 PR/SM Planning Guide.

2.5.1.4 RMF paging activity reportThe key indicators of central and expanded storage are Unreferenced Interval Count (UIC) and Migration Age. Figure 10 on page 23 shows the RMF paging activity report containing these indicators.

P A R T I T I O N D A T A R E P O R TPAGE 3

OS/390 SYSTEM ID SYS1 DATE 03/14/1999 INTERVAL 14.59.946REL. 02.07.00 RPT VERSION 2.7.0 TIME 16.30.00 CYCLE 1.000 SECONDS

MVS PARTITION NAME SYS1NUMBER OF CONFIGURED PARTITIONS 4NUMBER OF PHYSICAL PROCESSORS 6WAIT COMPLETION NODISPATCH INTERVAL DYNAMIC

--------- PARTITION DATA --------- -- LOGICAL PARTITION PROCESSOR DATA -- ---- AVERAGE PROCESSOR UTILIZATION PERCENTAGES ----NUMBER OF ----DISPATCH TIME DATA----- LOGICAL PROCESSORS ----- PHYSICAL PROCESSORS -----

NAME STATUS WEIGHTS CAPPING LOG PRCRS EFFECTIVE TOTAL EFFECTIVE TOTAL LPAR MGMT EFFECTIVE TOTALAQHO A 2 NO 1 00.00.27.776 00.00.31.109 3.09 3.46 0.06 0.51 0.58SYS1 A 75 NO 6 01.17.48.097 01.18.14.583 86.45 86.94 0.49 86.45 86.94AQXS A 3 NO 2 00.01.05.698 00.01.14.891 3.65 4.16 0.17 1.22 1.39XRFMCL A DED 2 00.06.08.005 00.06.08.351 20.49 20.85 0.12 6.80 6.90

20 NO 1 00.00.01.010 00.00.01.050 0.01 0.01 0.12 0.03 0.05*PHYSICAL* 00.00.49.387 0.91 0.91

------------ ------------ ------ ------ ------TOTAL 01.25.30.588 01.27.05.373 1.76 95.02 96.77


Figure 10. RMF paging activity report

You should regularly monitor the following:

• HIGH UIC AVG

The UIC is a measure of how central storage pages are referenced. This value should be at or near 254. The higher the number, the better. If it is in single digits, then the OS/390 image requires more central storage. If it is less than 50, you should consider increasing central storage.

• MIGR AGE AVG

The migration age is the length of time in seconds that a page of storage remains in expanded storage before migrating to a page data set. This number should be in the thousands of seconds. A low number means expanded storage is overcommitted, and a very high number means it is underutilized.

In Workload Manager goal mode, the storage used by address spaces is managed to achieve the velocity or response time goals requested. In compatibility mode, if you are storage-constrained, you will need to protect the

P A G I N G A C T I V I T Y

PAGE 2OS/390 SYSTEM ID ML96 START 11/04/1999-09.00.00 INTERVAL 000.44.59REL. 02.07.00 RPT VERSION 2.7.0 END 11/04/1999-09.45.00 CYCLE 1.000 SECONDS

OPT = IEAOPTML EXPANDED STORAGE MOVEMENT RATES - IN PAGES PER SECOND

-----------------------------------------------------------------------------------------------------------------------------ESF CONFIGURATION HIGH UIC MIGR AGE

INSTALLED ONLINE MIN 254 684517--------- ------ MAX 254 688471491520 491520 AVG 254.0 686492.8

MIGRATED FREEDWRITTEN TO READ FROM FROM WITHOUT *----- EXPANDED STORAGE FRAME COUNTS -----*EXP STOR EXP STOR EXP STOR MIGRATION MIN MAX AVG

TOTAL RT 46.97 41.65 0.00 0.00 145,607 150,677 148,511PAGES % 100.0% 100.0% 100.0%HIPERSPACE RT 0.00 0.00 0.00 1,442 1,442 1,442PAGES % 0.0% 0.0% 0.0%VIO RT 0.00 0.00 0.00 0 0 0PAGES % 0.0% 0.0% 0.0%SHARED RT 0.00 0.00PAGES % 0.0% 0.0%

-FRAME AND SLOT COUNTS

-----------------------------------------------------------------------------------------------------------------------------CENTRAL STORAGE EXPANDED STORAGE LOCAL PAGE DATA SET SLOT COUNTS

---------------------------- ---------------------------- ----------------------------------------------(273 SAMPLES) MIN MAX AVG MIN MAX AVG MIN MAX AVGAVAILABLE 110 1,666 440 340,843 345,913 343,009 AVAILABLE SLOTS 1,080,000 1,080,000

1,080,000SQA 8,822 8,860 8,842 0 0 0 VIO SLOTS 0 0 0LPA 6,172 6,172 6,172 0 0 0CSA 11,214 11,274 11,240 0 30 0 NON-VIO SLOTS 0 0 0LSQA 19,688 21,130 20,758 7,350 8,733 7,650REGIONS+SWA 469,466 472,320 471,000 145,568 150,623 148,483 BAD SLOTS 0 0 0TOTAL FRAMES 524,288 524,288 524,288 491,520 491,520 491,520 TOTAL SLOTS 1,080,000 1,080,000

1,080,000FIXED FRAMES SHARED FRAMES AND SLOTS

---------------------------- ------------------------------------------------NUCLEUS 5,833 5,833 5,833 CENTRAL STORAGE 260,658 260,661 260,659SQA 8,196 8,234 8,216 EXPANDED STORAGE 15 15 15LPA 66 66 66 FIXED TOTAL 1,000 1,000 1,000CSA 436 441 438 FIXED BELOW 16 M 0 0 0LSQA 6,888 7,026 6,967 AUXILIARY SLOTS 0 0 0REGIONS+SWA 53,940 54,116 53,999BELOW 16 MEG 260 370 300 TOTAL 264,944 264,946 264,945TOTAL FRAMES 75,406 75,654 75,520


Domino server address spaces by using storage isolation. This is discussed in 2.3.2.3, “Storage isolation” on page 13.

2.5.1.5 RMF workload reportsWhether running in goal mode or compatibility mode, you should define your workloads to report each Domino partition separately. You can run all the Domino partitions under the same priority level (Service Class or Performance Group), but should place each Domino partition in its own reporting group (Report Class or Report Performance Group). This allows you to monitor the resources used by each Domino partition.

Appendix A, “Workload Manager (WLM) example” on page 129 has an example of the Workload Manager goal mode definitions using Report Classes.

In Workload Manager goal mode, the SYSRPTS(WLMGL(SCLASS,RCLASS)) RMF report gives the CPU used by service classes and report service classes. Figure 11 shows an example of a report class.

Figure 11. RMF Workload Manager report class

Under the Service Rates column, you see the TCB and SRB (which are the CPU seconds used by this workload during the interval), as well as the APPL% (which is the percentage of the interval when the workload was processing). For a Domino server, you may see the APPL% above 100 percent, because the server has multiple address spaces that can be executing on more than one engine concurrently. This report can be used to obtain the CPU time used by a Domino server.

In compatibility mode, the REPORTS(WKLD(GROUP)) report provides similar information (excluding the velocity migration and execution delays). You need the performance group number or report performance group number used by the Domino partition to identify the correct set of data.

W O R K L O A D A C T I V I T Y PAGE 7

OS/390 SYSPLEX 96XCF01 START 11/04/1999-08.45.00 INTERVAL 001.00.00 MODE = GOALREL. 02.07.00 RPT VERSION 2.7.0 END 11/04/1999-09.45.00

POLICY ACTIVATION DATE/TIME 10/30/1999 22.36.37

REPORT BY: POLICY=PRISHFT REPORT CLASS=ML96NPML96 Notes servers

TRANSACTIONS TRANS.-TIME HHH.MM.SS.TTT --DASD I/O-- ---SERVICE---- --SERVICE RATES-- PAGE-IN RATES----STORAGE----

AVG 11.08 ACTUAL 26.560 SSCHRT 0.1 IOC 613809 ABSRPTN 660 SINGLE 0.0 AVG 7302.38MPL 11.08 EXECUTION 26.559 RESP 114.6 CPU 24513K TRX SERV 660 BLOCK 0.0 TOTAL 80887.0ENDED 68 QUEUED 1 CONN 0.9 MSO 0 TCB 2983.0 SHARED 0.0 CENTRAL 57467.6END/S 0.02 R/S AFFINITY 0 DISC 112.6 SRB 1191K SRB 145.0 HSP 0.0 EXPAND 23419.4#SWAPS 1356 INELIGIBLE 0 Q+PEND 1.1 TOT 26318K RCT 0.6 HSP MISS 0.0EXCTD 0 CONVERSION 0 IOSQ 0.0 /SEC 7310 IIT 0.0 EXP SNGL 0.5 SHARED 4630.91

STD DEV 1.06.254 HST 0.0 EXP BLK 0.0APPL % 86.9 EXP SHR 0.0

VELOCITY MIGRATION: I/O MGMT 91.5% INIT MGMT 91.5%

---RESPONSE TIME--- EX PERF AVG --USING%-- --------------- EXECUTION DELAYS % -------------------DLY%-- %

HH.MM.SS.TTT VEL INDX ADRSP CPU I/O TOTAL CPU


You can also obtain the central and expanded storage used by the address spaces in this workload group from the same report. However, these are weighted numbers based on the time during the interval that the address spaces were resident (not swapped out). The actual storage required by the address spaces may be higher when all the address spaces are resident.

2.5.2 Monitoring OS/390 with SMFSystem Management Facilities (SMF) is an important source of information on Domino resource use. SMF collects data about processor, storage, and I/O consumption for address spaces in OS/390.

Details of the SMF records and how to manage them are found in OS/390 MVS System Management Facilities (SMF), GC28-1783 (edition 07).

The RMF monitor also writes its records to the SMF data set as record types 70 through 79. The RMF record types are discussed in 2.5.1, “Monitoring OS/390 with RMF” on page 18.

RMF reports can be created using the RMF postprocessor, but no utility is provided to report on the other SMF records. Many installations have products that can report on SMF data. You can also write a program to produce the desired reports, although the format of SMF records is complicated.

Table 6 shows which sections of this redbook present details about some useful SMF record types.

Table 6. SMF record types

2.5.2.1 Interval recording of SMF recordsYou specify interval recording in the SMFPRMxx member of SYS1.PARMLIB, where xx is the suffix of the currently active member. You can have interval accounting for all workload types, or you can select specific workload types. You will need interval recording enabled for the OMVS workload type.

Figure 12 on page 26 shows an example of the SMFPRMxx parameters. You should use this as a guide only, since you may require your own values for other parameters. The important parameters are INTVAL, SYNCVAL, and INTERVAL. In the example, the interval is set to 15 minutes, but you can choose the frequency of reporting you require. If a job step starts at 10:50, interval records are written at 11:00, 11:15, 11:30 and so on until the step ends, when a final interval record is written. Each interval record shows the resources consumed since the previous interval record was created.

SMF record type Section

Record type 30 2.5.2.2, “Common address space work (record type 30)” on page 26

Record type 42 2.5.2.3, “DFSMS statistics and configuration (record type 42)” on page 27

Record type 92 2.5.2.4, “Open/MVS file system activity (record type 92)” on page 27

Record type 108 7.1, “SMF type 108 record” on page 97


Figure 12. SMFPRMxx example for SMF interval recording

With interval synchronization, you can directly compare SMF data with RMF data. We recommend the following options.

• SMF option: INTVAL(nn) SYNCVAL(00) where nn can be 05,10,12,15,20,30 or 60

• INTVAL(nn): SMF global recording interval

• SYNCVAL(mm): Synchronization with the hour

• RMF monitor I option: SYNC(SMF)

• SYNC(SMF): Synchronization with SMF. This means that records will be written as specified with INTVAL and SYNCVAL options.

For more information, see the following publications:

• OS/390 RMF User’s Guide, SC28-1949 (edition 04)

• OS/390 MVS System Management Facilities (SMF), GC28-1783 (edition 07)

2.5.2.2 Common address space work (record type 30)The type 30 record contains a lot of information about CPU and storage use by address space. These record types can be produced at specified intervals during an address space execution, which is very useful for the long-running Domino address spaces. We recommend that you enable interval recording for SMF type 30 records.

The type 30 records can be quite large. Therefore, if you are not already collecting them, you should plan on your SMF data sets filling more rapidly and allow for this in your operational procedures.

There are many fields with similar types of data (for example, CPU data) in the type 30 record. The fields that we found useful are:

SMF30JBN JOB name.

SMF30EXN Program name of the OS/390 UNIX System Services process. Do not use SMF30PGM, which is too short to hold the OS/390 UNIX System Services name. The program name in OS/390 UNIX System Services can be up to 16 characters, so you must use SMF30EXN. The accounting chapter in OS/390 MVS System

ACTIVE /* ACTIVE SMF RECORDING */DSNAME(SYS1.&SYSNAME..MAN1, /* SMF DATA SET NAMES */

SYS1.&SYSNAME..MAN2, /* SMF DATA SET NAMES */SYS1.&SYSNAME..MAN3) /* SMF DATA SET NAMES */

NOPROMPT /* DON'T PROMPT THE OPERATOR */REC(PERM) /* TYPE 17 PERM RECORDS ONLY */INTVAL(15) /* INTERVAL RECORDING EVERY 15 MIN */SYNCVAL(00) /* SYNCHRONIZE TO THE HOUR */MAXDORM(3000) /* WRITE AN IDLE BUFFER AFTER 30 MIN */STATUS(010000) /* WRITE SMF STATS AFTER 1 HOUR */JWT(0800) /* 522 AFTER 2 HOURS */SID(&SYSNAME(1:4)) /* SYSTEM ID IS &SYSNAME */LISTDSN /* LIST DATA SET STATUS AT IPL */LASTDS(MSG) /* DEFAULT TO MESSAGE */NOBUFFS(MSG) /* DEFAULT TO MESSAGE */

SYS(TYPE(0:255),EXITS(IEFU83,IEFU84,IEFU85,IEFACTRT,IEFUJV,IEFUSI,


Management Facilities (SMF), GC28-1783 describes the format of the program name.

SMF30CPT CPU time under a TCB.

SMF30CPS CPU time under an SRB.

SMF30OFR and SMF30OFW HFS reads and writes to regular files. There are also counts of reads and writes to HFS pipes and special files in the fields SMF30OPR, SMF30OPW, SMF30OSR, and SMF30OSW.

SMF30TEP Blocks transferred. The block count in the Domino address spaces is an indication of the blocks transferred for the Domino address space.

SMF30AIS Start Subchannel (SSCH). All I/O to the HFS is reported in the OMVS OS/390 UNIX System Services kernel address space. The I/O is performed in cross-memory mode under the TCB of the Domino address space. This field is zero for the OS/390 UNIX System Services Domino address spaces. The OMVS address space record contains the number of I/Os performed for all OS/390 UNIX System Services users. Also in the record are the connect, disconnect, and pending times for these I/Os.

2.5.2.3 DFSMS statistics and configuration (record type 42)The data set statistics in the subtype 6 record provide useful information about OS/390 data sets. The OS/390 UNIX System Services HFS file system is stored in OS/390 HFS data sets. This record therefore shows the activity to all of the UNIX files in each HFS data set.

By default, the records are written at dataset close time. For an HFS data set, this means from the time the dataset was mounted until it is unmounted. This is not useful for performance purposes. To get significant statistics for the Domino address spaces, you should have interval recording active. The type 42 records are written at the same time as the SMF type 30 records. These records contain useful information about I/O performance, including:

• Number of I/Os

• Average connect, disconnect, pending, and control unit queue time, as well as response time

• Cache performance, including number of cache candidates and cache hit rates

You will find these records useful when you have an I/O performance problem and need greater detail than the DASD volume information provided by RMF.

2.5.2.4 Open/MVS file system activity (record type 92)The type 92 record does not provide interval records, as types 30 and 42 do. Information is reported for the duration of the mount for HFS data sets, and for the duration of the open for UNIX files.

Record subtype 5 contains information about the HFS file system. It is written when the file system is unmounted, which should not be a frequent occurrence with a Domino server. Thus there may be no useful information from this record, because the activity it reflects covers a long period of time.


The UNIX file close record, subtype 11, provides useful information on the requests made to a UNIX file by the OS/390 UNIX System Services Domino address spaces. It contains:

• Read and write calls

• Directory I/O blocks

• Blocks read and written

• Bytes read and written

Unfortunately, the record does not contain the UNIX file name, but there is a way to obtain it. This is documented in OS/390: UNIX System Services Planning, SC28-1890, chapter 15. IBM intends to add the UNIX file name to this record in a future release of OS/390.

Collecting record type 92Collecting type 92 records, particularly subtypes 10 and 11, generates a large number of SMF records. You may want to suppress these subtypes (in SMFPRMxx in SYS1.PARMLIB) until you need this detailed level of information. However, you should note that to create these records:

• Subtype 5 must be collecting at the time the HFS file system is mounted and unmounted.

• Subtype 11 must be collecting at the time the UNIX file is opened and closed.


Chapter 3. DASD subsystem tuning

In this chapter we discuss the performance of the DASD subsystem, and how you can monitor and tune it.

I/O response time can be a significant part of application response time because, while processor activities take only nanoseconds to complete, I/O activities take milliseconds. Therefore, good DASD response time is an important part of Domino performance.

First we give some recommendations, then we discuss the subject in detail.

3.1 Recommendations

These are the key DASD tuning recommendations:

• We recommend the use of OS/390 Version 2 Release 7 or above to get the performance improvements of DFSMS 1.5.

• Monitor the system.

Look for ESCON channels at more than 45 percent utilization, or parallel channels at more than 30 percent utilization. Watch for DASD devices with high activity that are not performing well.

• Use cached DASD devices.

Use DASD Fast Write and Dynamic Cache Management Enhanced (DCME) to give the fastest I/O response time.

• Avoid multiple extents in HFS datasets.

HFS datasets can expand to multiple extents. As with all MVS datasets, consolidate them into one extent using DFSMSdss. This reduces the access time.

• Spread high-activity HFS datasets and volumes across multiple DASD control units.

Monitor the DASD subsystem and make sure that the activity is spread across the available volumes and control units. Keep user HFS datasets separate from system HFS datasets.

• Place UNIX files as close to the root as possible.

When accessing a file system, HFS locks all paths to the file. By placing files close to the root, you speed up HFS access.

• Avoid collecting unnecessary data.

Domino can collect a lot of statistics and log data, which can drive a lot of I/O activity. Make sure you collect only the data you are interested in.

3.2 DASD subsystem considerations

The S/390 platform supports a broad range of DASD subsystems, from the 3380 models to the Enterprise Storage Server product (popularly known as Shark). Newer products generally provide better performance characteristics; you should consider these for online workloads such as Domino.


A key element of the DASD subsystem is cache storage. Volumes that contain HFS datasets should be placed on cached DASD with DASD Fast Write for good performance.

The latest 3990/9390 control units are provided with Dynamic Cache Management Enhanced (DCME). DCME provides more comprehensive and responsive cache management techniques through the use of improved algorithms. This function determines which datasets are using cache effectively; it dynamically turns cache off for datasets that do not benefit from caching. DCME also provides additional cache activity and I/O activity statistics which DFSMS/MVS stores in the SMF dataset to be used for customer information reports.

We strongly recommend that you allocate Lotus Domino HFS datasets on cached DASD with the DASD Fast Write (DFW) feature enabled and under DCME-capable controllers. For more details about cache implementation, see IBM 3990/9390 Storage Control Planning and Installation, GA32-0100.

However, today DASD subsystems provide you with very large cache sizes and optimized caching algorithms. There are two subsystems, IBM RAMAC Virtual Array (RVA) and Enterprise Storage Server (ESS), that we discuss.

3.2.1 IBM RAMAC Virtual Array (RVA)The IBM RAMAC Virtual Array (RVA) DASD models are worth a special mention. RVA allows you to define up to 1024 virtual volumes for a storage pool, emulating any mixture of 3380 and 3390 models.

This allows you to place a high-activity dataset on a smaller dedicated volume. OS/390 will only start a single I/O at a time to a volume. With just one dataset on the volume, you will eliminate any queue time (called IOSQ time) in OS/390.

The RVA introduces new concepts in storage management. Thanks to its unique architecture, it offers an excellent environment for testing and development. However, in production systems, you should consider the impact of its features. A good example is Deleted Data Space Release (DDSR), a housekeeping utility that physically erases areas in the disk array after the data has been logically deleted from operating system. This utility can cause a temporary “freeze” of the RVA DASD subsystem, thus degrading performance of Domino I/O operations. DDSR can be invoked dynamically or in specified intervals. We recommend that DDSR be scheduled to run in non-peak hours.

RVA also supports SnapShot, a powerful backup/dump utility. This utility is automatically used by DFSMSdss for the COPY command. SnapShot is supported by the DUMP command if you have set up an environment for concurrent copy. This feature of RVA can significantly improve performance of the backup/restore processes. Be aware that you cannot use dataset level SnapShot for HFS files.

3.2.2 The Enterprise Storage Server (ESS)The IBM Enterprise Storage Server (ESS), also known as Shark, offers a new generation of enterprise disk storage systems. There are several new features that significantly enhance I/O performance. The combination of parallel access volumes, multiple allegiance, I/O priority queueing, and new CCWs, bring


substantial benefits to the S/390 customer. In particular, parallel access volumes (PAVs) allow the ability to issue multiple I/O requests to the same volume, nearly eliminating IOSQ time. With the logical volumes striped across the multiple disks in an array in the ESS, many cache misses will be able to be processed in parallel, too. With PAVs and the parallel access management support by WLM, the need to manually split data across multiple volumes is greatly reduced.

ESS can help Domino in all major areas of performance. We have typically seen improvements of 30 - 50% in elapsed time for write-intensive tasks, such as database compaction, replication, and view rebuilding, compared to traditional DASD architectures. Also, end-user response time can be improved by 20 to 30% if you use ESS for your databases.

ESS scales very well. This is true not only for capacity but also for response time, for an achievable I/O rate for given response time limits, and for a multisystem environment as well. This DASD subsystem can help you integrate Domino with other enterprise applications that use DB2, IMS, and CICS. However, there are certain limits in Domino internal design, as well as in environments outside Domino, that cannot be solved by a DASD subsystem. You can find a more detailed description of potential ESS benefits for Domino on URL:

http://www.s390.ibm.com/products/domino/domess

ESS gives you the most flexible DASD subsystem for Domino. You should get together with your storage management team to make a plan for Domino. This team should look into RMF reports for suitable control units and devices. Then you should select volumes which are not frequently accessed. As your installation grows, check and compare these measurements. It may be necessary to reconfigure ESS to support a heavier workload.

3.2.2.1 Recommendations • Use ESS in a production environment where possible. It is the fastest DASD

subsystem you can buy today, even without advanced features like PAV.

• Use as many channel paths as possible. ESS allows you to use up to 32 ESCON channels.

• Define as many LCUs as possible. ESS allows you to define up to sixteen 3990 model 6 control units in one ESS.

• Spread emulated CKD volumes across back-end loops. Use to set up and monitor basic functions of ESS.

• Monitor and evaluate collected data from ESS using Storwatch Enterprise Storage Server Expert. This tool, together with RMF reports, can help you better understand what is going on behind the scenes.

• Dedicate PAV-enabled volumes for HFS files containing frequently accessed Domino databases by many users, such as the Domino Directory (NAB) and other shared databases (/notesdata). This will speed up access to Notes directories and improve end-user response time.

• Dedicate PAV-enabled volumes for HFS files containing Domino transaction log files (/notesdata/noteslog).

• Spread mail databases for users evenly across HFS files, using many volumes. Use PAV-enabled volumes if possible.

Chapter 3. DASD subsystem tuning 31

• Use PAVs in mixed workload environments and when running multiple Domino partitions.

• If you have installed OS/390 V2R8 or later, run WLM in goal mode to take advantage of dynamic PAVs.

• In HCD definitions for DASD devices on ESS, specify WLMPAV=YES

3.3 Monitoring DASD I/O activity

You should monitor the DASD I/O activity for Domino files. You can do this at a number of levels.

3.3.1 RMFUse RMF to monitor the DASD response time for volumes that contain Domino HFS datasets. The RMF summary report that lists the average DASD response time and DASD I/O rate is discussed in 2.5.1.1, “RMF summary report -- system performance” on page 19.

Other reports provide you with information on:

• Channel path activity

• I/O queueing activity

• Cache subsystem activity

• DASD activity

These reports are useful for identifying performance issues such as:

• Poor response time from a DASD volume. Aim for at most six to ten milliseconds, although it will depend on the activity rate and the DASD device type.

• Too much activity on a single DASD volume

• Channels that are too busy. That is, more than 45 percent busy for ESCON channels with four paths; more than 30 percent busy for parallel channels.

• Insufficient paths. We recommend at least four paths to every control unit.


3.3.1.1 Monitoring DASD cacheThe RMF cache subsystem activity report and the IDCAMS LISTDATA STATUS command are useful tools to monitor cache use and effectiveness. Figure 13 shows an example.

Figure 13. RMF cache report

On the Cache subsystem activity report, note the following fields:

C A C H E S U B S Y S T E M A C T I V I T YPAGE 1

OS/390 SYSTEM ID SYS1 DATE 06/19/1999 INTERVAL 14.59.241REL. 02.07.00 RPT VERSION 2.7.0 TIME 12.35.59

SUBSYSTEM 3990-06 CU-ID 0D93 SSID 6001 CDATE 06/19/1999 CTIME 12.36.02 CINT 14.58TYPE-MODEL 9396-001

----------------------------------------------------------------------------------------------------------------------------------CACHE SUBSYSTEM STATUS

----------------------------------------------------------------------------------------------------------------------------------

SUBSYSTEM STORAGE NON-VOLATILE STORAGE STATUS

CONFIGURED 256.0M CONFIGURED 8.0M CACHING - ACTIVEAVAILABLE 254.9M PINNED 0.0 NON-VOLATILE STORAGE - ACTIVEPINNED 0.0 CACHE FAST WRITE - ACTIVEOFFLINE 0.0 IML DEVICE AVAILABLE - YES

----------------------------------------------------------------------------------------------------------------------------------CACHE SUBSYSTEM OVERVIEW

----------------------------------------------------------------------------------------------------------------------------------

TOTAL I/O 253160 CACHE I/O 252669 CACHE OFFLINE 0TOTAL H/R 0.985 CACHE H/R 0.987

CACHE I/O -------------READ I/O REQUESTS------------- ----------------------WRITE I/O REQUESTS---------------------- %

REQUESTS COUNT RATE HITS RATE H/R COUNT RATE FAST RATE HITS RATE H/R READ

NORMAL 138531 154.3 135658 151.1 0.979 2943 3.3 2943 3.3 2878 3.2 0.978 97.9SEQUENTIAL 110912 123.5 110609 123.2 0.997 283 0.3 283 0.3 263 0.3 0.929 99.7CFW DATA 0 0.0 0 0.0 N/A 0 0.0 0 0.0 0 0.0 N/A N/A

TOTAL 249443 277.8 246267 274.2 0.987 3226 3.6 3226 3.6 3141 3.5 0.974 98.7-----------------------CACHE MISSES----------------------- ------------MISC------------ ------NON-CACHE I/O-----REQUESTS READ RATE WRITE RATE TRACKS RATE COUNT RATE COUNT RATE

DFW BYPASS 23 0.0 ICL 487 0.5NORMAL 2873 3.2 65 0.1 2667 3.0 CFW BYPASS 0 0.0 BYPASS 4 0.0SEQUENTIAL 303 0.3 20 0.0 1875 2.1 DFW INHIBIT 0 0.0 TOTAL 491 0.5CFW DATA 0 0.0 0 0.0 ASYNC (TRKS) 1640 1.8

TOTAL 3261 RATE 3.6---CKD STATISTICS--- ---RECORD CACHING---

WRITE 18 READ MISSES 128WRITE HITS 18 WRITE PROM 0

C A C H E D E V I C E A C T I V I T YPAGE 2

OS/390 SYSTEM ID SYS1 DATE 06/19/1999 INTERVAL 14.59.241REL. 02.07.00 RPT VERSION 2.7.0 TIME 12.35.59

SUBSYSTEM 3990-06 CU-ID 0D93 SSID 6001 CDATE 06/19/1999 CTIME 12.36.02 CINT 14.58TYPE-MODEL 9396-001

----------------------------------------------------------------------------------------------------------------------------------CACHE SUBSYSTEM DEVICE OVERVIEW

----------------------------------------------------------------------------------------------------------------------------------

VOLUME DEV DUAL % I/O ---CACHE HIT RATE-- ----------DASD I/O RATE---------- ASYNC TOTAL READ WRITE %SERIAL NUM COPY I/O RATE READ DFW CFW STAGE DFWBP ICL BYP OTHER RATE H/R H/R H/R READ

*ALL 100.0 281.9 274.2 3.5 0.0 3.6 0.0 0.5 0.0 0.0 1.8 0.985 0.987 0.974 98.7*CACHE-OFF 0.0 0.0*CACHE 100.0 281.9 274.2 3.5 0.0 3.6 0.0 0.5 0.0 0.0 1.8 0.985 0.987 0.974 98.7D83STA 0D80 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N/A N/A N/A N/AD83STB 0D81 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.526 0.526 N/A 0.0D83STC 0D82 3.0 8.5 8.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.000 1.000 N/A 0.0D83STD 0D83 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N/A N/A N/A N/AD83STE 0D84 13.8 38.9 37.4 0.0 0.0 1.6 0.0 0.0 0.0 0.0 0.0 0.959 0.959 N/A 0.0D83STF 0D85 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N/A N/A N/A N/AD83STG 0D86 1.2 3.4 3.2 0.0 0.0 0.2 0.0 0.0 0.0 0.0 0.0 0.932 0.932 N/A 0.0D83STH 0D87 0.0 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.624 0.624 N/A 0.0D83STI 0D88 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N/A N/A N/A N/AD83STJ 0D89 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 N/A N/A N/A N/A


CACHE SUBSYSTEM STATUS Make sure all the fields under the STATUS column are active.

OFFLINE Shows the amount of cache disabled for use. This can indicate a hardware problem.

DEVICE OVERVIEW Take a look at the device overview section to see whether one of the volumes exhibits unusual behavior compared to the others.

3.3.2 SMFSMF provides you with more detailed information on I/O activity. See 2.5.2, “Monitoring OS/390 with SMF” on page 25 for details of the SMF records.

3.3.2.1 Record type 42SMF record type 42, subtype 6 provides information about HFS dataset performance. I/O statistics are provided for each HFS dataset.

3.3.2.2 Record type 30SMF record type 30 reports activity on a job and job step basis. The OS/390 UNIX System Services process section also provides information on file system lookups, which can consume significant resources on systems with hierarchical files. A lookup is an access to the directory path in order to access a file. Monitor the number of lookups on your system. To reduce them, keep frequently used files in the same directory and avoid too many levels on the file structure. If key jobs appear to be doing many lookups, your installation may be able to reduce this overhead by reorganizing the file system and mounting these key files closer to the root of the file system.

3.3.2.3 Record type 92SMF record type 92, subtype 5 collects information at the time the HFS is mounted and unmounted. SMF record type 92, subtype 11 collects information at the time the HFS is opened and closed.

3.3.3 UNIX statisticsYou also can obtain online statistics at the UNIX level by issuing the command /usr/sbin/kerninfo stats. The output is shown in Figure 21 on page 45.

3.4 DFSMS/MVS 1.5

DFSMS/MVS Version 1 Release 5 provides new functions and performance improvements. In this section we describe these improvements and show you how to tune the appropriate DFSMS parameters to maximize HFS performance.

3.4.1 HFS performance enhancementDramatic improvements in HFS performance are provided by DFSMS/MVS 1.5, which is an integral part of OS/390 Version 2 Release 7. Although Domino R5 can run on OS/390 Version 2 Release 6, you should consider moving to OS/390 2.7 or later to realize this performance advantage.


3.4.1.1 Deferred I/OPrior to DFSMS/MVS 1.5, HFS hardens all metadata changes to disk synchronously with user requests. For example, when an HFS file is created, the new file’s metadata (as well as the parent directory’s metadata and data) is written to disk before returning to the caller of the create function. This is true for all HFS functions which change HFS metadata, even though the POSIX and UNIX standards do not require that metadata must be hardened upon return from functions which change metadata.

In the HFS performance enhancement, a fundamental change was to move the disk hardening to a sync daemon, which runs periodically and writes out all metadata changes that have occurred since the last time the sync daemon ran. Therefore, multiple metadata changes are batched into a single I/O.

This technique obviously reduces I/O. The reduction can be very dramatic because the sync daemon interval will be on the order of a minute. If N files had been created in that minute, then a single I/O will harden them to disk, thus providing at least an N to 1 I/O reduction. This applies to any operation which currently hardens metadata.

This technique dramatically reduces path length. Trace analysis has shown that the majority of the path lengths for most HFS functions is for hardening the metadata. By deferring that processing to the sync daemon, the path length that is synchronous to a user request is dramatically reduced. There is reduced contention in the HFS layer, because HFS currently obtains a filesystem-wide latch every time HFS metadata is updated. This latch is held during the entire I/O. Since the number of metadata updates is reduced, the frequency of obtaining the latch is reduced.

3.4.1.2 Improved caching and bufferingWith DFSMS 1.4, HFS called another system component to obtain I/O buffers and perform lookaside on file data. Trace analysis has shown that there is excessive path length (thousands of instructions) to obtain a single buffer and to perform lookaside. With DFSMS 1.5, the HFS will maintain its own buffer pools and significantly reduce the cost of obtaining a buffer, as shown in Figure 14 on page 36. Additionally, HFS will cache file data attributes in HFS data areas.

The same techniques employed by DFSMS 1.5 to reduce buffering and caching costs for file data are now also used by the HFS to reduce buffering and caching costs for the indexes that the HFS maintains in its datasets.


Figure 14. Improved HFS performance

3.4.2 Setting DFSMS parametersThere are three kinds of parameters you can set for the Deferred I/O of DFSMS. You can put these parameters in the SYS1.PARMLIB(BPXPRMxx) member. See OS/390 MVS Initialization and Tuning Reference, SC28-1752 (edition 08), for more details. See Figure 15 on page 37 for an example.

3.4.2.1 Sync daemon intervalSync daemon interval is the amount of time, in seconds, between iterations of sync daemon. The purpose of the sync daemon is to harden the metadata for a given HFS file system. Setting this value to 0 indicates that the HFS should harden metadata synchronously with syscall requests. For best performance, we suggest that you use the default of 60 seconds. See Hierarchical File System Usage Guide, SG24-5482, for a further explanation of the meaning of this parameter.

You can set the sync interval as either a default value for all HFS file systems, or as a value specific to a given file system. Use the SYNCDEFAULT parameter on the HFS FILESYSTYPE statement in the BPXPRMxx parmlib member to set the default for all file systems. Use the SYNC parameter of the MOUNT statement for a given file system in BPXPRMxx to override any default. An example is shown in Figure 15 on page 37.

Unix ServicesHFS Layer

component

Unix ServicesHFS LayerHFS calls for bufferslong path lengthsstreamlined instructions

MemoryBuffers

SynchronousWrites

Before DFSMS 1.5

WRITES WRITESMemoryBuffers

DeferredWrites

DASD Controller CacheDASD Controller Cache

HFS calls for bufferslong path lengthsredundant instructions


3.4.2.2 Virtual storage sizeVirtual storage size is the maximum amount of virtual storage (in megabytes) that HFS data and metadata buffers should use. The minimum value that can be specified is 32 MB. If less is specified, an information message is issued and the maximum is set to 32 MB. The maximum limit can be changed dynamically by invoking the CONFIGHFS OMVS shell command. See OS/390 UNIX System Services Command Reference, SC28-1892 (edition 07), for more information about the CONFIGHFS OMVS shell command.

You can specify this value in the VIRTUAL parameter of the FILESYSTYPE TYPE(HFS) statement in BPXPRMxx. An example is shown in Figure 15.

3.4.2.3 Fixed storage sizeFixed storage size is the amount of virtual storage (in megabytes) that is fixed at HFS initialization time. It remains fixed even if HFS activity drops to zero.

This value cannot exceed 50 percent of the processor storage available to the system. If the allowed amount of storage is exceeded, an informational message is issued and this value is set to 50 percent of processor storage. The minimum limit can be changed dynamically by invoking the CONFIGHFS OMVS shell command. See OS/390 UNIX System Services Command Reference, SC28-1892 (edition 07), for more information about this command.

You can specify this value in the FIXED parameter of the FILESYSTYPE TYPE(HFS) statement. An example is shown in Figure 15.

Figure 15. BPXPRMxx setting for DFSMS deferred I/O

3.4.3 MonitoringThere is two-way of monitoring of HFS bufferpool statistics. One is a confighfs command, which gives you dynamic access to HFS bufferpool statistics. Another is RMF Postprocessor report, which is supported in OS/390 V2R7 and later.

3.4.3.1 confighfs commandYou can monitor the usage of these buffers using the confighfs -q command. Figure 16 on page 38 shows an output of the command.

The virtual and fixed parameter should be increased until the not already fixed values for pools 1 and 2 show relatively low values. Make sure you have enough storage to give.

FILESYSTYPE TYPE(HFS) /* Filesystem type HFS */ENTRYPOINT(GFUAINIT) /* Entrypoint for defining HFS */PARM('SYNCDEFAULT(60) VIRTUAL(100) FIXED(100)') /* */

MOUNT FILESYSTEM('OMVS.DOMINO5.R501A.DATA.HFS')TYPE(HFS) MODE(RDWR)MOUNTPOINT('/notesdata')PARM('SYNC(60)')

MOUNT FILESYSTEM('OMVS.DOMINO5.R501A.MAIL.HFS')TYPE(HFS) MODE(RDWR)MOUNTPOINT('/notesdata/mail')

( ( ) )


Figure 16. Output of confighfs -q command

3.4.3.2 RMF Postprocessor reportWith OS/390 2.7, new Resource Measurement Facility (RMF) reports are available for Monitor II and the Postprocessor. These reports provide performance information on HFS which allows you to tune your system and optimize HFS resources. You can use the storage information and utilization statistics in the new reports to determine the optimum size of the HFS buffer storage.

Setting up RMF Monitor IIIData gathering for HFS file statistics in the postprocessor report is performed in the Monitor III gatherer session. The Monitor III gathers SMF record 74 subtype 6. A new option, HFSNAME, is available for Monitor III data gathering to define the appropriate files. If you do not set option HFSNAME, you get only the HFS global statistics report of the Postprocessor.

Figure 17. Example of ERBRMFxx for Monitor III

You can dynamically enable or disable this option by using these OS/390 operator commands:

F RMF,F III,HFSNAME(ADD(OMVS.SC64.STYRES1))F RMF,F III,HFSNAME(DEL(OMVS.SC64.STYRES1))

HFS global statisticsThe HFS global statistics report shows you a summary of HFS activity for the entire system and each buffer pool. It can then be used for determining the VIRTUAL and FIXED values. The important point to note is that the CACHE, DASD, and HIT RATIO fields in the FILE I/O section show only the statistics for sequential I/O. Statistics for random I/O are not currently provided.

An example of the HFS global statistics report is shown in Figure 18 on page 39.

Virtual storage: _____25856(pages)_______101(MB)

Fixed storage: _____25600(pages)_______100(MB)

Lookup cache hit: ______________517247Lookup cache miss: ______________2077351st data page hit: ______________7121191st data page miss: _____________1380943

Pool Size #DS BP_pages Fixed Already_fixed Not_already_fixed1 ____1 __1 _____14636 _____14636 __21516073 _________02 ____4 __1 ______7316 ______7316 __23115873 _______8963 ___16 __1 ______3776 ______3648 ___1737682 ___1563161

HFSNAME(ADD(OMVS.SC64.STYRES1),ADD(OMVS.SC64.STYRES2))


Figure 18. Example of HFS global statistics report

The field descriptions shown in the report are listed in Table 7.

Table 7. RMF field descriptions -- HFS global statistics

Field Heading Meaning

Storage Limits - All fields are given in megabytes and show the values at interval end

VIRTUAL MAX Value of VIRTUAL(MAX) parameter.

VIRTUAL USE Total amount of virtual storage assigned to I/O buffers.

FIXED MIN Value of FIXED(MIN) parameter.

FIXED USE Total amount of permanently fixed storage assigned to I/O buffers. This number is included in the VIRTUAL USE field.

File I/O - The fields are given as COUNT and RATE (count per second)

CACHE The first page of a data file was requested and found in virtual storage (cache).

DASD The first page of a data file was requested and not found in virtual storage, and an I/O was necessary.

HIT RATIO Percentage of cache-found requests based on total number of requests.

Metadata I/O - The fields are given as COUNT and RATE (count per second)

CACHE The metadata for a file was found in virtual storage during file lookup.

DASD The metadata for a file was not found in virtual storage during file lookup, and an index call was necessary which may have resulted in an I/O.


Buffer Pool Statistics

POOL NUMBER HFS defines four buffer pools for processing. This number is used to refer to each of these pools.

NUMBER BUFFERS Number of buffers in this buffer pool currently residing in virtual storage.

H F S G L O B A L S T A T I S T I C S

OS/390 SYSTEM ID ML96 START 12/27/2000-11.30.00 INTERVAL 006.30.00REL. 02.10.00 RPT VERSION 02.10.00 END 12/27/2000-18.00.00 CYCLE 1.000 SECONDS

--- STORAGE LIMITS (MB) -- ----- FILE I/O ---- --- METADATA I/O --COUNT RATE COUNT RATE

VIRTUAL MAX 500USE 500 CACHE 1132K 48.358 255K 10.914

FIXED MIN 200 DASD 1020K 43.570 135K 5.767USE 200 HIT RATIO 52.60 65.43

------------------------------------------ BUFFER POOL STATISTICS ------------------------------------------POOL NUMBER BUFFER -------- POOL SIZE -------- DATA ---------- I/O ACTIVITY ---------

NUMBER BUFFERS SIZE PAGES BYTES %FIXED SPACES TOTAL FIXED %FIXED1 22212 1 22212 87M 68 1 40M 27M 672 1403 4 5612 22M 87 1 12M 12M 1003 3061 16 48976 191M 63 1 2167K 2167K 1004 800 64 51200 200M 0 1 8385K 0 0


HFS file system statisticsThe HFS file system statistics report shows statistics for each HFS file system. The important points to note are that the CACHE, DASD, and HIT RATIO fields in the FILE I/O section show only the statistics for sequential I/O. If your applications use random I/O to certain HFS file systems, those caching statistics are not currently available.

An example of the HFS global statistics report is shown in Figure 19.

Figure 19. Example of HFS file system statistics

The field descriptions shown in the report are listed in Table 8.

Table 8. RMF field descriptions - HFS file system statistics

BUFFER SIZE Size of each buffer in this pool (in pages).

POOL SIZE - PAGE Size of this buffer pool currently in virtual storage (in pages).

POOL SIZE - BYTES Size of this buffer pool currently in virtual storage (in bytes).

POOL SIZE -%FIXED Percentage of the size of the buffers which is permanently fixed.

DATA SPACES Number of data spaces comprising this buffer pool.

I/O ACTIVITY - FIXED Number of times a buffer was already fixed prior to an I/O request in this buffer pool.

I/O ACTIVITY - %FIXED Percentage of fixed I/Os.

I/O ACTIVITY - TOTAL Total number of buffers in this buffer pool for which I/Os were issued. This is not necessarily the number of actual I/Os issued since multiple buffers can be written in a single I/O request.


FILE SYSTEM NAME The name of the HFS file system which has been selected for reporting.

MOUNT DATE and TIME Date and time when the selected file system resonated.

Allocation - All fields are given in megabytes

SYSTEM Amount of storage allocated to this file system.



DATA Amount of storage internally used within HFS for data files, directories and HFS internal structures like the attribute directory (AD).

ATTR. DIR Amount of storage used for the attribute directory (AD). This number is included in the DATA field. The attribute directory is the internal HFS structure (index) which contains attribute information about individual file system objects as well as attributes of the file system itself.

CACHED Amount of data buffer storage cached by this file system.

File I/O - The fields are given as COUNT and RATE (count per second)

CACHE The first page of a data file was requested and found in virtual storage (cache).

DASD The first page of a data file was requested but was not found in virtual storage (cache) and an I/O was necessary.


SEQUENTIAL Sequential file data I/O requests.

A sequential I/O is one of a series of I/Os to read or write a data file, where the first I/O started at the first byte of the file and each subsequent I/O was for the next sequential set of bytes.

RANDOM Random file data I/O requests.

A random I/O is an I/O that does not read or write the start of a file, and was not preceded by an I/O that read or wrote the immediately preceding set of bytes.

Metadata I/O - The fields are given as COUNT and RATE (count per second)

CACHE The metadata for a file was found in virtual storage (cache) during file lookup.

DASD The metadata for a file was not found in virtual storage during file lookup and an index call was necessary which may result in an I/O.


Index I/O - The fields are given as COUNT and RATE (count per second)

CACHE Index page read/write hits.

DASD Index page read/write misses.


Index Events

NEW LEVEL Number of times HFS added a new level to its index structure.

The index statistics are relative to all of the indices in the HFS dataset. The attribute directory (AD) is one index.



3.4.4 RecommendationUse the default settings of DFSMS bufferpool parameters as a starting point, then monitor the values. The default value of fixed storage size is zero and the default value of maximum virtual storage size is half of the central storage size.

If you do not have the processor storage that is called for, you should avoid paging activities. Set the parameter of the maximum virtual storage size of HFS bufferpool to reduce the size of HFS bufferpool.

3.5 Hierarchical file system (HFS)

OS/390 provides a standard UNIX hierarchical file system (HFS) for applications that use the UNIX interfaces. The HFS on S/390 is made up of a number of OS/390 datasets, called HFS datasets, that are mounted into the HFS.

An HFS dataset is a standard OS/390 PDSE with DSNTYPE=HFS. The PDSE access method is used only at allocation time for an HFS dataset. After that, the PDSE access method is not involved in I/O operations.

For more information on the HFS, see Lotus Domino for S/390 Release 5: Installation, Customization and Administration, Chapter 2.

3.5.1 HFS dataset allocation and DFSMS/MVSHFS datasets must be allocated as system managed datasets on volumes managed by Data Facility System Managed Storage (DFSMS). When allocating these datasets, DFSMS uses four constructs, as shown in Table 9.

Table 9. DFSMS constructs

You must define at least one Storage Group and one Data Class. The other constructs are optional. However, in order to specify performance requirements, you should define and use the Storage Class. The Storage Class is the only way to manage performance in a DFSMS environment.

You also need to check your SMS ACS routines to ensure that appropriate constructs are assigned to HFS datasets for Domino use.

SPLITS Number of times an index page was split into two pages because new records were inserted. This gives an idea of how much insertion activity there has been for the index structure.

JOINS Number of times HFS was able to combine two index pages into one, because enough index records had been deleted in the two pages.

DFSMS Construct Definition

Storage Class Specify dataset performance and availability

Management Class Specify dataset backup and migration criteria

Data Class Specify dataset allocation parameters

Storage Group Specify volume named and free space thresholds



3.5.1.1 DFSMS storage classA sample storage class for Lotus Domino datasets is shown in Figure 20. This uses a “may cache” policy for those files, resulting in better performance and better use of the available hardware.

Figure 20. DFSMS storage class

The performance objectives parameters tell the I/O subsystem what response time you require for datasets that are assigned this storage class. A dataset may have different attributes for sequential access and for direct access. DFSMS will attempt to meet those requirements by allocating the dataset on a suitable device. If you have DCME-capable control units, DCME will also manage the control unit cache to try to meet the objectives of all of the datasets that it is managing.

DFSMS uses hiperspaces to automatically stage PDSE datasets to expanded storage when the MSR (millisecond response) of the SMS Storage Class indicates a “must cache” dataset. However, because HFS datasets are not conventional PDSE files, it does not use this feature.

See MVS/ESA SML: Managing Data, SC26-3124 (edition 02) for more details.

3.5.2 HFS dataset managementThis section provides some management recommendations for the HFS datasets.

3.5.2.1 Placement of HFS datasets on DASD volumesAs with other large subsystems, one of the keys to good I/O performance is to spread the I/O load across multiple DASD volumes, control units, and channels. You therefore need to identify the high-activity HFS datasets. Since each HFS dataset contains one or more UNIX directories and multiple files, you also need to understand which are the high-activity UNIX files.

STORAGE CLASS DISPLAY Page 1 of 2Command ===>

CDS Name . . . . . : ACTIVEStorage Class Name : NOTESDescription : NOTES STORAGE CLASS

Performance ObjectivesDirect Millisecond Response . . . :Direct Bias . . . . . . . . . . . :Sequential Millisecond Response . :Sequential Bias . . . . . . . . . :Initial Access Response Seconds . :Sustained Data Rate (MB/sec) . . . :

Availability . . . . . . . . . . . . : NOPREFAccessibility . . . . . . . . . . . : NOPREFBackup . . . . . . . . . . . . . . :Versioning . . . . . . . . . . . . :

Guaranteed Space . . . . . . . . : NOGuaranteed Synchronous Write . . : NOCache Set Name . . . . . . . . . :CF Direct Weight . . . . . . . . :CF Sequential Weight . . . . . . :


The likely high-activity files are:

• The Notes data directory (/notesdata by default). This directory normally contains the Domino Directory, help files, and the billing, log, journal, and statistics files.

• Notes databases that are shared by many users.

You should monitor your system regularly to check for bottlenecks in I/O activity.

3.5.2.2 Separating high-activity filesIf you have an HFS file system with a lot of activity and you wish to split it, you need to:

• Move the high-activity UNIX files into different directories. Domino can find these databases by the use of links; refer to 3.5.2.3 for more information.

• Make sure that those directories are in different HFS datasets. A single HFS dataset contains all of the files in a single directory and may or may not contain files in subdirectories.

• Make sure that the high-activity HFS datasets are placed on different volumes. You can move an HFS dataset using DFSMSdss.

3.5.2.3 Domino linksBy default, Domino expects databases to be in the Notes data directory (/notesdata, by default) or in a subdirectory of it. However, you can place a database in any directory on the system and point to it using a Notes directory or database link. Thus you can spread your databases across multiple directories and HFS datasets, while logically keeping them in a single directory for ease of management. In addition, this allows you to spread out those system databases that Domino expects to find in the Notes data directory.

For information on how to set up Notes directory and database links, see the Domino 5 Administration Help database.

3.5.2.4 Consolidate extentsHFS datasets can expand using multiple extents. As with other OS/390 dataset types, accessing a dataset with multiple extents is less efficient. Therefore, you should monitor the number of extents in your HFS datasets and consolidate datasets with more than one extent using DFSMSdss.

3.5.2.5 Minimize directory levelsWhen OS/390 UNIX System Services opens a UNIX file, it locks the entire path it uses to access the file. Therefore, by minimizing the number of directory levels, you minimize the number of directories that need to be locked, reducing contention.

3.5.2.6 Domino server partitioningDomino server partitioning is the capability to run multiple Domino servers on the same OS/390 image. Partitioned servers share the same Domino code but each partition has its own data files, including the Notes data directory and user databases. Files cannot be shared between Domino servers.

From a performance perspective, we recommend that you separate the files for each server into separate UNIX directories, HFS datasets, and DASD volumes.


This will help you avoid data access contention. It also provides the simplest management environment.

Figure 21. Output of kerninfo stats command

This display reports the number of reads, writes, blocks, cache count, and other variables for each HFS dataset.

3.5.3 Domino I/O statisticsDomino provides statistics on database use in the Notes log database, log.nsf. See the views “Database by sizes or usage.” This gives you information on documents read and written, data transferred, and number of transactions. These statistics are helpful for monitoring the most accessed databases, how full they are, the access mode pattern, and distribution across time. Based on these

MISAO @ SC67:/u/misao>/usr/sbin/kerninfo statsKERNINFO: 10 Nov 1999 13:38:28

Total lookups: 2314120Look-aside hits: 79%Age-out misses: 0%Latch set: BPXPRTB1.PPRA.LATCH.SETLatches: 12289 HWM: 128 Fast obtains: 8680420 Slow obtains: 14284

Latch set: BPXFSLIT.FILESYS.LATCH.SETLatches: 16384 HWM: 64 Fast obtains: 52958966 Slow obtains: 131

Latch set: BPXFSLIT.FILESYS.LATCH.SET.01Latches: 16383 HWM: 1536 Fast obtains: 19246787 Slow obtains: 20949

Misc file system slow obtains: 0NFS

** no vfs **CINET

AF_INETdev=7(7x) cache count=0 vnode count=65VFS slow obtains=0 Vnode slow obtains=0Reads=642788 Writes=706898 Dir I/O=0Read Blks=0 Write Blks=0

AUTOMNT** no vfs **

UDSAF_UNIX

dev=6(6x) cache count=0 vnode count=8VFS slow obtains=0 Vnode slow obtains=0Reads=15846 Writes=7990171 Dir I/O=0Read Blks=0 Write Blks=0

TFS/TMP

dev=5(5x) cache count=5 vnode count=47VFS slow obtains=0 Vnode slow obtains=0Reads=1638 Writes=208181 Dir I/O=1588Read Blks=0 Write Blks=0

HFSOMVS.DOMINO.SC67.DATA1.HFS

dev=29(1Dx) cache count=0 vnode count=2VFS slow obtains=0 Vnode slow obtains=0Reads=21898 Writes=13970 Dir I/O=156520Read Blks=28338 Write Blks=116762

OMVS.DOMINO.SC67.MAIL.HFSdev=28(1Cx) cache count=0 vnode count=1VFS slow obtains=0 Vnode slow obtains=0Reads=8533 Writes=306 Dir I/O=3475Read Blks=12029 Write Blks=451

OMVS.DOMINO.SC67.DATA.HFSdev=27(1Bx) cache count=0 vnode count=2VFS slow obtains=0 Vnode slow obtains=0Reads=734313 Writes=85919 Dir I/O=2181116Read Blks=938753 Write Blks=123041

OMVS.DOMINO.SC67.PROD.HFSdev=26(1Ax) cache count=10 vnode count=17VFS slow obtains=0 Vnode slow obtains=0Reads=81566 Writes=1569 Dir I/O=306792Read Blks=1775633 Write Blks=72111


statistics, you may consider repositioning a database. See 7.2, “Notes log” on page 111 for more information.

3.6 Multivolume HFS

The hierarchical filesystem (HFS) now supports allocation of a multivolume dataset. The most important advantage of the multivolume HFS is the capacity to allocate more space per HFS without stopping the Domino server for a long time. But care must be taken, or performance could degrade.

This section discusses how to improve performance on multivolume HFS datasets, including recommendations on tuning the appropriate parameters.

3.6.1 Recommendations Here are some recommendations to get better performance with a multivolume HFS.

• Use the IBM Enterprise Storage Server (ESS) for a significant improvement.

• Avoid using a large number of volumes with multivolume HFS datasets. It is preferable to define 5 or 6 GB per HFS. Several HFS files with 2 or 3 volumes is better for performance than one HFS with 15 or 16 volumes.

• Plan your HFS directory structure to avoid a lot of volumes in the same multivolume HFS. When you use a lot of volumes, the access time to a databases increases considerably. When using multivolume HFS files, use a small number of volumes.

• Do not place many databases or files in your /notesdata HFS. Leave /notesdata as small as possible. The Domino Directory (Name and Address Book) is the most important document in the Domino Server. It resides in /notesdata, so a small notesdata directory avoids bottlenecks when accessing the Directory database.

• If you need more space for the /notesdata directory, you have two options for optimizing performance:

• Allocate a single HFS on an IBM 3390 model 9 volume. For example, you can allocate an HFS with this mount point:

/notesdata/mail1; /notesdata/mail2;/notesdata/mail...;/notesdata/mailn

• Make a multivolume HFS file on an IBM 3390 model 3 with a few volumes, under the /notesdata directory, and place your database in this HFS.

• Make your transaction log HFS as small as possible. We do not recommend using a multivolume HFS for this log; but if you need to, use a small number of volumes.

• Do not use a multivolume HFS if it is unnecessary. It is more efficient to use several single HFS files on IBM 3390 model 9 volumes (8.4 GB). Or, use the ESS and you will have the best performance possible.


Figure 22. HFS structure for Domino for S/390

3.6.2 Planning the HFS structureDomino for S/390 is a UNIX application running on OS/390 UNIX, so it is important to organize your HFS directory structure for performance. You only need a multivolume HFS if your database increases in size constantly.

With multivolume capability, which allows you to allocate one super-HFS with 59 volumes, it is possible to think, “I just need one or two large HFS files for all my data.” However, this is not the best way to handle your HFS structure. A small HFS is faster and easier to manage, while a large HFS is difficult to maintain and makes it difficult to back up data. If you lose just one volume, you have lost everything.

Figure 23. Domino address space

3.6.3 Multiple extentsWith multiple extents, each access to the directories can take multiple I/Os, thus degrading performance. If you run out of space, for example on /notesdata or the system HFS datasets, you will experience problems such as hangs, crashes, corrupted databases, and unpredictable output results.

Even if you could put all of your data in a single UNIX directory (and therefore a single HFS dataset), you would not do that because we know from our

IBM 3390 model 3

IBM 3390 model 9

HFS directory

Root/

/bin /dev /notesdata /tmp /u /usr /var /was /etc


experience with very large IMS and DB2 databases that there are problems with performance and manageability, such as the time needed to back them up.

In practice, therefore, we split up very large data bases into multiple datasets in order to avoid these problems. For Notes databases, there are also design reasons to keep them smaller. Therefore, you will want to split your Notes data up into separate UNIX directories and HFS datasets anyway.

You can use the IDCAMS LISTCAT command to display the current number of candidate volumes:

LISTCAT ENTRIES(OMVS.DATA.HFS) VOLUME

3.6.3.1 Contention Contention for a given resource can cause the server to slow down or hang. These contentions may involve OS/390 resources such as HFS files or disk addresses. Console commands of the Global Resource Serialization (GRS) component of OS/390 may help to find those contentions. The command DGRS,C will display existing latch contentions, and you can take action to solve them.

3.6.3.2 Preventive actions Periodic or weekly runs of copytree should be done while the HFS is dormant, that is, while no updates are being made to the HFS. Customers should run it several hours for very large HFS files, 20 to 40 minutes per 2 GB. The copytree utility can be found at the following Web location:

http://www1.s390.ibm.com/unix/

At that site, choose toys and tools. Note that using copytree’s check mode function as an alternative to copying an HFS will not prevent problems from occurring.

An alternative approach, to reduce the risk of impact and the need to run copytree frequently, is to use smaller HFS datasets. With these, customers can provide a logical representation of one large HFS by using pointers to insulate the end users or application from having to know the real physical location of a specific file. Another example of this is the use of database links for Domino.

Within Domino, you define the maximum size of a database. You can increase the allowable size of the database by using the Domino Administrator. Provided there is room in the HFS dataset, the database will then be able to expand. DFSMS 1.5 provides some additional capabilities in terms of database size.

For more information about the copytree utility, see Hierarchical Filesystem Usage Guide, SG24-5482.

3.6.4 Recommended HFS structure for UNIX System Services Make a plan for everything necessary to install HFS in an OS/390 UNIX System Services production environment. The HFS directory structure must be adequate and flexible enough to accommodate all kinds of data and services. Determine the mount points; allocate the HFS as multivolume, if the data is not static.

If it is static data, do not allocate an unnecessary multivolume HFS. Remember to allocate a small number of volumes per HFS to obtain better performance. We


recommend that you use separate HFS datasets for the mount points /var, /tmp and /u, because they each have unique usage characteristics.

3.6.4.1 Root file systemThe root file system is an aggregation of read-intensive files, much like LINKLIB on OS/390. If you lose the root file system, the USS does not work. We recommend making a copy of the root file system into another dataset. This copied dataset can then be used for the product maintenance activities. If the root file system is damaged, you can use this backup. Try keep the root HFS as a single volume. If you desire, you can allocate an IBM 3390 model 9 for the root HFS.

3.6.4.2 The /etc directoryThe /etc directory contains important customized data, which is used by USS and other product initialization processes. Do not use multivolume for this directory unless it is necessary.

3.6.4.3 The /var directoryThe /var directory contains dynamic data used internally by products and by elements and features of OS/390. The needed files or directories are created during the execution of code. An example of this is caching data. IBM products will only create directories or files when code is executed.

A typical process that uses /var is the syslogd daemon. It writes log records from USS and TCP/IP applications to files in the /var directory, or sends those records to other systems through the network. Since these log record files continue to increase, a space management task to delete increasing HFS files should be performed automatically. The cron daemon can periodically run a shell script to delete files.

This file system is considered dischargeable data. If these log records are used as statistics data, you must make a choice. Allocate a large single volume HFS, perhaps a 3390 model 9. Or allocate a small single volume HFS and schedule an efficient cleanup routine.

3.6.4.4 The /tmp directoryYou should use the option of mounting a temporary file system (TFS) on /tmp rather than using an HFS file system for it, because this directory contains temporary data used by products and applications, such as the sort program. TFS enables you to get better performance. Domino Release 5 uses /tmp more extensively than before. You need to take it into account when designing your data structure.

At one location, support tried to connect with telnet sessions. One telnet session was able to reach the server, but was unable to execute UNIX commands. The /tmp directory was filling the root directory.

To avoid this, create an independent HFS for /tmp with automount capability of an appropriate value on a separate file system. When you have a large number of interactive users, the /tmp directory can sustain large amounts of I/O activity. There are two different approaches you can take:

• Mount a temporary file system (TFS) over /tmp in the HFS, so that you have a high-speed file system for temporary files. The temporary file system is an


in-memory file system that is not written to DASD. As many caches exist in the Domino for S/390 environment, this option is not yet considered as a potential performance improvement.

• Place the /tmp directory in its own mountable file system and put the file system on its own pack. This is the option that we e used on our test system.

If the /tmp HFS is full, the Domino server might not start.

3.6.4.5 The /u directoryThe /u directory contains individual USS user HFS datasets. Almost all USS users have their home directory under the /u directory. We recommend that you use the Automount Facility under the /u directory. The advantages of this arrangement are discussed in the following section.

3.6.4.6 Automount facilityWith automount active and the correct automount policy in place, there is no need to create a user directory with the mkdir command. Instead, the user directory will be dynamically allocated and the user’s HFS dataset will be automatically mounted at the /u/userid mount point. If the user’s file system is not accessed according to certain time criteria in your automount policy, the user’s HFS dataset will automatically be unmounted. Figure 24, “Automount facility” on page 51, illustrates the Automount facility.

Another advantage of using the Automount facility is to manage user file systems (as opposed to adding them to BPXPRMxx). The last access time of the dataset is updated only when the dataset is mounted automatically. If a user goes on vacation for two weeks, their HFS might not be mounted and therefore becomes a candidate for DFSMShsm migration. On the other hand, if the user HFS datasets are defined in the BPXPRMxx, the HFS datasets will always be mounted, so the last access date will always be updated.


Figure 24. Automount facility

3.6.5 Recommended HFS structure for DominoWhen Domino for S/390 is installed, three HFS datasets are allocated:

/notesdata - control files, databases and templates

/notesdata/mail - Notes mail

/usr/lpp/lotus - Product code modules

An HFS dataset should not be allocated on the mount point /notesdata/noteslog, because files under /notesdata/noteslog tend to increase in size. If these log records fill the dataset, other files under /notesdata cannot be updated, and this can stop the Domino server activities.

In general, the mail HFS dataset should be separated into several datasets, and the directory structures should be changed according to the recommendations in Domino for S/390. Figure 25, “Domino HFS directory structure - abstraction” on page 52 shows an abstraction of a USS HFS structure (not multivolume) for Domino.


Figure 25. Domino HFS directory structure - abstraction

3.6.5.1 Lotus HFS datasetThe Lotus HFS directory, mounted on /usr/lpp/lotus, contains several executable files, tools, and shell scripts related to the Domino server function. They are read-intensive files, but not totally static; for example, updates take place when a new server partition is added. If we lose this dataset, the Domino server will not run. We recommend making a copy of the Lotus HFS, using another dataset name, in case the Lotus HFS is damaged. It is not necessary to allocate a multivolume HFS for this directory.

3.6.5.2 Notesdata HFS datasetEvery Domino server has a master directory that contains the key server files, such as notes.ini and the Domino Directory. All the Notes databases for the server are stored in that directory or subdirectories of it. During server setup, you set the name of the notesdata directory for that server. The default directory on S/390 is /notesdata.

As you can imagine, the notesdata directory is the most active directory in a Domino environment. Special attention should be given to its size and placement. Try to keep this HFS with just one volume (an IBM 3390 model 9, for example), so you will have better performance. The Notesdata HFS dataset should be allocated in an independent storage group or a volume which is not influenced by other I/O activities.

Other tips for the /notesdata directory include:

• With multiple extents, each access to /notesdata can take multiple I/Os, thus degrading performance. These datasets should be reorganized to bring them


back to one extent, and to allow room for growth. Monitor and reorganize all HFS datasets regularly.

• Do not put all databases under /notesdata.

• Do not create a lot of multivolume files and a lot of volumes in the same HFS.

• To create an HFS dataset under the Notesdata directory, verify the appropriate directory (mount points) structure; the need to be under /notesdata; and the activity type, that is, static (single volume) versus dynamic (multivolume). Actually, even a dynamic database can be allocated in a single volume HFS; just ensure that it is monitored, pruned, and reorganized regularly.

3.6.5.3 Noteslog HFS datasetThe Noteslog HFS dataset, which is mounted on /notesdata/noteslog, consists of various log files. These log files continue to increase in size, so the space management to delete the increasing files can be performed automatically by the cron daemon, which can periodically run a shell script to delete files. This file system is considered dischargeable data. Try to allocate a single volume HFS.

3.6.5.4 Mail HFS datasetsThe mail HFS dataset is usually split into several datasets under separate directories. Previously, depending on the disk subsystem used, several separate volumes were used. From DFSMS/MVS 1.5 on, this dataset can be considered a multivolume allocation candidate, both to avoid running out of space and to support large capacities for Notes client databases.

The mail HFS dataset has the following characteristics:

• In general, it would be separated into several datasets after the Domino installation.

• It consists of many Notes client databases, which sometimes need a large amount of storage space.

• Most of the update I/Os in the Domino system are against this dataset.

Usually a mail HFS dataset is separated into several datasets under separate directories. You can also have separate storage groups, as needed.

3.6.5.5 The Noteslog HFS DatasetThe Noteslog dataset should ideally be allocated in an independent storage group, for better performance. From DFSMS 1.5 on, this dataset can be considered a multivolume allocation candidate.


Figure 26. HFS structure

3.6.6 LogsIn addition to Notes databases, take a look at files such as Debug_Outfile, log files (http.logs, notes.logs), CEE dump outputs, and nsd.sh outputs. Those files are placed in the notesdata directory by default. They become very large, so we recommend creating a separate HFS for them. Then set a corresponding path. The notes.log, output path, rc.notes, and DOMCON utilities provide a notes.log and archives in the noteslog directory under the NOTESDATA directory. We suggest creating one (or more) HFS file for this directory, so that you can move the files out of the notesdata HFS.

3.6.6.1 Debug_outfile pathRemember to specify Debug_outfile if you need to set debug notes.ini parameters but the output size is unpredictable.

As an example, a customer is using the DEBUG_OUTFILE=debug.out parameter in NOTES.INI to capture debugging information. The LotusScript agent is running information (in conjunction with LOG_AGENTMANAGER=1 and DEBUG_AMGR=*), but the debug.out file growth is at the rate of more than 200 MB per day and puts a strain on disk free space.

We recommend the output file be separate from the notesdata and system HFS files.

3.6.6.2 NSD Output PathYou can set the environment variable NSD_LOGDIR to point to the directory where you want your NSDs to be automatically saved. For more information regarding NSD, run /usr/lpp/lotus/bin/tools/diag/nsd.sh -help.

3.6.7 SMS-managed or notSingle volume HFS datasets now do not need to be managed by SMS. DFSMS 1.5 allows HFS datasets to span multiple volumes, but those HFS types must still


be SMS-managed. All HFS datasets must be cataloged, regardless of whether they are SMS-managed or not.

3.6.7.1 The fsview utilityThe fsview utility, which you can download from the Tools & Toys page, lists all the file systems mounted on the root file system and prompts you for more detailed information about those file systems. fsview can be used by any user or superuser, although ordinary users may not be able to view the full hierarchy. The fsview utility can run in TSO or in the OS/390 shell. It should be placed in a directory of executable programs, or in a PDS of REXX execs. Set the permission bits to 500 so that you can execute it. If you want anyone to be able use it, set the permissions to 555 for universal read/execute.

3.6.8 Monitoring HFS spaceFor the file system, we recommend that the utilization of disk space should not exceed 65% of the entire file system. This is to ensure that even during the maximum storage in user mail files over weekends and holidays, the file system will not run out of disk space. Otherwise the entire file system may crash or prevent users from accessing the servers. You should monitor HFS space utilization using several methods, for example HFS file system usage, HFS dataset extents, and SMS-managed volume usage.

3.6.8.1 Monitoring dataset utilization Because of the impact of an HFS dataset running out of space, you should monitor the space utilization on a regular basis. Standard OS/390 tools can be used to do this. There are several ways in which you can find information about HFS file systems.

The df UNIX Systems Services commandUse the UNIX command df - options to monitor HFS space. See Figure 27 for more details.The df shell command gives you information about space utilization in a file system, in units of 512-byte blocks. Note that the output of the df command may show different values from confighfs and ISPF. This is because the df command also includes space reserved for shadow writes of the metadata pages, and therefore will show more space being used. The values will be the most alike immediately after a sync operation.

If you issue df without parameters, it tells you about all mounted file systems.

Figure 27. monitoring HFS space in UNIX System Services with df command

MAXQS @ SC53:/>df -kP|grep -i notesNOTES.DATA2.HFS 432000 118592 313408 28% /notesdata2NOTES.MAIL2.HFS 432000 118592 313408 28% /notesdata2/mailNOTES.LOG1.HFS 360000 180 359820 1% /notesdata1/noteslogNOTES.MAIL1.HFS 360000 5976 354024 2% /notesdata1/mailNOTES.DATA1.HFS 432000 278324 153676 65% /notesdata1NOTES.PROD1.HFS 576000 226936 349064 40% /usr/lpp/lotusMAXQS @ SC53:/>===>


Figure 28. Another view of the df monitoring command

See the HFS Usage Guide, SG24-5482, for more details about this topic.

MAXQS @ SC53:/>df -vk|grep -i notes/notesdata2 (NOTES.DATA2.HFS) 313408/432000 4294966540 Available/notesdata2/mail (NOTES.MAIL2.HFS) 313408/432000 4294966540 Available/notesdata1/noteslog (NOTES.LOG1.HFS) 359820/360000 4294967290 Available/notesdata1/mail (NOTES.MAIL1.HFS) 354024/360000 4294967293 Available/notesdata1 (NOTES.DATA1.HFS) 153684/432000 4294966508 Available/usr/lpp/lotus (NOTES.PROD1.HFS) 349064/576000 4294966880 AvailableMAXQS @ SC53:/>.


Chapter 4. Network tuning

All client types are connected to a Domino server on S/390 through a TCP/IP network In this chapter we discuss the performance of the TCP/IP network, and how you can monitor and tune it.

Note: When deploying a Domino server on OS/390, TCP/IP is the only network protocol supported.

A client-server application such as Lotus Notes is designed so that parts of the application run on the client and parts run on the server. These parts must communicate with each other, and this is done through a set of network resources (hardware and software). The challenge is to ensure good performance along the entire path and consistent response time to the user. Network performance is an important part of Domino performance.

4.1 Recommendations

Here are some key network tuning recommendations:

• Consider the whole network.

The network consists of many hardware boxes and links. Monitor all the components in the network and address bottlenecks.

• Define REGION=0M for the TCP/IP region size.

TCP/IP development recommends setting the region size at 0. TCP/IP buffers are dynamically allocated in the private area. REGION=0M allows TCP/IP to allocate the maximum virtual storage above and below the 16 MB line. If you have implemented an IEFUSI exit, check that it will allow the specification of 0M for the region size.

If you use the sample TCP/IP start procedure available in the SEZAINST sample library, you will find that releases prior to V2R8 include a sample procedure with region size of 7500 KB. Ignore this setting and use 0 instead. Remember to consult the INFOAPARS II11710, II11711, and II11712 for this and further tuning information.

4.2 TCP/IP

Lotus Domino for S/390 uses the TCP/IP network protocol to communicate with any kind of client. This section discusses TCP/IP considerations on OS/390.

4.2.1 TCP/IP release levelLotus Domino for S/390 R5 requires eNetwork Communications Server (CS/390) R6 or higher. There is no longer a strongly recommended version of CS/390, but you might get some benefit from the latest version.

4.2.2 Monitoring TCP/IPTCP/IP provides several commands that are useful for monitoring the system and highlighting problems. In particular, we discuss:

• PING


• TRACERTE

• NETSTAT

• TCPIPSTATISTICS

You can also use Spool Display and Search Facility (SDSF) and Resource Measurement Facility (RMF) to monitor TCP/IP at the OS/390 level. Monitor CPU usage by the TCP/IP address space, paging, and I/O rates for communication devices, for example. See 2.5.1, “Monitoring OS/390 with RMF” on page 18.

In addition, you should use a network monitor product for more detailed monitoring, tuning, and troubleshooting.

4.2.2.1 PINGThe PING command sends an echo to an IP host to determine whether the host is accessible. The response shows how long the round trip message took. This is the first tool to use to identify connectivity problems, and it can also show performance problems.

The PING command can be issued from any system that supports TCP/IP, including a Notes workstation client or a Web browser, where you would issue it from a command prompt. PING can also be issued from OS/390 TSO, as shown in Figure 29.

Figure 29. TCP/IP PING command

In this example, the PING command is sending 481 bytes of data five times. The time for the data to be sent to the remote host with IP address 9.12.2.17 and then returned to this system was 0.0184 seconds. High values for the response time can show performance problems somewhere between the issuing system and the target system.

4.2.2.2 TRACERTEThe TRACERTE command is similar to the PING command except that it reports the round trip time by intermediate hosts. It is helpful to check how each connection between the issuing system and the target system is performing and to determine bottlenecks along the route. You must be defined in the OBEY statement of the TCP/IP profile data set to issue this command. Figure 30 on page 59 shows an example of this command.

ISPF Command ShellEnter TSO or Workstation commands below:

===> ping 9.12.2.17 (length 481 count 5

Ping CS V2R6: Pinging host 9.12.2.17. Use ATTN to interrupt.PING: Ping #1 response took 0.016 seconds. Successes so far 1.PING: Ping #2 response took 0.018 seconds. Successes so far 2.PING: Ping #3 response took 0.019 seconds. Successes so far 3.PING: Ping #4 response took 0.020 seconds. Successes so far 4.PING: Ping #5 response took 0.019 seconds. Successes so far 5.***


Figure 30. TCP/IP TRACERTE command

For more information on the TRACERTE command, see OS/390 SecureWay Communications Server IP User’s Guide, GC31-8514.

4.2.2.3 D NET commandTo monitor TCP/IP buffer usage, you can use the VTAM command D NET:

• D NET,CSM (see Figure 31) • D NET,BFRUSE (see Figure 32)

To interpret the output of these commands, refer to VTAM Network Implementation Guide, SC31-8370.

Figure 31. Output from D NET,CSM

D NET,BFRUSEIST097I DISPLAY ACCEPTEDIST350I DISPLAY TYPE = BUFFER POOL DATAIST920I IO00 BUFF SIZE 1018 EXP INCREMENT 51IST921I TIMES EXP 0 EXP/CONT THRESH 50 / *NA*IST922I CURR TOTAL 1002 CURR AVAILABLE 870IST923I MAX TOTAL 1002 MAX USED 360IST989I EXP LIMIT 3000 BUFFS REQUESTED 0IST924I -------------------------------------------------------------IST920I BS00 BUFF SIZE 256 EXP INCREMENT 15IST921I TIMES EXP 0 EXP/CONT THRESH 15 / *NA*IST922I CURR TOTAL 30 CURR AVAILABLE 30IST923I MAX TOTAL 30 MAX USED 0IST924I -------------------------------------------------------------


===> tracerte 9.12.2.8

Trace route to 9.12.2.8 (9.12.2.8)1 (9.12.14.75) 6 ms 6 ms 7 ms***

D NET,CSMIVT5508I DISPLAY ACCEPTEDIVT5529I PROCESSING DISPLAY CSM COMMAND - OWNERID NOT SPECIFIEDIVT5530I BUFFER BUFFERIVT5531I SIZE SOURCE INUSE FREE TOTALIVT5532I ------------------------------------------------------IVT5533I 4K ECSA 204K 180K 384KIVT5533I 16K ECSA 48K 464K 512KIVT5533I 32K ECSA 320K 192K 512KIVT5533I 60K ECSA 0M 0M 0MIVT5533I 180K ECSA 0M 1080K 1080KIVT5535I TOTAL ECSA 572K 1916K 2488KIVT5532I ------------------------------------------------------IVT5533I 4K DATA SPACE 288K 160K 448KIVT5533I 16K DATA SPACE 864K 160K 1MIVT5533I 32K DATA SPACE 224K 1312K 1536KIVT5533I 60K DATA SPACE 0M 0M 0MIVT5533I 180K DATA SPACE 0M 360K 360KIVT5535I TOTAL DATA SPACE 1376K 1992K 3368KIVT5532I ------------------------------------------------------IVT5536I TOTAL ALL SOURCES 1948K 3908K 5856KIVT5538I FIXED MAXIMUM = 120M FIXED CURRENT = 5168KIVT5539I ECSA MAXIMUM = 100M ECSA CURRENT = 3634KIVT5559I CSM DATA SPACE 1 NAME: 00000CSMIVT5599I END

Chapter 4. Network tuning 59

IST920I LP00 BUFF SIZE 2032 EXP INCREMENT 50IST921I TIMES EXP 0 EXP/CONT THRESH 50 / *NA*IST922I CURR TOTAL 76 CURR AVAILABLE 73IST923I MAX TOTAL 76 MAX USED 7IST924I -------------------------------------------------------------IST920I XD00 BUFF SIZE 697 EXP INCREMENT 5IST921I TIMES EXP 0 EXP/CONT THRESH 4 / *NA*IST922I CURR TOTAL 5 CURR AVAILABLE 5IST923I MAX TOTAL 5 MAX USED 0IST924I -------------------------------------------------------------IST920I LF00 BUFF SIZE 120 EXP INCREMENT 30IST921I TIMES EXP 1 EXP/CONT THRESH 18 / 78IST922I CURR TOTAL 120 CURR AVAILABLE 31IST923I MAX TOTAL 120 MAX USED 97IST924I -------------------------------------------------------------IST920I CRPL BUFF SIZE 144 EXP INCREMENT 25IST921I TIMES EXP 0 EXP/CONT THRESH 18 / *NA*IST922I CURR TOTAL 475 CURR AVAILABLE 306IST923I MAX TOTAL 475 MAX USED 205IST924I -------------------------------------------------------------IST920I SF00 BUFF SIZE 112 EXP INCREMENT 32IST921I TIMES EXP 0 EXP/CONT THRESH 15 / *NA*IST922I CURR TOTAL 352 CURR AVAILABLE 337IST923I MAX TOTAL 352 MAX USED 15IST924I -------------------------------------------------------------IST920I SP00 BUFF SIZE 176 EXP INCREMENT 21IST921I TIMES EXP 0 EXP/CONT THRESH 1 / *NA*IST922I CURR TOTAL 21 CURR AVAILABLE 21IST923I MAX TOTAL 21 MAX USED 1IST924I -------------------------------------------------------------IST920I AP00 BUFF SIZE 56 EXP INCREMENT 56IST921I TIMES EXP 0 EXP/CONT THRESH 3 / *NA*IST922I CURR TOTAL 56 CURR AVAILABLE 56IST923I MAX TOTAL 56 MAX USED 0IST924I -------------------------------------------------------------IST920I TI00 BUFF SIZE 632 EXP INCREMENT 60IST921I TIMES EXP 145 EXP/CONT THRESH 120 / 300IST922I CURR TOTAL 540 CURR AVAILABLE 540IST923I MAX TOTAL 1140 MAX USED 983IST924I -------------------------------------------------------------IST920I CRA4 BUFF SIZE 4080 EXP INCREMENT 10IST921I TIMES EXP 0 EXP/CONT THRESH 20 / *NA*IST922I CURR TOTAL 50 CURR AVAILABLE 47IST923I MAX TOTAL 50 MAX USED 9IST924I -------------------------------------------------------------IST920I CRA8 BUFF SIZE 8176 EXP INCREMENT 6IST921I TIMES EXP 0 EXP/CONT THRESH 2 / *NA*IST922I CURR TOTAL 12 CURR AVAILABLE 12IST923I MAX TOTAL 12 MAX USED 1IST924I -------------------------------------------------------------IST449I CSALIMIT = 115250K, CURRENT = 4544K, MAXIMUM = 4989KIST790I MAXIMUM CSA USED = 4989KIST1667I SYSTEM CSA LIMIT = 115250KIST449I CSA24 LIMIT = 1000K, CURRENT = 47K, MAXIMUM = 48KIST790I MAXIMUM CSA24 USED = 48KIST595I IRNLIMIT = NOLIMIT, CURRENT = 0K, MAXIMUM = 0KIST981I VTAM PRIVATE: CURRENT = 5320K, MAXIMUM USED = 10057KIST924I -------------------------------------------------------------IST1565I CSA MODULES = 1672K

IST1565I CSA24 MODULES = 36KIST1565I PRIVATE MODULES = 5459KIST314I END

Figure 32. Output from D NET,BFRUSE

4.2.2.4 NETSTATThe NETSTAT command can be used to monitor byte counts and window sizes. It has many parameters; use NETSTAT HELP to display them. Figure 33 on page 61 shows an example of the NETSTAT command.


Figure 33. TCP/IP NETSTAT CONFIG command

4.2.2.5 TCPIPSTATISTICSTCPIPSTATISTICS provides TCP/IP statistics when TCP/IP is shut down. To get these statistics, specify this parameter on the ASSORTEDPARMS statement in the TCP/IP profile data set. Figure 34 on page 62 shows an example of these statistics.

Note: If you have OS/390 V2R8, you should use the GLOBALCONFIG keyword for this function.


===> netstat config

MVS TCP/IP NETSTAT CS V2R6 TCPIP NAME: TCPIPMVS 19:17:06TCP Configuration Table:DefaultRcvBufSize: 00016384 DefaultSndBufSize: 00016384DefltMaxRcvBufSize: 00262144MaxReTransmitTime: 120.000 MinReTransmitTime: 0.500RoundTripGain: 0.125 VarianceGain: 0.250VarianceMultiplier: 2.000 MaxSegLifeTime: 60.000DefaultKeepALive: 0.120 LogProtoErr: 00TcpFlags: 40UDP Configuration Table:DefaultRcvBufSize: 00065535 DefaultSndBufSize: 00065535CheckSum: 00000001 LogProtoErr: 01UdpFlags: 80

IP Configuration Table:Forwarding: 00000 TimeToLive: 00064 RsmTimeOut: 00060FireWall: 00000 ArpTimeout: 01200 MaxRsmSize: 65535IgRedirect: 00000 SysplxRout: 00000 DoubleNop: 00000StopClawEr: 00000 SourceVipa: 00000 VarSubnet: 00000MultiPath: 00000

SMF Parameters:InitType: 00 TermType: 00 ClientType: 00


Figure 34. TCP/IP shutdown statistics

See OS/390 SecureWay Communications Server IP User’s Guide for more details.

4.2.3 Tuning TCP/IPThis section contains some actions that you can take to tune TCP/IP. However, detailed TCP/IP tuning is a specialist skill. If available, refer to IBM MVS TCP/IP Performance Tuning Tips and Capacity Planning, Share 92, Session 3915, for more detailed tuning information.

4.2.3.1 TCP/IP priority in OS/390We recommend that you set the MVS dispatching priority of TCP/IP to be approximately the same as VTAM. Other TCP/IP servers, such as FTP and Telnet, should be set at a slightly lower priority than TCP/IP.

Also, make sure that TCP/IP is getting the OS/390 resources that it needs by monitoring it with RMF. If TCP/IP is constrained, it will not perform well. See Chapter 2, “OS/390 tuning” on page 5 for more information.

4.2.3.2 Specify REGION=0M for TCP/IPTCP/IP buffers are dynamically allocated in the private area. REGION=0M allows TCP/IP to allocate the maximum virtual storage above and below the 16 MB line, and thus does not constrain the number of buffers it can use.

4.2.3.3 OS/390 UNIX System Services network parametersThere are two parameters related to the network in the OS/390 UNIX System Services startup parameters in SYS1.PARMLIB(BPXPRMxx):

• NETWORK DOMAINAME(AF_INET) MAXSOCKETS

TCP/IP supports AF_INET sockets for communication with OS/390 UNIX System Services applications. Notes clients use AF_INET sockets to communicate with the Domino server. This parameter should be set to 35000. Starting with OS/390 V2R7, this parameter applies to each TCP/IP stack instead of across the entire image.

ipInReceives = 622 vmcfSendsOK = 188ipOutRequests = 371 vmcfSendsAbnormal = 0ipForwDatagrams = 0 vmcfSendsFatal = 5ipReasmReqds = 0 iucvReplies = 0ipReasmFails = 0 iucvReceives = 0ipFragCreates = 0 iucvSends = 0ipFragFails = 0 iucvRejects = 0ipInHdrErrors = 0 ioReads = 320ipInAddrErrors = 9 ioWrites = 224icmpInMsgs = 1 ioInOctets = 234860icmpOutMsgs = 4 ioOutOctets = 134482arpInRequests = 58arpOutReplies = 0arpOutRequests = 3tcpInSegs = 449tcpOutSegs = 360tcpRetransSegs = 72udpInDatagrams = 0udpOutDatagrams = 0


• NETWORK DOMAINAME(AF_UNIX) MAXSOCKETS

AF_UNIX sockets are used when two OS/390 UNIX System Services applications establish a connection. Use a value of 10000 for Domino.

For more information about these parameters, see Lotus Domino for S/390 Release 5: Installation, Customization and Administration.

There are two parameters related to storage in SYS1.PARMLIB(IEASYSxx):

• CSA (3000,262144)

• SQA (8,448)

There are three parameters related to the Communication Storage Manager (CSM) in SYS1.PARMLIB(IVTPRMxx):

• FIXED MAX (60MB)

Defines the maximum amount of storage dedicated to fixed CSM buffers.

• ECSA MAX (40MB)

Defines the maximum amount of storage dedicated to ECSA CSM buffers.

• POOL

One POOL definition can be specified for each CSM buffer pool of a particular bufsize and bufsource combination.

For more information about these parameters, see OS/390 MVS Initialization and Tuning Reference, SC28-1752 (edition 08). Helpful redbooks include the three-volume series entitled OS/390 eNetwork Communications Server V2R7 TCP/IP. The publication numbers are SG24-5227, SG24-5228, and SG24-5229.

There is a Web site for the latest TCP/IP migration hints and tips flashes, including W98019 (OS/390 V2R5), W98042 (V2R6), N3192 (V2R7), and N3196 (V2R8):

www.ibm.com/support/techdocs

4.3 Network connection

The S/390 server can connect to the IP network using:

• S/390 Open System Adapter (OSA) family of network adaptors

• 3174 Networking Server

• 3745/3746 Communications Controller

• 3172-3 Interconnect Controller

You should monitor the connection to ensure good performance.

4.3.1 Monitoring connection activityYou can monitor the connection from an OS/390 perspective, using RMF postprocessor or MONITOR-II. To use RMF MONITOR-II:

1. Start Mon-II data collection.

2. From TSO, enter RMFMONITOR.


3. Enter DEV COMM in the input area of the primary menu. This creates an I/O activity report for communication devices.

4. Enter D ON in the input area. This includes only the data since the last report, rather than a cumulative total.

5. Press Enter to begin.

6. Look at a couple of sequential intervals and see if there are any constraints (see the DELAY columns).

Figure 35 shows a sample report.

Figure 35. Sample RMF report to monitor communications device

In this example, address 102 is an OSA device. The address number matches the DEVICE statement in the TCP/IP profile data set.

For more information on OSA, see OSA Planning, GC23-3870 (edition 08).

4.4 Network performance

In addition to considering TCP/IP and the network connection on the S/390 server, you also need to consider all of the other network components. That is beyond the scope of this book. However, here we discuss the option of splitting the network.

4.4.1 Splitting the networkUnder certain circumstances you may consider splitting the network, in order to improve performance or to separate different types of workloads. For example, you could have a separate network for Domino traffic. This is likely to be more costly than running all traffic over a single network, but it can offer these benefits:

• It allows Domino work to be isolated from other workloads. Therefore, Domino will not be impacted if the other workloads generate a lot of network traffic.

• It allows you to tune the separate networks for their specific workloads. You could, for example, customize the TCP/IP parameters to a specific application pattern usage, such as Domino.

Note: Lotus recommends that you run a separate network for inter-server traffic if you use Domino clustering, to ensure good performance for that traffic.

Some of the ways that you could split the network traffic are:

• Add more network interfaces to the host.

CPU= 13 UIC=254 PFR= 0 DEV T13:15:17 DEV ACTV RESP IOSQ ---DELAY--- PEND DISC CONN %DEV %DSTG GRP VOLSER NUM LCU RATE TIME TIME DPB CUB DB TIME TIME TIME UTIL RV I%

00F --- 0.000 0 0 0.0 0.0 0.0 0.0 0.0 0.00 0 61102 --- 15.21 42 0 0.0 0.0 0.2 63.5 0.1 96.8 0 61158 --- 0.000 0 0 0.0 0.0 0.0 0.0 0.0 0.00 0 61159 --- 0.000 0 0 0.0 0.0 0.0 0.0 0.0 0.00 0 619CE --- 0.000 0 0 0.0 0.0 0.0 0.0 0.0 0.00 0 619CF --- 0.000 0 0 0.0 0.0 0.0 0.0 0.0 0.00 0 61DAA --- 0.000 0 0 0.0 0.0 0.0 0.0 0.0 0.00 0 61


By adding more network interfaces, you can balance the network traffic arriving at the S/390 Server. If the interface, such as an OSA card, is shared between several MVS images, you may consider adding additional ones and dedicating them to an MVS image.

• Run multiple copies of TCP/IP.

This allows you to tune each TCP/IP for the specific applications that it supports.

• Define additional PORT addresses.

Many LAN gateway interfaces support more than one port per gateway, such as OSA-2 and 3172-3.

• Use the equal-cost multipath function.

This is provided by OSPF dynamic routing protocol, supported by the OMPROUTE daemon. It allows you to balance the outbound traffic over multiple network interfaces. OSPF protocols in the router network provide for inbound traffic balancing over equal-cost routes.

• Use MVS virtual IP addressing (VIPA).

VIPA allows you to define more than one IP address on the same TCP/IP stack. The purpose is to aid in configuring a fault-tolerant network and for security reasons.

It is also useful if you want to drive the network traffic through different IP addresses for workload balancing. For inbound traffic, if you have a single TCP/IP stack on an OS/390 image and multiple VIPA addresses, you can use RIP to split the inbound traffic destined for each of the VIPAs across different network interfaces. Outbound traffic can also be split with RIP, but only by manipulating metrics and filters in the network routers outside of OS/390. Consult Communications Server V2R7 TCP/IP Implementation Guide, Vol. 1: Configuration and Routing, SG24-5227 for more information on this subject.

4.5 Domino network statistics

Domino provides statistics on network use in the Notes log database log.nsf. (see 7.2, “Notes log” on page 111). See the views “Usage By Date” or “Usage By User.” This gives you information on the amount of data transferred across the network. It is helpful to analyze and forecast network traffic. You can monitor the network usage throughout the day to determine peak times and bottlenecks.


Chapter 5. Domino tuning

This chapter discusses the Domino parameters you can set that affect the performance of your system. It shows how you specify which tasks the Domino server will run, and provides information on the relative resource use of those tasks. We also discuss useful Domino information that you should monitor.

5.1 Recommendations

These are the key recommendations for tuning Domino:

• Do not change the following parameters in Lotus Domino for S/390 Release 5:

• Server_Max_Concurrent_Trans

• Server_MaxSessions

Leave the values at their defaults. Otherwise, you may cause problems with the Domino server on S/390.

Note: In Lotus Domino for S/390 Release 5, the default value of Server_Max_Concurrent_Trans is -1, which is the disabled setting. Do not change this default value.

• Do not run unnecessary Domino tasks.

Check the notes.ini file and remove any server tasks that you do not need. For example, you may not need tasks such as logging and billing. You can run multiple versions of some tasks, but verify whether you need them.

• Schedule tasks for non-peak times.

Some tasks run on a scheduled basis. Schedule them to run at non-peak times. Examples are compact, index (to create a new full text index), update, updall, and fixup. The schedule is set by the Notes administrator but should be agreed on by the OS/390 operations staff to make sure that it does not interfere with other peak workloads or administration tasks on the server.

• Balance work between the server and the clients.

Lotus Notes is a client-server application. It runs on a combination of the Domino server and the Notes client. (If you are using a Web browser as the client, the work that can be done on that client is limited.) Consider running some functions on the client rather than on the server, to minimize the server load. For example, you could tell your users to:

• Put all users that they send mail to in their local Domino Directory, to avoid searching the Domino Directory on the server when sending mail.

• Replicate their mail and other databases to their Notes client, to offload processing from the server.

As you consider these ideas, keep in mind how much server offload you might get. For example, you may not be doing your mail on the server (no load), but you do need to replicate with the server (a load). Also consider the management aspects of users having database replicas on their workstations.

• Educate users on replication.

If users replicate databases to their Notes clients, educate them on the replication interval. We recommend a value of 15 to 30 minutes or more. As


each replication uses server resource, even if there is no data to transfer, a longer interval will reduce the server load.

Also educate users to use scheduled replication only on databases for which it makes sense. Within IBM we recommend that some applications be run locally by replicating them to the client. However, for applications used infrequently (for example, to submit expense claims), we recommend that those databases do not replicate automatically. Instead the application reminds the user to replicate the database manually whenever they update information in the local replica.

5.2 Domino initialization parameters

The notes.ini file provides the parameters for the initialization of the Domino server. During server startup, this file is read to set up the proper Notes environment and load the predefined tasks and programs.

The notes.ini file contains parameters to define:

• Server tasks to start

• Tuning parameters

• Other parameters

New for Release 5, the server configuration document in the Domino Directory allows you to set the same parameters that have always been set in notes.ini. Parameters which have existed prior to Release 5 can be entered on the notes.ini settings page. Newer parameters for Release 5, such as the number of mail.boxes to use, are generally integrated as fields within other pages of the configuration document. Some of these fields will be written through to the notes.ini file when the server is started, so don't be surprised to see that changes you made in the configuration document show up there.

In the following sections, we discuss the Domino server tasks and some of the performance parameters. We include a list of the Domino tuning parameters that can be set in notes.ini in Appendix B, “Domino parameters that affect performance” on page 133. For further information on the notes.ini file and the server configuration document, see the Domino 5 Administration Help database.

5.2.1 Changing parameters in notes.iniTo modify parameters in notes.ini you can:

• Use a server configuration document • Use the Set Configuration server command at the console • Edit the server document • Edit the file directly with an editor

These are all described in the Domino 5 Administration Help database and Lotus Domino for S/390 Release 5: Installation, Customization and Administration, SG24-2083 (edition 02).

5.2.2 The Notes_SHARED_DPOOLSIZE parameterThe NOTES_SHARED_DPOOLSIZE parameter is a UNIX environment variable that you can set before you start the Domino server. This variable tells the


Domino server how much UNIX shared memory to ask for in each request to OS/390. The default is 8 MB (8388608 bytes); we recommend that you use this value. This recommendation is different from Domino R4, where we recommended that you use a 32 MB dpoolsize in order to minimize the number of shared memory segments that are allocated. In Domino R5, the 8 MB size provides the best balance between the number of segments required and the internal serialization that Domino performs when using these segments.

You set this as a UNIX environment variable by putting the statement exportNotes_SHARED_DPOOLSIZE = 8388608 in the .profile file.

5.2.3 Thread parameterThe following notes.ini parameters are related to the number of physical threads per process.

• SERVER_ENABLE_THREADPOOL=1

This parameter controls whether the server will exploit the R5 thread processing model, where a single small pool of physical threads provided by the operating system is used to support a larger set of Domino virtual threads. Using this model, each user who connects to the server gets a Domino virtual thread. This is serviced as required by one of the physical threads in the thread pool.

• SERVER_POOL_TASKS

This defines a total number of physical threads. The default value of this parameter is 100. The number that you define in this parameter corresponds to the SMF108SLTT value of the SMF108 record. SMF108 records provide information about the maximum number of physical threads in use during the interval (SMF108SLPTIUMAX). If this value often reaches your defined maximum, we recommend considering additional threads.

For more information on monitoring the number of threads, see 7.1.2, “RMF Lotus Domino server report” on page 106.

5.2.4 Make Domino server tasks non-swappableIf you use RACF, you may improve performance by making the address spaces of some Domino server tasks non-swappable. This prevents the system from swapping the memory for an address space out of processor storage, so that

This environment variable is closely related to the OS/390 SYS1.PARMLIB(BPXPRMxx) parameter IPCSHMMPAGES. Do not change Notes_SHARED_DPOOLSIZE without first checking the value of this parameter.

If IPCSHMMPAGES is not large enough to accommodate the specified Notes_SHARED_DPOOLSIZE value, the server will not start. The Domino for S/390 5.0 Install Guide recommends that IPCSHMMPAGES should be set to the maximum value (25600).

See Lotus Domino for S/390 Release 5: Installation, Customization and Administration, SG24-2083, for more information.

Important

Chapter 5. Domino tuning 69

paging activity for Domino executables is reduced. A new notes.ini variable, NONSWAP_SERVER_TASKS, is available with Lotus Domino for S/390 Release 5.

We recommend that you mark as non-swappable only those parts that you put into LPA that are not dynamic load libraries (DLLs). The libnotes, decsext, domliba, and ftgtr are examples of DLLs that we recommend putting in LPA, while executables such as server, router, replica, and update are program executables that represent tasks and cause an address space to be created on their behalf. See 2.4.3, “Place program modules in LPA” on page 16 for more information about our LPA recommendations.

To make Domino server tasks non-swappable:

1. Permit the user ID that starts the server to access the BPX.STOR.SWAP facility class of RACF. At least READ access authority is required.

2. Put the modules into an APF-authorized data set. There are several methods for this, however we strongly recommend putting the modules into the dynamic LPA library.

Note: If the server is correctly set up according to the Domino S/390 R5 Install Guide, several modules (including libnotes, ftgtr, and decsext) should be in the dynamic LPA.

3. Turn on the sticky bits for the modules.

Note: If the server is correctly set up according to the Domino S/390 R5 Install Guide, the sticky bits of libnotes, ftgtr, domliba, and decsext should be turned on.

4. Specify “export Notes_NONSWAP_SERVER_TASKS” as a UNIX environment variable in the .profile file. List the module names in it, separated by commas. The following is a example of setting four tasks (server, replica, update, and router) to non-swappable.

export Notes_NONSWAP_SERVER_TASKS=server,replica,update,router

5. List the module names (server task names) in the NONSWAP_SERVER_TASKS notes.ini variable. All the task names should be separated by commas. The following is a example of setting four tasks (server, replica, update, and router) to non-swappable.

NONSWAP_SERVER_TASKS=server,replica,update,router

The following steps describe how to remove the setting of “non-swappable” for a task. These steps assume that you use the dynamic LPA library.

1. Shut down the server.

Make sure that the libnotes, ftgtr, domliba, and decsext modules are placed in LPA. If you forget to do this, you will get an authorization error from OS/390.

This procedure is closely related to 2.4.3, “Place program modules in LPA” on page 16.

Important


2. Remove the name of the server task from the Notes_NONSWAP_SERVER_TASKS statement. If you need to remove all the non-swappable tasks, remove the entire statement from the .profile file.

3. Remove RACF permission to BPX.STOR.SWAP for the user starting the server if you remove all the non-swappable tasks.

Note: Do not remove the permission if you remove only some of the tasks in the NONSWAP_SERVER_TASKS statement from your settings.

4. Restart the server.

The modules are still in dynamic LPA, though the tasks are not non-swappable. If you need to remove the modules from LPA, the following additional steps are required.

1. Remove the modules from the dynamic LPA library.

2. Turn off the sticky bits for the modules.

5.3 NSF bufferpool size

This parameter specifies the maximum size of the NSF buffer pool (in bytes), a section of storage dedicated to buffering I/O transfers between the NIF indexing functions and disk storage.

NSF_BUFFER_POOL_SIZE_MB is an important parameter to support a heavy workload. This parameter specifies the maximum size (in MB) of the NSF buffer pool, a section of memory dedicated to buffering I/O transfers between the NIF indexing functions and disk storage. It can take as much as one-third of the physical memory. You will benefit from a high value for storing views.

5.3.1 Setting the parameterYou may set up the NSF_Buffer_Pool_Size parameter by editing your notes.ini file. See 5.2.1, “Changing parameters in notes.ini” on page 68 to change parameters in your notes.ini file. You can also specify this setting in the NOTES.INI Settings tab of the Configuration Settings document in the Domino Directory.

Syntax: NSF_Buffer_Pool_Size=<value>

Note: You can also use NSF_Buffer_Pool_Size_MB to set the maximum size of the NSF buffer pool. This is the same as NSF_Buffer_Pool_Size, except it sets the size in megabytes instead of bytes. There is no Domino limit to how big you can set it. The default NSF_BUFFER_POOL_SIZE is 171.

5.3.2 MonitoringThe following steps show the monitoring procedure.

1. Use the show stat command to view the HFS bufferpool statistics. See Figure 36 on page 72.

2. Check Database.Database.BufferPool.Maximum.Megabytes to confirm your current NSF bufferpool size.


3. Check Database.Database.BufferPool.PerCentReadsInBuffer. This value should generally be more than 90 percent. If it is less than 90 percent, you should consider increasing your current NSF_Buffer_Pool_Size.

Figure 36. Using the show stat command to monitor HFS_Buffer_Pool_Size

The following describes the statistics revealed by the show stat command.

Database.BufferPool.Maximum.Megabytes The maximum allowed size of the NSF Buffer Pool. It defaults to 256 MB. If the notes.ini parameter NSF_BUFFER_POOL_SIZE=<bytes> is set, this parameter overrides the default.You may also use the NSF_BUFFER_POOL_SIZE_MB variable to set the size of the buffer pool in megabytes. It's a simpler way than using the NSF_BUFFER_POOL_SIZE variable to set the size of the buffer pool.

Note: Do not be alarmed if your statistics show that the values for the Peak and Used exceed the Maximum value by a small percentage. This is due to the way the server reads some data structures into the pool.

Database.BufferPool.Peak.Megabytes This value shows the maximum size to which the pool has grown since the server was last started.

Database.BufferPool.PerCentReadsInBuffer The percentage of file read requests made by Domino which were successfully obtained from the BSF buffer pool. The higher this number, the better.

Database.BufferPool.MM.Reads The cumulative number of times the buffer has been read.

Database.BufferPool.MM.Writes The cumulative number of times that data has been written to the buffer.

5.3.3 Tuning NSF_Buffer_PoolThe parameter NSF_Buffer_Pool_Size must be specified for a machine running a single server partition, or for a machine running more than one DPAR. It should always be specified for partitioned servers, since the physical memory in the machine has to be shared between multiple servers.

Other entries in the notes.ini file may be ignored by the R5 server. Many of these now have an equivalent setting option on a server configuration document and do not need to be listed explicitly in the notes.ini file. They will not impact server performance if left in the file.

Some Domino parameters may greatly influence the performance of a UNIX Domino server. For instance, the load Buffer_Pool value needs to be chosen very carefully.

> show stat:

Database.Database.BufferPool.Maximum.Megabytes = 172Database.Database.BufferPool.MM.Reads = 0Database.Database.BufferPool.MM.Writes = 0Database.Database.BufferPool.Peak.Megabytes = 33Database.Database.BufferPool.PerCentReadsInBuffer = 97.63

::


Setting it to a high value will allow Domino to cache its data easily, but may cause OS/390 paging overhead if central and expanded storage are in short supply. Setting this variable too low may eliminate paging by OS/390 at the expense of the Domino caching.

The correct setting of this variable is completely dependent on the amount of central and expanded storage, as well as on the number of paging packs that are available. When changing or setting NSF_BUFFER_POOL_SIZE_MB, a good rule of thumb is to monitor SMF/RMF data for system paging, and Domino database buffer pool statistics. Spread your databases over several file systems.

If you have a large server with lots of databases, I/O bottlenecks can become an issue, either at the hardware level or within OS/390 itself. SMF provides useful information for determining if you have a problem. Lotus Domino for S/390 Release 5: Installation, Customization and Administration, SG24-2083 (edition 02), provides a very good description on how to structure Notes directories for the best performance.

You can find an article about NSF_Buffer_Pool entitled “Optimizing Server Performance—Port Encryption and Buffer Pool Settings” at:

http://www.notes.net

5.3.4 RecommendationOur tests indicate that the default NSF_BUFFER_POOL_SIZE VALUE is a good starting point when configuring the Release 5 server.

To determine if you have made the buffer pool size too big, look for paging. Ensure that you have enough central storage available before increasing NSF_BUFFER_POOL_SIZE, or you will cause paging.

Another indicator is the Database.BufferPool.PerCentReadsInBuffer statistic. As you increase NSF_BUFFER_POOL_SIZE, you should see this value increase. If it decreases, you may have grown the buffer beyond the point of benefit.

NSF_DBCACHE_MAXENTRIESThe storage used for the dbcache comes out of the NSF buffer pool. Therefore, the higher this value, the less space for other Domino caching activity. If this value is not set, Domino sets it by default to three times the number of NSF_BUFFER_POOL_SIZE megabytes (for example, an NSF_BUFFER_POOL_SIZE of 171 Megabytes would result in a default NSF_DBCACHE_MAXENTRIES setting of 513).

The key indicator to look at when determining the size of NSF_DBCACHE_MAXENTRIES is the Database.DbCache.HighWaterMark statistic. If your high water mark reaches the max entries setting, then you should consider increasing the NSF_DBCACHE_MAXENTRIES setting, either by

It is very important to specify NSF_Buffer Pool in the notes.ini file. However, use care when changing the value of this variable.

Attention


increasing NSF_BUFFER_POOL_SIZE or by setting NSF_DBCACHE_MAXENTRIES itself.

5.4 Transaction logging

Domino R5 supports transaction logging and recovery. With this feature enabled, the system captures database changes and writes them to the transaction log. Then, if a system or media failure occurs, you can use the transaction log and a backup utility to recover your databases.

A single transaction is a series of changes made to a database on a server—for example, a transaction might include opening a new document, adding text, and saving the document.

Transaction logging provides three main benefits:

• In most situations, you no longer need to run the Fixup task to recover databases following a system failure. When transaction logging is not used, the server will scan all databases it can find at startup time to see if any are in an inconsistent state. When it finds one, it performs a type of fixup to remove the inconsistencies. This could require a great deal of time for large installations that have many databases.

When transaction logging is employed, only those databases that have updates in the log need to be processed at server startup. This greatly reduces the processing time, because the number of databases that need attention is much smaller, and the number of updates required to make a given database consistent is well known ahead of time. There is no need to scan all of the documents in the database to see if they are all correct.

• Transaction logging saves processing time because it allows Domino to defer database updates to disk during periods of high server activity. Transactions are recorded sequentially in the log files, which is much quicker than database updates to random, nonsequential parts of a disk. Because the transactions are already recorded, Domino can safely defer database updates until a period of low server activity.

• Using transaction logging simplifies your daily backup procedure. You can use a backup utility to perform daily incremental backups of the transaction logs, rather than perform full database backups.

Transaction logging is only valid for databases in the Domino R5 format. Earlier Domino formats do not support logging; they require backup and recovery to be done at the full database level. In the event of a system failure, Domino automatically performs database recovery against logged R5 databases during restart. The system uses the transaction logs to apply and undo database transactions not written to disk for databases that were open during the failure.

Additionally, Domino runs its standard Fixup utility against R4 databases, or against R5 databases with transaction logging disabled. In the event that a database becomes corrupted or lost, a backup utility must be invoked to recover the database from a full backup and archived transaction log files. Tivoli Data Protection for Lotus Domino R5 is IBM's solution for this backup utility.


See Chapter 14 of Lotus Domino for S/390 Release 5: Installation, Customization and Administration, SC24-2083 (edition -02) for more information on backup and recovery using transaction logging.

5.4.1 Setting up transaction loggingThe following procedure shows you how to set up transaction logging:

1. Ensure that all databases to be logged reside in the Domino data directory, either at the root or in subdirectories.

2. From the Domino Administrator client, click the Configuration tab.

3. In the “Use Directory on” field, choose the server's Domino Directory.

4. Click Server Configuration, and then click Current Server Document.

5. Click the Transactional Logging tab.

6. Decide if you will use circular or archival logging. Circular (the default) is easier to begin with and includes a slight performance gain. Whichever method you choose, ensure that you have enough disk space.

7. Make Transactional logging Enabled. See Figure 37.

8. Complete other fields as appropriate, and then save the document. See Domino 5 Administration Help database for more details.

9. Stop and restart the Domino server for the changes to take effect.

Note: The initial setup for transaction logging can result in a long period of unavailability while the server assigns DBIIDs to each database. This could take several hours.

Figure 37. Transactional logging tab in Domino Directory

5.4.2 Test result of transaction loggingTable 10 lists our test results, running with and without Transaction Logging.


Test environment:

• 9672-ZZ7 with three dedicated CPs

• OS/390 Release 2.6

• DFSMS 1.5 with DFSMS virtual storage 250 MB, fixed storage 250 MB

• Lotus Domino for S/390 Release 5.01

• Number of partitions: 1

• NSF_Buffer_Pool_Size_MB = 320

• Number of mail boxes: 8

• Number of connected users: 5000

• Application: Mail with calendar and scheduling

The test results are shown in Table 10.

Table 10. Test results of transaction logging

5.4.3 RecommendationWe recommend that you use transaction logging. A side benefit is a small reduction in the amount of CPU time required to support a given number of users, but the real benefit is seen in server restart time if the server crashes. You must, however, be very certain that you follow the instructions for putting the transaction log on an appropriately isolated file system on single volume. The transaction log can potentially receive an extremely large volume of I/O, and it will become a bottleneck if not configured properly.

5.5 Domino server tasks

Domino consists of a number of server tasks. These are shown in Table 11.

Table 11. Description of Domino server tasks

Case Recovery time Response time CPU utilization

Disable 01:40:40 (100%) 233ms (100%) 48.9% (100%)

Standard 00:17:43 (17%) 216ms (93%) 46.3% (95%)

Favor Runtime 00:33:24 (33%) 172ms (74%) 41.0% (84%)

Favor Recovery 00:12:35 (12%) 223ms (96%) 46.7% (96%)

Task name Description

AdminP Administration Process

AMgr Agent manager

Billing Billing

Calconn Calendar Connector

Catalog Cataloger

Chronos Updates full-text indexes

Cladmin Cluster Administration Process

Cldbdir Cluster Database Directory Manager


Table 11 notes:

1. Some of these tasks are required, while others are optional. Some, such as Fixup, Updall, Compact, and Catalog, can be scheduled to run at non-peak times.

2. Look at the tasks that your server is running and see if there are some that you do not need or that you can schedule at non-peak times. Stats, Update, AdminP, and Billing are some possible candidates that you may not need to run at peak times.

Clrepl Cluster Replicator

Collect Statistic Collector. Collects statistics for multiple servers.

Compact Database compactor

Design Designer

DIIOP Allows Domino and the browser client to use the Domino Object Request Broker (ORB) server program.

Dircat Directory Cataloger

Event Event monitor

Fixup Database fixup

HTTP HTTP Server

IMAP IMAP server

ISpy Sends server and mail probes and stores the statistics.

LDAP LDAP server

MTC Reads log files produced by the router and writes summary data about message traffic to a database for message tracking purpose.

NNTP NNTP server

Object Object store manager

POP3 POP3 server

Replica Replicator

Router Route mail to other servers

Sched Schedule manager

Server Used for the Domino user address space.

SH Used for Server Base address space.

Statlog Statistics. Records database activity in the log file.

Update Updates changed views and/or full-text indexes. Permanent task.

Updall Updates all changed views and/or full-text indexes for all databases. Temporary task.

Web Web retriever

Task name Description


3. The Reporter task no longer exists in Release 5. The file statistics tasks performed by the Reporter task in Release 4 are handled by the Directory Catalog.

For a detailed description of the various server tasks, see the Domino 5 Administration Help database.

5.5.1 Domino tasks startedWhen the Domino server starts, it will start a number of tasks, as specified in notes.ini. If you issue the server command show tasks from the Domino server console, you will see the output shown in Figure 38 on page 78.

Each task listed in this example is a thread in a process, and each process has an associated address space. For more details, see 2.2.1, “OS/390 address spaces” on page 6.

Figure 38. Output from the show tasks command

5.6 Replication

Replication is one of Domino's key functions. Replication ensures that all users on a global network can access the latest version of a database without having access to one centrally located database. By accessing a local copy, users can get better response times.

> sh ta

Lotus Domino (r) Server (Release 5.0.1a for UNIX) 10/21/99 05:55:46 PM

Server name: Sonomi/ITSO - System Under TestServer directory: /r50dataPartition: .r50dataElapsed time: 00:02:26Transactions/minute: Last minute: 0; Last hour: 0; Peak: 0Peak # of sessions: 0 atTransactions: 0Availability Index: 100 (state: AVAILABLE)Message Tracking: Not EnabledShared mail: Not EnabledNumber of Mailboxes: 4Pending mail: 0 Dead mail: 0Waiting Tasks: 0Transactional Logging: Not Enabled

Task Description

Database Server Consistency check will start in 21 secondsDatabase Server Perform console commandsDatabase Server Listen for connect requests on TCPIPDatabase Server Load Monitor is idleDatabase Server Idle taskDatabase Server Idle taskDatabase Server Perform Database Cache maintenanceDatabase Server Idle taskDatabase Server Idle taskDatabase Server Idle taskDatabase Server Idle taskDatabase Server Idle taskDatabase Server Idle taskDatabase Server Idle taskDatabase Server Idle taskRouter Idle


During replication, copies of a database on different systems exchange modifications in one or both directions. There are two types of replication:

• Replication between servers

• Replication between a client and a server

5.6.1 Replication between serversReplication between servers is managed by the Notes administrator. An administrator can set up the system so that databases will be automatically replicated at intervals.

In a large Domino network, replication is a important factor in planning network traffic and system load. You should carefully plan the times, frequency and type of database replication. To optimize this area, the Notes administrator should work closely with the OS/390 system programmer and the network specialist.

For more information about settings for server-to-server replication, see the Domino 5 Administration Help database.

5.6.1.1 Number of replicator tasksOn a processor with a single engine, run a single replicator task. The task will handle replication requests one at a time.

On a processor with multiple engines, we recommend that you start with a single replicator task. If you have a replication backlog you can start more replicator tasks, but the maximum number should be one less than the number of engines. This will ensure that one engine is always available for time-critical work.

For more information, see the Domino 5 Administration Help database.

5.6.2 Replication between a client and a serverDatabases can also be replicated between a client and a server. Creating a local copy of a database on your workstation will significantly increase performance and availability. It will reduce the amount of time it takes to open and access the database (such as your mail), as well as let you continue to process when the network or server is temporarily unavailable.

Client replication is managed from the Notes client. The Notes administrator cannot control this type of replication, other than by restricting databases from being replicated and by providing guidelines to the users.

To tune client-to-server replication, consider the following:

Large databases A single replication can take a large amount of CPU time and do many I/Os to the HFS, depending on the size of the database.

Resources used Each replication uses resources even when there are no changes to the databases.

5.6.2.1 Replication frequencyUsers could use a significant amount of resources replicating databases between the clients and the server even when there are no changes to the data. Therefore, you should encourage your users to not do unnecessary replication.


The replication frequency is specified in the repeat every keyword in the Notes client's location document. This value should not be set below 15 minutes. Thirty minutes or higher is recommended.

5.6.2.2 Monitoring replication activityThere is no administration tool to limit the frequency of replicating databases to a client (the minimum value is one minute). However, the Notes administrator can see the replication activity in the log.nsf database. You can request that client session events are logged at server setup time, or with the notes.ini parameter Log_Sessions, or with the Log_Replication parameter.

We recommend using the Log_Replication parameter and setting it to 1. This will reduce the amount of message traffic at the server console (compared to Log_Sessions), since there won't be messages logged every time a user opens a database on the server for non-replication purposes. Log_Replication will show more details about replication, depending on the value you select.

You will see messages in the Notes log for each client-to-server replication. Figure 39 shows that documents for the user Ingo ik Karge are checked for replication every minute.

Figure 39. Client replication events in notes log

Thus you can see how often your users have set up replication to run on their Notes clients.

5.6.2.3 Database access on server versus localThe decision on whether it is more efficient to access a database on the server or replicate it to the client and access it locally depends on many factors, such as:

• Will the user access all of the data in the database, or just a subset of it?

• How often does the user need to replicate to have up-to-date information?

• Will all users access the server at the same time, resulting in a large workload peak on the server which could be eliminated if replication is used?

5.7 Indexing

Indexing server tasks are used to update and repair the views and the index of a database after the database has been changed. Update and Updall are the two tasks responsible for this process.

Only a user with designer or manager access to the database can create a new index for a database. For example, you can create an index on your own mail database because you have manager access level. For the common databases,

03/10/98 06:01:02 PM Opened session for Ingo ik Karge/Poktest2 (Release03/10/98 06:01:04 PM Closed session for Ingo ik Karge/Poktest2Databases accessed: 3 Documents read: 0 Documents written:03/10/98 06:01:32 PM Database mail/draisch.nsf created by Ingo ik Karge/03/10/98 06:02:05 PM Opened session for Ingo ik Karge/Poktest2 (Release03/10/98 06:02:06 PM Closed session for Ingo ik Karge/Poktest2Databases accessed: 3 Documents read: 0 Documents written:03/10/98 06:02:48 PM Database mail/bmacfade.nsf created by Ingo ik Karge03/10/98 06:03:08 PM Opened session for Ingo ik Karge/Poktest2 (Release03/10/98 06:03:09 PM Closed session for Ingo ik Karge/Poktest2

b d d i


especially large databases, the Notes administrator limits the number of people who can create new indexes and full text indexes. This is important because indexing can use a significant amount of system resource.

Chronos is another server task that Domino runs hourly to update the index.

5.7.1 Update and Updall tasksUpdate is a standard server task, and can be started from notes.ini. This task is usually automatically started when the server is started. Update runs continually, checking its work queue and folders that require updating. When a view or folder change is recorded in the queue, Update waits approximately 15 minutes before updating all view indexes in the database so that the update can include other database changes made during the 15-minute period.

To improve view-indexing performance, you can run more than one update task. However, each update task will use additional resources, so plan this carefully.

Updall updates all views that have been accessed once and all full text indexes for all databases on the server. It runs as a temporary server task. After finishing its work, the task shuts down. Updall can also be started from notes.ini. Check whether you need this function in your environment.

Note: The Updall server task has a number of parameters or arguments. We recommend that you run Updall without any arguments specified. In particular, beware of the -R argument. It rebuilds from scratch all database views that have been accessed at least once and updates all full text indexes. This can be time-consuming and affect server performance, so use it only as a last resort to solve corruption problems.

Both Update and Updall can be scheduled using the ServertasksATx= Taskname parameter in notes.ini.

When you create an index for a database, you can select an update frequency from the list for the option “update frequency”. The options you can select are shown in Table 12. You can change this option after creating the index. The default is “Immediate”, but this default option consumes high amounts of CPU in the update process. We don’t recommend using the default option. A less frequent schedule is recommended. For more information, see Domino 5 Administration Help database.

Table 12. Update frequency options

5.7.2 Full text indexA full text index is a collection of files that lets users search database information. Only a user with designer or manager rights can create a full text index.

Frequency option Updates occur

Daily Nightly, when the Updall server program runs by default at 2 AM

Hourly Every hour, as scheduled by the Chronos server task

Immediate As soon as possible after you close the database

Scheduled As scheduled by a Program document for the Updall server task in the Domino Directory


The creation of a full text index takes system resources:

• CPU cycles are used to create the index.

• I/Os are generated to build the index.

• DASD space is used to hold the index.

Depending on the size of the database, and the amount of text, indexing can take a significant amount of resource. Therefore we recommend:

• Running indexing at times of low system load.

• Limiting which users can do indexing.

Overall, expect up to a 10 percent overhead in CPU consumption required to support full text index processing once the index has been created. Memory consumption may grow by a similar amount, and DASD activity will also climb slightly, depending on how much the users of the system make use of the indexes.

5.7.3 Number of indexing tasksOn a processor with a single engine, run a single indexing task. The task will handle indexing requests one at a time.

On a processor with multiple engines, we recommend that you start with a single indexing task. If you have an indexing backlog you can start more indexing tasks, but the maximum number should be one less than the number of engines. This will ensure that one engine is always available for other, more time-critical work.

5.7.4 Indexing parametersWe recommend the following parameter settings for indexing:

Update_Suppression_TimeThis parameter specifies the delay time between full text index and view updates. Increase the value to index less often.

Update_Suppression_LimitThis parameter overrides Update_Suppression_Time and forces an update of the views when a specified number of updates have been received.

5.8 Compact

When documents and attachments are deleted from a Notes database there are blocks of unused space left within the database. Compact is the Domino task to get back this unused space. Databases should be compacted periodically to reclaim this space. Statistics reports can be used to monitor databases sizes.

There are several ways to run the compact task. For example, you can start it with the load compact command from the Domino console. Compact is not automatically started with the default notes.ini. It runs as a temporary task.

There are three styles of compacting used in Release 5:

• In-place compacting with space recovery

• In-place compacting with space recovery and reduction in file size


• Copy style compacting

Table 13 compares the three styles of compacting.

Table 13. Comparing the three styles of compacting

Compaction can use a lot of processor time and create a large amount of I/O activity, depending on the amount of data being compacted. Therefore, you should run it at a quiet period on the system.

We recommend that you run compact at least monthly to recover unused space, and more often if there are a lot of deletes in your databases. You should also run compact for a specific database after a lot of delete activity. Note that Release 5 is much better at reusing space in R5-format databases, so compacting is needed less often than in past releases.

Once you install Domino R5 on a server, you can upgrade the format of databases on that server. Compacting a database using Domino R5 upgrades the database format (On-Disk Structure, or ODS) to Release 5. When migrating a name and address book database from Domino Release 4 to Release 5, it is extremely important to compact this database. For more information on migration of a Domino server, see Domino 5 Administration Help database.

Notes:

1. Compacting a database using copy style compacting creates a temporary copy of it in the same directory as the original. Make sure there is enough disk space available for the temporary copy in the HFS data set, especially if the database is large.

2. By default, compacting converts Release 4 databases to the Release 5 file format or ODS. To prevent this, use the -R option with the load compact

command. However, it is recommended that you convert Release 4 file format databases to the Release 5 format, because there are many advantages.

Characteristics In place, space recovery

In place, space recovery with file size reduction

Copy style

Databases that use it when compact runs without options

R5 logged databases with no pending structural changes

R5 unlogged databases with no pending structural changes

R4 databases; R5 databases with pending structural changes

Databases you can use it on R5 R5 (Need -B option to enable size reduction)

All (Need -C for R5 databases)

Relative speed Fastest Medium Slowest

Users can read databases during compacting

Yes Yes No (Unless -L option used)

Users can edit databases during compacting

Yes Yes No

Reduction in file size No Yes Yes

Extra disk space required No No Yes


3. Compacting a database on a server with transaction logging enabled causes the DBIID to change. This database then requires a full backup. This is handled for you if you are using the Tivoli Data Protection tool.

See the Domino 5 Administration Help database for more information on compact.

5.9 Unread marks

One performance enhancement made to the R5 server was to give administrators and users the ability to disable unread marks processing for specific databases. Unread marks may be disabled via a selection from the advanced features page of the database properties. When disabled, the server stops maintaining a list of documents that each user has not yet read.

For Release 4, unread marks were maintained for all databases. This created a bottleneck for the server, because it had to maintain the unread marks for the public name and address book of the domain, for each user who opened this database. The problem would became especially acute when a user opened the public NAB on the server for the first time. This caused the server to lock the NAB until the unread marks list was built for the requesting user. This blocked anyone who simply wanted to connect to the server to read or replicate their mail. While the issue was not a big one for small to moderately-sized domains, it became a problem for enterprise-wide domains.

For Release 5, the template for the Domino Directory (public NAB) has the “Don't maintain unread marks” property turned off by default, so this will not be a problem unless you enable unread marks processing explicitly. However, you may want to consider disabling unread marks for other large databases which receive a high volume of traffic, or which are accessed with a Web browser, where unread marks are not seen by the end user anyway.

It is important to keep in mind, though, that unread marks are a useful feature of Domino, and you may not want to disable them. Databases where it is probably desirable to maintain unread marks include the mail databases of your users and discussion databases where users want to know which documents are new.

5.10 Multiple MAIL.BOX database

When a Domino server is used as a Notes mail server, the router process is used to route mail between users. This process makes use of a special database called MAIL.BOX, which serves as a temporary location for notes that are in transit. Prior to Release 5, the router could make use of only one MAIL.BOX database, and this would become a bottleneck when the server got to be heavily loaded.

For Release 5, it is now possible to create several MAIL.BOX databases, and to have the Router use multiple execution threads so that mail routing can take place in parallel. This significantly improves the scale and performance characteristics of the server.

Disk contention is rarely an issue for MAIL.BOX, so you usually do not need to put the multiple MAIL.BOX databases on different disks.


You can specify the number of MAIL.BOX databases in the configuration settings document. The maximum number of MAIL.BOX databases is 10.

To create multiple MAIL.BOX databases:

1. Make sure you already have a Configuration Settings document for the server to be configured.

2. From the Domino Administrator client, click the Configuration tab and expand the Messaging section.

3. Click Configurations.

4. Select the Configuration Settings document for the mail server or servers you want to restrict mail on and click Edit Configuration.

5. Click the Router/SMTP - Basics tab.

6. Input “number of mailboxes” field. The default is 1 and the maximum is 10.

7. Save the document.

You can confirm how many MAIL.BOX databases you currently use on the server from sh tasks command output. You can see sample output from this command in Figure 38 on page 78.

We think that a single MAIL.BOX is sufficient for a small installation; for example, fewer than 1,000 registered users. Above this number, we recommend adding one or more MAIL.BOX databases. As a very general rule, if pending mail starts to accumulate, one possible cause of the problem could be contention on MAIL.BOX databases.

From our test experiences, more than one MAIL.BOX was a big help, but adding more than two had a much less positive impact on router performance. However, the optimal number of MAIL.BOX databases varies depending on router usage in your system.

For more information, see the Domino 5 Administration Help database.

5.11 Web server

Domino has a built-in Web server, which we discuss in the following section.

5.11.1 Domino Web serverThe Lotus Domino Server Release 4.5 and later provides Web server capability. Web browsers can connect to the Domino server and access Notes databases. Most functions available to a Notes client are available to a Web browser, although some, such as database replication, are not supported due to technology differences between the two user interfaces. The Domino server provides Web server function, and also provides a conversion between the Web HTTP and HTML protocol standards and the Notes internal standards. This is shown in Figure 40 on page 86.


Figure 40. How the Domino Web server works

The Web server task is called http. You can start it by adding it to the ServerTasks parameter in notes.ini, or by issuing the Domino server command load http. OS/390 creates an address space for the http task.

5.11.2 Domino Web server parametersThe http task reads the parameters in the HTTP server section of the server document in the Domino Directory. There you can specify the minimum and maximum number of threads that you want the http task to run. When a request is received from a Web browser, it is run on one of these threads. When the response is sent to the browser, the thread is released and is available for the next request. There is no long-running user connection.

If there are no available threads when a request is received, the request is queued until a thread is released. Therefore, you should ensure that you have the maximum number of threads set sufficiently high. You can see the number of threads used by issuing the command show stat domino on the Domino server console.

You should set the minimum number of threads high enough to handle a typical workload. This reduces the need to create threads, and thus minimizes the system load.

For more information on the Web server task and related tuning parameters, see the Domino 5 Administration Help database.

5.11.3 Domino for IBM HTTP server connectorDomino for IBM HTTP Server Connector is a new feature included with Lotus Domino for S/390 Release 5.0.1. It allows the IBM HTTP server to process HTTP

Domino

ServerNotesClient

HTTPPage Rendering

WebBrowser

NSFDatabases


requests for Domino databases. When this configuration is in effect, the IBM HTTP server receives all URL requests from browser clients and passes to Domino any requests that include the file extension .NSF, which indicates a request for data in a Domino database. The IBM HTTP server processes all other requests (HTML files, CGI scripts, and others).

Domino for IBM HTTP Server Connector is supported on OS/390 Version 2 Release 7, in conjunction with the IBM HTTP Server for OS/390 Version 5.1.

All information and setup instructions for the Domino IBM HTTP server are included on the Domino software distribution CD at:

/APPS/webconnector/connector.nsf


Chapter 6. Domino tuning (advanced functions)

In July 1998, the Domino Advanced Services license was incorporated into the Domino Enterprise Server (DES) offering. The logical features provided by the DES license remain unchanged.

The Domino Enterprise Server features consist of:

• Partitioning

• Clustering

• Billing

In this chapter we discuss the Domino tuning of partitioning and clustering.

6.1 Partitioning

Partitioning allows you to run multiple Domino servers on a single OS/390 system. There are several benefits of partitioning, including:

• Isolation of workloads from each other

• Consolidation of multiple Domino servers quickly and easily

• Minimizing costs of running multiple servers

• Increasing scalability, enabling many users to run on a single OS/390 server

• Increasing availability by isolating workloads from each other

For those readers who know S/390 logical partitioning, it may be helpful to summarize the differences between Domino partitioning (DPAR) and S/390 logical partitioning (LPAR). With logical partitioning, we can run up to 15 different operating systems on a single S/390 processor. These “images” can be any combination of supported operating systems. On the other hand, with Domino partitioning, we can run multiple Domino servers on a single OS/390 server. Partitioned servers share the same binary files (Domino code), but have individual data directories which allows for independent configuration of resources.

Note: The limit of six supported partitioned servers on one system has been removed in Domino Release 5. System resource was the limiting factor.

See Lotus Domino for S/390 Release 5: Installation, Customization and Administration, SG24-2083 (edition 02) for a more detailed discussion of partitioning. Now we look at the performance aspects of partitioning.

6.1.1 Costs of partitioningRunning partitioned servers uses increased resources in some areas, such as:

• Disk space to hold the server files and multiple copies of databases. Each server must have its own set of files and databases. There is no sharing between servers.

• Processor storage to run the server. There is a base storage requirement for each server.


• CPU resources for communication between servers. This includes mail routing and database replication. In a single server, these tasks are not needed.

The following sections describe our test results from multiple Domino partitioned servers on a single OS/390 server.

6.1.1.1 Test result with two Domino partitioned serversThis is our test result running two Domino partitioned servers on a single OS/390 system.

Test environment: • 9672-ZZ7 with three dedicated CPs

• Storage configured: Central storage 2 GB, Expanded storage 1.94 GB.


• DFSMS 1.5 with DFSMS virtual storage 250 MB, fixed storage 250 MB


NSF_Buffer_Pool_Size_MB = 320

• Application: Mail with calendar and scheduling.

Data11: 1 Domino partitioned server (DPAR), 2500 connected users


Data13: 2 Domino partitioned servers (DPARs), 5000 connected users. Each server has 2500 users.

Results:The test results are shown in Table 14 and in Figure 41 on page 91.

CPU usage of Data12 and Data13 was nearly the same; therefore the CPU cost for two DPARs was small in this test. However, Data13 used an additional 612 MB of central storage over Data12.

Table 14. Test results of two Domino partitioned servers

CPU utilization(%)

Central storage used (MB)

Expanded storage used (MB)

Total storage used (MB)

Data 111 DPAR2500 users

24 1213 23 1236

Data 121 DPAR5000 users

50 1436 23 1459

Data 132 DPARs5000 users

51 2024 47 2071


Figure 41. CPU utilization and total storage usage for test with two DPARs

6.1.1.2 Test result with four Domino partitioned serversThis is our test result running four Domino partitioned servers on a single OS/390 server.

Test environment: • 9672-ZZ7 with 12 dedicated CPs

• Storage configured: central storage 1887 MB, expanded storage 14 GB.


• DFSMS 1.5 with DFSMS virtual/fixed storage 350 MB (data21,data22), 400 MB (data23, data24)


NSF_Buffer_Pool_Size_MB = 160

• Application: Mail with calendar and scheduling.





Results:The test results are shown in Table 15 and Figure 42. The CPU cost for three or four DPARs is relatively high.

Table 15. Test results of four Domino partitioned servers

CPU utilization (%)

Shared storage used (MB)




1 DPAR 10 871 1736 33 1769

Chapter 6. Domino tuning (advanced functions) 91

Figure 42. CPU utilization and total storage used for four partitioned servers

6.1.2 Efficiency of partitioningApart from the benefits of partitioning already discussed, there is also a potential performance benefit. Domino is still a relative newcomer to very large systems. Lotus and IBM continually look for improvements that can be made to enhance the performance of Domino servers, and design changes are made accordingly.

Tests on Lotus Domino for S/390, run with Domino Release 5 and many thousands of users, have shown that you need to run multiple Domino partitions (DPARs) to fully utilize the larger S/390 servers. Therefore, when you plan to support a large number of users, we recommend you consider running multiple Domino servers. There are two ways to do this in a single S/390 system: Domino partitioning (DPAR) and logical partitioning (LPAR).

As you see in Table 15, the CPU cost of more than two DPARs is high. Therefore, if you need to run a third server, we recommend you run it in a new LPAR, rather than adding a third DPAR to an LPAR. This recommendation is provided as a general guideline only.

Domino Release 5 can support more users than previous releases. For more information about the capacity of one server, see 8.4, “The use of Domino partitioned servers” on page 122.

2 DPARs 21 1732 1886 960 2846

3 DPARs 38 2660 1887 2226 4113

4 DPARs 61 3520 1887 3266 5153

CPU utilization (%)

Shared storage used (MB)





6.2 Clustering

A cluster is a group of Domino servers that work together. The servers pass database updates among themselves, using real-time replication, so that the data on the various servers is kept in synchronization. Up to six Domino servers can be in a single cluster, and they can be running on different platforms.

The benefits of clustering are:

• Higher availability. If one server fails, users can automatically be switched to another server in the cluster.

• Workload balancing across multiple servers.

• Scalability, by allowing multiple servers to act as one

See Lotus Domino for S/390 Release 5: Installation, Customization and Administration, SG24-2083 (edition 02) for a longer discussion of clustering. Now we look at the performance aspects of clustering.

6.2.1 Cluster componentsThe following additional tasks run on each server in a cluster:

• The Cluster Administration process (Cladmin) performs many of the housekeeping tasks associated with a cluster. It runs temporarily.

• The Cluster Database Directory Manager task (Cldbdir) on each server creates the Cluster Database Directory and keeps it up-to-date with the most current database information. You can get status information with the show

cluster command.

• The Cluster Replicator (Clrepl) task constantly synchronizes data among replicas in a cluster. Whenever a change occurs to a database in the cluster, the Cluster Replicator immediately pushes the change to the other replicas in the cluster.

• The Internet Cluster Manager task manages failover and load balancing of Web client requests for data stored on clustered servers.

You will also see additional databases created:

• CLUSTA4.NSF is the Cluster Analysis database that contains analysis data on the configuration of the cluster. You can use Cluster Analysis to verify that a cluster is set up correctly. Cluster Analysis creates reports about the cluster configuration.

• CLDBDIR.NSF is the Cluster Database Directory that contains a document about each database and replica in the cluster. This document contains information, such as the database name, server, path, replica ID, and other replication and access information.

When a cluster is first defined on a server, the two cluster tasks Cldbdir and Clrepl are started. They are also added to the ServerTasks parameter in notes.ini so that they are started automatically when the server is started in the future.

You can also start additional cluster replicator tasks. The Domino 5 Administration Help database recommends that on each server in a cluster you should run one fewer cluster replicator tasks than there are servers in the cluster.


In this way there is one cluster replicator task to handle replication to each other server in the cluster.

6.2.2 Parameters that affect clusteringThe Server_Restricted parameter in notes.ini affects clustering. See the Domino 5 Administration Help database for more details.

6.2.3 Costs of clusteringRunning clustered servers uses increased resources in the following areas:

• CPU resources are used to send database updates to other servers and apply them to all copies of the database. A cluster with one primary and one secondary server takes 40 to 80% more CPU than a non-clustered configuration.

• Storage (memory) is 2.5 to 3.5 as much as a non-clustered configuration, depending on the number of DPARs and LPARs used.

• DPARs: four times as many will be needed if you use a primary and backup pair. The maximum recommended number of concurrently active users for a primary cluster DPAR is 1000 vs. 2000 for a non-clustered DPAR. Add to this a backup DPAR for each primary DPAR.

However, only 2.5 times as many DPARs will be needed if you put 800 active users in each DPAR and back it up with three other primary servers with 200 users each.

• LPARs: two to four times as many will be needed, depending on how many DPARs you put in each LPAR.

• I/O activity doubles since updates are done multiple times, once on each server.

• Disk space doubles if you have two copies of each database. Each server must have its own set of files and databases. It triples if you elect to have three copies of each database. There is no sharing between servers.

• Network traffic increases to pass the updates to all servers in the cluster. Lotus recommends providing a separate network for the server-to-server traffic in a cluster, to ensure good performance.

For each database, you can decide which servers in the cluster contain a replica, from only one server to all servers. Make that decision carefully for each database, in order to limit the number of updates being passed through the cluster and therefore minimize the resource use.

Depending on your configuration, the cost of clustering can be two or more times that of a non-clustered configuration. Compare this to the high availability of the S/390 and the fast (5-minute) restart capability provided with transaction logging to determine if this added cost is justified. If so, consider clustering only those users that need higher availability.

6.2.4 Monitoring clusteringYou can get statistics on clustering from the log database log.nsf and from the output of the Domino server. The following commands can be used to obtain clustering statistics:


•show tasks

•show statistic replica

•show statistic <servername>

To look for replication backlogs, issue the server command show statistic

replica. Look at the Replica.Cluster.WorkQueueDepth value, which shows the current number of modified databases waiting for replication. If this value is consistently greater than 10, you should consider running additional cluster replicator tasks.


Chapter 7. Domino monitoring

Lotus Domino for S/390 provides a set of tools for monitoring the server. Some of the tools run automatically. Others have to be set up, or started using a server command or started task. We discuss the various sources of monitoring information in this chapter, including the following tools:

• SMF type 108 record

• Notes log

• Domino statistics

Note: Do not collect information that you do not need. Data collection takes system resources.

See the Domino 5 Administration Help database for more details about monitoring a Domino server.

7.1 SMF type 108 record

The SMF type 108 record is a new feature for Lotus Domino for S/390 Release 5. It provides data about a Domino server running on an OS/390. It is a powerful tool for performance monitoring, capacity planning, and problem diagnosis.

To enable this feature:

• Include SYS(TYPE(108)) in the SMFPRMxx PARMLIB member.

• OS/390 Security Server (RACF) users must define the user ID of the server to have at least READ access to the BPX.SMF FACILITY class.

You can get the record format from the Lotus Domino for S/390 Web page at:

http://www.s390.ibm.com/products/domino

From the “Technical information” link, you can get the following documents about the SMF type 108 record. Make sure that you select the document corresponding to your Domino release:

• SMF Type 108 Support for Lotus Domino for S/390

Note: This documentation will be included in future revisions of the MVS System Management Facilities, GC28-1783.

• ASM Record Mapping Macro


7.1.1 Record mappingTable 16 shows the record mapping of SMF type 108 records. There are two sections: a common section which appears on all subtypes, and a unique section for each subtype.

Table 16. Record mapping of SMF type 108 records

Note: Make sure that the document corresponds to your Domino release level because the record mapping varies. Release 5.0 provided subtype 1. Release 5.02 added some fields to the subtype 1. A transaction type table was added in Release 5.01. Release 5.02 added fields to the subtype 1 Product Section and also added subtype 3 tables. Release 5.03 provided subtype 2 and subtype 6.

Section Section name

Description

Common Sections

Header Section

The common SMF record headers and triplet fields (offset/length/number) that locate the other sections on the record

Product Section

The general information about the server and the system that it is running on

Subtype 1 (Server Load)

Self-Defining Section

The triplet fields (offset/length/number) that locate the specific sections for this subtype on the record

Server Load Section

The counters showing activity at the server level. This is generated at the expiration of the SMF Global Interval (combination of INTVAL and SYNCVAL parameters in the SMFPRMxx PARMLIB member)

Transaction Section

The data being reported for each transaction (by type) that is requested from the server. Only transactions with non-zero activity are included. The transaction types shown in this section are Domino internal transactions. One user action, such as opening a database, performs one or more Domino transactions

Port Activity Section

The data being reported for each TCP/IP port to which the server is connected

Subtype 2 (User Activity)



User Activity Data Section

The data for users by IP address and connection type

Subtype 3 (Monitoring and Tuning)



Monitoring and Tuning Data Section

Statistics and configuration parameters for tuning the Domino server

Subtype 6 Data Base Activity



Data Base Activity Data Section

The data for Domino Data Base activity


7.1.1.1 Common SectionsThe following sections appear on each of the Type 108 subtype records and are included in the documentation once.

Header SectionThis section contains the common SMF record headers fields and the triplet fields (offset/length/number) that locate the other sections on the record.

Table 17. Fields of Header Section

Offsets Name Length Format Description

0 SMF108LENSMF108SLDMSENTRAS

2 binary Record length. This field and the next field (total of four bytes) form the RDW (record descriptor word).

2 SMF108SEG 2 binary Segment descriptor (see record length field).

4 SMF108FLG 1 binary System indicator:

Bit Meaning When Set0 Reserved1 Subtypes used2 Reserved3-6 Version indicators*7 Reserved.

5 SMF108RTY 1 binary Record type 108 (X'6C').

6 SMF108TME 4 binary Time since midnight, in hundredths of a second, that the record was moved into the SMF buffer.

10 SMF108DTE 4 packed Date when the record was moved into the SMF buffer, in the form 0cyydddF.

14 SMF108SID 4 EBCDIC System identification (from the SID parameter).

18 SMF108SSI 4 EBCDIC Subsystem identification.

22 SMF108STP 2 binary Record Subtype

SubType Description1 Server Load2 User Activity3 Monitoring and Tuning6 Data Base Activity

24 SMF108PRO 4 binary Offset to Product Section

28 SMF108PRL 2 binary Length of Product Section

30 SMF108PRN 2 binary Number of Product Sections (should be '1')

32 SMF108SSO 4 binary Offset to Self-Defining Section

36 SMF108SSL 2 binary Length of Self-Defining Section

38 SMF108SSN 2 binary Number of Self-Defining Sections (should be '1')

Chapter 7. Domino monitoring 99

Product SectionThis section contains the general information about the server and the system that it is running on.

Table 18. Fields of Product Section

7.1.1.2 Subtype 1 - Server LoadThis subtype contains counts of activity done by the server running on the OS/390 system.

Self-Defining SectionThis section contains the triplet fields (offset/length/number) that locate the specific sections for this subtype on the record.

Table 19. Fields of Product Section


0 SMF108PRRVN 4 binary Record Version Number (starting with '1' for release 5.0a) (Set to 2 for release 5.01) (Set to 3 for release 5.02). (Set to 4 for release 5.03).

4 SMF108PRPVN 8 EBCDIC Product Version ('5.0' for example). This is the first eight bytes of the Product Version string and may contain text or other characters after the number.

12 SMF108PRSVN 32 EBCDIC Server Name (used to identify partitioned servers)

44 SMF108PRSPN 8 EBCDIC Sysplex Name (ECVTSPLX field in cvt/ecvt)

52 SMF108PRSYN 8 EBCDIC System Name (CVTSYSN field in cvt/ecvt)

60 SMF108PROSL 8 EBCDIC OS/390 System Level (CVTPRODN field in cvt/ecvt)

68 SMF108PRISTARTT 8 binary STCK format

Interval Start Time

76 SMF108PRIENDT 8 binary STCK format

Interval End Time

84 SMF108CVTTV 4 binary CVTTV GMT offset time.


0 SMF108SLO 4 binary Offset to Server Load Section

4 SMF108SLL 2 binary Length of Server Load Section

6 SMF108SLN 2 binary Number of Server Load Sections (should be '1')

8 SMF108TRO 4 binary Offset to Transaction Section

12 SMF108TRL 2 binary Length of Transaction Section

14 SMF108TRN 2 binary Number of Transaction Sections (1 per transaction type processed)

16 SMF108PTO 4 binary Offset to Port Activity Section

20 SMF108PTL 2 binary Length of Port Activity Section

22 SMF108PTN 2 binary Number of Port Activity Sections (1 per TCP/IP port)


Server Load SectionThis section contains the counters showing activity at the server level (globally).

Table 20. Fields of ServerLoad Section


0 SMF108SLCU 4 binary Current number of users

4 SMF108SLUA 4 binary Number of currently connected users that are currently active

8 SMF108SLUA1M 4 binary Number of currently connected users that have been active within the last 1 minute

12 SMF108SLUA3M 4 binary Number of currently connected users that have been active within the last 3 minutes




28 SMF108SLDMSENTL 4 binary Number of Domino mail messages delivered to local users

32 SMF108SLDMSENTLAS 4 binary Average size of Domino mail and SMTP messages delivered to local users

36 SMF108SLDMSENTR 4 binary Number of Domino mail and SMTP messages sent to other servers

40 SMF108SLDMSENTRAS 4 binary Average size of Domino mail messages sent to other servers

44 SMF108SLSMREC 4 binary Number of SMTP messages received from other servers during interval

48 SMF108SLSMRECAS 4 binary Average size of SMTP messages received from other servers during interval

52 SMF108SLSMSENT 4 binary Number of SMTP messages sent to other servers during interval

56 SMF108SLSMSENTAS 4 binary Average size of SMTP messages sent to other servers during interval

60 SMF108SLTRANS 4 binary Total number of transactions processed during interval [Note 1]

64 SMF108SLSVREPL 4 binary Number of replications initiated by this server

68 SMF108SLNWSESIN 4 binary Number of incoming (to the server from clients) sessions established during the interval [Note 4]

72 SMF108SLNWSESOUT 4 binary Number of outgoing sessions established during the interval [Note 4]

76 SMF108SLNWBR 4 binary Number of network bytes/1024 received during interval [Note 4]


Table 20 notes:

1. This “transaction” means Domino internal transactions. One user action, such as opening a database, performs one or more Domino internal transactions. The transaction section in SMF type108 subtype1 provides additional information for Domino transactions, such as the number of each transaction processed during the interval and the response time of each.

2. This corresponds to the value defined in SERVER_POOL_TASKS parameter in notes.ini.

3. These fields contains no data when you use Domino for the IBM HTTP Server Connector.

4. For Release 5.01 or higher, these fields will be set to zero and recorded in the Port Activity Section.

Transaction SectionThis section contains the data being reported for each transaction (by type) that is requested of the server. Only transactions with no-zero activity counts are included.

Table 21. Fields of Transaction Section

80 SMF108SLNWBS 4 binary Number of network bytes/1024 sent during interval [Note 4]

84 SMF108SLTT 4 binary Total number of physical thread pool threads [Note2]

86 SMF108SLVTIU 2 binary Number of virtual thread pool threads currently in use

88 SMF108SLAIOR 2 binary Number of async I/O reads during interval

92 SMF108SLAIOW 4 binary Number of async I/O writes during interval

96 SMF108SLPOP3R 4 binary Number of POP3 reads during interval

100 SMF108SLIMAPR 4 binary Number of IMAP reads during interval

104 SMF108SLHTTPR 4 binary Number of HTTP reads during interval [Note 3]

108 SMF108SLHTTPW 4 binary Number of HTTP writes during interval [Note 3]

112 SMF108SLVTIUMAX 2 binary Maximum number of virtual thread pool threads in use during interval

114 SMF108SLTASKS 2 binary Number of tasks currently in use

116 SMF108SLTASKSMAX 2 binary Maximum number of tasks in use during interval

118 SMF108SLPTIU 2 binary Number of physical thread pool threads currently in use

120 SMF108SLPTIUMAX 2 binary Maximum number of physical thread pool threads in use during interval


0 SMF108TRTYPE 4 binary Transaction type

4 SMF108TRTYPENP 4 binary Number of transactions of type processed during interval



Port Activity SectionThis section contains the data being reported for each TCP/IP port to which the server has a connection.

Table 22. Fields of Port Activity Section

7.1.1.3 Subtype 2 - User ActivityThis subtype will report Domino User activity for the different protocols Domino supports.


Table 23. Fields of Self-Defining Section

User Activity Data SectionThis section contains the data for users by IP address and connection type.

Table 24. Fields of User Activity Data Section

8 SMF108TRTYPETA 4 binary Total accumulated response time, in milliseconds, for all transactions of type that completed during interval

12 SMF108TRTYPENW 4 binary Total accumulated net wait time, in milliseconds, for all transactions of type that completed during interval. This is the time the server has been waiting for clients to respond.


0 SMF108PTNAME 8 EBCDIC The first eight bytes of the TCP/IP port. ('TCPIP' for example)

8 SMF108PTNWSESIN 4 binary Number of incoming sessions processed during the interval (client to server connection)

12 SMF108PTNWSESOUT 4 binary Number of outgoing sessions processed during the interval

16 SMF108PTNWBR 4 binary Total number of bytes/1024 received for this port during the interval

20 SMF108PTNWBS 4 binary Total number of bytes/1024 sent for this port during the interval


0 SMF108UDO 4 binary Offset to data section

4 SMF108UDL 2 binary Length of data section

6 SMF108UDN 2 binary Number of data sections


0 SMF108UIPA 16 EBCDIC IP address presenting the request for service



7.1.1.4 Subtype 3 - Monitoring and TuningThis subtype will monitor some statistics and certain configuration parameters used by the server.



Monitoring and Tuning Data SectionThis section contains some statistics and certain configuration parameters for tuning the Domino server.

Table 26. Fields of Monitoring and Tuning Data Section

16 SMF108UTYPE 4 EBCDIC Type of connection to the Domino server

User/Type/DescriptionNRPC/Domino/mail and database serverHTTP/Domino/http serverIMAP/IMAP/mail serverPOP3/POP3/mail serverSMTP/SMTP/server

20 SMF108UNAME 32 EBCDIC Notes user name for NRPC clients

52 SMF108UCPU 8 binary STCK format

CPU time used by this user

60 SMF108UBR 4 binary number of bytes read this interval

64 SMF108UBW 4 binary number of bytes written this interval


0 SMF108MTO 4 binary Offset to Data Section

4 SMF108MTL 2 binary Length of Data Sections

6 SMF108MTN 2 binary Number of Data Sections (should be ‘1’)


0 SMF108MTMAXUSERS 4 binary Maximum number of users

4 SMF108MTMAXCONTR 4 binary Limit for number of concurrent transactions

8 SMF108MTMAXCONSES 4 binary Maximum number of sessions to run concurrently

12 SMF108MTSESTIMEOUT 2 binary Number of minutes in timeout

14 SMF108MTUPMAX 2 binary Maximum number of concurrent update tasks

16 SMF108MTMAILBOXES 2 binary Maximum number of mail.boxes

18 SMF108MTREPMAX 2 binary Maximum number of replicators (concurrent)

20 SMF108MTNSFPOOL 4 binary Maximum size of nsf buffer pool (bytes/4096).

24 SMF108MTNSFPOOLIU 4 binary Number of bytes in nsf buffer pool (in use)



7.1.1.5 Subtype 6 - Data Base ActivityThis subtype will report Domino specific data for Domino data bases.



28 SMF108MTDBCENAB 4 binary dbcache enabled = 1, 0 if disabled

29 RESERVECHAR 1 N/A Reserved for alignment

32 SMF108MTDBCMAXE 3 binary Maximum number of dbcache entries

36 SMF108MTDBCCE 4 binary Number of dbcache (current entries)

40 SMF108MTDBCIDBO 4 binary Number of dbcache (initial db opens)

44 SMF108MTDBCOCR 4 binary Number of dbcache (overcrowding rejections)

48 SMF108MTDBCHITS 4 binary Number of dbcache (hits)

52 SMF108MTDBCHWM 4 binary dbcache (high water mark)

56 SMF108MTSATH 2 binary Server availability threshold

58 SMF108MTSAX 2 binary Server availability index

60 SMF108MTNIFS 2 binary Database.NIFPool.Size (in bytes)

64 SMF108MTNIFN 4 binary Database.NIFPool.Used

68 SMF108MTNSFS 4 binary Database.NSFPool.Size (in bytes)

72 SMF108MTNSFN 4 binary Database.NSFPool.Used

76 SMF108MTDBPR 4 binary Number of Database.BufferPool Reads (no longer set by the server).

80 SMF108MTDBPW 4 binary Number of Database.BufferPool Writes (no longer set by the server).

84 SMF108MTMMXFER 4 binary Maximum number of mail transfer threads

86 SMF108MTMMXDLV 2 binary Maximum number of mail delivery threads

88 SMF108MTMMXCONXFR 2 binary Maximum number of concurrent mail transfer threads


0 SMF108DBO 4 binary Offset to Data Section

4 SMF108DBL 2 binary Length of Data Sections

6 SMF108DBN 2 binary Number of Data Sections



Data Base Activity Data SectionThis section contains the data for Domino Data Base activity.


7.1.2 RMF Lotus Domino server reportIn OS/390 V2R10, RMF has enhanced the Postprocessor. It now accepts SMF record type 108 subtypes 1 and 3 written by Lotus Domino R5. It also builds reports that provide feedback on server load, as well as the number and type of messages that the server handles.

The Lotus Domino server report provides information about the activities of a server. The information can be used to analyze the activities of the server in case of problems.

The report consists of two parts:

• Lotus Domino server summary

The summary contains one line for each server which is part of the report.

• Lotus Domino server details

This part consists of the following sections:

• Definition data (provided by record type 108-3)

• Performance data (provided by record type 108-3)

• Load data (provided by record type 108-1)

Figure 43 on page 107 is sample JCL to generate a Lotus Domino server report.


0 SMF108DBNAME 64 EBCDIC Last 64 characters of the Data Base Name.

64 SMF108DBINDEX 4 binary Number of indexing operations started on this data base by the server.

68 SMF108DBREPS 4 binary Number of replications on this data base initiated by this server.

72 SMF108DBDOCADDS 4 binary Number of documents added to this data base.

76 SMF108DBDOCDELS 4 binary Number of documents deleted from this data base.


Figure 43. Sample JCL for RMF Lotus Domino server report

7.1.2.1 Lotus Domino server summaryFigure 44 is an example of a Lotus Domino server summary report.

Figure 44. Lotus Domino server summary report

Table 29. Fields in the Domino server summary report


SERVER NAME Name of the Domino server

AVAILABLE Total time (hhh.mm.ss) the server was available during the interval

USERS CONNECTED Average number of currently connected users

USERS ACTIVE Average number of currently active users

TASKS Average number of tasks currently in use

TRANSACTION RATE Rate of all transactions processed during the interval

ASYNC I/O RATE - READS Rate of asynchronous reads

ASYNC I/O RATE - WRITES Rate of asynchronous writes

MAIL RATE - DELIVERED Rate of Domino mail messages delivered to local users

MAIL RATE - SENT Rate of Domino mail messages sent to other servers

//xxxxxxRA JOB 'xxxxxx,?','xxxxxx',CLASS=J,REGION=0M,// MSGCLASS=H,MSGLEVEL=(1,1),NOTIFY=xxxxxx//RMFSORT EXEC PGM=SORT//SORTIN DD DISP=SHR,DSN=xxxxxx.xxxxxx.SMF//SORTOUT DD DISP=(NEW,PASS),UNIT=SYSDA,SPACE=(TRK,(200,200))//SORTWK01 DD DISP=(NEW,DELETE),UNIT=SYSDA,SPACE=(CYL,(10))//SORTWK02 DD DISP=(NEW,DELETE),UNIT=SYSDA,SPACE=(CYL,(10))//SORTWK03 DD DISP=(NEW,DELETE),UNIT=SYSDA,SPACE=(CYL,(10))//SYSPRINT DD SYSOUT=*//SYSOUT DD SYSOUT=*//SYSIN DD *SORT FIELDS=(11,4,CH,A,7,4,CH,A),EQUALSMODS E15=(ERBPPE15,500,,N),E35=(ERBPPE35,500,,N)

//RMFPP EXEC PGM=ERBRMFPP//MFPINPUT DD DISP=(OLD,DELETE),DSN=*.RMFSORT.SORTOUT//MFPMSGDS DD SYSOUT=*//SYSOUT DD SYSOUT=*//SYSIN DD *SYSOUT(H)DINTV(0055)STOD(1425,1520)RTOD(1425,1520)REPORTS(DOMINO)SYSRPTS(WLMGL(POLICY,SCLASS,SCPER,RCLASS))/*

L O T U S D O M I N O S E R V E R S U M M A R Y

PAGE 1

OS/390 SYSTEM ID ML96 START 12/27/2000-12.45.00 INTERVAL 005.30.00

RPT VERSION 02.10.00 END 12/27/2000-18.15.00

SERVER NAME AVAILABLE ---- USERS ----- TASKS TRANSACTION ASYNC I/O RATE MAIL RATE SMTP RATE

HHH.MM.SS CONNECTED ACTIVE RATE READS WRITES DELIVERED SENT READS WRITES

D01MLC96/01/M/IBM 005.30.00 801 10 946 13.54 37.55 22.23 0.29 0.06 0.04 0.01D01MLC83/01/M/IBM 005.30.00 103 1 221 0.94 2.65 1.46 0.03 0.00 0.01 0.00


7.1.2.2 Lotus Domino server detailsFigure 45 is an example of a Lotus Domino server details report.

Figure 45. Lotus Domino server details report

Table 30. Fields in the Lotus Domino server details report

SMTP RATE - READS Rate of SMTP messages received from other servers

SMTP RATE - WRITES Rate of SMTP messages sent to other servers


NAME Name of Domino server

User Activity

MAX Maximum number of users that are allowed to access the server. The value 0 means that there is no limit.

CONNECTED Number of current users (connections)

ACTIVE Number of active users

WITHIN n MIN Number of currently connected users that have been active within the last 1, 3, 5, 15, and 30 minutes

Tasks


L O T U S D O M I N O S E R V E R D E T A I L SPAGE 2

OS/390 SYSTEM ID ML96 START 12/27/2000-12.45.00 INTERVAL 005.30.00RPT VERSION 02.10.00 END 12/27/2000-18.15.00

NAME: D01MLC96/01/M/IBM--- USER ACTIVITY --- ------ TASKS ------ ------------- MESSAGES ------------- --- ACCESS RATES --- --- DATABASE CACHE ---MAX 0 MAX 1169 MAILBOXES 3 AS I/O READ 102.9 STATUS OKCONNECTED 812 CURRENT 956 COUNT RATE AVG SIZE AS I/O WRITE 60.44 MAX ENTRIES 1152ACTIVE 10 MAX UPDATES 0 MAIL DELIVERED 12742 0.79 32 POP3 READ 0.00 CURRENT ENTRIES 1098WITHIN 1 MIN 78 MAX REPLICS 0 MAIL SENT 2691 0.17 41 IMAP READ 0.00 HIGH WATER MARK 1530WITHIN 3 MIN 142 COUNT REPLICS 0 SMTP RECEIVED 1740 0.11 16 HTTP READ 0.00 INITIAL DB OPENS 3661WITHIN 5 MIN 195 SMTP SENT 658 0.04 62 HTTP WRITE 0.00 REJECTIONS 0WITHIN 15 MIN 380 WITHIN 30 MIN 1056WITHIN 30 MIN 561

- VIRTUAL THREADS - - PHYSICAL THREADS - --- AVAILABILITY --- -- NSF BUFFER POOL ---MAX 1025 MAX 15 TRANSACTION 0 MAX 98304CURRENT 812 CURRENT 1 INDEX 90 CURRENT 162889K

TOTAL 2250---------------------------------------------------------------------------------------------------------------------------------

TRANSACTION ACTIVITY---------------------------------------------------------------------------------------------------------------------------------MAXIMUM NUMBER OF CONCURRENT SESSIONS: NO LIMIT----------------------- TOP-10 BY COUNT ----------------------- --------------------- TOP-10 BY R/T TOTAL ---------------------TYPE NAME COUNT %TOTAL RATE R/T R/T TYPE NAME COUNT %TOTAL RATE R/T R/T

AVG TOTAL AVG TOTALTOTAL 596979 100.0 36.85 6192 3696533K TOTAL 596979 100.0 36.85 6192 3696533K

6 OPEN_NOTE_RQST 76279 12.78 4.71 5079 387409K 142 START_SERVER_RQST 38483 6.45 2.38 49990 1923759K1 OPEN_DB_RQST 75001 12.56 4.63 2320 174023K 6 OPEN_NOTE_RQST 76279 12.78 4.71 5079 387409K48 CLOSE_DB_RQST_ALT 68962 11.55 4.26 3.14 216509 8 UPDATE_NOTE_RQST_ALT 24388 4.09 1.51 11006 268420K142 START_SERVER_RQST 38483 6.45 2.38 49990 1923759K 55 READ_ENTRIES_RQST 37930 6.35 2.34 5234 198538K55 READ_ENTRIES_RQST 37930 6.35 2.34 5234 198538K 34 READ_OBJECT_RQST 17054 2.86 1.05 11264 192094K8 UPDATE_NOTE_RQST_ALT 24388 4.09 1.51 11006 268420K 35 WRITE_OBJECT_RQST 8659 1.45 0.53 21636 187346K75 NAME_LOOKUP_RQST 20784 3.48 1.28 125.3 2603674 1 OPEN_DB_RQST 75001 12.56 4.63 2320 174023K112 GET_SERVER_STATS_LITE_ 20039 3.36 1.24 36.53 732114 59 NIFOPENNOTE_RQST 4761 0.80 0.29 18349 87359648122 SERVER_AVAILABLE_LITE_ 19959 3.34 1.23 0.49 9704 53 UPDATE_COLLECTION_RQST 6921 1.16 0.43 12550 8685848051 OPEN_COLLECTION_RQST 18652 3.12 1.15 163.6 3050857 9 DELETE_NOTE_RQST 2597 0.44 0.16 33319 86530704

---------------------------------------------------------------PORT ACTIVITY

---------------------------------------------------------------MAXIMUM NUMBER OF CONCURRENT SESSIONS: 65535SESSION TIMEOUT: 38320NAME -- SESSIONS IN -- -- SESSIONS OUT -- SENT RECEIVED

COUNT RATE COUNT RATE (KB) (KB)TCPIP0A 41922 2.59 2646 0.16 2920215 497776


MAX Maximum number of tasks in use

CURRENT Number of tasks currently in use

MAX UPDATES Maximum number of concurrent update tasks

MAX REPLICS Maximum number of concurrent replicator tasks

COUNT REPLICS Number of replications initiated by this server

Messages

MAILBOXES Number of mail boxes

MAIL DELIVERED The number, rate and average size of Domino mail messages delivered to local users

MAIL SENT Domino mail messages sent to other servers

SMTP RECEIVED SMTP messages received from other servers

SMTP SENT SMTP messages sent to other servers

Access Rates

AS I/O READ Rate of asynchronous I/O reads

AS I/O WRITE Rate of asynchronous I/O writes

POP3 READ Rate of POP3 reads

IMAP READ Rate of IMAP reads

DOMINO READ Rate of Domino reads

DOMINO WRITE Rate of Domino writes

Database Cache

STATUS Status of the database cache: either OK or ? (=undefined)

MAX ENTRIES Maximum number of database entries allowed in cache at any one time

CURRENT ENTRIES Number of current entries

HIGH WATER MARK High water mark

INITIAL DB OPENS Number of initial database opens

REJECTIONS Number of overcrowding rejections

HITS Hits in database cache

Virtual Threads [Note 1]

MAX Maximum number of virtual thread pool threads

CURRENT Number of virtual thread pool threads currently in use

Physical Threads

MAX Maximum number of physical thread pool threads in use



Table 30 notes:

1. Lotus Domino for S/390 Release 5 exploits a new thread mechanism called the thread pool model. When a user connects to the server, a client session is created with a unique socket descriptor and a virtual thread ID is assigned.

CURRENT [Note 3] Number of physical thread pool threads currently in use

TOTAL [Note 2] Total number of physical thread pool threads

Availability

THRESHOLD Server availability threshold

INDEX Server availability index

NSF Buffer Pool

MAX Maximum size (in bytes) of the NSF (Notes Storage Facility) buffer pool

CURRENT Number of bytes of the NSF buffer pool currently in use

Transaction Activity [Note 4]

MAXIMAL CONCURRENT Limit for number of concurrent transactions on a server

Top-10 List of Transaction Types—Sorted by COUNT and by R/T TOTAL.

TYPE Transaction type

NAME Transaction name

COUNT Number of transactions processed during interval

%TOTAL %Percentage based on all transactions

RATE Rate of processed transactions

R/T AVG Average response time (milliseconds)

R/T TOTAL Total response time (milliseconds) of all transactions that completed during the interval

Port Activity

MAX CONCURRENT SESSIONS

Maximum number of sessions that can run concurrently on the server

SESSION TIMEOUT Time limit (minutes) after which idle connections are terminated

NAME Port name

SESSIONS IN Count and rate of incoming sessions (from clients to theserver) established during the interval

SESSIONS OUT Count and rate of outgoing sessions established during the interval

SENT (KB) Number of K bytes sent to the network

RECEIVED (KB) Number of K bytes received from the network



When a client performs I/Os, a physical thread is associated with the virtual thread and does the associated work on behalf of the client session.

For more information of the thread pool model, see 2.2.1.1, “Address spaces for user connections” on page 7.

2. This value corresponds to the value specified in SERVER_POOL_TASKS parameter in notes.ini.

3. If this value often reaches the definition value (SERVER_POOL_TASKS=125, in this sample), we recommend you increase the SERVER_POOL_TASKS value.

4. This “transaction” refers to Domino transactions. One user action, such as opening a database, performs one or more Domino transactions.

The transaction rate should be an indication of the use of the server. The transaction rate allows you to check whether the load across several servers is balanced.

7.2 Notes log

Every Domino server records information about server activities in the log database (log.nsf). The information includes:

• Database usage by user

• Database size

• Mail routing, replication and other events

• Usage of the system by user, including:

• User name

• Connect time

• Number of documents read and written, by database

• Amount of data transferred across the network

• Number of transactions run

You can choose to collect replication and client session event records when you initially set up the server. You can change these settings later with theLog_Replication and Log_Sessions parameters in notes.ini.

7.3 Statistics and events log

The statistics and events database events4.nsf is used to configure Notes server event handling, statistic monitoring, ACL monitoring, and replication monitoring. The database contains the names of all statistics monitored by the server, and thresholds for producing event records. It also contains error and status messages from the server.

This database can be used to:

• Configure Notes server event handling

• Look up information about a specific statistic or event message

• Set statistic thresholds


• Assign types and severities to server events

You run events by running the server task Event.

7.4 Statistics and reporting database

If you run the reporter task on the server, a set of reports will be created at scheduled intervals. These are put in the statistics and reporting database statrep.nsf. You can specify the interval between records. For information on setting up the reporter task, see the Domino 5 Administration Help database.

7.4.1 Reporting periodBe careful about the reporting period for the various statistics. There are several types of statistics in the interval record:

• A sample at the end of the interval

• Accumulation for the duration of the interval

• Maximum value since the server started

Unfortunately, there are not many values accumulated during the interval.

7.4.2 Domino statisticsStatistics that you can use for performance monitoring are:

Server.Trans.PerMinute.PeakThis is the peak number of transactions during a one-minute interval since the server started. It is not the peak during this interval. You are given the date and time this occurred.

Server.UsersThis is the number of users with connections to the server at the end of the interval when statistics were collected.

There are also counters for the number of calendar and mail requests, and also agent processes. These may be of use if you are investigating the average profile of your mail users.

7.4.3 Coordinating SMF, RMF and Domino data collectionIf you wish to correlate the Domino statistics with the SMF and RMF data, you should try to coincide the start of Domino statistics collection with the start of the SMF and RMF intervals. SMF and RMF have controls to synchronize the time of interval collection to the time of day, for example at zero and 30 minutes after each hour. Domino has no such control. It records statistics every so many minutes after the statistics task is started. Therefore, to get Domino to record its records at the same time as SMF and RMF, you must start the statistics task at the desired time.

7.4.4 Statistics commandsYou can also use the load stats <servername> command to create statistics on demand for a remote server and the show statistics server command to display the complete list of statistics for the server.


A sample of a show statistics command is shown in Appendix D, “Sample output of the show statistics command” on page 137.

7.5 Processing Domino statistics

To process Domino statistics, you can:

• View them in the database.

• Use a tool to analyze the data and produce reports.

• Export the data into another format for further processing.

7.5.1 Exporting the contents of a databaseIf you wish to combine Domino statistics with SMF data, for example, you will probably need to extract the Domino statistics into a sequential OS/390 file. You can then use currently available tools to analyze the data.

To export the contents of a database into a sequential file on your workstation:

1. Open the database on the server.

2. Select the view that you want.

3. Select the documents that you want to extract based on the collection time.

4. Select File/Export.

5. Set the file name to a file on your workstation.

6. Save as type Structured Text.

7. Click the Export Button.

8. Choose:

• Selected documents

• Char code 12 as the interdocument delimiter

• Wrap words at 75 characters per line

9. Then click OK.

The file will be written on your workstation. You can then upload it to OS/390 and input it into your analysis program.


Chapter 8. Capacity planning

This chapter provides capacity planning guidelines in the areas of:

• Processor capacity

• Processor storage (memory)

• The use of Domino partitioned servers (DPARs)

• The use of S/390 logical partitioning (LPARs)

• DASD

• Space

• I/O rate

• Network capacity

We also provide information on a large IBM production Domino server on S/390, and show how you can get assistance with your capacity planning.

8.1 Recommendations

These are our recommendations for capacity planning:

• Estimate carefully.

Initial estimates of the resources required to support a Domino server are based on estimates of:

• Number of registered users

• Number of connected users

• Transaction rate generated by the users

• Average CPU time used per transaction

• Amount of data you will have, including the amount of data an average user will have in their mail file

The more accurately you estimate these values, the more accurate your estimate of resource needs will be.

• Make use of the capacity planning assistance available.

See 8.10, “Capacity planning assistance” on page 126.

• Base your capacity planning on your own measurements as soon as possible.

Every Domino environment is different. Therefore, you should monitor your system as soon as you have a significant Domino workload, and then revise your plans based on that. If you have other Domino servers, you can collect information on your workload profile from them and use that to help validate your assumptions.

• If you will have more than 5,000 Notes client or POP3 registered users, 700 IMAP registered users, or 1,500 WebMail client registered users, consider using Domino server partitioning.

See 8.4, “The use of Domino partitioned servers” on page 122.

• Remember that these guidelines may change.


These guidelines were put together in November 1999 for Domino Release 5.0, OS/390 Version 2 Release 6, and 9672 generation 6 processors and modified November 2000 to reflect the changes that have taken place during that time. We have seen a significant increase in capacity with Release 5 of Domino for S/390, and with the reduction in the number of DPARs made possible with R5.

• Beware of running your own performance tests and misinterpreting the results.

It is possible to set up some seemingly simple performance tests. However, we have seen very misleading results when trying to use the run time of a single Domino function to estimate the performance of a production workload. Unless you understand exactly which system resources the function uses, and can accurately relate that to the resources your production workload will use, you can get erroneous results. In addition, unless you understand how Domino performs the functions, it is possible that you are measuring something quite different from what you think. You could, for example, be measuring network performance instead of server performance.

Realistic tests are, unfortunately, complex and expensive to run. Use the information in this document (and the other information sources listed) to get capacity estimates for production workloads.

8.2 Estimating processor capacity

The processor capacity required for Domino depends on:

• The number of users

• The number of server transactions generated by the users

• The CPU time used per transaction

8.2.1 Calculating the processor capacity neededBased on the number of registered users, transaction rate per user, and CPU time used per transaction (all discussed in this section), you can calculate the processor capacity needed. It is the number of registered users times the transaction rate per user per hour times the CPU time used per transaction. This is the number of CPU seconds per hour needed on a specific processor. The percentage utilization of the processor is: (CPU seconds per hour / number of engines / 3600) x 100%.

If you do not currently have Domino on a S/390 platform, begin with 8.2.6, “Making an initial estimate” on page 118.

8.2.2 Number of usersYou can probably estimate the number of registered users on your server relatively easily: for mail applications, it is the number of users whose mail files are located on that specific server. For a Notes application, it is the number of users authorized for each database. If you allow anonymous browser access, you will need to estimate how many users might be logged on at a time.

8.2.3 Transaction rateThe rate at which transactions arrive at the Domino server depends on the number of active users and the rate at which they are working. For example, if an


installation with 1,000 registered users has 200 active users working at a rate of 90 server transactions per hour, then the server would need to process 18,000 transactions per hour.

On a production system, you can easily measure the transaction rate. The number of transactions run in an interval can be calculated by figuring the difference between the Domino transaction count at the ends of two consecutive intervals. The Domino transaction count is recorded in the SMF 108 record as discussed in 7.1.1, “Record mapping” on page 98.

You can calculate the transaction rate per registered user by dividing the number of transactions by the number of registered users.

Note that the transaction rate per user could be higher during a pilot when users are “experimenting” with this new tool. Alternatively, the transaction rate per user could be lower until you have a critical mass of users on the server.

8.2.4 CPU time used per transactionThere are many variables that affect the CPU time used by a server transaction. The size of the address book used for address resolution, the size of the items being sent, and the number of items in the user's mail file are just a few of the variables that affect the CPU time per transaction.

You can measure the CPU time used per transaction on your system. For a time interval you select, you can get the total CPU time used by Domino by adding up the CPU time used by each of Domino's address spaces. You can get that information from the SMF records; see 2.5.2, “Monitoring OS/390 with SMF” on page 25. You should include a proportion of the CPU time used by OS/390 and TCP/IP, depending on the workload mix on the system. Divide the CPU time used by the number of Domino transactions recorded for the same interval, as described above.

If Domino is running in its own LPAR, then divide the CPU time used by the LPAR for a given time interval by the number of transactions recorded.

8.2.5 Consider the peak timesThere are times during the day when the capacity needed by Domino is higher than the average and there are times when it is lower.

If Domino will represent less than 50% of the total workload on the S/390 server, and where the other workload is of lower priority, you can probably use the average workload for planning. The OS/390 Workload Manager will manage the overall resources and make them available to Domino at the peak times if you define the Domino workload to have a high priority.

For dedicated Domino servers, or for S/390 systems where Domino is not the highest priority workload, a peak load factor should be considered to ensure consistently good response times.

Determining the peak load factor to use requires some understanding of the work schedule of the users. If the users are teachers who can only use the system between classes, or at the beginning or end of the day, then the peak load factor could be much higher than two. If the majority of the users are using a local copy

Chapter 8. Capacity planning 117

of their mail database on their workstation, then the peak load factor could be much less than two.

The use of a local copy by a high percentage of users results in a more uniform workload throughout the business day, but could also extend it. Users may start their replication earlier so that it finishes before they arrive at work.

If Domino will share a S/390 server with other high priority workloads, consider the Domino peak in relation to the other workloads. If the users of your Domino system are also the users of your CICS or IMS system, they are unlikely to be using both systems heavily at the same time of day. Since the peaks of the two workloads occur at different times of the day, the total capacity you need is less than the sum of the peaks for the two workloads. By recognizing this, many S/390 installations will require less capacity than if they ran each workload on a separate server.

8.2.6 Making an initial estimateSee 8.10.1, “TechXpress” on page 126 for information on how to request an initial sizing for a planned Domino for S/390 installation.

8.2.7 Capacity planningProjecting the CPU time that you will need for your Domino servers can be as simple as projecting your future requirements based on your past usage. By measuring the CPU cost per registered user, you can plan for additional users. Measuring the transaction rate and CPU time per transaction allows you to plan for additional applications.

Capacity planning is not optional. You need to periodically measure the resources that your Domino installation is using and adjust as needed. Whether or not you do capacity planning at the transaction level is up to you.

8.3 Estimating processor storage

The measured and estimated storage in the following sections is the total amount of storage, in megabytes, needed on the S/390 to support Domino. It includes the storage required for OS/390 and other prerequisite software as well as for Domino. This storage is the sum of processor storage and paging storage. However, paging to auxiliary storage should be avoided since it will increase CPU usage as well as response time.

For notes mail, the amount of storage needed depends on the number of connected users. The maximum number of connected users would be less than the total number of registered users. A conservative estimate is that 70 percent of the registered users would be connected during peak activity periods.

The amount of storage needed also depends on the number of physical threads acquired by a DPAR, as specified in the SERVER_POOL_TASKS parameter in the notes.ini file, and the size of the DFSMS buffer pool as specified in the FILESYSTYPE TYPE(HFS) statement in SYS1.PARMLIB(BPXPRMxx).

The following sections describe how to estimate the storage needed by Domino for the following types of clients:


• Notes

• POP3

• IMAP

• WebMail

8.3.1 Estimating processor storage for Notes clientsTable 31 shows the amount of processor storage that is needed to support Domino on a S/390 server, based on the number of connected Notes client users. These are the results of our Laboratory measurements. The “Measured Storage” is the actual storage that was used in the series of tests show in the table. The “Estimated Storage” was calculated using the following formula:

Total Storage per LPAR (MB) = 700 + n (300 + 0.12 CU)

where n is the number of Domino partitions (DPARs) and CU is the number of connected users per DPAR.

Table 31. Estimated server storage in megabytes when using Notes clients

Table 31 notes:

1. The measured and estimated storage is the total amount of storage, in megabytes, needed and includes OS/390 and other prerequisite software.

2. The estimates are based on Domino Release 5.0 running on OS/390 Version 2 Release 6. Storage requirements may change for later releases.

3. The Domino components in these storage estimates are comparable to the storage estimates for other Domino platforms.

Also note that if you use the equation above to estimate your storage requirements, you arrive at a minimum value of 1000 MB. Our lab measurements have demonstrated that the server can actually be started on a system with less than 512 MB of processor storage. However, by the time 1,000 users are simulated against the server, the memory requirements closely match those predicted by the equation. This is displayed graphically in Figure 46 on page 120.

Connected Users

Number of DPARs

Measured Storage (MB)

Estimated Storage (MB)

3,500 1 1,285 1,420

4,000 1 1,358 1,480

4,500 1 1,407 1,540

5,000 1 1,462 1,600

8,000 1 1,810 1,960

9,000 1 1,885 2,080

10,000 1 2,037 2,200

16,000 2 2,920 3,220

24,000 3 4,220 4,480

32,000 4 5,290 5,740


Figure 46. Storage usage for Notes clients

If you intend to experiment with an R5 server or run a server with a very light user load, you may be able to run with less than the amount of storage predicted by the equation. However, you should closely monitor RMF to be sure that paging is not an issue. When you are ready to run in production mode, you should use the equation to calculate an appropriate storage value, and use that value. In both cases, you should closely monitor your storage usage to be sure the system isn't constrained.

8.3.2 Estimating processor storage for POP3 clientsTable 32 shows the amount of processor storage that is needed to support POP3 mail users on a single Domino for S/390 server, based on the number of active users. The “Measured Storage” is the actual storage that was used in the series of tests show in the table. The “Estimated Storage” was calculated using the following formula:

Total Storage per LPAR (MB) = 700 + 0.024 APU

where APU is the maximum number of POP3 users that are active during a 15-minute interval.

Table 32. Estimated server storage in megabytes when using POP3 clients

Table 32 notes:

1. The measured and estimated storage is the total amount of storage needed, in megabytes, and includes OS/390 and other prerequisite software.

2. NSF buffer pool size: 256 MB

Active Users Measured Storage (MB)

Estimated Storage (MB)

800 702 719

1600 747 738

3200 774 777

4800 804 815


3. DFSMS VIRTUAL: 100 MB

4. DFSMS FIXED: 0 MB

Note: We recommend setting the same values in DFSMS virtual and fixed. For more information, see 3.4.2, “Setting DFSMS parameters” on page 36.

5. All the test runs were with one DPAR. The storage estimates reflect the use of one DPAR. See Section 8.9.2, “A large POP3 installation” on page 126 for information on a large POP3 production system with multiple DPARs per LPAR.

8.3.3 Estimating processor storage for IMAP clientsTable 33 shows the amount of processor storage that is needed to support Domino on a S/390 server, based on the number of active IMAP client users. The “Measured Storage” is the actual storage that was used in the series of tests show in the table. The “Estimated Storage” was calculated using the following formula:

Total Storage per LPAR (MB) = 1,050 + n (440 + 0.35 AIU)

where n is the number of Domino partitions (DPARs) and AIU is the number of users per DPAR that are active during a 15-minute interval.

At this time, the maximum number of active IMAP users per DPAR is 700.

Table 33. Estimated server storage in megabytes when using IMAP clients

Table 33 notes:

1. The measured and estimated storage is the total amount of storage, in megabytes, needed and includes OS/390 and other prerequisite software.

2. NSF buffer pool size: 128 MB (default)

3. DFSMS VIRTUAL: default

4. DFSMS FIXED: default

8.3.4 Estimating processor storage for WebMail clientsRecent tests have shown that the number of active WebMail users per DPAR can be significantly increased by increasing the size of the NSF buffer pool to 512 MB and increasing the size of the DFSMS buffer pool by 200 MB over the default. Based on these tests, we recommend one DPAR per LPAR with 2 GB of storage per LPAR.

At this time, the maximum number of active WebMail users per DPAR is 1,500.

Active Users Numberof DPARS

Measured Storage (MB)

Estimated Storage(MB)

400 1 1629 1630

600 1 1699 1700

2689 4 3733 3751


8.4 The use of Domino partitioned servers

If you plan to have more than 5,000 registered Notes client users on your Domino for OS/390 system, you may need to run multiple Domino servers to support them.

We have observed no degradation in performance with up to 10,000 connected Notes client users in a single Domino partition (DPAR) running in a controlled environment with adequate resources. Going beyond 10,000 connected users results in the inability of new users to establish a connection to the server.

The largest DPAR in production as of September 2000 has over 7,000 registered users. However, we suggest that you monitor your system as it grows beyond 2,000 registered users to determine the maximum number of active users per DPAR. A reasonable target is 2,000 active users per DPAR for Notes clients.

8.5 The use of S/390 LPARs

For users with Notes clients, POP3 clients, or IMAP clients, we recommend running no more than two Domino partitions (DPARs) per S/390 logical partition (LPAR)—one DPAR per LPAR would be even better. For WebMail users, we recommend only one DPAR per LPAR. Although each new LPAR requires another copy of all the prerequisite software, each additional DPAR in an LPAR increases the amount of CPU used. For more information on the cost of a DPAR and the measurement results, see 6.1.1, “Costs of partitioning” on page 89.

Another consideration is the limit of 2 GB of central storage for a single LPAR. The storage estimates in Table 31 on page 119 show that more than 9,000 connected Notes client users per LPAR will exceed this 2 GB limit and introduce paging. Even though a small amount of page movement to expanded storage will have a minimal affect on performance, paging to auxiliary storage will significantly increase CPU usage as well as degrade performance.

You can track the storage use on your system using the high UIC value - see 2.5.1.4, “RMF paging activity report” on page 22.

You can run up to 15 OS/390 systems on a single S/390 server using logical partitioning (LPAR).

8.6 Estimating DASD required

Configuring the DASD I/O subsystem required to support your Domino server(s) requires that you estimate the amount of space that you will need and the I/O rate that you will need to sustain. The following sections provide some guidance in estimating your DASD space and I/O rate requirements. As with any system, you should measure the DASD space used and I/O rates on your system periodically and base your follow-on estimates on these measurements.

8.6.1 Estimating DASD spaceThe amount of DASD space that is needed for Domino depends on the amount of data that you will store. It is the same as for other Domino platforms.


The amount of DASD space required for mail databases depends on the amount of data that each user will store. This is normally in the range of 40 to 100 megabytes per user and will typically grow as the users become more advanced in their use of Domino. Installations with thousands of users will therefore require hundreds of gigabytes (GB), if not terabytes, of DASD space. Add to this 17 GB for each DPAR and 15 GB for each LPAR.

OS/390 provides a standard UNIX hierarchical file system (HFS) for applications such as Domino that use OS/390 UNIX . You should plan your use of the HFS carefully to avoid the need to move data around later. See section 2.12, “Notes Directory Structure” in Lotus Domino for S/390 Release 5: Installation, Customization and Administration, SG24-2083 for more information on the HFS.

8.6.2 Estimating DASD I/O ratesThe DASD I/O rate for Domino will vary by client type, number of connected users, and the amount of work being done by the users. We recommend that you use the following for your initial estimate. These results are from our laboratory measurements; the rate may vary in your environment.

• 0.5 to 1 I/O per second per active user for Notes, POP3, and IMAP clients

• 1.5 to 3 I/Os per second per active user for WebMail clients

8.7 Network considerations

Proper planning of the network will minimize delays due to an overloaded network. As users begin to exploit the advanced functions of Lotus Notes, such as attaching documents, spread sheets and presentations to notes, the amount of data flowing through the network will grow.

One difference with a large Domino server, such as S/390, is that you will have a lot of network traffic into one point. S/390 users already understand how to do that, but it is different from an implementation where there are many small servers. Make sure that you configure at least two paths into the S/390 server, for both performance and availability.

A large server also results in less network traffic overall. It eliminates the server-to-server network traffic for mail routing, database replication and calendar tasks.

We recommend that you use the following for your initial estimate. These results are from our laboratory measurements; the rate may vary in your environment.

• 3 to 4 Kbps of bandwidth per active user for Notes, POP3, and IMAP clients

• 5 to 6 Kbps of bandwidth per active user for WebMail clients

For more information on designing Domino networks, we recommend reading Lotus Notes and Domino Scalable Network Design by John Lamb and Peter Lew, ISBN 007913792X, McGraw Hill, 1999.

8.8 The production Domino for S/390 servers running in Poughkeepsie

Many people have shown an interest in the production Domino servers on S/390 that IBM has been running in Poughkeepsie, New York, USA since November


1996. These are true production Domino servers, used for IBM's internal rollout of Lotus Notes. They have also been used as a production proving ground for Domino for S/390 since 1996. We provide this information as of November 2000 to show you an example of a large Domino for S/390 system.

8.8.1 Hardware and software environmentThe Poughkeepsie production system is currently supporting approximately 18,000 registered users with two S/390 server machines: 10,000 registered users on an IBM 9672-ZZ7, and 8,000 registered users on an IBM 9672-YX6. The IBM 9672-ZZ7 has four logical partitions (LPARs), three with two Domino mail partitions (DPARs) each and one with two application DPARs. The IBM 9672-YX6 has two LPARs, with two mail DPARS each. This is shown in Figure 47.

We saw a significant reduction in resources needed when we migrated from Lotus Domino Release 4.6.3 to Release 5.0 and consolidated eight DPARs into three DPARs—over 20 percent, in both cases. All of the mail LPARs are running OS/390 2.8 and Release 5.05 of Domino, except LPAR 1, which is running OS/390 2.10 and Domino Release 5.06. The IBM 9672-ZZ7 is about 77% utilized and the IBM 9672-YX6 is about 83% utilized, during the day.

LPAR 1 on the IBM 9672-ZZ7 is used as a production test environment. It is the first to run with a new release of Domino or any of the prerequisite software. One of the DPARs in this LPAR has over 2,800 registered users. Increasing the number of registered users per LPAR has allowed us to significantly reduce the number of DPARs that we need to administer, as well as reduce the amount of CPU used.

Figure 47. Poughkeepsie production Domino for S/390 servers

All six LPARs have 2 GB of central storage each. LPARs 1, 2, and 3 have 4 GB of expanded storage each, LPARs 5 and 6 have 6 GB of expanded storage each, and LPAR 4 has 1920 MB of expanded storage. LPARs 1 and 3 each have seven logical CPs, LPAR 2 has 8, LPAR 4 has 2, and LPARs 5 and 6 each have 10.

A total of 4,500 GB of DASD has been allocated to the six Domino LPARs. Of that, a total of 2,400 GB of DASD is being used. Although each registered user is allocated 100 MB of DASD space, some are currently using over 800 MB of

9672 - ZZ7 9672-YX6

LPAR 5 LPAR 6

OS/390

Domino MailServer

Domino MailServer

OS/390

Domino MailServer

Domino MailServer

LPAR 1 LPAR 2

OS/390

Domino MailServer

Domino MailServer

OS/390

Domino MailServer

Domino MailServer

LPAR 3 LPAR 4

OS/390

DominoApplication

Server

DominoApplication

Server

OS/390

Domino MailServer

Domino MailServer


DASD space. We started out with 25 mail files per HFS data set and then moved some of the larger mail files to other HFS datasets.

A total of 1131 HFS datasets are backed up nightly using ADSM. This takes 6 to 10 hours per night running 6 to 19 ADSM clients per LPAR, each backing up 18 to 40 HFS datasets.

With the implementation of transaction logging, our restart time following an unscheduled outage was significantly reduced. Prior to the implementation of transaction logging, it would take 30 to 40 minutes to recover a DPAR with over 500 connected users from an unscheduled outage. With transaction logging, using the “favor recovery” option, it takes about 5 minutes. The implementation of transaction logging did not make a noticeable change in CPU usage.

8.8.2 WorkloadThe S/390 server currently supports 18,000 registered users doing mail, calendar, and scheduling. A typical day's workload consists of:

• About 60 percent of the registered users are connected during the peak hour.

• More than 1000 server transactions per registered user are run per day.

• More than 100 server transactions per registered user per hour are run during the nine prime-time hours.

8.9 Other production Domino for S/390 servers

This section provides an update on the status of some of the larger Domino for S/390 installations outside of IBM.

8.9.1 A large production DPARGiven adequate resources, a single DPAR can support over 2,000 concurrently active users generating over 2 million transactions during the business day. One customer installation has a single DPAR in production that has peaked at over 1,550 concurrently active users, and has exceeded 1.7 million transactions during their 10-hour business day.

Their single DPAR supports 5,000 registered users doing mail, calendar, and scheduling. A typical day's workload consists of:

• About 30 percent of the registered users are connected during the peak hour.

• More than 300 server transactions per registered user are run per day.

• More than 25 server transactions per registered user per hour are run during the 10 prime-time hours.

Domino is the only application running on their IBM 9672-R25. Although the CPU utilization will periodically peak at 100%, the average CPU utilization during the business day is typically less than 80%. They are using about 1,100 MB of central storage and 17 MB of expanded storage. Their I/O subsystem is averaging 370 I/Os per second, with peaks over 700 I/Os per second, and averages 4 ms per I/O.


8.9.2 A large POP3 installationA customer has 30,000 production POP3 users running on a 9672-R86 with 10 GB of storage. They are averaging 65% CPU utilization during the day and peak at 75%. Their average daily workload consists of 440,000 sessions, 78,000 mail items received, 72,000 mail items sent, and 3.4 TB of data transferred. They have 8 mail DPARs (7 with 4,000 users each and 1 with 2,000 users) and 4 mail LPARs, 2 DPARs/LPAR.

8.9.3 Leading edge installationsIf you’d like to share your Domino for S/390 success story with others, you can submit a brief description of your installation to us in an Internet note to:

redbooks.ibm.com

In your note, be sure to include the number of this redbook, SG24-5149 (edition -02).

8.10 Capacity planning assistance

An initial capacity planning estimate for a Domino workload is available through your IBM or Lotus Marketing Representative. A more detailed analysis is available through various chargeable IBM offerings.

8.10.1 TechXpressYour IBM or Lotus Marketing Representative can request an initial sizing for Domino on S/390 by requesting a form from TechXPress, completing this form with your help, and submitting the completed form to TechXPress.

The following information is needed to complete this form:

• Estimate the peak number of registered, connected, and active users that you expect Domino to support.

• The number of registered users is the total number of users that you plan to authorize to use the servers that you are requesting the sizing for.

• The number of connected users is the maximum number of users that you expect to be in session with Domino at one time. A reasonable starting point is 70% of the total number of registered users. This is used to estimate the amount of storage (memory) that will be needed for your server(s).

• The number of active users is the maximum number of users that you expect to be doing productive work during a 15-minute interval. A reasonable starting point is somewhere in the range of 20% to 40% of the number of registered users. This is used to estimate the size of the processor that will be needed.

• State the percent of active users that are Notes, POP3, IMAP, or WebMail clients.

• For Notes and WebMail clients, state the percentage that are doing light mail, experienced mail, experienced mail plus calendaring/scheduling, experienced mail plus calendaring/scheduling, and applications. If applications are planned, then an estimate of the application workload, as a percent of the mail workload, is requested.


• For POP3 and IMAP clients, state the percentage that are doing light mail and the percent that are doing experienced mail.

• Tell us the processors that you would like sized, if any; for example, an existing S/390 that has excess capacity.

Upon receipt of this form, TechXPress will provide you with an estimate of the processor size you will need to support this workload. IBM and Lotus employees can submit a TechXpress request on the IBM intranet or by phone. Find out more at this Web site:

http://w3-1.ibm.com/support/techxpress.html

8.10.2 SNAP/SHOTHost SNAP/SHOT Services System Capacity Planning supports many applications, including Domino for S/390. SNAP/SHOT can show you the effect of adding Domino to your existing S/390 workload. It can help you determine what changes, if any, should be made to your S/390 and network configuration to accommodate this additional workload. For more information, visit the Web site:

http://www.as.ibm.com/asus/performance.html

You can also call toll-free at the following numbers:

Telephone (USA)1-888-474-TESTTelephone (outside)(301) 240-8535

8.10.3 IBM Testing ServicesIBM Testing Services provides the hardware, software, drivers, network, and expertise to perform comprehensive stress testing. Whether for proof-of-concept testing, capacity and performance testing, or full stress testing, there are times when only a “real” test will do. For more information, visit the Web site:

http://www.as.ibm.com/asus/svs.html

You can also call toll-free at the following numbers:

Telephone (USA) 1-888-474-TESTTelephone (outside)(301) 240-8535


Appendix A. Workload Manager (WLM) example

This appendix contains an example of the Workload Manager goal mode definitions for a Lotus Domino production environment. There are six Domino servers: ML96NP, ML97NP, ML98NP, ML99NP, DBL90NP and DB99NP. These servers run in three LPARs of a sysplex and are the prime workloads for these LPARs.

Use the ISPF administrative application to enter or change your WLM goal mode definitions.

A.1 Workload and service class descriptions

Table 34 shows the workloads that the Workload Manager will use. A workload is a named collection of work to be reported as a unit.

Table 34. Workload descriptions

Table 35 shows the service classes that the different workloads will be assigned to. A service class is a named group of work within a workload with similar performance characteristics. For Domino, we specified two service classes called NOTESLOW and NOTESSRV. The default service class for Domino is NOTESSRV. Service class NOTESLOW is not used for normal operation.

Table 35. Service class descriptions

Workload Description

PRIMEAPP APPC workload

PRIMEBAT Batch workload

PRIMESTC STC workload

PRIMETSO TSO workload

PRIMOMVS UNIX System Services workload

Service class Description

BATCHLOW Low priority batch

HOTTSO Support TSO users

NOTESLOW Lotus Notes Server ResetToClass

NOTESSRV Lotus Notes server

OMVSLOW Low priority OMVS Work

OMVSTASK OMVS Support Tasks

OMVSUSER UNIX System Services users

STCLOW Low priority STC

STCMED Better performance for ADSM

TSOPRIME Normal TSO users


A.2 Report classes

Table 36 shows the report classes set up to provide more granular reporting. Report classes allow this reporting for individual Domino servers while managing all the Domino servers the same in a single service class (NOTESSRV).

Table 36. Report classes

A.3 Classification rules

Classification rules specify how a workload, such as a Domino address space, is assigned to a service class. It can also be assigned to a report class if required. In this example all Domino servers have a service class of NOTESSRV but have different report classes.

The Qualifier type column shows on what input the classes are selected. In our example (Table 37), Workload Manager assigns service classes and report classes to defined Domino servers based on job name (qualifier type TN, values are in column Qualifier name). There are other workloads, identified by user ID (qualifier type UI).

Table 37. Classification rules

Workload Description

ADSM ADSM report class

DFHSM DFHSM report class

JES2 JES2 report class

ML96NP ML96NP Notes server





DB99NP DB99NP Notes server

MVSNFS MVSNFS report class

OMVS OMVS kernel report class

RMFIII RMF Monitor III report class

SYSBMAS SYSBMAS report class

TCPML TCP/IP report class

Level Qualifiertype

Qualifiername

Service class Report class

1 Default BATCHLOW

1 TN ADSM OMVSLOW ADSM

1 UI OMVS OMVSTASK

1 UI SERVER NOTESSRV


Note: A blank in the Service Class column means “started task default” (also blank) to get into SYSTEM and SYSSTC classes.

A.4 Service definition goals for service classes

Having put workloads in the defined service classes, you now need to define the performance goals required for these service classes. These goals may be velocity (the rate at which you expect work to be processed), response times (a percentage completed within a specified time), or discretionary (WLM-defined goal for workload for which you do not have any specific requirement). You also assign a number to specify how important it is that the workload achieves its goal.

You can also specify the relative importance of given workload and describe your performance requirements. Service class NOTESSRV specifies business importance of 1.

OS/390 Workload Manager recognizes importance in five levels: 1 to 5, where 1 is the highest importance. Service class NOTESSRV also specifies velocity 60 as

1 TN ML96NP NOTESSRV ML96NP





1 TN DB99NP NOTESSRV DB99NP

1 TN MVSNFS MVSNFS

1 TN OMVS OMVS

1 TN SYSBMAS SYSBMAS

1 TN TCPML TCPML

1 TN RMFGAT RMFIII

1 TN DFHSM DFHSM

1 TN JES2 JES2

1 TN ADSM STCLOW ADSM

1 UI D60SXM1 HOTTSO

1 UI M330475 HOTTSO

1 UI GOLDIJ HOTTSO

1 UI GARRIGN HOTTSO

1 UI S036 HOTTSO

1 UI THEWAY HOTTSO

1 UI NEILS HOTTSO

Level Qualifiertype

Qualifiername

Service class Report class

Appendix A. Workload Manager (WLM) example 131

its performance goal. Velocity is percentage from 1 to 99 and specifies how fast a given workload should run when ready, without being delayed by waiting for WLM-managed resources.

In our example, we specified all Domino server performance goals within one period only. Other workloads have periods specified, because they have variable resource requirements, so their goals change as they use more resources.

Occasionally you may need to limit resources used by Domino servers because of a more important workload in the system. In that case, you can use the NOTESLOW service class with business importance of 3 and velocity 20. Use service policy overrides to adjust WLM goals temporarily or during certain periods like weekends or holidays. Table 38 shows our example.

Table 38. Service definition goals

Serviceclass

Workload Period Duration Importance Goal

BATCHLOW PRIMBAT 1 Discretionary

HOTTSO PRIMETSO 1 400 1 90% 00:00:00.500

2 1400 1 80% 00:00:01.000

3 40000 1 70% 00:00:03.000

4 2 Velocity 30

NOTESLOW PRIMOMVS 1 3 Velocity 20

NOTESSRV PRIMOMVS 1 1 Velocity 60

OMVSLOW PRIMOMVS 1 Discretionary

OMVSTASK PRIMOMVS 1 2 Velocity 30

OMVSUSER PRIMOMVS 1 400 2 80% 00:00:05.000

2 2000 2 Velocity 30

3 Discretionary

STCLOW PRIMETSO 1 3 Velocity 20

STCMED PRIMETSO 1 1 Velocity 40

TSOPRIME PRIMETSO 1 200 1 90% 00:00:00.500

2 400 1 90% 00:00:02.000

3 1 Velocity 30


Appendix B. Domino parameters that affect performance

This appendix lists the parameters in the notes.ini file and in the configuration settings that affect Domino performance. They are sorted by function. For detailed information, see Chapter 42 and Appendix E in Lotus Domino 5—Administering the Domino System. This is published in two volumes and can be ordered from Lotus using part numbers CT6T7NA and CT6T8NA.

B.1 System parameters

These parameters affect the Domino server overall:

• Fixup_Tasks • NSF_Buffer_Pool_Size and NSF_Buffer_Pool_Size_MB • NSF_DbCache_Maxentries • Replicators • Server_Enable_Threadpool • Server_Max_Concurrent_Trans (do not change the default) • Server_MaxSessions (do not change the default) • Server_MaxUsers (do not change the default • Server_Pool_Tasks • Server_Session_Timeout • ServerTasks • Translog_Status • View_Rebuild_Dir

B.2 Monitoring parameters

These parameters affect monitoring:

• Console_Loglevel • Log • Log_Replication • Log_Sessions • Log_Tasks • Log_Update • Log_ViewEvents • RTR_Logging

B.3 Administration automation parameters

These parameters affect automation of administration activities:

Do not change the following parameters:

• Server_Max_Concurrent_Trans • Server_MaxSessions

Leave the values at their defaults. Otherwise you may cause problems with your Lotus Domino for S/390 server.

Caution


• AdminPInterval • AMgr_DocUpdateAgentMinInterval • Amgr_DocUpdateEventDelay • Amgr_NewMailAgentMinInterval • Amgr_NewMailEventDelay • Amgr_SchedulingInterval • Amgr_UntriggeredMailInterval • DominoAsynchronizeAgents

B.4 Indexing parameters

These parameters affect indexing:

• FT_Intl_Setting • Update_No_Fulltext • Updaters • Default_Index_Lifetime_Days • Update_Suppression_Time • Update_Suppression_Limit

B.5 Mailing parameters

These parameters affect mailing:

• MailCompactDisabled • MailMaxConcurrentXferThreads • MailMaxDeleveryThreads • MailMaxThreads • Mail_Number of Mail_Boxes • MinNewMailPoll • NoMsgCache • POP3_Config_Update_Interval • SecureMail • SharedMail


Appendix C. Sample notes.ini file

This is an example of a notes.ini file.

[Notes]Directory=/test01KitType=2SetupDB=setupweb.nsfSERVER_POOL_TASKS=125SERVER_ENABLE_THREADPOOL=1PhoneLog=2Log=log.nsf, 1, 0, 7, 40000TCPIP0A=TCP,0,15,0,,45056,Ports=TCPIP0ATCPIP0A_TCPIPADDRESS=0,x.xxx.xxx.xx:1352SPX=SPX, 0, 15, 0Serial1=XPC,1,15,0,Serial2=XPC,2,15,0,$$HasLANPort=1DisabledPorts=SPX,Serial1,Serial2LOG_REPLICATION=1LOG_SESSIONS=1SETUP_PERCENTDONE=100SETUP_STATUS=Updating network settingsKeyFilename=xxxxx.idMailServer=CN=test01/OU=01/OU=M/O=ITSODomain=IBMITSOServerKeyFileName=xxxxxx.idAdmin=CN=ITSO/OU=Pxxxxx/O=ITSO, CN=ITSO2/ OU=Pxxxxx/O=ITSOServerTasks=Replica,Router,Update,Stats,AMgr,Adminp,Sched,CalConn,Event,mtbf,CollectServerTasksAt2=UpdAll,TemplateSetup=55Setup=59ServerSetup=50Timezone=5DST=0CleanupScriptPath=/usr/lpp/lotus/notesapi/rc.notes.fatalMTEnabled=0SCHEDULE_VERSION=3SERVER_SESSION_TIMEOUT=30MAILTIMEOUT=1ADMINPINTERVAL=15ADMINPMODIFYPERSONDOCUMENTSAT=0LOG_AGENTMANAGER=1SHOW_TASK_DETAIL=1LOG_VIEW_EVENTS=1Fixup_Tasks=75names=names,pdlSERVER_CLUSTER_ON=0server_restricted=0DSTLaw=4 1 1 10 -1 1AgentManagerVerboseMode=1da_reload_interval=60log_update=1maildynamiccostreset=15AMgr_DisableMailLookup=1


VIEW_REBUILD_DIR=/test01/viewbuild/CleanSetup=1EventSetup=500SERVER_NAME_LOOKUP_NOUPDATE=1SERVER_SHOW_PERFORMANCE=1Passthru_LogLevel=0Console_LogLevel=2DDETimeout=10NAMEDSTYLE0=030042617369630000000000000000000000000000000000000000000000000000000000000001010100000A0000000000000100A0050A0000006400A0050A0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000009404000000000000NAMEDSTYLE0_FACE=Default Sans SerifNAMEDSTYLE1=030042756C6C657400000000000000000000000000000000000000000000000000000000000001010100000A000000000000000008070A000000640008070A0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000049404000000000000NAMEDSTYLE1_FACE=Default Sans SerifNAMEDSTYLE2=0300486561646C696E6500000000000000000000000000000000000000000000000000000000010101010B0C0000000000000100A0050A0000006400A0050A0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000009404000000000000NAMEDSTYLE2_FACE=Default Sans Serif$$$OpenSpecial=NotesNIC$$$NotesNIC=CN=Home/OU=Notes/O=NET, welcome.nsf, Notes NIC Welcome, NotesNetwork Information Center on the InternetDefaultMailTemplate=mail50.ntfPreferences=32ServerTasksAt1=Catalog,StatlogTRANSLOG_AutoFixup=1TRANSLOG_UseAll=0TRANSLOG_Style=0TRANSLOG_Performance=2TRANSLOG_Status=0NSF_BUFFER_POOL_SIZE_MB=256server_availability_threshold=0ServerTasksAt3=FileDlgDirectory=/test01TCPIP1A=TCP,0,15,0,,45056,COLLECT_DB_LOCK_WAITS=0


Appendix D. Sample output of the show statistics command

The following is a sample output of the show statistics command.

> sh stat

Database.Database.BufferPool.Maximum.Megabytes = 256Database.Database.BufferPool.MM.Reads = 20Database.Database.BufferPool.MM.Writes = 6Database.Database.BufferPool.Peak.Megabytes = 255Database.Database.BufferPool.PerCentReadsInBuffer = 88.18Database.DbCache.CurrentEntries = 1152Database.DbCache.HighWaterMark = 1152Database.DbCache.Hits = 29384Database.DbCache.InitialDbOpens = 37074Database.DbCache.Lookups = 36084Database.DbCache.MaxEntries = 768Database.DbCache.OvercrowdingRejections = 3894Database.ExtMgrPool.Peak = 65,406Database.ExtMgrPool.Used = 160Database.FreeHandleStack.FreeHandleStackHits = 1116435Database.FreeHandleStack.HandleAllocations = 1149330Database.FreeHandleStack.MissRate = 2Database.NAMELookupCacheCacheSize = 5,605,036Database.NAMELookupCacheHashSize = 451,409Database.NAMELookupCacheHits = 561,940Database.NAMELookupCacheLookups = 574,446Database.NAMELookupCacheMisses = 12,506Database.NAMELookupCacheNoHitHits = 0Database.NAMELookupCachePool.Peak = 11,534,336Database.NAMELookupCachePool.Used = 10,643,200Database.NAMELookupCacheResets = 0Database.NAMELookupMisses = 2,497Database.NIFPool.Peak = 23,068,672Database.NIFPool.Used = 350,048Database.NSFPool.Peak = 113,246,208Database.NSFPool.Used = 21,642,688Database.RM.Sys.Logged = DisabledDisk./.Free = 184,614,912Mail.AverageDeliverTime = 3Mail.AverageServerHops = 1Mail.AverageSizeDelivered = 5Mail.CurrentByteDeliveryRate = 6458Mail.CurrentByteTransferRate = 0Mail.CurrentMessageDeliveryRate = 1Mail.CurrentMessageTransferRate = 0Mail.DBCacheEntries = 767Mail.DBCacheHits = 47045Mail.DBCacheReads = 306999MAIL.Dead = 1Mail.Delivered = 307434Mail.Deliveries = 306894Mail.Domain = ITSOMAIL.Hold = 0Mail.MaximumDeliverTime = 563Mail.MaximumServerHops = 2Mail.MaximumSizeDelivered = 10


Mail.MinimumDeliverTime = 1Mail.MinimumServerHops = 1Mail.MinimumSizeDelivered = 1Mail.PeakByteDeliveryRate = 130245Mail.PeakMessageDeliveryRate = 24Mail.PeakMessageDeliveryTime = 10/22/1999 04:08:38 EDTMail.PeakMessagesDelivered = 1481Mail.PeakTotalBytesDelivered = 7814739MAIL.TotalFailures = 350Mail.TotalRouted = 307434Mail.TotalRouted.NRPC = 307434MAIL.Waiting = 0MAIL.WaitingForDNS = 0MAIL.WaitingRecipients = 0MailByDest.CN=SONOMI/O=ITSO.TotalFailures = 350MailByDest.[$LocalDelivery].Delivered = 307434MailByDest.[$LocalDelivery].TotalRouted = 307434Mem.Allocated = 568533860Mem.Allocated.Process = 37541598Mem.Allocated.Shared = 530992262Mem.Availability = PlentifulMem.PhysicalRAM (a hard coded value for S/390) = 536870912NET.Log.Sonomi/ITSO.UnwrittenEntries = 81NET.TCPIP.BytesReceived = 6,754,000,318NET.TCPIP.BytesSent = 16,304,736,216NET.TCPIP.Sessions.Established.Incoming = 48637NET.TCPIP.Sessions.Established.Outgoing = 0NET.TCPIP.Sessions.Limit = 65535NET.TCPIP.Sessions.LimitMax = 65535NET.TCPIP.Sessions.LimitMin = 10NET.TCPIP.Sessions.Peak = 5005NET.TCPIP.Sessions.Recycled = 0NET.TCPIP.Sessions.Recycling = 0Server.AvailabilityIndex = 100Server.AvailabilityThreshold = 0Server.BootID = 7879806Server.BusyTimeQuery.ReceivedCount = 0Server.CPU.Count = 3Server.Monitor.Start = 10/21/1999 17:53:20 EDTServer.Name = CN=Sonomi/O=ITSOServer.OpenRequest.MaxUsers = 0Server.OpenRequest.PreV4Client = 0Server.OpenRequest.Restricted = 0Server.OpenRequest.V4Client = 186020Server.Path.Data = /r50dataServer.Ports = TCPIPServer.PoweredBy = NotesServer.Sessions.Dropped = 28922

:::

Server.Tasks = 5127:::

Server.Time.Start = 10/21/1999 17:53:20 EDTServer.Title = System Under TestServer.Trans.PerMinute = 1138


Server.Trans.PerMinute.Peak = 9698Server.Trans.PerMinute.Peak.Time = 10/21/1999 19:56:15 EDTServer.Trans.Total = 5884522Server.Users = 471Server.Users.1MinPeak = 2241Server.Users.1MinPeakTime = 10/22/1999 08:58:13 EDTServer.Users.5MinPeak = 4141Server.Users.5MinPeakTime = 10/22/1999 08:58:13 EDTServer.Users.active = 0Server.Users.active15min = 2Server.Users.active1min = 0Server.Users.active30min = 2Server.Users.active3min = 0Server.Users.active5min = 0Server.Users.Peak = 5005Server.Users.Peak.Time = 10/21/1999 20:39:42 EDTServer.Version.Notes = Release 5.0.1aServer.Version.OS = OS/390Server.WorkThreads = 6553601Stats.Time.Current = 10/22/1999 09:08:33 EDTStats.Time.Start = 10/21/1999 17:53:18 EDT

Appendix D. Sample output of the show statistics command 139

Appendix E. Special notices

This publication is intended to help Domino administrators and OS/390 system programmers to tune Domino for OS/390. The information in this publication is not intended as the specification of any programming interfaces that are provided by Domino for OS/390 V5R5. See the PUBLICATIONS section of the IBM Programming Announcement for Domino for OS/390 V5R5 for more information about what publications are considered to be product documentation.

References in this publication to IBM products, programs or services do not imply that IBM intends to make these available in all countries in which IBM operates. Any reference to an IBM product, program, or service is not intended to state or imply that only IBM's product, program, or service may be used. Any functionally equivalent program that does not infringe any of IBM's intellectual property rights may be used instead of the IBM product, program or service.

Information in this book was developed in conjunction with use of the equipment specified, and is limited in application to those specific hardware and software products and levels.

IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to the IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785.

Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact IBM Corporation, Dept. 600A, Mail Drop 1329, Somers, NY 10589 USA.

Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of a fee.

The information contained in this document has not been submitted to any formal IBM test and is distributed AS IS. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer's ability to evaluate and integrate them into the customer's operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk.

Any pointers in this publication to external Web sites are provided for convenience only and do not in any manner serve as an endorsement of these Web sites.


The following terms are trademarks of the International Business Machines Corporation in the United States and/or other countries:

The following terms are trademarks of other companies:

Tivoli, Manage. Anything. Anywhere.,The Power To Manage., Anything. Anywhere.,TME, NetView, Cross-Site, Tivoli Ready, Tivoli Certified, Planet Tivoli, and Tivoli Enterprise are trademarks or registered trademarks of Tivoli Systems Inc., an IBM company, in the United States, other countries, or both. In Denmark, Tivoli is a trademark licensed from Kjøbenhavns Sommer - Tivoli A/S.

C-bus is a trademark of Corollary, Inc. in the United States and/or other countries.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and/or other countries.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States and/or other countries.

PC Direct is a trademark of Ziff Communications Company in the United States and/or other countries and is used by IBM Corporation under license.

ActionMedia, LANDesk, MMX, Pentium and ProShare are trademarks of Intel Corporation in the United States and/or other countries.

UNIX is a registered trademark in the United States and other countries licensed exclusively through The Open Group.

SET, SET Secure Electronic Transaction, and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC.

Other company, product, and service names may be trademarks or service marks of others.

AS/400ATCICSCTCurrentDB2DFSMS/MVSDFSMSdssDFSMShsmDominoEnterprise Storage ServerESCONIBM �Intelligent MinerLotusLotus NotesMVS/ESANetfinityNotes

OpenEditionOS/390PR/SMPR/SMRACFRAMACRedbooksRedbooks Logo RMFRS/6000S/390SecureWaySNAP/SHOTSPSXMSystem/390VTAMXT400


Appendix F. Related publications

The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this redbook.

F.1 IBM Redbooks

For information on ordering these publications see “How to get IBM Redbooks” on page 145.

• Lotus Domino for S/390 Release 5: Installation, Customization and Administration, SG24-2083

• Lotus Domino for S/390 Release 5: Problem Determination Guide, SG24-5599

• Lotus Domino for S/390 Release 5: Running a Large Domino Server, SG24-5984 (available as a redpiece)

• Debugging UNIX System Services, Lotus Domino, Novell Network Services, and other Applications on OS/390, SG24-5613

• OS/390 Version 2 Release 6 UNIX System Services Implementation and Customization, SG24-5178

• Hierarchical File System Usage Guide, SG24-5482

• OS/390 eNetwork Communications Server TCP/IP, Vol. 1, SG24-5227



F.2 IBM Redbooks collections

Redbooks are also available on the following CD-ROMs. Click the CD-ROMs button at ibm.com/redbooks for information about all the CD-ROMs offered, updates and formats.

F.3 Other resources

These publications are also relevant as further information sources:

• OS/390 UNIX System Services Messages and Codes, SC28-1908

• OS/390 UNIX System Services Planning, SC28-1890

• OS/390 UNIX System Services Users Guide, SC28-1891

CD-ROM Title Collection Kit Number

IBM System/390 Redbooks Collection SK2T-2177IBM Networking Redbooks Collection SK2T-6022IBM Transaction Processing and Data Management Redbooks Collection SK2T-8038IBM Lotus Redbooks Collection SK2T-8039Tivoli Redbooks Collection SK2T-8044IBM AS/400 Redbooks Collection SK2T-2849IBM Netfinity Hardware and Software Redbooks Collection SK2T-8046IBM RS/6000 Redbooks Collection SK2T-8043IBM Application Development Redbooks Collection SK2T-8037IBM Enterprise Storage and Systems Management Solutions SK3T-3694




• OS/390 UNIX System Services Command Reference, SC28-1892

• OS/390 RMF Diagnosis Guide, SC33-6592

• OS/390 RMF Messages and Codes, GC28-1948

• OS/390 RMF Programmers Guide, SC28-1952

• OS/390 RMF Performance Management Guide, SC28-1951

• OS/390 RMF Report Analysis, SC28-1950

• OS/390 RMF Reference Summary, SX22-0044

• OS/390 RMF User’s Guide, SC28-1949

• OSA Planning, GC23-3870

• OS/390 MVS System Management Facilities (SMF), GC28-1783

• OS/390 MVS Initialization and Tuning Reference, SC28-1752

• OS/390 SecureWay Communications Server IP User’s Guide, GC31-8514

• S/390 PR/SM Planning Guide, GA22-7236

• IBM 3990/9390 Planning, Installation and Storage Administration, GA32-0100

• SML: Managing Data, SC26-3124

• Lotus Notes and Domino Scalable Network Design, J. Lamb and P. Lew, Mcgraw-Hill, 1999. ISBN 007913792X

F.4 Referenced Web sites

These Web sites are also relevant as further information sources:

• IBM site for Lotus Domino for S/390 -- http://www.s390.ibm.com/products/domino

• OS/390 UNIX System Services -- http://www.s390.ibm.com/oe/

• S/390 Independent Software Vendors -- http://www.s390.ibm.com/products/s390da/

• IBM Home Page -- http://www.ibm.com/

• Lotus Home Page -- http://www.lotus.com/

• IBM Networking -- http://www.networking.ibm.com/

F.5 Lotus Documentation

These books are shipped with the Lotus Domino for S/390 Release 5.0 server code CD-ROM:

• Lotus Domino for S/390 Install Guide for Servers (part number AB0999)

• Lotus Domino for S/390 Release 5.0 Release Notes (part number AA0814)

• R5 Domino Administration Doc Pack

• Administering the Domino System: Volume 1 & Volume 2

• Moving to Notes and Domino Release 5

• Setting up a Domino Server

• Configuring the Domino Network


How to get IBM Redbooks

This section explains how both customers and IBM employees can find out about IBM Redbooks, redpieces, and CD-ROMs. A form for ordering books and CD-ROMs by fax or e-mail is also provided.

• Redbooks Web Site ibm.com/redbooks

Search for, view, download, or order hardcopy/CD-ROM Redbooks from the Redbooks Web site. Also read redpieces and download additional materials (code samples or diskette/CD-ROM images) from this Redbooks site.

Redpieces are Redbooks in progress; not all Redbooks become redpieces and sometimes just a few chapters will be published this way. The intent is to get the information out much quicker than the formal publishing process allows.

• E-mail Orders

Send orders by e-mail including information from the IBM Redbooks fax order form to:

• Telephone Orders

• Fax Orders

This information was current at the time of publication, but is continually subject to change. The latest information may be found at the Redbooks Web site.

In United States or CanadaOutside North America

e-mail [email protected] information is in the “How to Order” section at this site:http://www.elink.ibmlink.ibm.com/pbl/pbl

United States (toll free)Canada (toll free)Outside North America

1-800-879-27551-800-IBM-4YOUCountry coordinator phone number is in the “How to Order” section at this site:http://www.elink.ibmlink.ibm.com/pbl/pbl

United States (toll free)CanadaOutside North America

1-800-445-92691-403-267-4455Fax phone number is in the “How to Order” section at this site:http://www.elink.ibmlink.ibm.com/pbl/pbl

IBM employees may register for information on workshops, residencies, and Redbooks by accessing the IBM Intranet Web site at http://w3.itso.ibm.com/ and clicking the ITSO Mailing List button. Look in the Materials repository for workshops, presentations, papers, and Web pages developed and written by the ITSO technical professionals; click the Additional Materials button. Employees may access MyNews at http://w3.ibm.com/ for redbook, residency, and workshop announcements.

IBM Intranet for Employees



http://www.elink.ibmlink.ibm.com/pbl/pbl



mailto: [email protected]

http://w3.itso.ibm.com/

http://w3.ibm.com/





IBM Redbooks fax order formPlease send me the following:

We accept American Express, Diners, Eurocard, Master Card, and Visa. Payment by credit card notavailable in all countries. Signature mandatory for credit card payment.

Title Order Number Quantity

First name Last name

Company

Address

City Postal code

Telephone number Telefax number VAT number

Invoice to customer number

Country

Credit card number

Credit card expiration date SignatureCard issued to


Glossary

ACL. Access control list. A list of database users (individual users, Lotus Domino servers, and groups of users and/or servers) created and updated by the database manager. The ACL specifies which users can access the database and what tasks they can perform.

Address space. An OS/390 construct used to represent a batch job, TSO/E user, started task, UNIX Systems Services user, or forked USS process. An address space control block (ASCB) points to many other control blocks used to manage the virtual storage, I/O resources and programs used by the work unit. A USS process creates an address space.

ADSM. ADSTAR Distributed Storage Manager. IBM product for distributed file backup, restore, archive and more. The products have been rebranded to Tivoli Data Manager (TDM).

APAR. Authorized Program Analysis Report. IBM record of a software problem.

Canonical Format. Format for storing hierarchical names that displays the hierarchical attribute of each component of the name. For example, the canonical format for the name Reuben D. Smith/ Ottawa/Acme/CA is: CN=Reuben D. Smith/OU=Ottawa/O=Acme/C=CA where: CN is the common name, OU is the organizational unit, O is the organization, and C is the country code

DECS. Domino Enterprise Connection Services. A forms-based interface that allows integration to external data from Domino applications.

DNN. Domino Named Network.

DNS. Domain Name System. A function to associate names and addresses on Internet domain servers.

Domain. A Domino domain is a collection of Domino servers and users that share a common Domino Directory. The primary function is mail routing. Users' domains are determined by the location of their server-based mail files.

GRS. Global Resource Serialization. An OS/390 base component that manages serialization.

HTTP. Hypertext Transmission Protocol. An Internet protocol used to transfer files from one computer to another.

ICM. Internet Cluster Manager. Domino component for failover and load balancing of HTTP clustering.

IIOP. Internet Inter-ORB Protocol. An Internet protocol that implements CORBA solutions over the Web. IIOP lets browsers and servers exchange complex objects, unlike HTTP, which only supports transmission of text.

IIS. Internet Information Server. Microsoft Internet Information Server is a Web server that lets you

© Copyright IBM Corp. 2000

browse HTML and Active Server pages. Domino includes an IIS product extension that lets you browse Domino databases using IIS.

IMAP. Internet Message Access Protocol. A mail protocol that allows clients running it to retrieve mail from a host mail server also running the protocol. IMAP is similar to POP3 but has additional features. For example, it supports three modes of mailbox access. You can enable IMAP on a Domino server.

I/O. Input/output.

IPCS. Interactive Problem Control System. An OS/390 tool for examining operating system and subsystem dumps. The OS/390 flavor of IPCS is written in all caps.

ipcs. Inter-Process Communication System. A UNIX programming model to coordinate and serialize separate threads. The UNIX ipcs is usually written in lower case characters.

IPL. Initial program load. The initialization procedure that causes an operating system to start operation.

IRIS. A wholly owned division of Lotus. They developed the Domino/Notes product.

ITSO. International Technical Support Organization.

JES2. Job Entry Subsystem/2. A component of OS/390 that manages the initiation and termination of batch jobs, TSO users and started tasks as well as printed output.

JES3. Job Entry Subsystem/3. An optional component of OS/390 that manages the scheduling, device allocation, initiation and termination of batch jobs, TSO users and started tasks as well as printed output. JES3 evolved in parallel with JES2 to satisfy unique requirements of some MVS customers.

LDAP. Lightweight Directory Access Protocol. A set of protocols for accessing information directories. LDAP is based on the X.500 protocol, but supports TCP/IP, which is necessary for Internet access. Because it's a simpler version of X.500, LDAP is sometimes called X.500-lite. You can enable LDAP on a Domino server to allow LDAP clients to access information in Domino Directory, for example, e-mail addresses.

LMBCS. Lotus Multibyte Character Set. The format in which Notes stores all internal text, except file attachments and objects. As a result, any user can edit, forward, and mail documents and work with databases in any language. All text leaving the system (displayed, printed, and exported) is translated from LMBCS to the appropriate character set. LMBCS supports Western and Eastern European, North American, and Asian languages.

147

Logical partition. In LPAR mode, a subset of the processor unit resources that is defined to support the operation of a system control program (SCP).

Logically partitioned mode (LPAR). A mode that allows the operator to allocate hardware resources of the processor unit among several logical partitions.

Lotus. A wholly-owned subsidiary of IBM, and the owner of the Domino/Notes product.

LotusScript. A version of BASIC that offers not only standard capabilities of structured programming languages, but a powerful set of language extensions that enable object-oriented development within and across products. Its interface to Notes is through predefined object classes.

LPAR. Logical Partition. System/390 capability to divide up the processors, memory and channels so that several operating system images can run independently on the same machine.

LS:DO. The ODBCConnection, ODBCQuery, and ODBCResultSet classes, collectively called the LotusScript Data Object (LS:DO), provide properties and methods for accessing and updating tables in external databases through the ODBC (Open Database Connectivity) Version 2.0 standard.

MIB. Management Information Base. Defines the scope of what can be managed by SNMP.

MIME. Multipurpose Internet Mail Extensions. Software that allows you to attach non-text files to Internet mail messages. Non-text files include graphics, spreadsheets, formatted word-processor documents, and sound files.

MTA. Mail transfer agent. A message transfer agent, also called a gateway, is a program that translates messages between mail formats.

NAB. Notes Name and Address Book. Now called the Domino Directory or Personal Address Book.

NNN. Notes Named Network. Now DNN or Domino Named Network.

NNS. Notes Name Service. Domino service for converting Fully Qualified Domain Names or Host names to a TCP/IP address.

NNSC. Notes Name Service Cache. Hidden fields in the user’s personal NAB location document that are used when accessing the Domino server in question or in the server document of the server that initiated the connection to the other Domino server.

NNTP. Network News Transfer Protocol. Protocol that supports reading newsgroups, posting new articles, and transferring articles between news servers. When enabled on a Domino server, allows NNTP clients to access newsgroups on the server and allows the Domino server to exchange news with other NNTP servers.

Notes RPC. Notes remote procedure call. This is the architectural layer of Notes used for all Notes-to-Notes

communication. You can set up either the HTTP or the SOCKS proxy to work with RPC.

NSF. The file extension for a Notes database file (.NSF). A database file contains the data for an application. Its structure is composed of forms, fields, folders, views, and other presentation features, such as a navigator and a database icon.

NTF. The file extension for a Notes template file (.NTF). A template file contains the structure for the database--that is, forms, folders, and views--but does not contain documents. Domino Designer comes with a collection of templates that you can use to create system and application databases.

NTI. Notes Transport Interface. The component of Domino Server that manages communications.

ODBC. Open Database Connectivity. A standard developed by Microsoft for accessing external data. ODBC has four components: the ODBC-enabled application, the ODBC Driver Manager, ODBC drivers, and data sources. Lotus Notes is an ODBC-enabled application.

ODS. On Disk Structure. The format in which a Notes database is physically stored on disk. Also called a database format. In Release 5, the ODS version of a database is listed on the Info tab of the Database Properties box.

POP3. Post Office Protocol (level 3). A mail protocol that allows clients running it to retrieve mail from a host mail server also running the protocol. You can enable POP3 on a Domino server.

PTF. Program Temporary Fix. A temporary solution or by-pass for a problem diagnosed by IBM as resulting from an error in a current unaltered release of the program.

QMR. Quarterly Maintenance Release. Lotus software upgrade package, distributed quarterly, to give customers the latest release level of Domino/Notes code. The entire package must be installed, as it is a complete code replacement.

QMU. Quarterly Maintenance Upgrade. Lotus software upgrade package distributed between QMRs and designated by a letter, for example 4.5.4a. It is for critical or widespread fixes to Domino code.

RMF. Resource Measurement Facility. Monitors the activity of the OS/390 system. It is useful for capacity planning, performance tuning, and problem diagnosis. RMF gives information about system resources, workload groups, DASD controllers, and DASD volumes.

SCIP. Session Control Inbound Processing facility

SIMS. Support Information Management System. Lotus internal databases for tracking problems

SMF. System Management Facilities. OS/390 facility for recording performance and capacity data.


S/MIME. Secure/MIME. A secure version of the MIME protocol that allows users to send encrypted and electronically signed mail messages, even if users have different mail programs.

SDSF. Spool Display Search Facility. OS/390 tool for JES2 systems that allows a TSO user to view and route output from the JES2 spool and to view the SYSLOG and issue commands to OS/390 if authorized.

SMTP. Simple Mail Transfer Protocol. The Internet standard host-to-host mail transport protocol. It traditionally operates over TCP/IP using port 25. SMTP does not provide a mailbox facility, or any special features beyond basic mail transport.

SNMP. Simple Network Management Protocol. A TCP/IP protocol to enable managing remote hosts.

SPR. Software Problem Record. Lotus software problem tracking record.

SSL. Secure Sockets Layer. Security protocol for the Internet and intranets that provides communications privacy and authentication for Domino server tasks that operate over TCP.

TCB. Task Control Block. An OS/390 construct that represents a dispatchable unit of work. One address space can create multiple tasks. A USS thread creates a TCB in OS/390.

TCP/IP. Transmission Control Protocol/Internet Protocol. Network protocols that define the Internet. Originally designed for UNIX, TCP/IP software is now available for every major computer operating system.

TSO/E. Time Sharing Option/Extended. The principal interactive interface to OS/390

UNIX. An operating system developed at Bell Laboratories (trademark of UNIX System Laboratories, licensed exclusively by X/Open Company, Ltd.).

URL. Uniform Resource Locator

Webmail. Allowing a browser to access a mail file as HTML.

149

Index

Symbols/etc 49/notesdata 52/u 50/usr/lpp/lotus 52/var 49

Numerics3172-3 Interconnect Controller 633174 Networking Server 633380 293745/3746 Communications Controller 633990/9390 309672 116, 124

Aaccess level 80ACS routines 42active users 117, 123activity

cache 32DASD 32

address space 5, 6, 26name 7reusing 6your naming convention 8

addressing, virtual IP (VIPA) 65administrator 1Administrator client 48, 85AdminPInterval 134Advanced Services, Domino 93AF_INET sockets 62ALOCPROD 16AMgr_DocUpdateAgentMinInterval 134Amgr_DocUpdateEventDelay 134Amgr_NewMailAgentMinInterval 134Amgr_NewMailEventDeley 134analysis program 113APPL% 24assistance, capacity planning 126ASSORTEDPARMS 61authorization failure 17automount facility 50availability 89, 93average workload 117

Bbackground 6backup utility 74batch 12billing 67

file 44blocks transferred 27bottlenecks 44BPX.SMF FACILITY class 97

© Copyright IBM Corp. 2000

BPX.STOR.SWAP facility class 70BPXAS 6, 7BPXPRMxx 5, 9, 20, 21, 36, 62, 69, 118browser, Web 67buffer, TCP/IP 60

Ccaching 18, 29, 43

activity 32DASD 29performance 27

CALLS390 126capacity 9

planning 1, 115recommendations 115

processor 11, 116capacity planning 118capping, LPAR 22case, program name 9central storage 14, 119, 120, 121, 124channel

path 32utilization 29, 32

characteristics, Domino 1Chronos 81CICS 1, 12, 118Cladmin 93class

report 130service 11, 129

classification rules 130cldbdir.nsf 93client

replication 79Web 67

client-server 57, 67Clrepl 93CLUSTA4.NSF 93cluster administration process 93Cluster Database Directory Manager 93clustering 64, 93

components 93cost of 94monitoring 94performance 93replication 93

CMD field 9command

show tasks 78statistics 112

compact 67, 82compatibility mode 6, 11, 12, 23confighfs 37connections, user 7Console_Loglevel 133consolidation 89contention 7, 48

151

copytree utility 48cost of partitioning 89CPU 12

busy 20resource 90, 94time 90, 94, 115, 117utilization 19, 20, 58, 82

CSA 63CSM buffer pool 63

DD A,L 7, 8D OMVS,A=ALL 8D OMVS,U=userid 8DASD

3380 29activity 32administrator 1average response time 20cache 33DCME 29fast write (DFW) 30I/O rate 115, 123monitor 32performance 29recommendations 29response time 19, 32space 83, 89, 94, 115, 122, 124tuning 29volume 43

virtual 30data

class 42collection 29directory, Notes 44

Data Facility System Managed Storage (DFSMS) 42data set

HFS 29, 42data transfer 45database

corruption 74index 80link 44mail 123maximum size 48monitor 45size 111statistics and reporting 112usage 111view 80

Database.Database.BufferPool.Maximum.Megabytes 71Database.Database.BufferPool.PerCentReadsInBuffer 72dataset

HFS 29, 42, 43monitoring utilization 55

dataspaces 5DB2 1DCME 30, 43debug_outfile 54

decsext 17, 70Default_Index_Lifetime_Days 134definitions, workload manager 129delete document 82Deleted Data Space Release (DDSR) 30demand paging 20designer access 80df command 55DFSMS 12

buffer manager 12statistics 27

DFSMS/MVS 34, 42DFSMSdss 29, 30, 44DFW 30DINTV parameter 21directory 83

level 44link 44Notes data 44root 29UNIX 44

disk space 83, 89, 94, 122, 124document

delete 82read/write 45

DOMAINAME, NETWORK 62Domino

Administrator client 48, 75, 85Advanced Services 89, 93capacity

network 123partitioned servers 122

CD 87characteristics 1clustering 64customer results 125Enterprise Server (DES) 89expertise 5HTTP server 86

connector 86I/O statistics 45initialization parameters 68interfaces to OS/390 5link 44mail.box 84monitoring 97multiple servers 89network statistics 65parameters 67, 133partitioned server (DPAR) 9, 11, 44, 89, 115, 122Poughkeepsie server 123production server 123program module 16recommendations 67response time 5server tasks 76sizing help 126, 127statistics 97tasks 78team 1


tuning 67Web server 85workload 1

Domino Directory 84domliba 17, 70DP 13DPAR (see Domino partitioned server) 9, 122DPOOLSIZE 68, 69duplicate job name 8Dynamic Alias Management 12dynamic cache management enhanced (DCME) 30, 43dynamic LPA (DLPA) 70

EECSA MAX 10, 63efficiency of partitioning 92ELPA 16eNetwork Communications Server (CS/390) 57engines

multiple 79, 82single 79, 82

Enterprise Storage Server (ESS or Shark) 29, 30Storwatch Enterprise Storage Server Specialist 31

ESCON channel 29, 32ESQA 14, 119, 120, 121ESS 12

PAV feature 12estimating

processorcapacity 116storage 118

resources 115event 111

database 111task 112

events4.nsf 111example, workload manager 129expanded processor storage 2expanded storage 14, 124export 113extended link pack area (ELPA) 16extents

multiple 44

Ffactor, peak load 117failover 94file

activity 27high-activity 43name 28performance 27system lookup 34UNIX 29, 44

file systemhigh speed 49

FILESYSTYPE TYPE(HFS) 118FIXED

MAX 10, 63

FIXUP utility 67, 74Fixup_Tasks 133flashes 63flexibility 9FORKCOPY 10fsview utility 55FT_Intl_Setting 134ftgtr 17, 70FTP 62full text indexing 81

Gglobal network 78global resource serialization (GRS) 48GLOBALCONFIG 61goal mode 6, 11goals 131

Hhelp file 44HFS

backing up 125data set 27, 29, 42, 43planning 123

HFS (see hierarchical file system) 42hierarchical file system (HFS) 16, 42

directory structure 47multi-volume dataset 46

high-speed file system 49hiperspaces 43host 64HTML 85HTTP 17, 85, 86

server 86connector 86

task 86

II/O

activity 94blocks transferred 27Domino statistics 45performance 27priority management 12priority queueing 30queue 32rate 32response time 29, 32, 43

IBM 3390 model 9 49IBM HTTP Server Connector 86ID, user 7IDCAMS LISTDATA 33IEAICSxx 13IEAIPSxx 12, 13IEFUSI 57

exit 57IMAP 17, 115, 119

storage of clients 121

153

importance 131IMS 1, 118independent, platform 1indexing 67, 80

tasks 82initialization parameters, Domino 68Internet 17

Cluster Manager (ICM) 93interval 25

recording (SMF) 25synchronization 26

INTVAL 25IOSQ time 30IP

network 63virtual addressing (VIPA) 65

IPCSEMNIDS 10IPCSEMNOPS 10IPCSHMMPAGES 10, 69IPCSHMNIDS 10IPCSHMNSEGS 10IPCSHMSPAGES 10IPS 12isolation

storage 13workload 89

Jjob

scheduler 2job name

duplicate 8jobname, duplicate 7journal file 44

Kkernel activity report 20kerninfo 34

Llatch 35libnotes 17, 70link

database 44directory 44Domino 44

link pack area (LPA) 16, 17LISTDATA 33load factor, peak 117load http 86load stats 112local access 80lock path 44log 29, 111

Notes 45, 65, 80statistics and events 111

log.nsf 45, 65, 80, 94, 111Log_Replication 80, 111, 133

Log_Sessions 80, 111, 133Log_Tasks 133Log_ViewEvents 133logical control unit (LCU) 31logical partitioning (LPAR) 9, 21, 89lookaside 35lookup, file system 34Lotus Domino 2lower case 9LPA 16, 17, 70LPAR 6, 9, 14, 122, 124

RMF report 21

Mmail 79

database 123dataset 53routing 111workload 14

mail.box 84MailCluster_Failover 94MailCompactDisabled 134MailMaxThreads 134management class 42manager access 80marks

unread 84MAXASSIZE 10MAXCPUTIME 10MAXFILEPROC 10maximum

transactions 67users 92

MAXQUEUEDSIGS 10MAXSHAREPAGES 10MAXSOCKETS 10, 62MAXTHREADS 10MAXTHREADTASKS 10MAXUSER 10may cache policy 43MDFYPROD 16measurements 115memory 7

processor 13metadata 35microcode level 5migration age 22millisecond response (MSR) 43MONITOR-II, RMF 63monitoring 32, 71, 97

DASD 32cache 33

database 45Domino 97network connection 63replication 80statistics 111TCP/IP 57

mount 9mount points 49


MP 9MPL parameter 12multiple

allegiance 30Domino servers 9, 89engines 79, 82extents 44processors 9threads 9

multiprocessor 9, 79, 82MVS 2

virtual IP addressing (VIPA) 65

Nname

address space 7program 8UNIX file 28

NETSTAT 58, 60network

administrator 1capacity 115connection, monitoring 63DOMAINAME 62global 78interfaces 64performance 64planning 123protocol 57recommendations 57specialist 79splitting 64statistics, domino 65traffic 79, 94, 123tuning 57

NONSWAP_SERVER_TASKS 70non-swappable 69Notes

administrator 1, 67, 80, 81client 119

number of users 115client storage 119data directory 44log 45, 65, 80, 97, 111mail.box 84ports dialog box 11

notes.ini 6, 9, 67, 68, 69, 78, 80, 92, 102, 133, 135change 68performance parameters 133sample 135

Notes_SHARED_DPOOLSIZE 68, 69Notesbench 2notesdata 44noteslog 53NSD_LOGDIR 54NSF buffer pool 71

size 71NSF_Buffer_Pool_Size 71, 133NSF_Buffer_Pool_Size_MB 71number of users 112, 115, 116, 122

OOBEY 58ODS (on-disk structure) 83OMVS 12, 13

kernel activity report 20OMVSKERN 11, 13Open System Adapter (OSA) 63OS/390

address space 6image 9interfaces 5memory 5monitoring 1parameter 9performance specialist 1recommendations 5Security Server (RACF) 97system programmer 79tuning 5UNIX System Services 1, 2

OSA 63OSPF dynamic routing protocol 65overruns 20, 21

Ppaging 5, 14, 118

data sets 15demand 20rate 13storage 119, 120, 121use 19

Parallel Access Volumes (PAVs) 31parallel channel 29, 32parameter

clustering 94Domino 67, 133Domino initialization 68OS/390 9recommended value 9settings 1Web server 86

PARMLIB 5, 12, 69, 118partition data report 21partitioning 89

cost of 89Domino 9, 11, 44, 122efficiency 92logical 122, 124resource 89server 122

pathactivity 32channel 32lock 44

PAV feature of ESS 12PC

operating systems 2PDSE 42peak

155

load factor 117time 67transaction rate 112workload 118

Perform Locked Operation 116performance 81, 92

clustering 93constraining 1DPARs 89goals 131group 24I/O 27network 64OS/390 UNIX System Services 1parameters 133specialist, OS/390 1test 116

physical thread 7, 69pilot 117PING 57, 58planning

assistance 126capacity 115DASD 122HFS 123network 123storage 118

platform independent 1PLO 116POOL definition 63POP3 17, 115, 119

client storage 120POSIX

standards 35postprocessor 38Poughkeepsie

Domino server 123PR/SM Logical Partition (LPAR) 9, 122, 124priority 5, 11, 12

TCP/IP 62processor

capacity 11storage 89, 118

processors, multiple 9production mode 120production server 123profile, TCP/IP 58program

module, Domino 16name 8

protocolnetwork 57

Public Address Book 44PUTINLPA 16, 17

Qqueue, I/O 32

RRACF 70RAMAC Virtual Array (RVA) 30rate, transaction 111, 116real memory 5real-time replication 93recommendations 1

capacity planning 115DASD 29Domino 67network 57OS/390 5

recovery 74REGION 57

TCP/IP 62registered user 125registered users 116, 125replica 17Replica.Cluster.WorkQueueDepth 95replication 67, 78, 111

client 79frequency 80monitor 80real-time 93servers 79

replicator task 79report 24

class 24, 130LPAR 21performance group 24summary 19workload 24

reporter 112requirements, Domino 1resource

compact 83contention 7estimate 115indexing 82partitioning 89

Resource Measurement Facility (RMF) 18, 58RMFMON (Monitor II) 19RMFWDM (Monitor III) 19

response time 5, 10, 23, 131DASD 32I/O 29, 32, 43

restart timereduced 125

restricted, server 94RMF 12, 32

cache subsystem activity 33LPAR report 21MONITOR-II 63summary report 32workload report 24

RMFMON (Monitor II) 19RMFMONITOR 63RMFWDM (Monitor III) 19root directory 29router 9, 17, 84


routing, mail 111rules, classification 130RVA 30

SS/390

hardware 2logical partitioning (LPARs) 115

scalability 89, 93scheduled task 67, 79, 81SecureMail 134server

configuration document 68console traffic 80process 7production 123replication 79tasks 76

Server.Trans.PerMinute.Peak 112Server.Users 112Server_Cluster_On 94SERVER_ENABLE_THREADPOOL 69Server_Max_Concurrent_Trans 67, 133Server_MaxSessions 67, 133Server_MaxUsers 92SERVER_POOL_TASKS 69, 102, 118Server_Restricted 94ServertasksATx 81service

class 11, 24, 129rates 24

services, testing 127Set Configuration 68SEZAINST sample library 57shared memory 69SHARED_DPOOLSIZE 68SharedMail 134sharing 89Shark (see Enterprise Storage Server) 29show cluster 93show statistic 86, 95, 112

sample 137show statistic replica 95show tasks 78, 95simple tests 2single engine 79, 82sizing

Domino 126SMF 25, 34, 117

type 108 record 69, 97type 30 record 26, 34type 42 record 34type 92 record 27, 34

SMF type 108 recordenabling 97field mappings 98

SMF30AIS 27SMF30CPS 27SMF30CPT 27SMF30EXN 26

SMF30JBN 26SMF30OFR 27SMF30OFW 27SMF30OPR 27SMF30OPW 27SMF30OSR 27SMF30OSW 27SMF30PGM 26SMF30TEP 27SMFPRMxx 28, 97SMP (see symmetric multiprocessors) 9SMTP 17SNAP/SHOT capacity planning tool 127SnapShot backup/dump utility 30socket descriptor 7space

DASD 89, 122, 124unused 82

splitting the network 64Spool Display and Search Facility (SDSF) 58SQA 63SRB 24SSCH 27Start Subchannel 27started task 97statistics 29, 111

and events log 111and reporting database 112command 112database 111file 44I/O 45monitoring 111, 112network 65TCP/IP 61UNIX 34

statrep.nsf 112sticky bit 16storage

central 14class 42, 43expanded 14extended private 13group 42have enough 5IMAP clients 121isolation 13Notes clients 119POP3 client 120processor 118use 19, 26virtual 13WebMail clients 121

storage group 42Storwatch Enterprise Storage Server Specialist 31stress testing 127summary report, RMF 19, 32swapping 12symmetric multiprocessors (SMPs) 2, 9sync daemon interval 36

157

SYNCVAL 25SYS1.PARMLIB 5, 9, 12

BPXPRMxx 118SYSBMAS 12syslogd daemon 49SYSRPTS 24System Management Facilities (SMF) 18, 25system resources manager (SRM) 12

Ttask control block (TCB) 7tasks

Domino 6, 67non-peak times

adminp 77billing 77stats 77update 77

non-swappable 69replicator 79scheduled 67, 79, 81

catalog 77compact 67fixup 67index 67updall 67update 67

show 78TCB (see also task control block) 24TCP/IP 9, 12, 57

AF_NET sockets 62buffer 60dispatching priority 62monitor 57priority 62profile 58region size 57, 62session timeout 11statistics 61tuning 62

TCPIPSTATISTICS 58, 61team 1

Domino 1TechXPress 126telnet 49, 62template 84temporary file system (TFS) 49testing

performance 116services 127simple 2stress 127

thread 6, 69, 78multiple 9physical 69pool model 7

timeCPU 115, 117peak 117response 131

timing out 5, 12tips, OS/390 UNIX Services 5Tivoli Data Protection for Lotus Domino R5 74TRACERTE 58tracing 35traffic, network 79, 123transaction

maximum 67number of 45rate 115, 116

transaction logging 74, 75, 125benefits 74setup 75

transfer, data 45TSO 12tuning 1

DASD subsystem 29Domino 67network 57OS/390 5partitioning and clustering 89TCP/IP 62

type 30 SMF record 34type 42 SMF record 34type 92 SMF record 34

UUNIX

directory 44file 28, 29, 44standards 35statistics 34System Services 1

unmount 9unread marks 84unreferenced interval count (UIC) 22unused space 82updall 67, 80, 81update 67, 80, 81Update_No_Fulltext 134Update_Suppression_Limit 82, 134Update_Suppression_Time 82, 134updaters 134upper case 9URL requests 87useful Domino statistics 112user

registered 125user connections 7user ID 7users

active 117, 123maximum 92number of 112, 115, 116, 122registered 116, 125

utilization, channel 29, 32

Vvelocity 11, 23, 131


view, database 80virtual

DASD volume 30storage 13thread 7

ID 7Virtual Array, RAMAC 30virtual IP addressing (VIPA) 65volume, DASD 43

virtual 30

Wwasted space 82Web 5

browser 67, 85server 85, 86

WebMail 119client storage 121

weight, LPAR 22working set 13, 14workload

average 117Domino 1isolation 89mail 14manager 6, 11

goal mode 11, 23, 129peak 118RMF report 24

Workload Manager (WLM) 12workload manager (WLM)

compatibility mode 11goal mode 11

159


IBM Redbooks review

Your feedback is valued by the Redbook authors. In particular we are interested in situations where a Redbook "made the difference" in a task or problem you encountered. Using one of the following methods, please review the Redbook, addressing value, subject matter, structure, depth and quality as appropriate.

• Use the online Contact us review redbook form found at ibm.com/redbooks • Fax this form to: USA International Access Code + 1 845 432 8264 • Send your comments in an Internet note to [email protected]

Document NumberRedbook Title

SG24-5149-02Lotus Domino for S/390 Release 5: Performance Tuning and Capacity Planning

Review

What other subjects would you like to see IBM Redbooks address?

Please rate your overall satisfaction:

O Very Good O Good O Average O Poor

Please identify yourself as belonging to one of the following groups:

O CustomerO Business PartnerO Solution DeveloperO IBM, Lotus or Tivoli EmployeeO None of the above

Your email address:The data you provide here may be used to provide you with information from IBM or our business partners about our products, services or activities.

O Please do not use the information collected here for future marketing or promotional contacts or other communications beyond the scope of this transaction.

Questions about IBM’s privacy policy?

The following link explains how we protect your personal information.ibm.com/privacy/yourprivacy/



http://www.ibm.com/privacy/yourprivacy/


http://www.ibm.com/privacy/yourprivacy/

(0.2”spine)0.17”<->

0.473”90<->

249 pages

Lotus Domino for S/390 Release 5: Perform

ance Tuning and Capacity PlanningLotus Dom

ino for S/390 Release 5: Performance Tuning and Capacity Planning

®

SG24-5149-02 ISBN 0738421146

INTERNATIONAL TECHNICALSUPPORTORGANIZATION

BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE

IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment.

For more information:ibm.com/redbooks

Lotus Domino for S/390 Release 5:Performance Tuning andCapacity PlanningHow to plan for Domino capacity on an OS/390 system

How to tune an OS/390 system for Domino performance

How to tune and monitor Domino for OS/390

This IBM Redbook helps OS/390 system programmers and Domino administrators to monitor and tune Domino for S/390. It identifies factors that affect Domino R5 performance on S/390, and supplies recommendations to tune the configuration and parameter settings for optimal performance.

In addition, it gives guidelines for capacity planning. A background in OS/390 and experience with Domino is assumed.

This updated redbook describes the new SMF type 108 records and covers clustering, shared CPs, and RPCs.




Lotus Domino for S/390 Release 5: Performance Tuning … Tuning and Capacity Planning Mike Ebbers...

Documents

Transcript of Lotus Domino for S/390 Release 5: Performance Tuning … Tuning and Capacity Planning Mike Ebbers...