Lotus Domino

What is the difference between running NSD and NSD -info?

Technote (FAQ)

ProblemWhat is the difference between running "NSD" and "NSD -info"?

SolutionThe difference is in the information that is collected:

NSD -info Running an "NSD -info" on a Domino server displays:

system information notes.ini environment variables data and program directory contents disk drive system memory statistics."

Bottom line: When you want to collect only system information, run "NSD -info".

NSD Running "NSD" captures:

all the system information that you get with "nsd -info" as well as the current processes that are running

Bottom line: When you see a performance issue and need to get a sense of both the system information and what is running, run an NSD.

Another way to position the distinction To get the list of processes that are running, NSD uses ("touches") the PID.NBF file, which is the Process ID table of all Notes/Domino processes.

The "NSD -info" does not touch the "pid.nbf" file.

For more information on the pid.nbf file, refer to the technote titled, "What is the PID.NBF file?" (#1229311)

Troubleshooting Windows NSD:

When NSD is not generating a ".log" file successfully, it is a good idea to run "NSD -info".

Running NSD -info avoids touching the "pid.nbf" file. If it runs to completion, it shows that the nsd executable is capable of running. Perhaps there is a problem with the PID.NBF file. Refer to the technote titled, "What is the PID.NBF file?" (#1229311).

What is the PID.NBF file?

Technote (FAQ) ProblemWhat is the PID.NBF file? Solution

http://www.ibm.com/support/docview.wss?rs=203&uid=swg21229311


The PID.NBF file is the Process ID table of all Notes/Domino processes. For the lifetime of a Notes client/server, each process start or shutdown causes the process ID (PID) to be added to or removed from the PID.NBF.

In a Win32 environment, NSD uses the pid.nbf to determine, in case of a crash, if the failing process is a Notes/Domino process. The PID.NBF file is also used by Domino in a partitioned server environment. The file allows the processes associated with each partitioned server to be identified.

The PID.NBF is located in the Domino\Data directory. The PID.NBF file can be renamed or deleted. The file is recreated the next time the Domino server is restarted. If the PID.NBF file is deleted while the Domino server is running, NSD will be unable to determine the Notes processes to attach to until the server is restarted (when the new PID.NBF is created).

How does the Compact function behave when using Database Properties

Technote (FAQ)

ProblemIn Lotus Domino®, if you use the File menu --> Database --> Properties -> Info tab --> Compact option, how does Compact work and behave?

SolutionWhen triggered from the Database Properties Info panel, three kinds of compacting can occur: copy-style compaction (-L), in-place compaction (-b), and in-place with file size reduction compaction (-B). Note the case-sensitivity of the commands.

NOTE: Compacting a database does not necessarily reduce its size.

Types of compacting:

-- If the Notes database requires structural changes (typically a change of ODS [on-disk structure]). then a copy-style compaction will be performed (equivalent to -L).

-- If the Notes database is logged (Transactional Logging enabled), then a simple in-place compaction will be performed (equivalent to -b).

-- If the Notes database is not logged (Transactional Logging disabled), an in-place with file size reduction compaction will be performed (equivalent to -B).

NOTE: In Domino 7 releases, the request from the Notes Client to compact a database was changed. This functionality is now handled by the Update task so that the connection to the Domino server is not tied up waiting for a mail file to be compacted. The Update task places the Compact request in a deferred queue and processes it when it has free cycles.

Compact operation not performed immediately on a busy server

Technote (FAQ)

QuestionYou want to execute a compact operation on a Lotus Notes®/Lotus Domino® database from within Notes. In the Notes database, from the File menu, you choose Database --> Properties --> Info panel --> Compact. The following message appears in the status bar:

"Your request to compact this database is now being carried out."

However, after waiting several minutes, you notice that the size of the database reported on the Properties Info panel has not changed.

You verify that the Notes database is not logged (Transactional Logging is enabled) and that you have access to the Compact button (manager access in the ACL) since these could be reasons why the database size has not reduced.

AnswerThis issue was reported to Quality Engineering as SPR# JLBS6QKTAF and has been addressed in Domino release 7.0.3.

Excerpt from the Lotus Notes and Domino Release 7.0.3 MR fix list (available at http://www.ibm.com/developerworks/lotus ):

SPR# JCHN6VM6G9 - Added the new Notes.ini variable "Debug_Compact_Immediately" to allow compact requests from clients to be performed immediately (like in R6) rather than queued. (Regression in 7.0.)

Refer to the Upgrade Central site for details on upgrading Notes/Domino.

Additional InformationIn Domino 7 and 8 releases, the request from Notes to compact a database was changed. This functionality is now handled by the Update (updall) task so that the connection to the Domino server is not tied up waiting for a mail file to be compacted. The Update task places the Compact request in a deferred queue and processes it when it has free cycles.

On a busy Domino server, the Update task might take from minutes to hours to process requests for compacting, especially during peak hours.

You can issue the following Domino console command (without quotes) to see how many compact requests were executed by Update: "show stat Update.PendingList.Processed.Compactions"

In some cases, increasing the number of Update tasks and configuring Update to use a separate thread to do full text indexing reduces the time involved.

To enable multiple Update tasks, add the following parameter to the server's notes.ini file: Updaters = # (number of desired Update tasks to run on the server).

To configure Update to use a separate thread to do full text indexing, add the following parameter to the server's notes.ini file: Update_Fulltext_Thread = 1.

NOTE: For the Update_Fulltext_Thread parameter, setting this variable to 1 allows use of a separate thread to do full text indexing so that long full text indexes don't delay view updates. By default, view updates and full text index updates are driven by the same thread. View updating will still work in "updall."

What is a Notes database?

A Notes database is a single file that contains multiple documents. A database generally contains information about a particular area of interest, such as the forms and policy documents for a department, or it might contain documents of a similar type, such as e-mail messages. In addition, some companies create "discussion databases," where employees can

http://www.ibm.com/software/lotus/support/upgradecentral/index.html

http://www.ibm.com/developerworks/lotus

post responses to particular topics. Documents in Notes databases contain rich text, pictures, objects, and many other types of information.

Most databases are stored on one or more Domino servers, accessible by many users. These are called shared databases. Databases that are used only by you and reside on your computer are called local databases.

Notes comes with templates you can use to create your own databases. These templates have the file extension NTF. The Notes databases have the extension NSF.

Notes databases also have access control lists (ACLs) that control the actions people, groups, and servers can perform in the database. For example, one person may be able to create and read documents in a database, where another person may be allowed only to read documents in the same database.

Known limitations of Notes and Domino 7.x

Technote (FAQ)

ProblemWhat are some of the known limitations of Notes and Domino 7.x?

Please refer to this document for similar information in Notes/Domino 6.x: "Known limitations of Notes and Domino 6.x" (#1159187).

SolutionThe following tables summarize the known maximum limits of various Notes and Domino 7.x features as noted in the Notes/Domino documentation. Note: Specifically, the first table originated in the Notes 7 Client Help. The second table originated in the Domino 7 Designer Help, and has been edited to contain only unique items that are not also listed in the Notes Client table.

GETTING STARTED

Limits of Notes

Item Limit

What is the maximum size of a database?

The maximum OS file size limit (up to 64GB)

What is the maximum size of text fields?

32KB (storage); 32KB displayed in a view's column

http://www.ibm.com/support/docview.wss?uid=swg21159187

What is the maximum size of a rich text field?

Limited only by available disk space up to 1GB

What is the maximum size of a single paragraph in a rich text field?

64KB

What is the maximum amount of Text (Summary) data per document?

64KB

How many levels of responses in a hierarchical view; how many documents at each level?

31 levels; 300,000 documents

How many columns can be included in one table?

64

How many rows can be included in one table?

255

How many views can be added to a database?

No limit; however, as the number of views increases, the length of time to display other views also increases

How many columns are allowed in a view?

289 ten-character columns; dependent upon # or characters per column

How many documents can be imported into a view?

Documents totaling at least 350K

How many cascading views are allowed in a database?

200

What is the maximum value (in inches) you can enter for margin size?

22.75

What is the maximum value (in inches) you can enter for page size cropping?

46

What is the maximum point size you can select/print?

250

How many documents are allowed in one view?

Up to the size of the database, with a maximum of 64GB

What is the maximum number of documents that can be exported to Tabular Text?

Limited only by available disk space

What is the maximum number of entries in an Access Control List?

~950 names (total ACL size is limited to 32767 bytes)

What is the maximum number of roles in an Access Control List?

75 Roles

What's the maximum password length allowed on an ID?

64 bytes (63 characters for single-byte character sets but only 21 characters for some double-byte languages)

What is the maximum number of contacts allowed in a group in the Personal Address Book?

32K of names in the Members text field

What is the maximum number of recipients in a single mail message?

For individual names and private groups which expand locally, 15KB; for public groups which expand on a server, 5MB

APPLICATION DESIGN

Table of Notes and Domino known limits

The following table summarizes the known maximum limits of various Notes and Domino features. Item Maximum limit

Characters in names Database Title: 96 bytes

Filenames: On Windows® and UNIX® platforms minimum of 255 and/or OS limits; on local Macintosh workstation 31 Field names: 32 View names: 64 Form names: 32 Agent names: 32

Fields in a database ~ 3000 (limited to ~ 64K total length for all field names). You can enable the database property "Allow more fields in database" to get up to 22,893 uniquely-named fields in the database.

Forms in a database Limited only by database size.

Margin size (in inches) 46

Page cropping size (in inches)

46

Point size to select or print

250

Documents in a view Up to the maximum size of the database

Documents that can be exported to tabular text


Authorized users on a multiple password ID

8 users

Outline entries in an outline

~21,000 entries

Notes allows you to paste a maximum of 255 rows into a table. Exceeding this limit will yield the error, "No more than 255 rows allowed in a table".

Related informationKnown limitations of Notes and Domino 6.x

Cross Reference information

Segment Product Component Platform Version Edition

Messaging Applications

Lotus Domino

Lotus Domino Designer

7.0

Support & downloads >

Known limitations of Notes and Domino 6.x Technote (FAQ)

ProblemWhat are some of the known limitations of Notes and Domino 6.x?

Please refer to this document for similar information in Notes/Domino 7.x: "Known limitations of Notes and Domino 7.x" (#1222807).

Solution

The following table summarizes the known maximum limits of

http://www.ibm.com/support/docview.wss?uid=swg21222807

http://www.ibm.com/support/us/

http://www-1.ibm.com/support/docview.wss?rs=0&uid=swg21159187

various Notes and Domino 6.x features as noted in the Notes Domino documentation: Item Maximum limit

Database size The maximum OS file size limit -- (up to 64GB)

Text field size 32KB (storage); 32KB displayed in a view's column

Rich text field size Limited only by available disk space up to 1GB

Response levels in a hierarchical view; number of documents per level

31 levels; 300,000 documents

Characters in names Database Title: 96 bytes Filenames: On Windows® and UNIX® platforms minimum of 255 and/or OS limits; on local Macintosh workstation 31 Field names: 32 View names: 64 Form names: 32 Agent names: 32

Fields in a database ~ 3000 (limited to ~ 64K total length for all field names). You can enable the database property "Allow more fields in database" to get up to 22,893 uniquely-named fields in the database.

Columns in a table 64

Rows in a table 255

Views in a database No limit; however, as the number of views increases, the length of time to display other views also increases

Forms in a database Limited only by database size.

Columns in a view 289 ten-character columns; dependent upon # or characters per column

Documents imported into a view

Documents totaling at least 350K

Cascading views in a database

200

Margin size (in inches) 46

Page cropping size (in inches)

46

Point size to select or print

250

Documents in a view Views are architecturally limited to 130GB. However, since databases are limited to 64GB, the views cannot grow larger than the db. So in practice, views can grow as large as the db size limit of 64GB. **

Documents that can be exported to tabular text


Entries in an Access Control List (ACL)

~950 names (ACL size is limited to 32767 bytes)

Roles in an Access Control List

75 Roles

ID password length 63 characters

Authorized users on a multiple password ID

8 users

Outline entries in an outline

~21,000 entries

** Note: The information in the Help documentation for release 6.5.2 (and below)incorrectly states the "Documents in a view" specification. The documentation error was reported to Quality Engineering (SPR SVRO3JV3UN)

and addressed in the documentation available with release 6.5.3 and above.

Note: This table originated in the Domino 6.x Designer Help.

Related informationKnown limitations of Notes and Domino 5.xKnown limitations of Notes and Domino 7.x

Cross Reference information

Segment Product Component Platform Version Edition

Messaging Applications

Lotus Domino

Lotus Domino Designer

Platform Independent

6.5, 6.0

Results of the First Server SetupRunning the setup of the first server modifies the notes.ini file so that it contains all of the required information to start the server. It also creates several files, Domino databases, and Domino documents. The following tables outline in more detail what is created, where it is stored, and what its purpose is.

The following tables discusses some of the settings that are added by the setup program to the notes.ini file:



Setting Example Description

NotesProgram NotesProgram=c:\program files\lotus\domino

Executable directory name.

Directory Directory=c:\program files\lotus\domino\data

Data directory name (for non-partitioned servers).

ServerKeyFileName

ServerKeyFileName=server.id

Server ID file name.

Domain Domain=Adventure Domain name.

ServerTasks ServerTasks=Update, Replica, Router, Stats, AMgr, Adminp, Sched, CalConn, RunJava ISpy, HTTP

Tasks to start at server startup.

ServerTasksAtx

ServerTasksAt1=Catalog, Design

Tasks to run once per day at scheduled times.

Port TCPIP=TCP, 0, 15, 0 Configuration settings for the port driver.

Ports Ports=TCPIP Enabled ports.

DisabledPorts DisabledPorts=LAN0,SPX,COM2,COM4,COM5

Disabled ports.

$$HasLanPort $$HasLanPort=1 The server has or does not have a LAN port.

Log Log=LOG.NSF,1,0,7,40000 Settings for the log file: file name, log option (default is 1, log to the console), 0 (the third setting not used at this time), number of days to retain log information, maximum number of bytes log file can contain.

LOG_task Log_Replication=1 Level of detail to be logged for that task.

Passthru_LogLevel

Passthru_LogLevel=0 Level of detail to be logged about network connections, including passthrough connections

The setup creates the following Domino databases, all stored in the data directory:

Database File Name Description

Certification Log

certlog.nsf

Logs all registrations of servers and users.

Domino Directory

names.nsf Used to administer the Domino network.

Server Log log.nsf Stores the log of messages that have been displayed on the server console in the last seven days.

Reports reports.nsf

Generates and stores mail usage reports.

Administrator's mail file

mail\msmith.nsf (for example)

The first administrator's mail database is also created, in a sub-folder of the data directory called mail. The file name is based on the administrator's name.

The following documents are created in the Domino Directory:

Document Description

Person document Person document for the first administrator.

Server document Server document for the first server.

Group document Group document for the LocalDomainAdmins group. The name of the first administrator is automatically added to this group.

Group document Group document for the LocalDomainServers group. The name of the first server is automatically added to this group. Any additional servers registered later will also automatically be added to the group. The group type is initially set to "Multi-purpose". This should be changed to "Servers".

Group document Group document for the OtherDomainServers group. This group is initially empty. It is intended to list servers in other Domino domains that connect with this domain. The group type is initially set to "Multi-purpose". This should be changed to "Servers".

Certifier document

Certifier document for the organization.

Certifier document

Certifier document for the first organizational unit. This document is only created if the option to create an OU was selected.

The following IDs are created:

ID Stored in the...

Default File Name

Description

Organization

Data directory cert.id Certifier ID for the entire Domino organization.

Organizational Unit

Domino Directory, attached to the Certifier document

oucert.id Certifier ID for the first OU. This file is only created if the option to create an OU was selected.

Server Data directory server.id Server ID for the first server.

Administrator

Domino Directory, attached to the Person document. May also be stored as a file, if that option was selected

user.id User ID for the first administrator.

The following tables describe the options you can use with the Compact server task. The first column lists the options as they appear when you run Compact using the Task - Start tool or the Files tab in the Domino Administrator. The second column lists the equivalent command-line options that you use when you run Compact using a console command or using a Program document.

Compact - Basics

Option Command-line equivalent

Description

Compact only this database or folder

(To specify databases to compact using the Files tab, select the databases in the files pane.)

database path

Specify any additional options after the database path.

To compact a database in the Domino data folder, enter the file name, for example SALES.NSF. To compact databases in a folder within the data folder, specify the database path relative to the data folder. For example, to compact all databases in the folder DATA\SALES, specify SALES.

If you choose "Compact all databases" (or don't specify a database path at the command line) Compact compacts all databases in the data folder and in folders within the data folder.

For more information on database path, see (1)Running Compact using a console command.

Compact - Options


Description

Compact database only if unused space is greater than x percent

-S percent Compacts all databases with a specified percent of unused space. For example, if you specify 10, databases with 10% or more recorded unused space are compacted. Note that the unused space calculation is not always a reliable measure of unused space.

Discard any built view indexes

-D Discards built view indexes. Use this option to compact databases just before you store them on tape, for example. Does copy-style compacting.

Keep or revert database to previous format

-R Compacts databases without converting to the current release file format of the server that stores the databases or reverts databases in the current release file format to the previous release file format. For example, on Domino 6 and more recent servers, this option compacts Domino 5 databases without converting them to the Domino 6 file format and converts Domino 6 databases to the Domino 5 file format. This option uses copy-style compacting.

Compact - Style


Description

In-place (recommended) -b Uses in-place compacting and recovers unused space without reducing the file size, unless there's a pending structural change to a database, in which case copy-style compacting occurs. This is the recommended method of compacting.

In-place with file size reduction

-B Uses in-place compacting, recovers unused space and reduces file size, unless there's a pending structural change in which case copy-style compacting occurs. If you use transaction logging, do full database backups after compacting completes.

Copy-style -c Uses copy-style compacting. Use this option, for example, to solve database corruption problems.

Copy-style: Allow access while compacting

-L Enables users to continue to access databases during compacting. If a user edits a database during compacting, compacting is canceled. This is useful only when copy-style compacting is done.

Copy-style: Ignore errors and proceed

-i Enables compacting to continue even if it encounters errors such as document corruption. Only used for copy-style compacting.

Compact - Advanced

The advanced compact options are not available through the Compact tool in the Files tab of the Domino Administrator.

Option* Command-line equivalent

Description

Document table bitmap optimization: Off

-f Disables "Document table bitmap optimization" database property. Does copy-style compacting.

Document table bitmap optimization: On

-F Enables "Document table bitmap optimization" database property. Does copy-style compacting.

Don't support specialized response hierarchy: Off

-h Disables "Don't support specialized response hierarchy" database property; in other words, support specialized response hierarchy. Does copy-style compacting.

Don't support specialized response hierarchy: On

-H Enables "Don't support specialized response hierarchy" database property; in other words, do not support specialized response hierarchy. Does copy-style compacting.

Enable transaction logging: Off

-t Disables transaction logging.

Enable transaction logging: On

-T Enables transaction logging. Use Compact - T when a database is open or closed. If you use Compact - T on a database that is closed, logging is enabled but the Compact is not logged until the database is opened; therefore, logging is not available until you reopen the database.

Don't maintain unread marks: Off

-u Disables "Don't maintain unread marks" database property; in other words, maintain unread marks.

Don't maintain unread marks: On

-U Enables "Don't maintain unread marks" database property; in other words, do not maintain unread marks.

* Select "Set advanced properties" before you enable or disable any of these properties.

Compact - Archive

When you use the document archiving tool to archive and delete documents in a database, you can use the following Compact options to archive documents if the database is located on a server and you've chosen the advanced archiving option "Automatically on server."

For more information on the server archiving option, see Archiving locally or on a server if you

have installed Lotus Notes 7 Help. Or, go to www.lotus.com/ldd/doc to download or view Lotus Notes 7 Help.

Option* Command-line equivalent

Description

Archive only -A Archives and deletes documents from a database without compacting the database.

Archive and then compact -a Archives and deletes documents from a database and then compacts the database.

Delete and then archive -j Deletes documents from a database and then compacts the database.

*The Compact tool in the Files tab of the Domino Administrator provides only the option "Archive database;" this option archives and then compacts.

See also(2)Ways to compact databases(3)Compacting databases(4)Database properties that optimize database performance(5)Running Compact using a console command

.TXN Files

ContentLotus Domino supports transaction logging and recovery. With this feature enabled, the system captures database changes and writes them to the transaction log. Then if a system or media failure occurs, you can use the transaction log and a third-party backup utility to recover your databasesIMPORTANT: Enabling transaction logging can improve server performance in most cases. Transaction logging saves processing time because it allows Domino to defer database updates to disk during periods of high server activity. Transactions are recorded sequentially in the log files, which is much quicker than database updates to random, nonsequential parts of a disk. Because the transactions are already recorded, Domino can safely defer database updates until a period of low server activity.

What is transaction logging?

Transaction logging keeps a sequential record of every operation that occurs to data. If a database becomes corrupted, you can "roll back" the database to a point before it was corrupted and replay the changes from the transaction log.

A single transaction is a series of changes made to a database on a server -- for example, a transaction might include opening a new document, adding text, and saving the document.

Transaction logging provides three main benefits:

- In most situations, you no longer need to run the Fixup task to recover databases following a system failure. Excluding Fixup results in quicker server restarts, since Fixup must check every document in each database, while transaction log recovery applies or undoes only those transactions not written to disk at the time of the system failure.

- Transaction logging saves processing time because it allows Domino to defer database updates to disk during periods of high server activity. Transactions are recorded sequentially in the log files, which is much quicker than database updates to random, nonsequential parts of a disk. Because the transactions are already recorded, Domino can safely defer database updates until a period of low server activity.

- Using transaction logging simplifies your daily backup procedure. You can use a third-party backup utility to perform daily incremental backups of the transaction logs, rather than perform full database backups.

IMPORTANT: Transaction logging works with databases in format ODS 41 or higher but not with databases that use formats from earlier releases (ODS 20 will not work). After you enable transaction logging, all databases are automatically logged. To check database formats, use the Files tab in Domino Administrator.

NOTE: To use all of the features of transaction logging and recovery, you need a third-party backup utility that supports Domino transaction logging.

What is considered a transaction?

A transaction is a single API call. It includes creating, modifying, reading (unread marks change) or deleting documents. A transaction is considered COMPLETE when the change has been saved to disk by the user. For example, if a user makes a change to the database, and does not save that change before the server crashes, that transaction is not considered COMPLETE. The transaction would have been COMPLETE only if the user had saved the change before the server had crashed. COMPLETE transactions are "commited" to the transactional log.

What is a Transaction log?

A transactional log is a binary file where transactions are written. The transactional log has a .txn file extension. These .txn files should never be deleted. The maximum size of each log extent (.txn file) is 64 MB. You can have several .txt logs based on the size specified in the Server document. The maximum total of .txn files is 4 GB.

What is the Database Instance ID (DBIID)

When you enable transaction logging, Domino assigns a Database Instance Identifier (DBIID) to each Domino database. When Domino records a transaction in the log, it includes the DBIID. During recovery, Domino uses the DBIID to match transactions to databases (it identifies which database the changes should be applied to). The DBIID is stored in the file header, along with the database ID and the Replica ID. Note: There is no relation to the Replica ID or the DBID.

Some database maintenance activities, such as compaction with options, cause Domino to assign a new DBIID to a database. From that point forward, all new transactions recorded in the log use the new DBIID; however, any old transactions still have the old DBIID and no longer match the database's new DBIID. As a result, Domino cannot restore these old transactions to the database.

To avoid losing data, you should immediately perform a full database backup whenever a database receives a new DBIID. When you perform this backup, you capture all the database transactions up until that point and ensure that Domino needs only the new transactions (with the new DBIID) to restore the database. If the DBIID changes and a backup is not taken after the fact, the database cannot be successfully restored (backup will have the old DBIID and the transactional log will not "know" the old DBIID.

NOTE: The DBIID has no relation to the REPLICAID or DBID.

Domino assigns a new DBIID to Domino databases when:

You enable transaction logging for the first time.- System logging is disabled then re-enabled.- The database is compacted using copy-style compaction.- The database has had Fixup -J applied to it.

IMPORTANT NOTES: If a database is logged, the default for Compact with no switches is -b (lowercase) If a database is un-logged, the default for Compact with no switches is -B (uppercase). Compact with no switches and Compact -b (lowercase b) are the only times Compact does

not change the DBIID.

The DBIID changes when a database is copy-style compacted because a copy-style essentially creates an entire new NSF with a new structure, which basically does not match the structure in the logs for the "old" NSF anymore. Note: -L, -c, and -i are switches that enable copy style compaction. -B at times uses copy style compaction.

Compact -B may change the DBIID. This option uses in-place compaction unless there is a pending structural change in which case copy-style compacting occurs. So when using this option and transaction logging, do full database backups after compacting completes.

Note: For additional information on the compact tasks switches refer to:Switches for COMPACT Server Task for Domino (#1084388)

Fixup is forced on the database (fixup -j) You move a Notes database from one logged server to another logged server or from an

unlogged server to a logged server.

NOTE: Changing the log path or maximum log size (after initial set up and use) does not trigger a DBIID change.

How to set up Transaction logging

1. Ensure that all databases to be logged reside in the Domino data directory, either at the root or in subdirectories.

2. From the Domino Administrator, click the Configuration tab.3. In the "Use Directory on" field, choose the server's Domino Directory.4. Click Server Configuration, and then click Current Server Document.5. Click the Transactional Logging tab.

6. Complete these fields, and then save the document.

Field Enter

Transactional Logging Choose Enabled. The default is Disabled.

Log path Path name location of the transaction log. The default path name is \LOGDIR in the Domino data directory, although it is strongly recommended to store the log on a separate, mirrored device, such as a RAID (Redundant Array of Independent Disks) level 0 or 1 device with a dedicated controller.The separate device should have at least 1GB of disk space for the transaction log. If you are using the device solely for storing the transaction log, set the "Use all available space on log device" field to Yes.

Maximum log space The maximum size, in MB, for the transaction log. Default is 192MB. Maximum is 4096MB (4GB).Domino formats at least 3 and up to 64 log files, depending on the maximum log space you allocate.

Use all available space on log device Choose one: Yes to use all available space on the device

for the transaction log. This is recommended if you use a separate device dedicated to storing the log. If you choose Yes, you don't need to enter a value in the "Maximum log space" field.

No to use the default or specified value in the "Maximum log space" field.

Automatic fixup of corrupt databases

Choose one: Enabled (default). If a database is corrupt and

Domino cannot use the transaction log to recover it, Domino runs the Fixup task,


assigns a new DBIID, and notifies the administrator that a new database backup is required.

Disabled. Domino does not run the Fixup task automatically and notifies the administrator to run the Fixup task with the -J parameter on corrupt logged databases.

Runtime/Restart performance This field controls how often Domino records a recovery checkpoint in the transaction log, which affects server performance. To record a recovery checkpoint, Domino evaluates each active logged database to determine how many transactions would be necessary to recover each database after a system failure. When Domino completes this evaluation, it:

Creates a recovery checkpoint record in the transaction log, listing each open database and the starting point transaction needed for recovery.

Forces database changes to be saved to disk if they have not been saved already.

Choose one: Standard (default and recommended).

Checkpoints occur regularly. Favor runtime. Domino records fewer

checkpoints, which requires fewer system resources and improves server run time performance.

Favor restart recovery time. Domino records more checkpoints, which improves restart recovery time because fewer transactions are required for recovery.

Logging style Choose one: Circular (default) to continuously re-use the

log files and overwrite old transactions. You are limited to restoring only the transactions stored in the transaction log.

Archive (recommended) to not re-use the log files until they are archived. A log file can be archived when it is inactive, which means that it does not contain any transactions necessary for a restart recovery. Use a third-party backup utility to copy and archive the existing log. When Domino starts using the existing file again, it increments the log file name. If all the log files become inactive and are not archived, Domino creates additional log files.

How to disable Transaction Logging for a specific database

In most cases, disabling Transaction Logging (on a server or database level) is not recommended because you lose all of the benefits of transaction logging (there are no ill side effects of disabling, you simply lose the benefits). One of the benefits of transaction logging is fast server restart. Disabling transaction logging will cause Fixup to run on the database (or all databases on the server), creating the potential for slow restart.

After you set up transaction logging, all databases that are in Domino Release 5 or higher format are logged. You can disable transaction logging of specific databases.

Attachments are transactionally logged; however, attachments are logged redo only. Therefore, if the database is recovered using media recovery you will get back the last copy of the attachment (once they are done they stay done). If, however, the server crashes with uncommited attachment

updates, they will not be undone since an undo record is never created for them .

Views are not logged, so after media recovery, you will need to rebuild views.

First, perform any of the following: When creating a new database, choose "Disable transaction logging" on the Advanced

Databases Options dialog. For an existing database, choose "Disable transaction logging" on the Database Properties

box, Beanies tab. In Domino Administrator, select a database on the Files tab, choose Tools - Database -

Advanced Properties, then choose "Disable transaction logging" Use the Compact task with the -t parameter.

Second, ensure that all users have closed the database. Next, use the DBCACHE command with the "flush" parameter to close the database in the database cache. Finally, open the database.

How to schedule backups or Transaction logs and logged databases

Backups are essential for recovering from a media failure, which is a failure of the server's disk or disks. If you have a third-party backup utility, you should:

- Schedule daily incremental backups of the transaction log. Use the backup utility daily to back up the transaction log. - Schedule archiving of transaction log files. If you use the archive logging style, use a third-party backup utility to schedule archiving of log files.- Schedule weekly full database backups. Each week, it is recommended to run the Compact task with the option to reduce file size. Because this compaction style changes each database's DBIID, you should schedule compaction with a full database backup.

How to fix corrupted databases

Corrupted databases don't occur frequently when you use Release 5 or higher databases and transaction logging. When you use transaction logging to log changes to Release 5 or higher databases, a server automatically uses the transaction log to restore and recover databases after a system failure, for example after server failures or power failures. If a disk failure occurs, you use the transaction log along with a certified backup utility to restore and recover the databases.

Using Transaction logging for recovery

Transaction logging is an integral part of recovering from system and media failures. A system failure causes the server to stop and requires you to restart the server. During restart, Domino automatically performs database recovery. The system uses the transaction logs to apply or undo database transactions not flushed to disk for databases that were open during the system failure.

Domino also runs the Fixup task on databases that use formats from earlier releases, databases that are in Release 5 or higher format but have transaction logging disabled, and on corrupt databases if you have the "Auto fixup of corrupt databases" field in the

Server document set to Yes.

Fixup -J

Causes Fixup to run on databases that are enabled for transaction logging. Fixup -j should only be run if a database is corrupt and you have no backup of the database to roll forward from.

Without this -j option, Fixup generally doesn't run on logged databases. The Fixup task interferes with the way transaction logging keeps track of databases. If you are using a backup utility certified for Domino, it's important that you schedule a full back up of the database as soon after Fixup finishes as possible.

Notes.ini parameter: Translog_Status

The TRANSLOG_Status NOTES.INI parameter is used to enable transaction logging for all databases on the server. "0" is disabled, "1" is enabled.

ContentTransactional Logging and How it Operates

Transactional Logging is a new feature in Domino R5. It is essentially a new method of writing out database changes to improve performance and to ensure data integrity. Its main purpose is three-fold:

to improve performance on the R5 server through sequential writes to the Transactional Logs

better data integrity by avoiding inconsistencies and data corruption faster server restart and crash recovery

A Transactional Log is simply a binary file where transactions are written. The transactions are saved in log extents that have a .TXN extension. Each log extent is 64MB and will fill before a new extent is created or a spare one reused. Multiple log extents collectively can grow to a maximum size of 4GB with circular logging or available disk space for archive logging.

Performance Improvement

When transactional logging is enabled on the server, the performance improvement is mainly due to the nature of how transactional logging operates. The writes to the Transactional Log are sequential. This is faster since there is less head movement and there is never a need to search for a place on the disk to write as there is in R4 or if transactional logging is not enabled.

The Transactional Logs must be on a separate physical drive for there to be any performance improvement. It is not sufficient to simply redirect the logs to a separate partition or a separate logical drive. In general, if the transactional logs are on a separate drive, a 10-20% improvement should be seen. However, if the logs are put on the same drive, it is likely that there will be approximately a 60% degradation.

In R4, writing to disk was time consuming. Modifications could occur across multiple databases or different parts of one database. As a result, the head had to move over various areas of disk to change or update data. This means there was a significant amount of transaction time committing data to the actual NSF (database). Without the benefit of transactional logging in R4, fixup relies on the fact that 99.9% of the data is present in the NSF to correct integrity problems.

In R5, when transactional logging is enabled, complete transactions are "committed" to the Transactional Log. All writes are done to the transactional

log before they are ever written to the database. The writes are done sequentially at least after each transaction so the Transactional Log is up to date generally to the hundredth of a second. Again, since the writes are sequential, there is less I/O and performance is improved. Please note that view indexes are not transactionally logged in R5 so views may need rebuilding after a restart or media recovery. Attachments, however, are transactionally logged. It is important to note that attachments are logged redo only, so if the database is recovered using media recovery, you will get back the last copy of the attachment (once done they stay done). If, however, the server crashes with uncommitted attachment updates, they will not be undone since an undo record is never created for them.

When transactional logging writes transactions to the logs, an undo record and a redo record are usually committed for each transaction. First an UNDO log record is generated in the event of a system outage. This is done before a change is written to a database. Before committing a transaction, a REDO record is also generated. It is used to re-apply a transaction from the transactional log to the database in the event that it did not get flushed to the NSF before a server outage. Undo and redo records ensure that if a change is half done it will be fully undone, and if a change was completely done then it will be fully re-done to the NSF.

Flushing and Hardening

Once changes are put into the Transactional Log, the changes must also eventually be hardened to the database. This occurs through a process called flushing. Any open database has an in-memory version of the database that is held in the UBM (Unified Buffer Manager). Flushing moves all changes that were made to the database but only kept in memory (UBM), to the actual NSF file. There is no set interval for this as the UBM determines when flushing will occur. It is usually done when there is a lull in the server activity. The DBIID (Database Instance Identifier) is used to correlate the updates in the Transactional Logs and in-memory to the respective database. It is important to note, however, that the Transactional Logs are not read from during this process since the Transactional Logs are mainly a write-only object. The updates are read and flushed from the UBM. They are only read from the Transactional Logs during crash recovery. Transactional logging is more expedient because of the fact that there are not a lot of read/writes to it during server production. Otherwise performance would suffer and it would defeat one of the purposes of transactional logging.

The Runtime/Restart Performance field in the Server document determines how many MB of changes are kept in memory. The amount of space used is bound by the "performance" level chosen in the Server document. There are three choices: Standard (default), Favor Runtime, and Favor Restart Recovery Time. If Standard is selected, the Redo.Limit is 49MB. This means that checkpoints during runtime are minimal and 49MB worth of changes are held in the UBM before they are flushed and hardened to databases. The Favor Runtime choice has a Redo.Limit of the smaller of 500MB or half the defined log size. This means that more information is held in the UBM and hardened to the database less frequently. Allowing UBM to manage more of the database changes will allow larger machines to better utilize memory and reduce unnecessary flushing. The Favor Restart Recovery Time choice allows for more checkpoints during runtime. There is less information held in the UBM and data is hardened to databases more frequently. The trade-off is that production time is slower but server restart is faster.

Crash Recovery

After a server outage the Transactional Logs are played back. The Recovery Point is determined for each NSF requiring log updates, it is the oldest log information that needs to be re-applied to databases. The databases are restored to the exact moment of the outage, guaranteed to restore any data from a completed transaction. Partial transactions will be undone and rolled back to the last good state in an effort to avoid corruption in the database. The partial work will be removed from the database before restart completes and the database is made available for use.

Customers should not move databases away from a server and copy them over from another server after a server crash. The database may be missing a significant amount of changes that are only stored in the transaction log

and if the database is not found during restart the changes will not be restored. Recovery restart must be performed first and can be trigged, for example, by a request to compact a nonexistent database.

Transactional Logging NOTES.INI Parameters

Transactional logging is enabled in the Server document. All the fields in the Server document map to specific NOTES.INI parameters.

The parameters are as follows:

TRANSLOG_AutoFixup= Tells whether autofixup is Enabled (1) or Disabled (0)

TRANSLOG_UseAll= To use all available space or not. 0=Disable, 1=Enable

TRANSLOG_Style= Circular (0) vs. Archive (1)

TRANSLOG_Performance= Favor runtime (1), Standard (2), Favor restart recovery (3)

TRANSLOG_Status= Whether transactional logging is Enabled (1) or Disabled (0)

TRANSLOG_Path=XXX Specifies the path to the .TXN files

QuestionWhat are the Checkpoint Frequencies and the Flushing Thresholds for Transactional Logging?

Answer

Setting Checkpoint Frequency Flushing Threshold

Favor Restart 8MB of data in buffer 25MB of data in buffer

Standard 40MB of data in buffer 121MB of data in buffer

Favor Runtime 250MB of data in buffer 500MB of data in buffer

ProblemWhat switches are available for the COMPACT server task?

SolutionIn Notes Release 5.x and later, the default compaction style is In-place compaction.

Note: When the Compact task runs it opens the database to find out the amount of % Used (as opposed to looking in the log.nsf).

When running COMPACT from the server console, the syntax is as follows:

load compact databasename -switch

Compact - Style


Description Notes/Domino Release

Compaction style: In-place (recommended)

-b Uses in-place compaction and recovers unused space without reducing the file size, unless there's a pending structural change to a

Release 5.x, 6.x, 7.x

database, in which case copy-style compaction occurs. This is the recommended method of compacting.

-b is the only switch that will not change the DBIID of a database if transactional logging is enabled.

Compaction style: In-place with file size reduction

-B Uses in-place compaction, recovers unused space and reduces file size, unless there's a pending structural change in which case copy-style compacting occurs. If you use transaction logging, do full database backups after compacting completes.


Compaction style: Copy-style

-c or -C Uses copy-style compaction. Use this option, for example, to solve database corruption problems


Compaction style: Copy-style: Allow access while compacting

-LUpper or lower case 'L' will work.

Enables users to continue to access databases during compacting. If a user edits a database during compacting, compacting is cancelled. This is useful only when copy-style compacting is done.


Compaction style: Copy-style: Ignore errors and proceed

-iUpper and lower case 'I' will work in R4, but only lower case 'i' will work in R5.

Note: the -i option has to be used together with -c or -L otherwise you will get the error message << "Ignore errors" is only allowed with copy-style compaction >>

Enables compacting to continue even if it encounters errors such as document corruption. Only used for copy-style compaction.

Release 4.x, 5.x, 6.x, 7.x

Check for overlapping obects

-o or -O If using in place compaction checks for overlapping objects and switches to copy style if any found.

Release 6.x, 7.x

Compact - Options Option Command-line

equivalentDescription

Compact database only if unused

-S percent Compacts all databases with a specified percent of unused space. For example,

space is greater than x percent

if you specify 10, databases with 10% or more recorded unused space are compacted. Note that the unused space calculation is not always a reliable measure of unused space.

Discard any built view indexes

-D Discards built view indexes. Use this option to compact databases just before you store them on tape, for example. Does copy-style compacting.

Keep or revert database to previous format

-R Compacts databases without converting to the current release file format of the server that stores the databases or reverts databases in the current release file format to the previous release file format. For example, on Domino 6 servers, this option compacts Domino 5 databases without converting them to the Domino 6 file format and converts Domino 6 databases to the Domino 5 file format. This option uses copy-style compacting.

Compact - Advanced Option* Command-

line equivalent

Description Notes/Domino Release

No-Execute -eUpper or lower case 'E' will work.

Show effective arguments without execution since some arguments affect others. For example -i only works with -c, so by executing the compact command first with the -e on the end, is a good way to check syntax without actually compacting a database.


Document table bitmap optimization: Off

Note: Prior to 5.0.3 this Option description was incorrect in the online documentation

-f Disables "Document table bitmap optimization" database property. Does copy-style compaction.


Document table bitmap optimization: On


-F Enables "Document table bitmap optimization" database property. Does copy-style compaction.


Don't support specialized response hierarchy: Off

Note: Prior to 5.0.3 this

-h Disables "Don't support specialized response hierarchy" database property. Does copy-style compaction.


Option description was incorrect in the online documentation

Don't support specialized response hierarchy: On


-H Enables "Don't support specialized response hierarchy" database property. Does copy-style compaction.


Enable transaction logging: On

-T Enables transaction logging. Release 5.x, 6.x, 7.x

Enable transaction logging: Off

-t Disables transaction logging. Release 5.x, 6.x, 7.x

Maintain unread marks: On

-u Disables "Don't maintain unread marks" database property.


Maintain unread marks: Off

-U Enables "Don't maintain unread marks" database property.


Allow for a larger UNK table

-K Allow for a larger UNK table in a database past 64 K.NOTE : not available through the Compact tool in the Files tab of the Domino Administrator.


Disable large (>64 KB) UNK table

-k Removes large UNK table in database.NOTE : not available through the Compact tool in the Files tab of the Domino Administrator.


No equivalent option.

-ZU Converts document attachments using compression type Huffman to LZ1.

Note: The database property "Use LZ1 compression for attachments" must be enabled for this switch to be effective.

For Domino 6.5.6 only the following Notes.INI parameter must also be set: DEBUG_ENABLE_LZ1_REPAIR=1

Domino 6.5.6, 7.0.1 and later.

No equivalent option.

-ZD Converts document attachments using compression type LZ1 to Huffman.

For Domino 6.5.6 only the following Notes.INI parameter must also be set: DEBUG_ENABLE_LZ1_REPAIR=1

Domino 6.5.6, 7.0.1 and later.

Compact - Archive

When you use the document archiving tool to archive and delete documents in a database, you can use the following Compact options to

archive documents if the database is located on a server and you've chosen the advanced archiving option "Automatically on server." Option* Command-line

equivalentDescription Notes/Domino

Release

Archive only -A Archives and deletes documents from a database without compacting the database.


Archive and then compact

-a Archives and deletes documents from a database and then compacts the database.


Delete and then archive

-j Deletes documents from a database and then compacts the database. NOTE : not available through the Compact tool in the Files tab of the Domino Administrator.

New in Release 6.x, 7.x

Domino 6 console help (produced by lo compact -?) lists the following options which have no effect :-v Make (v)iews big (do not compress indexes)-V Make (V)iews small (compress indexes)This issue was reported to Quality Engineering, through SPR # WBLD5XZHL3. The -v and -V flags were never implemented in Domino.

QuestionMail delivered prior to Compact -c running against a database appears as READ in the user's mail file. This mail was delivered after the user closed the session to the server.

AnswerThis issue was reported to Lotus Quality Engineering and has been addressed in Domino 5.0.9.Excerpt from the Lotus Notes and Domino Release 5.0.9 MR fix list (available at http://www.notes.net ):

Server-Compact SPR# HHAA4WUBAS - Fixed a problem with notes being marked as

read incorrectly after the database was compacted.

To avoid this problem in Notes/Domino prior to 5.0.9, use in-place compaction with file-space reclamation (Compact -B).

The problem occurs because of an issue related to how Profile documents are copied into the new database during copy-style compaction. It is unrelated to a change in the ODS of the database, and it is unrelated to purging deletion stubs.

The problem is really caused by copy-style compaction, and will occur anytime copy-style compaction takes place. COMPACT will perform copy-style compaction regardless of the in-place style compact parameters if the mail file has undergone a structural change, such as the mail file design replacement/conversion. A copy-style compact will be used, also, under the following conditions:

database ODS is less than the current version the following Compact parameters are used: -D/d, -R/r, -M/m, -C/c, -

F/f, -U/u, -L/l

http://www.notes.net/

local encryption is being enabled

Any of the advanced database properties are being changed, such as: Do not support specialized response hierarchy Document table bitmap optimization Do not overwrite free space Do not maintain unread marks Disable Transaction Logging

Allow more fields in database

Lotus Domino

Documents

Transcript of Lotus Domino