IBM Technical Exchange

© 2010 IBM Corporation

IBM Technical Exchange

August 19th, 2010

Session Title: Sizing, Tuning, and Use of the Recovery Log and DB with TSM V6

Speaker Name: Dave Canan, ATS


© 2010 IBM Corporation2

ATS Team

Dave Canan

–[email protected]

Dave Daun


Tom Hepner


Randy Larson




Agenda

How the recovery log works in TSM V6

Sizing the recovery log for your V6 environment

Server/Client parameters and commands that affect the recovery log

Performance, tuning and monitoring of the DB and recovery log

References



August 19th, 2010

How the Recovery Log Works in TSM V6



TSM V6 Components

TSM Server

Active Log

Log Mirror (optional)

Archive Log

Failover Archive Log (optional)

TSM DB

TSM STGPools (disk, tape)

ActiveLogDir

MirrorLogDir

ArchiveLogDir

ArchFailoverLogDir



TSM / DB2 Terminology

Both TSM and DB2 use the terms of “active logs”, “archive logs” , and “failover archive logs”.

TSM collectively refers to these 3 as the “recovery log”. Sometimes DB2 administrators understand this term, sometimes they don’t. They usually refer to them just as “transaction logs”.

TSM uses the term “directory” for where these are to be written to. DB2 administrators refer to use “container”. They mean the same thing.



TSM Active Log

Contains current in-flight transaction data.

Roll-forward mode only.

Use is required.

Sequential IO access

Initial directory of active logs determined by ActiveLogDir parameter (on dsmserv format / loadformat); can be changed later in dsmserv.opt

Active log files created in 512 MB sized files (defined by the Log File Size of DB2). Number of logs created is determined by ActiveLogSize / 512.

If the ActiveLogSize is increased, customers may see a delay on the restart of TSM while the new 512MB logs are being formatted.

If a transaction is not committed and all active log files are filled, then TSM halts.

Default ActiveLogSize is 16GB (changed in 6.2 from 2GB), maximum value is 128GB



TSM Active Log

As soon as an active log file gets full, it is copied to archive log space. It is not renamed, however, until all transactions in the active log are committed and a new active log file is needed.

A transaction can span active log files.

Customer Tip: recommend having several emergency 2GB dummy files (this can be done with dsmfmt) in the location of ActiveLogDir space. If the location referenced by ActiveLogDir gets full, you can create an emergency amount of space by deleting these dummy files, changing ActiveLogSize, and restarting TSM.



TSM Active Log (Mirror)

Used to contain mirrored copies of active transaction data

Sequential IO access – use faster disks for active logs and active log mirrors.

Use is optional but recommended.

Initial directory of active log mirrors determined by MirrorLogDir parameter (on dsmserv format / loadformat); can be changed later in dsmserv.opt

Change of MirrorLogDir requires TSM restart.

If mirror log directory becomes full, message issued to activity log and db2diag.log, TSM continues



TSM V6 Log Mirroring

If error writing to either primary or log mirror

– Failing path to log is marked as bad.

– Message written to TSM activity log and also db2diag.log

– Writes continue to remaining good log volume until current log volume is filled. When DB2 needs to open the next log file, then the path is retested and reused if it is OK.

If error occurs in the remaining good path, TSM shuts down.



TSM Archive Log

Contains committed transaction data.

DB2 is configured to keep the logs around for 2 DB backups. This will be changed in a future service level so that only enough log space will be required for 1 full DB backup. Addressed In 6.1.4.0,Tentatively for 6.2.2, subject to change. The appropriate DB2 parameters will be changed as part of the service pack install.

Sequential IO access – slower disks can be used for archive logs.

Use is required.

Initial directory of archive logs determined by ArchiveLogDir parameter (on dsmserv format / loadformat); can be changed later in dsmserv.opt

To change the location where archive logs are placed, the ArchiveLogDir parameter in dsmserv.opt file should be updated. TSM will need to be restarted.

If archive log directory becomes full, and no fail over archive log location has been specified, then TSM just keeps logs in the ActiveLogDir location and creates new ones. If THIS fills, then TSM halts.



TSM Failover Archive Log

Use is optional but highly recommended. Consider use of large NFS mountpoint or large “cheap” disk for this. For Windows, a CIFS share could also be used for the failover archive log.

Sequential IO access

Set with ArchFailOverLogDir parameter (on dsmserv format / loadformat), or added later in dsmserv.opt

Log files are removed after DB backup.

To change the location where failover archive logs are placed, the ArchFailOverLogDir parameter in dsmserv.opt file should be updated. TSM will need to be restarted.



August 19th, 2010

Sizing the Recovery Log for Your V6 Environment



Sizing the Logs for TSM V6

Consider what your workload is and which functions you will use. TSM data deduplication, for example, uses considerably more log space, as do in-line table reorganizations.

For new TSM customers, refer to following table.

Keep current on TSM V6 maintenance. Many APARs have been addressed in current service levels (see references at end of presentation).

Monitor log usage with examples provided in this presentation.



Initial Minimum Recommended Sizes for Active Log and Archive Log (For a New TSM Customer)

Storage Pool Deduplication Enabled?

Minimum recommended size for Active Log Directory

Minimum recommended size for Archive Log Directory

No 16GB 48GB

Yes 32GB 96GB

This is from the TSM Wiki. But let’s look closer at how to size it

more accurately if TSM is already in use.



Initial Minimum Recommended Sizes for Active Log and Archive Log (For an Existing TSM Customer)

Use the following approach as an initial estimate for the ActiveLogSize value for the upgrade to TSM V6:

– Before the daily full DB backup issue a “show logv” command (look for the log head LSN value with a format of xxxxxx.yyy.zzzz)

– Do the normal daily DB full backup

– Before the next scheduled daily full DB backup, again issue the show command above and record the new log head LSN value.

– Subtract the two xxxxxx values. This is the amount of log space, in MB, that represents the transaction workload for a daily backup cycle. Multiply this by a value of 3 for ActiveLogSize.

– For a conservative initial estimate of the amount of archive log space needed (in MB), take this value and multiply by 3.



A Closer Look at Sizing the Log – Backups and Recovery Log Usage

Factors to consider in log space used with backup and restore:

– Number of client nodes backing up each night

– Number of files stored per transaction (value of TXNGROUPMAX)

– Length of filename (for a filename between 12 and 120 bytes, lab measurements indicate a value of 3053 bytes of log space is needed)

– Resourceutilization value

– Is setting for simultaneous write set for storage pools?

Formula: Active Log space needed for backups/archives in bytes =

(number of clients performing operations) x (sessions per client) x (files stored per transaction) x (log space needed per file)

Archive Log space needed for backups/archives in bytes = 3 * above value

Formula discussed in detail in Techdoc 21389352 at :

http://www-01.ibm.com/support/docview.wss?uid=swg21389352



A Closer Look at Sizing the Log – Migration and Recovery Log Usage

Factors to consider in log space used with migration:

– Number of client nodes backed up each night

– Number of files for each client

– Each file migrated takes approximately 110 bytes of log space

Formula: Log space needed for migration in bytes =

(# of clients) * (# of files/client) * (110 bytes per file)



A Closer Look at Sizing the Log – TSM Deduplication and Recovery Log Usage

Factors to consider in log space used with TSM data deduplication:

– Amount of data to be deduplicated

– When the “Identify Duplicates” process is run (during backups or after backups).

– Number of “Identify Duplicates” processes

– Sizes of files being deduplicated. Large files require more active log space.

– Deduplication ratio. The higher the ratio, the more log space that is required. Approximately 1500 bytes of active log space is used for each file chunk that is identified by “Identify Duplicates” process.

Formula discussed in detail in Techdoc 21389352 at :

http://www-01.ibm.com/support/docview.wss?uid=swg21389352



August 19th, 2010

Server/Client Parameters and Commands that Affect the Recovery Log



Client / Server Parameters affecting the Recovery Log (TSM 6.2)

ActiveLogSize

– Default was changed with TSM V6.2

• Was 2GB before, now the default is 16GB

AllowReorgTable Yes|No (Default Yes)

– Table reorganizations will use large amounts of log space. See Documentation APAR IC68310 This will be in a future Topic titled “Performing an on-line table reorganization requires additional archive log space” in the Administrators Guide. Text of doc APAR provided in reference at end of presentation.

– In V6.2 TSM, runstats are now being done by DB2 instead of TSM issuing them. Reorgs can still potentially be done by TSM/DB2 on a daily basis unless disabled.

– Many APARs in this area – refer to references at end of presentation.

• DB backups, large transactions, data dedup can all affect behavior of Reorgs.



Client / Server Parameters affecting the Recovery Log (TSM 6.2)

Two new server options

– ServerDedupTxnLimit

• Sets the maximum size of objects that can be deduplicated on the server

– ClientDedupTxnLimit

• Sets the maximum size of a transaction when client-side deduplicated data is backed up or archived

Can be set dynamically with setopt server command

Use default values for now – changes may be coming in upcoming maintenance

– Note: To control which objects are deduplicated, you can also use the MAXSIZE parameter of the DEFINE STGPOOL and UPDATE STGPOOL commands. Using the MAXSIZE parameter, you can force large objects to the NEXT storage pool for storage.



Other Parameters affecting the Recovery Log (not a complete list)

Maxsessions

Resourceutilization

Number of Identify Duplicate Processes run if deduplication is enabled

Compressalways

Txngroupmax, txnbytelimit



Client / Server Commands affecting the Recovery Log – Automatic DB Backup Function (enabled in 6.1.2)

Set DBRECOVery device_class_name

– Enables TSM Automatic DB Backup capability

– If not enabled, message ANR0295I issued indicating a DB backup is needed

– DB monitored and checked every 30 minutes for various conditions

Enabling TSM Automatic DB Backup causes TSM to trigger a DB backup based on either of the following criteria:

– Active log space consumed since last backup (triggers a FULL DB backup) – issues message ANR4531I

– Active Log utilization ratio (triggers an incremental DB backup) – issues message ANR4529I

Information gotten thru DB2 API call

– Can use “db2 get snapshot for db on db_alias” to see information (see following example)



Automatic TSM Full DB Backup Trigger Example

Assume the ActiveLogSize is set to default of 16GB.

Example: get snapshot for DB on TSMDB1 gives:

File number of first active log = 5108

File number of last active log = 5141

Formula for full DB backup trigger is:

(last active log file number – (first active log file number +1) * 512MB

= (5141 – 5109)*512M = 16GB (so an automatic full backup of DB would be started)



Automatic TSM Incremental DB Backup Trigger Example

Based on ratio of total active log space used

Calculated as:

– Total Active Log Space Used / ( total Active Log Space Used + total Active Log Space Available )

– If greater than 80% (not configurable), then an incremental DB backup is started

– Values used can be seen with same DB snapshot command from previous slide. Look for the following in the output:

Log space available to the database (Bytes)= xxxxxxxxLog space used by the database (Bytes) = yyyyyyyy



Automatic DB Backup for Archive Log Utilization – Possible Future Enhancement

Currently, automatic DB backup triggers backups based on active log consumption/utilization

Does not look at the archive log, only the active log

APAR IC67387 addresses possible future enhancement to provide monitoring of the archive log and would trigger an automatic Full DB backup when a threshold of 80% utilization in the archive log filesystem is reached. (Only will monitor archive log, not failover archive log.) Checked every 30 minutes.

Contained in 6.1.4.0, tentatively scheduled for 6.2.2, subject to change.



August 19th, 2010

Performance, Tuning and Monitoring of the DB and Recovery Log



Tuning Tips for the TSM V6 DB

Use faster disks for the DB. Don’t use the slower internal disk included by default in most AIX servers, or using consumer grade PATA/SATA disk in a Linux or Windows system. These will slow everything down.

Use multiple database containers. Recommended to use at least 4 containers initially for the DB, spread across 4 LUNs / physical disks. Larger TSM servers should have up to 8 containers.

Plan for growth with additional containers up front. Adding containers later can initially result in an imbalance of IO and create hot spots. Adding containers will automatically rebalance the data across the new containers (done as a low priority background task).

If possible, place each database container is in a different filesystem. This improves performance; DB2 will stripe the database data across the various containers. TSM supports up to 128 containers for the DB.



Tuning Tips for the TSM V6 DB (cont.)

Separate your TSM Components (DB LUNs, Log LUNs, Storage Pool LUNs)

The block size for the DB varies depending on the tablespace, most are 16K, but a few are 32K. Segment sizes on disk subsystems should be 64K or 128K.

If using RAID, then define all your LUNs with the same size and type. (For example, don’t mix 4+1 RAID5 and 7+1 RAID5)

RAID10 outperforms RAID5 (when doing large numbers of writes) but comes at a cost of 50% more disk being needed.

Smaller capacity disks are better than larger ones if they have the same rotational speed.

Recommended best practice to have containers on disks that have the same capacity and IO characteristics.



Tuning Tips for the TSM V6 DB (cont.)

If you’re using SVC or a RAID controller, you may be able to further improve I/O parallelism by setting the DB2 registry variable DB2_PARALLEL_IO. This is set using the DB2SET command, as in

db2set DB2_PARALLEL_IO=value

The value is a comma separated list of tablespace ids or '*', followed by a colon, followed by a number which indicates the number of physical disks backing your database container (directory). For TSM, it is recommended to specify '*' for the table space ids, and then whatever number of disks you have in your raid configuration.

Example: For LUNs of 4+1 RAID5, you would use:

db2set DB2_PARALLEO_IO=*:4

This requires a restart of DB2.



Tuning Tips for the TSM V6 Logs

Use faster disks for active logs. Do not mix active logs with disks containing the DB, archive logs, or system files such as page or swap space.

Can use slower disks for archive logs and failover archive logs.

Subsystem readahead is good to use for the active logs; it helps in archiving them faster.

RAID1 is good for active logs.

Highly recommended that FailoverArchiveLog space be set aside for possible emergency use. Slower disks can also be used for FailoverArchiveLog space.

Look to see if write latency for active log pages written faster than .005 seconds (5 ms.) See Server Instrumentation



Common questions about DB / Log Usage and performance

How fast am I consuming log space?

When is my log usage highest?

How can I monitor to see if I’ve started to use my failover archive log space?

How can I determine the maximum active log utilization that used to be in TSM V5?

How can I monitor log usage? What tools are available for UNIX / Windows to check log usage?

How can I now monitor performance of my DB and Log volumes with DB2 / TSM?

Are there any rules of thumb for DB / Log performance?



How fast am I consuming log space?

Examine the DB2DIAG.LOG. Here is a sample entry for an archive log:

2010-05-27-06.52.57.794000-420 I250127F372 LEVEL: Warning

PID : 3680 TID : 4868 PROC : db2syscs.exe

INSTANCE: SERVER1 NODE : 000

EDUID : 4868 EDUNAME: db2logmgr (TSMDB1)

FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogFile, probe:3108

MESSAGE : Started archive for log file S0005226.LOG.

2010-05-27-06.53.00.185000-420 I250501F500 LEVEL: Warning

PID : 3680 TID : 4868 PROC : db2syscs.exe

INSTANCE: SERVER1 NODE : 000

EDUID : 4868 EDUNAME: db2logmgr (TSMDB1)

FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogFile, probe:3180

MESSAGE : Completed archive for log file S0005226.LOG to

J:\Server1\ArchiveLogDir\archmeth1\SERVER1\TSMDB1\NODE0000\C0000001\

from E:\Server1\ActiveLogDir\.

To filter entries out of log, see the following tip



How fast am I consuming log space (cont.) ? How fast am I creating archive logs?

– Log usage is tracked in the db2diag.log file. Following is an example of a script that can be used to track log usage. This provides a timestamp and log file name for each log file created within a 24 hour period.

Windows: db2diag -H 1d -g msg:="Completed archive" -fmt "Time : %%{ts} Message : @{msg}" | grep Time | gawk -F\ -f "C:\Program files\tivoli\tsm\baclient\loginfo.awk" >> c:\temp\tsm-files\\loginfo.txt

UNIX: db2diag -H 1d -g msg:="Completed archive" -fmt "Time : %{ts} Message : @{msg}" |grep Time |awk -f dclogs.awk

Sample output:– Time : 18.18.43 S0003158.LOG

Time : 18.19.55 S0003159.LOGTime : 18.20.46 S0003160.LOGTime : 18.21.29 S0003161.LOGTime : 18.22.25 S0003162.LOGTime : 18.23.24 S0003163.LOGTime : 18.24.19 S0003164.LOGTime : 18.49.38 S0003165.LOGTime : 18.50.18 S0003166.LOGTime : 18.51.00 S0003167.LOGTime : 18.51.43 S0003168.LOGTime : 18.52.33 S0003169.LOG

– Time : 03.20.32 S0003170.LOGTime : 07.21.04 S0003171.LOGTime : 11.56.06 S0003172.LOGTime : 12.17.13 S0003173.LOGTime : 12.39.28 S0003174.LOGTime : 13.00.44 S0003175.LOGTime : 13.20.44 S0003176.LOG

Enhancement to above could be done to detect when Failover Archive log location was being used

Note: Contents of dclogs.awk script above is {print substr($1,1,4) ": " substr($3,12,8) " " substr($11,1,13) }

Note the large number of logs that were created in just 6 minutes. Might

want to look in the actlog to see what was happening here. (In this case, a re-org had been started)



When is my log utilization highest?

Nmon AIX example – look at tab marked jfsfile in nmon analyzer spreadsheet

Chart shows filesystem utilization of 3 log file spaces over time

Log Usage

0

20

40

60

80

100

120

10:2

0:1

0

10:2

0:4

0

10:2

1:1

0

10:2

1:4

0

10:2

2:1

1

10:2

2:4

1

10:2

3:1

0

10:2

3:4

0

10:2

4:1

0

10:2

4:4

0

10:2

5:1

1

10:2

5:4

1

10:2

6:1

1

10:2

6:4

0

10:2

7:1

0

10:2

7:4

0

10:2

8:1

0

10:2

8:4

1

10:2

9:1

1

10:2

9:4

1

10:3

0:1

0

10:3

0:4

0

10:3

1:1

0

10:3

1:4

0

10:3

2:1

1

10:3

2:4

1

10:3

3:1

1

10:3

3:4

1

Time

Utilization

Active Log

Archive Log

Failover Archive Log

Highest archive log util. DB Backup taken here

Increase in Failover archive log indicates it is being used



How can I determine the maximum active log utilization?

In TSM V5, a ‘q log’ command displayed a column titled “maxlogutilization”. This is not reported in V6.x

Used in V5 to determine the maximum size that the log had gotten to since last DB backup.

In V6, do the following from a db2 command prompt (db2 => ):

– db2 => connect to tsmdb1

– db2 => select int(total_log_used/1024/1024) as "Used Space(MB)",

int(total_log_available/1024/1024) as "Free Space(MB)", int((float(total_log_used) / float(total_log_used+total_log_available))*100) as "Pct Used", int(tot_log_used_top/1024/1024) as "MaxLog Used (MB)" from table(snapshot_database('TSMDB1',-1))



How can I determine the maximum active log utilization (cont)?

The column “Max Log Used (MB)” is the maximum log utilization since TSM was last restarted.

Value is reset on TSM restart, it is not reset on a DB backup.

Example:

Used Space(MB) Free Space(MB) Pct Used MaxLog Used (MB)

------------- ------------------- ----------- -----------------

5 2032 0 6



How do I now monitor my DB and Log volumes for performance (UNIX)?

With TSM V6.1, server instrumentation no longer shows performance of DB and Log volumes.

For UNIX, use IOSTAT and look at the column %tm_acct. This should not be more that 40%. A higher value than that indicates processes are waiting for IO to complete and that there is an IO issue.



How do I now monitor my DB and Log volumes for performance (Windows)?

For Windows, use perfmon. (look at “TOP 6 FAQ on Windows Disk Performance” in reference on how to interpret)

Use the Avg. Disk Queue Length and %Idle Time counters and look at the DB and log volumes



Possible Future Enhancement – Enhanced Server Instrumentation Report

Following slide illustrates a future enhancement being considered.

Statistics gotten using DB2 api snapshot information

Information is still gathered with the same commands as before (“instr begin”, “instr end”)

If log writes exceed 5 ms., message issued. See Example here

This enhancement is tentatively scheduled for a future service release of TSM V6, and is subject to change.



Possible Future Enhancement – Enhanced Server Instrumentation Report

This enhancement is tentatively scheduled for a future service release of TSM V6, and is subject to change.



August 19th, 2010

References



Other Information About the TSM Recovery Log References:

– TSM V6.1 or 6.2 might crash if log is not sized appropriately:

• http://www-01.ibm.com/support/docview.wss?uid=swg21389352

– Deduplication of large files on Tivoli Storage Manager Version 6.1 may cause active log to run out of space.


– Tivoli Storage Manager V6.1.2 automatic DB backup

• http://www.ibm.com/developerworks/wikis/display/tivolistoragemanager/Tivoli+Storage+Manager+V6.1.2+automatic+DB+backup

– Active and archive log directory are full - TSM server will not start.




Other Information About the TSM Recovery Log (cont.)

References:

– Active log fills when a transaction runs for a long time :• http://www-01.ibm.com/support/docview.wss?uid=swg21406035&myns=swgti

v&mynp=OCSSGSG7&mync=E

– Archive log space for Tivoli Storage Manager server fills because long-running online table reorganization prevents completion of database backup• http://www-01.ibm.com/support/docview.wss?uid=swg214229

82&myns=swgtiv&mynp=OCSSGSG7&mync=E

– Tivoli Storage Manager Server 1024 Database Connection Limit on AIX • http://www-01.ibm.com/support/docview.wss?uid=swg214285

57&myns=swgtiv&mynp=OCSSGSG7&mync=E



Other Information About the TSM Recovery Log (cont.)

References:

– IC65197: ONLINE REORG ON A LARGE TABLE CAN CAUSE THE ARCHIVE LOG SPACE TO BE CONSUMED :

• http://www-01.ibm.com/support/docview.wss?uid=swg1IC65197– IC62978: ARCHIVE LOGS GROW RAPIDLY AND CAN CAUSE THE

SERVER TO KEEP CRASHING DUE TO THE DB2 REORG OPERATION RUNNING TOO FREQUENTLY :

• http://www-01.ibm.com/support/docview.wss?uid=swg1IC62978– TOP 6 FAQ on Windows Disk Performance

• http://www.oreillynet.com/pub/a/network/2002/01/18/diskperf.html– Best Practices for DB2 on AIX 6.1 for Power Systems

• http://www.redbooks.ibm.com/redbooks/pdfs/sg247821.pdf– TSM Wiki

• http://www.ibm.com/developerworks/wikis/display/tivolistoragemanager/Home



Text of APAR IC68310 (In future Admin Guide)Performing an on-line table reorganization requires additional archive log space.

The space that is required for an on-line table reorganization is determined by the number of rows being reorganized, the number of indexes, the size of the index keys, and the current organization of the table. It is a good idea to establish a typical benchmark for log space consumption associated with your tables.

Typically, every row in a table is moved twice during an on-line table reorganization. For each index each table row must update the index key to reflect the new location. After all accesses to the old location have completed, the index key is updated again to remove references to the old location. When the row is moved back, updates to the index key are performed again. All of this activity is logged to make on-line table reorganization fully recoverable. There is a minimum of two data log records (each including the row data) and four index log records (each including the key data) for each row (assuming one index). Clustering indexes are prone to filling up the index pages, causing index splits and merges that must also be logged. A number of the tables implemented by the server have more than one index, and a table that has four indices would require sixteen index log records for each row being moved for the reorganization.

The server monitors characteristics of the database, the active log, and the archive log to determine if a database backup is needed. For example, during on-line table reorganization, if the filesystem used for the archive log space begins to fill up, the server triggers a database backup in order to relieve the situation. When a database backup is started, any on-line table reorganization in progress is paused so that the database backup can operate without contending for resources with the reorganization.



Questions?

IBM Technical Exchange

Documents

Transcript of IBM Technical Exchange