Compact, Compress, De-Duplicate

Ulrich Krause DanNotes, 23. – 24. November 2011,

Comwell Klarskovgaard, Korsør Denmark

About: Ulrich Krause

Administrator & Developer since 1993

– Business Application Developer

– is@web, Ratingen, Germany

OpenNTF http://openntf.org

• !!HELP!!, Cebulon, Trigger Happy

• Various XPages Custom Controls

IBM Champion 2011• https://www-304.ibm.com/connections/blogs/socialbusiness/entry/

announcing_the_2011_ibm_champions_for_collaboration_solutions4?lang=de

Blog http://www.eknori.de

Notes Forum (http://www.atnotes.de)

How did I get here?

How did I get here?

How did I get here?

Overall Distance 839 km

Flight SK 626

– Düsseldorf Kopenhagen

Train

– Kopenhagen Korsor

Car

– Korsor Venue

– Thanks to Tobias !!

How is today‘s situation?

More and bigger documents and attachments

More expensive storage is needed

The average size of an application increases

Duplicate attachments ( n times )

Duration of standard tasks increases

More time for Backup

Network traffic increases

I/O and performance issues

What do we want to achieve?

Leverage existing storage

Reduce size of applications

Avoid attachment redundancies

Speed-up standard tasks

Reduce backup time

Reduce network traffic

Reduce disc I / O

Application model ( simplified )

Design

Documents

– Text

– Attachments

– Embedded objects

Compact Secrets

DEBUG_ENABLE_COMPACT_8_5=1

– W nn - Only compact databases which haven't been compacted successfully in the last nn days.

– w - Exclude system databases (e.g., log.nsf, names.nsf).

– X nn - Limit each database compaction to no more than nn minutes. (applies to in-place compaction only)

– x nn - Limit total compaction time to nn minutes.

– # nn - Execute up to nn compactions in parallel using different threads.

• Don’t use more than 8 threads in parallel

– ODS performs a copy-style compact only on databases with an ODS smaller than the ODS provided with this parameter.

• ODS 48 would convert to ODS 51

Compact Secrets (Sample)

Load compact -# 8 mail\

Compacts all applications in the mail subdir in 8 parallel Threads

Compact via application properties

Compacting a database does notnecessarily reduce its size

If the application requires structural changes (typically a change of ODS [on-disk structure]). then a copy-style compaction will be performed (equivalent to -L).

If the application is logged (Transactional Logging enabled), then a simple in-place compaction will be performed (equivalent to -b).

If the application is not logged (Transactional Logging disabled), an in-place with file size reduction compaction will be performed (equivalent to -B).

Bug Alert!!!

In-place compact does not upgrade ODS when Create_R8_Databases is set in notes.ini

In Lotus Domino 8, a copy-style compact will automatically occur when a database needs a structural change, regardless of the compact options specified. However, you notice that databases compacted with the -b switch are not upgraded to ODS 48, even when Create_R8_Databases=1 is set in the notes.ini file.

Two notes.ini parameters are required to upgrade a database to ODS 48 using an in-place compact (compact -b):

– 1. Create_R8_Databases=1

– 2. Dont_Inplace_Compact_ODS43=1

Copy Style Compaction

The compact -C (copy-style compaction) (1) creates a new empty application container on the same server with the same replica ID and a .tmp suffix, (2) copies all of the documents from the original application to the new application container.

After all documents have been populated in the new application, (3) the old .nsf application is deleted and the (4) .tmp file is renamed to the original filename.

Defrag

File fragmentation saps Domino server I/O performance

Even a slightly fragmented file can cause a 50% decrease in non-cached read performance

Load compact –C does not allocate space needed before compacting the application

Applications are fragmented during copy-style compaction

Full Text Indexes are fragmented too

Even NLO files in DAOS repository are fragmented over the time

Use a tool to defrag your files speed up your Domino server !

Defrag

DefragNSF

• Adam Osborne

• http://www.preemptive.com.au/defrag

Domino Defrag

• Andrew Luder, OpenNTF

• http://www.openntf.org/Projects/pmt.nsf/ProjectLookup/DominoDefrag

Both products usethe Microsoft defragmentation API

Defrag

Sample: Lotus Software KnowledgeBase ( 5 * compact –c )

Before

– 68 fragments

– 7550 ms

After

– 1 fragments

– 125 ms

Defrag

Domino Defrag uses IBM Lotus Notes C API

Open Source

FREE

– There is also a professional support plan available

Well documented

32/64 Bit support

New in Version 3: Free Space Defragmentation

– Uses jkdefragcmd.exe (http://kessels.com/jkdefrag/)

Domino Defrag

Xpages based administration application, logs, charts

Uses http://www.openntf.org/projects/pmt.nsf/ProjectLookup/Java%20ChartsUses http://www.openntf.org/projects/pmt.nsf/ProjectLookup/Java%20Charts

FT Index and Defrag

When the index is updated it just about always fragments the index files

Little benefit for lots of work!

Adam Osbourne

– http://ab1osborne.blogspot.com/2011/11/defragging-ft-indexes-little-benefit.html

– http://ab1osborne.blogspot.com/2011/11/defragging-ft-indexes-part-2.html

Design Note Compression

Design Note Compression

New feature as of Domino 8

Needs ODS 48 or above

– Create_R8_Databases=1

– or Create_R85_Databases=1

– load compact -c

– load compact -c –n

Check “Compress database design”

Reduces size of design by 55 – 60 %

Reduces disc I / O

Design Note Compression

Document Body Compression

Document Body Compression

New feature as of Domino 8.0.1

Reduces size of Rich Text up to 60%

Needs ODS 48 or above

load compact -c –v

Check „Compress document data“

Document Body Compression

LZ1 Attachment Compression

LZ1 Attachment Compression

As of release 6

Replaces Huffmann algorithm

NOT active when creating a new applications.

Recompress using

• load compact -c –ZU

• In 6.5.6 also set DEBUG_ENABLE_LZ1_REPAIR=1

To avoid problems with JAVA applets in <= 8.5 set

• DEBUG_ENABLE_LZ1_HOST_TYPES=0

• fixed in 8.5.1

LZ1 Attachment Compression

There have been problems reported that the fulltext search does not work as expected using LZ1 compression.

To revert back to Huffmann use

– Load compact –c –ZD

– In 6.5.6 also set DEBUG_ENABLE_LZ1_REPAIR=1

Avoiding compression conversions

There are circumstances that compression conversion is explicitly avoid.

Overall, it is to avoid the performance penalty of attempting a compression and failing back to uncompressed for file types that are already compressed.

The common trick is applied to observe the file extensions and avoid these.

– NONCOMPRESSABLE_FILES=<comma delimited replacement list>

– Default list: zip,rar,jpg,gz,z,zz,lzo,zoo,rz,lzma,bz2

LZ1 Attachment Compression

Design

Text

Attachments

DAOS Estimator

DAOS Estimator

Free tool to plan your DAOS rollout

• Download: http://www-01.ibm.com/support/docview.wss?rs=463&uid=swg24021920

DAOS Estimator (notes.ini)

DAOS Estimator by default has a large gap in attachment size evaluation between 64k and 1 MB

– DAOSEST_BUCKETS=16,64,128,256,512,768,1024,2048,3072,4096

– The above line sets to:

– 16K, 64K, 128K, 256K, 512K, 768K, 1MB, 2MB, 3MB and 4MB

Source: Lotus Software Knowledge Base

• http://www-01.ibm.com/support/docview.wss?rs=0&q1=1418102&uid=swg21418102

LZ1 != HUFFMAN

Db1 – LZ1 enabled

Db2 – LZ1 disabled

Result is 2 .NLO Files !!

D A O S

Detailled scientific study

DAOS

Domino Attachment Object Service

Attachments are no longer stored in application

Attachments are stored as NLO (Notes Large Object )

Document contains only a ticket

Separates static and dynamic data

Only one copy of a bit-like attachment

Transparent to users and 3rd-Party Products

Prerequisites

Domino 8.5 ( ODS 51)

SHARED MAIL OFF

Transaction protocol

Create DAOS Repository

Activate DAOS on server

Activate DAOS on application

Switch OFF Shared Mail

Serverdocument – Shared Mail

Field Shared Mail = None

„Load object unlink shared.nsf“

Transaction Protocol

Server document – Transactional Logging

Required for DAOS – no discussion!!

Use a separate disk drive.

Create log file folder

Choose method

• Circular» 4GB limit

• Archive» No limit, backup will delete logs

• Linear» Like circular, NO size limit

CREATE_R85_LOG=1

Restart server

DAOS Repository

Use separate disk drive

Can be SAN, NAS

Each server has it‘s own repository

– Max.1.000 subdirectories

– Max 40.000 .NLO / subdirectory

– = 40.000.000 unique .NLOs

– Daos.cfg ( in Domino Data Directory)

– Daoscat.nsf

.NLO files are encrypted with server.id

– DAOS_ENCRYPT_NLO=0

Include repository in backup!!

DAOS Repository

How do you change the number of DAOS NLO's per directory?

DAOS_MAX_FILES_PER_SUBCONTAINER=40000

Requires a server restart

Will only affect the current DAOS subdirectory (older subdirectories are not changed).

This will also affect the maximum number of files in the DAOS store which is currently this values multiplied by 1000 (this value is not configurable currently).

• 1.000 * DAOS_MAX_FILES_PER_SUBCONTAINER = MAX_NLO

Activate DAOS on server

Goto Serverdocument – DAOS

Set Minimum size of an object

• recommended 64kB (= DEFAULT in 8.5.1 )

• Evaluate with DAOS Estimator

Set DAOS base path

• Put it outside the data path!

Set Delete objects in repository

• Depends on your backup strategy

• 0 Days = immediate delete after the refcount = 0

• Manually from console: TELL DAOSMGR PRUNE 0

Restart the server

show server

Activate DAOS on applications

Application properties „propeller hat“ for single application

Use Admin client for multiple applications

Valid for new attachments only!

To move existing attachments into the DAOS repository

– Load compact –C –DAOS ON <Path\NSFs>

Bug Alert !!!

load compact -c -DAOS ON mail (w/o trailing backslash) enables DAOS on ALL applications, not only on those in mail subdirectory !!

Documented Lotus Software Knowledge Base #1410926

SPR AHOE7XHM9J (fixed in 8.5.2, no regression in 8.5.3)

Bug Alert !!!

"tell daosmgr status path/DatabaseName.nsf„

– Notice the forward slash (/) instead of a back slash (\)

A programming error was found but will not be corrected. It will be a permanent restriction. Use the correct syntax

LO50060: ENHANCEMENT: "THIS DATABASE IS NOT REGISTERED WITH DAOS"

tell daosmgr status path/DatabaseName.nsf tell daosmgr status path/DatabaseName.nsf

The Result

Transaction Protocol Trap

load compact –c –DAOS ON

NO attachment in DAOS repository

tell daosmgr dbsummary count = 0

load compact -C -T -DAOS on

DAOS and replication

DAOS is a server feature!

On a client replica, all attachments are stored in the application.

Replicas on a NON-DAOS server store attachments in the application

Each server in a cluster has it‘s own DAOS repository.

DAOS and replication (cont.)

Client Server (as of 8.5.1)

– „known” attachments are no longer transferred to server = „smart replication“

• Forward mail

• Reply with history

Server Server (as of 8.5.1)

– “known” attachments are no longer replicated to target server

8 new statistics

– show stat DAOS

DAOS and replication (cont.)

Domino 8.5

Domino 8.5.1

WARNING!!: ADMINP_ACCELERATED_REPLICA_OVERRIDE=4 !!

DAOS and replication (last slide )

DAOS and mail.box

Remove RM_NO_LOG_OBJECTS_IN_MAILBOX from notes.ini.

Enable DAOS on all mail.box files

Enable DAOS on all mail files

Increases performance

DAOS and Encryption

Increases number of NLO files!

– User A writes an encrypted mail with attachment to User B

– User B writes an encrypted mail with same attachment to User A

– Results in 4 NLO Files in repository

Does not save space but separates static from dynamic data

DAOS and Quotas

Technote: 1405456

Depends on the quota enforcement setting that is in place

Check space used in file when adding a note

– Acts on the physical size of the application (without attachments)

– Possibly a problem when switching DAOS off!!

• 64GB limit

Check file size when extending the file

Check file size when adding a note

– Both acts on the logical size of the application (with attachments)

Keep daoscat.nsf in sync

DON‘T use DDM !!

Use program document

Tell daosmgr resync

SPR# DROO7ZNPZ8Added a new DAOSMGR resync switch to perform a quick resync. This new switch updates the DIT and DOI but does not run through the applications or updates the NLO refcounts. ( as of 8.5.1 IF1 )

„Needs Resync“

DDM points to the first application that causes the DAOS catalog to go into „Needs Resync“ state.

If you cannot find a cause for the issue set

debug_threadid=1console_log_enabled=1daos_logging=<path/daos.txt>STATE_CHANGE

Many improvements in 8.5.2 that reduces the number of state changes (SPR# RCOR83NLQK)

DAOS resync time window feature

The following two notes.ini parameters control when resync will be allowed to run

– DAOS_RESYNC_START_TIME=12:00:00 AM

– DAOS_RESYNC_STOP_TIME=04:00:00 AM

Note that these parameters do not launch resync automatically; they just control whether or not resync may run at a given time, and also tell resync when to stop.

A program document could be created to launch resync just after midnight. If the catalog is SYNCHRONIZED, then the resync will not take place. However, if a resync is needed, then the resync process will begin. When the end time is reached, the resync threads will finish up the databases they are processing and exit.

DAOS OFF

What happens when DAOS is disabled / switched OFF?

In application properties

– No documents with attachments in the application

– Application has documents with attachments

• Status in Admin Client changes to „Read Only“

• DAOS still works for already stored attachments but no new attachments are written to the repository for the specific application.

In server document

– Same as 1

DAOS and missing NLO

NLO has been deleted on the OS level

TELL DAOSMGR LISTNLO MISSING <application.nsf>

Output to <datadir> by default

Daosmgr in Domino 8.5.2

daosmgr options have been enhanced in Domino 8.5.2

You can now do a search for missing NLO on all applications in a subdirectory

– tell daosmgr listnlo missing mail

DAOS and missing application

NSF has been deleted on the OS level

Message on server startup

• Informational - The DAOS catalog is not synchronized. Deletions will be postponed. Please run 'tell daosmgr resync' at the next convenient opportunity to re-synchronize.

Has the application been deleted by accident?

– NO

• TELL DAOSMGR RESYNC oder RESYNC FORCE

– YES

• Restore from backup

• Create a new replica from a cluster mate

Same behaviour when moving an application (on OS level)

Can we save another 50%

If we disable encryption, then we could put NLO for all servers into a single repository …

Yes, in theory

It‘s only experimental

Does not work because each server has its owndaoscat.nsf.

Would cause problems with pruning

DAOS and Nagios ( Statistics )

Catalog.Status

Repository.Size

Plugin

http://www.eknori.de/2009-07-11/nagios-centreon-and-daos/http://www.eknori.de/2009-07-12/more-nagios-more-daos/

DAOS and Statistics

8 new statistics (8.5.1)

– show stat DAOS

8.5.1

Debug_NSF_Show_Allstats=1

DAOS and Statistics

Windows:@echo off

setlocal

for /f "tokens=*" %%a in ('dir/w/s %1 ^|findstr "Datei(en)"') do (

set info=%%a

)

echo %info%

Linux:#!/bin/sh

REPOSITORY=/local/daos

OUTFILE=/local/daos.txt

fCount=`find $REPOSITORY -type f | wc –l`

fSize=`du -ksb $REPOSITORY`

IFS=”/”

array=($fSize)

fDate=$(date +”%d-%m-%Y”)

echo $fDate / $fCount / ${array[0]}>> $OUTFILE

Use XML

DAOS and daos.cfg

Contains information on file count and file size of .NLO files

Created on server startup when it has been deleted

Updated on server shutdown

Do not backup or restore!!

DAOS and daoscat.nsf

Contains index and refcount of NLO

Contains index of all applications using DAOS

Does NOT contain any view

Re-Created on re-sync

Created on server startup when it has been deleted

It should not be replicated, compacted, indexed, or otherwise manipulated

DAOS and daoscat.nsf

You receive an error when opening daoscat.nsf

– Error, special noteid not found

– Database is not fully initialized yet. (It does not contain any views.) you will have to use 'Replicate...' on the 'File, Replication' menu to initialize it.

The operation that incurred the error should not be performed

("Doctor, it hurts when I do this" "Then don't do that")

Admin Client ( Files tab )

The application now has sizes

– Logical size

– Physical size

– DAOS status

– DAOS count

– DAOS size

Quotas use the logical size.

New columns regarding compression

Backup & Restore

No change in backing up NSF files.

Additional backup job for NLO files

NLO files do not require any special agent for backup

IMPORTANT: the backup sequence

– server is up during backup

• NSF before NLO

– server is down during backup

• it’s your choice

IBM Technote # 1358548 - DAOS Backup and Restore

FT Indexes in a different location

Migrate existing FT Indexes by bringing down server, moving all the indexes into matching subdirectory in new FTBasePath= and restart Domino Server

Works on the Notes Client as well.

– Run updall -f to rebuild all FT indexes

FT Indexes in a different location

You can move FTs to local drives, while the dbs are located on a network share/homeshare, e.g. for Citrix XenApp installations

If you set the notes.ini and don't move the indexes, a new index will be built from scratch in the new location and the old index will be deleted... No migration but fairly user friendly when it comes to cleanup..

If for some reason the FTBasePath= set in the notes.ini does not exist or server does not have proper access to, the code defaults back to the same location as the database as it always did.

ODS Tuning

Automatic Client ODS Update

– As of release 8.5.2

– Can be configured in „Desktop Policy

• Desktop Policy Mail MailSettings

• Or via notes.ini NSF_UpdateODS=1 + (CREATE_R85_DATABASES=1)

• NSF_UpdatedToODS=51

– Applications that are in use will be upgraded on next client startup

ODS Tuning

Technote: Upgrading multiple local databases to a new ODS

– Unfortunately the admin help that is delivered with 8.5.2 does not include any information about this feature.

– https://www-304.ibm.com/support/docview.wss?uid=swg21429889

ODS Tuning

ns6, ns7 … instead of .nsf

compact -c does not change the ODS !!

– Even, if you revert the suffix to .nsf

– Technote: 1086780

– Create new copy or replica

Notespeek Download :https://www-304.ibm.com/support/docview.wss?uid=swg24005686

ODS Tuning

Upgrading the ODS of templates

– load compact -c

• Upgrades all.nsf files, but not templates

– load compact –c *.ntf

• ERROR: "Database compactor error: File does not exist.„

– load compact –c cois_crm.ntf

• Works

– Workaround: use .ind files

• dir *.ntf /b > templates.ind

• load compact -c templates.ind

Agent Manager Retry on Compact

There is a new setting starting with 8.5.3 and 8.5.2 FP3 to retry to run an agent after the agent has been blocked out thru a compact previously.

SPR# KMUR63DF3V - Fix introduces an ini DEBUG_AMGR_ENABLE_RETRY_ON_COMPACT to allow an agent to run on a time interval once database compact is complete. Previously when a database was being compacted and an attempt was made to load a scheduled agent it would fail and the agent would be marked to not run again unless the user restarted the agent manager or the agent cache refreshed. This fix introduces a notes.ini variable to allow the agent in question to be retried on it's subsequent time interval.

JavaScript/CSS Aggregation

Dynamically aggregates multiple Dojo modules, or multiple CSS into a single file.

This results in the following performance improvements:

– a decrease in requests sent from the browser to the server

– an increase in user performance, particularly in the context of networks with high latency

– an increase in the speed of JS/CSS parsing from the browser.

– the freeing up of server connections to fulfill other requests.

Server level in xsp.properties:xsp.resources.aggregate=true

JavaScript/CSS Aggregation

XPages Preload

New notes.ini variable as of release 8.5.3

– Server and Client

– XPagesPreload=1

• LoadsXPages runtime components at server startup

– XPagesPreloadDB=Server!!Db.nsf/XPage.xsp,myLocalDb.nsf

• Loads XPages runtime Java classes for 1 to n applications

Resources

DAOS Wiki

• http://www-10.lotus.com/ldd/dominowiki.nsf/archive?openview&title=DAOS&type=cat&cat=null&tag=DAOS

DAOS Backup and Restore

• http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daosbackup-and-restore

DAOS Estimator

• http://www.ibm.com/support/docview.wss?rs=463&uid=swg24021920

DAOS Technical Overview

• http://www.nelotus.org/A55CBA/nelotus.nsf/87fde291d7608ada852564c9006eeba5/b6523e7f4b15585b862575b5004d3488/$FILE/DAOSTechnicalOverview_NELotus.pdf

Har du nogen spørgsmål

• Mail: eknori@eknori.de

• Blog: http://www.eknori.de

• Sametime: bleedyellow.com

• Twitter: eknori

Tak for

jeres opmærksomhed