MAIL MANAGEMENT

Gabriella DavisTechnical Director

The Turtle Partnershipgabriella@turtlepartnership.com

Why Mail Management?

Use of mail whether it’s Notes, Web, Traveler or even IMAP/POP is probably the most significant interaction with Domino that your users have

Many things affect Mail performance and the user experience of mail and you can do a lot to optimise it and make user’s happier

Much of the work is done on the server and to the databases and server configuration and is nothing to do with their client

Mail Databases

Database Size

Database size isn’t just about disk space, it’s also about performance

It takes exponentially more work for a Domino server to perform a view rebuild or update a full text index on a 5GB database than it does on a 500MB one

This is also reflected in client performance, slowness to open views and move between folders as well as create new folders and file messages

Maintaining a managed database size can be done using quotas but you should never deploy a quota without a warning level and the warning level should be no more than 80% of the quote size

Previously we used quotas and archiving to maintain databases sizes

both of these things punished the users

DAOS is a better alternative

Inbox MaintenanceKeeping down the number of messages in an inbox also helps performance

Once someone has 6000 inbox messages they really aren’t going to file them in folders ever

Inbox Maintenance is an agent in the Mail file

Run by Adminp on Mail Server

Defined by Server Document or Mail Policy

Inbox Maintenance - Requirements

Mail8 template

Directory on server based on v8 template

Adminp process to be running

Server document schedule defined

Mail Policy ConfigurationDifferent inbox maintenance settings can be applied to users via the polices Mail settings - Mail - Basics*

Tell Adminp Process mb - forces an immediate application of the policy for each mail user

The schedule is still determined by the server document

DDM.NSF

LOG.NSF

ADMINP_VERBOSE_POLL_TASK=2

*Overridden by Server document settings

Design

Maintaining up to date database designs impacts not just the user experience but also performance

Updated mail templates have more efficient code and behaviour

OOO for example in v8 mail designs or later

The design version should never be newer than either the server or client versions but can always be older

Database Properties

Keeping databases on the newest ODS (currently 51) enables you to take advantage of the newest server features and performance improvements

Some database properties won’t be set in older files that have never been changed. You’ll need to set those as soon as you can

LZ1 , replicating unread marks

Always compact and rebuild the indexes when you upgrade the ODS of a database

compact -c (copy style)

updall -RX (rebuild view indexes and full text indexes)

Managed Mail ReplicasLocal replicas managed by desktop profiles

Users don’t need to understand the concepts of replication or outboxes

Behaves as though you were on a server replica

Better performance as it avoids network latency issues

The local replica is automatically created by the policy

If it needs fixing or compacting it does this automatically too and fails the user over to a server replica

Existing users with local replicas can be changed to managed replicas via a policy settings

For remote users it’s a better alternative to trying to access server based mail over a WAN connection

DAOS

Domino Attachment Object Store (DAOS)

DAOS was new with Domino 8.5

but really started working well in 8.5.1 and later

In a DAOS enabled mail file, all the attachments are removed and stored as flat encrypted files on the file system called NLOs

the Domino mail file contains a file reference to that file in each message

DAOS is controlled by DAOSManager which is a Domino task that acts at a “sub Domino” level.

to the user and to the Domino administrator nothing really has changed

Some DAOS Behaviours

If I send a message to 10 people with a file attachment, that message will be removed and stored as a NLO only once

Each person’s copy of the message will have a pointer to the same file

Only when everyone who had a pointer to that file, deletes the message, will the file itself be marked for purging

The default is to prune deleted / unused NLOs after 30 days

If I send a message to Chris with a file attachment, that file attachment will be removed from the message and stored as a NLO if the file size is above a minimum size limit set in the server document

A local replica of a DAOS enabled database will have all the attachments still in it. As the replica is created the replication task creates each document with its attachment taken from the DAOS store

File Sizes

DAOS enabled files have both a physical and logical file size

The logical file size is how big the database would be if it contained all attachments. Such as if you replicated it locally or to a non DAOS server

The physical file size is how big the NSF actually is

The difference between the two is your DAOS disk saving

I have seen anywhere from 40 - 90% reduction in NSF size

Do you need to enable archiving if you can simply reduce the NSF size using DAOS?

Backups

Backups are the one thing that changes when you enable DAOS

If you backup a DAOS enabled file you are only backing up the NSF, unless you backup software is “DAOS aware”.

The NLOs are read only files so they would usually only be backed up incrementally and they sit outside the Domino data directory so your regular backup routine will miss them

By default deleted NLOs are only pruned after 30 days so any backups restored within 30 days will still have NLOs in place on the server

If you don’t have DAOS aware backup software then you have to backup both the NSFs and the NLOs and then match them up on restore

DAOS aware backup software will backup both the NSF and all its matching NLOs and can restore the two together

How To Enable DAOSMake sure your servers are 8.5.1 or later (you can do it on 8.5 but I wouldn’t recommend it)

DAOS needs a disk separate from the location of NSFs

You will kill the server with disk I/O and performance issues otherwise

DAOS needs transaction logs enabled. I often use the same disk for DAOS and transaction logs

The database needs to be ODS51 or later

Add the value Create_R85_Databases=1 to the server notes.ini and then compact the databases with a -c switch to upgrade the ODS

Update the server document with the location of the transaction logs and where the NLOs should be stored

Compact the required database with the -DAOS ON switch

Useful DAOS Commands

Tell DAOSMgr Status

Tell DAOSMgr Status Catalog

Tell DAOSMgr Resync

if Catalog needs rebuilding

Tell DAOSMgr Status DBSummary

all databases being managed by DAOS

Tell DAOSMgr Status mail\gdavis.nsf

how many NLOs etc in use in my mail file

Tell DAOSMgr LISTNLO ALL mail\gdavis.nsf (-o filename.txt)

List of all NLOs referenced in my mail file (for restore purposes)

Some Things To Remember

Replicating a DAOS enabled database to another server that doesn’t have DAOS will result in all the file attachments being stored inside the replica on that destination server

The disk space required will be the logical file size

The server document has a setting for the minimum attachment size before the attachment is removed. The default is 64k

on 8.5 it was 4k so verify your settings if you deployed on 8.5 originally

Quotas apply to the logical, not physical file size

In a cluster NLOs won’t be resent with messages if they already exist on the destination server

Standardising on LZ1 compression for attachments as a database property means that duplicate NLO files won’t be created for both Huffman and LZ1

More Things To Remember

Servers don’t all have to be DAOS enabled

even servers in a cluster

Not all databases on a DAOS enabled server need to be enabled for DAOS

The design of the database doesn’t matter to DAOS which is sub domino so you can DAOS enable a v6 database as long as its ODS is 51

Performance for Domino server tasks is greatly improved when the databases physical size is so much smaller

view indexes, user interaction, folder moves and searches all quicker

Directories

What Have Directories To Do With Mail?

Addressing and duplicate addressing

Vulnerability for directory harvesting or spam attacks

Mostly though - it’s about client performance

Type ahead addressing uses the directories, if the directories are slow so will the “Notes client” be

Finding matches for addresses or duplicate addresses is reliant upon the Directories

Make sure you complete the Directory server field on location document

Avoid local replicas of names.nsf

If you must use a mobile directory catalog but only for “offline” users otherwise it will become out of sync and conflict with the server replica of names.nsf

Directory Assistance

If you use Directory Assistance, make sure your directories are all working using the settings you expect. Not using a replica on a server in Singapore

Sh XDir

Authentication-only / authorisation-only secondary directories prevents them being used for mail addressing or routing

You have both a Domino™ and an LDAP directory that contain some identical names.

You do not want to use the LDAP directory's names for mailing.

Your mail clients are experiencing "Ambiguous name" dialog boxes when sending mail.

DA Modified Form for LDAP

Dirlint

Server Task (Load Dirlint)

Scans a directory for inconsistencies

inconsistencies in naming hierarchy

invalid syntax in directory names

valid group members

Load Dirlint -<directory> <directory>

Load Dirlint -NoDAorCascaded

Traveler and Syncing

Traveler Under the Hood

What’s All This Syncing About?

It is how the Traveler server scans/syncs data to devices

Consists of three different tasks

Server thread

Prime sync thread

Worker threads

Note

From 8.5.2 onwards, threads are mostly dynamic and now rarely need modifying

We have had to do this, however, when doing large Traveler server moves

From one server to another

Server ThreadThere is a SINGLE thread on the Traveler server that scans target servers

Target servers are servers that house users utilizing the Traveler service

Traveler issues a call to each server in turn

NSFGetChangedDB

Lists all changed databases since last scan

Very fast/efficient request

Traveler is served the list of all changed databases

Parses list and keeps changed databases that it is interested in

I.e., mail files

Passes the list of changed mail files to the Prime Sync Thread

By default, Traveler will scan the same server at a minimum of three seconds

Prime Sync Thread

Scans the target mail files

One prime sync can work with one mail file at a time

Identifies what has changed in the mail file

I.e., what is out of sync

Passes to device sync thread/worker thread

By default, there are 20 prime sync threads on a Traveler server

Can be increased using NTSConfig.xml

TSS_PrimeSync_Threads=n

Once complete, passed to device sync thread/worker thread

Device Sync Thread/Worker Thread

The thread that does the work

Sends changed data to device

Retrieves changed data from device

Touchpoint thread between mail file and device

Limit of 5,000 device threads

Worker thread is for internal Traveler communication

Limit of 5,000 worker threads

Traveler Threads

Possible Traveler Problems

Traveler uses a non-replicating Derby database that is critical, specific to a server, and ensures only incremental updates are sent to devicesThe Traveler task is a sub-task that is reliant on HTTP. It should be started after HTTP and stopped before HTTP.Lookups are performed against each user's mail server and honor Directory Assistance, but can instead be done entirely on the Traveler server to improve performanceSince the server threads will query for every changed database on every home mail server the Traveler server knows of for its users, keeping minimal hops between the Traveler server and the mail servers is important

As is having mail servers without thousands of non mail databases on themThe worker thread is reliant upon disk performance for writing and reading data from the Derby database

beware of disk fragmentationpoor performing disk affects traveler performance

Delete ntsclcache.nsf if you have cluster failover, routing or corruption issues. It will recreate itself

Common Mail Problems

Mail Delivery - What Do We Want

Mail from who you want to where you want

SMTP servers just managing inbound mail intended for you

A significant reduction of receiving spam from something like 90% of mail traffic to less than 10%

Ability for POP or IMAP users to relay mail without compromising security and risking blacklisting

From people you want delivered to the servers you want

Why is this a problem

Your MX records determine the inbound route of externally addressed mail but they can also be a bottleneck on your network

Any server with a SMTP listener which is running under default configuration can vulnerable

If you’re inside a firewall and one of your servers starts relaying mail you entire ip range can be blacklisted

Dictionary attacks and partial matches for SMTP addresses overload your servers and users with mail you don’t want and haven’t anticipated

Allowing authenticated relaying exposes your server to authentication by any identity with a person document and http password

Spammers know this and will try password cracks against common names

Your users hold the mail system responsible for delivering them unwanted mail and making it more difficult for them to find good mail

You network, disk space and bandwidth are not there to process mail you don’t even want

What Can We Do To Fix?Enable ‘Internet Sites’ on each server document so SMTP can’t be started by accident

Have multiple MX entries and at least one ‘failover’ entry with a very low priority

Server Configuration Document - Router - SMTP Inbound Restrictions

Restrict connecting hosts if you can

Configure fullname lookups only not partial matches and verify that name exists in a Directory before receiving mail

Reject ambiguous names and if you can group names

Perform anti relay checks for authenticated users

SMTPVerifyAuthenticatedSender=1

Even with this, Domino has to handle rejecting invalid inbound mail. Consider moving that initial SMTP listener to another source such as the Postini service , Lotus Protector or similar

What Do We Want?

Controlling where the mail comes into allows you to load balance traffic around your network.

Multiple MX records are part of your DR solution, once mail is received anywhere on your network you can route it or deliver it to the user

Being responsible in how you handle outbound mail protects you from being blacklisted

Pre-emptively blocking or denying all but complete valid addresses makes you unappealing for spammers

What Could Be Going Wrong?Spammers rely on connecting to your server and sending mail to users without knowing their addresses in advance (a DHA)

Domino servers that are enabled for SMTP listening have port 25 listening for any connecting hosts wanting to deliver mail

Server configuration documents accept ‘fuzzy matches’ on usernames unless you configure otherwise

Group names (which are often very common) are accepted for inbound mail unless you specify otherwise

If your servers are found to be sending out spam mail especially that didn’t originate from your domain you will find yourself quickly blacklisted

Enabling authenticated users to relay validates any user with a HTTP Password in any of your directories

If one of your servers behind your firewall does this then the NAT address (which could be the address of all your servers) will be blocked

What Can We Do To Fix?

Enable ‘Internet Sites’ on each server document so SMTP can’t be started by accident

In the server configuration document

Restrict connecting hosts if you are using a spam filter service in the server configuration document

Configure fullname lookups only not partial matches and verify that name exists in a Directory before receiving mail

Reject ambiguous names and, if you can, group names

Perform anti relay checks for authenticated users

SMTPVerifyAuthenticatedSender=1

Why This Works

Pre-emptively blocking or denying all but complete valid addresses makes you unappealing for spammers

Being responsible in how you handle outbound mail protects you from being blacklisted

Busy Mail Servers?

You have a constantly busy mail server and you don’t know if it’s running as efficiently as it could.

What settings can you change to improve mail routing speeds and could they make things worse?

What Do We Want?

The right number of mailboxes for the router to perform optimally

The right number of delivery threads to ensure local mail is being delivered and transfer threads to ensure mail is pushed to destination servers

Making it easy for you to track mail routing problems

Notifications of any mail bottlenecks

What Could Be Going Wrong?

The default configuration for a new Domino server is a single mailbox created and used by the router task

The Domino server will automatically set the number of Delivery and Transfer threads on your server according to memory availability

but that assumes your server is performing other Domino tasks too and isn’t dedicated for mail

the default setting for transfer threads is for one thread per destination at a time

If you chose to hold undeliverable mail as a way of combatting Spam you are overloading the mailbox with dead mail which effects how well the router task can process the live mail

You don’t have delayed mail notifications configured or mail probes configured to tell you if there are bottlenecks or problems

How Do We Fix It?In the server configuration document

Set number of mailboxes to be 2, 3 or 4

Mail.Mailbox.AccessConflicts / Mail.Mailbox.Accesses > 2% means time for a new mailbox

Delivery and Transfer threads can be set here but if not are calculated by the router based upon server memory and resources

Tell Router Sh Queues displays current settings

If you set the server to hold undeliverable mail you will need to write a mailbox agent to clear that mail out periodically so it doesn’t effect router performance

Setting notifications for delayed mail can let you know early if there are routing problems to a particular destination

Using Notes Named Networks means there are no connection documents and no easy to follow mail topology

Configure the DDM probes for messaging such as the

“Mail flow statistic check”

“Transfer Queue check”

Why This WorksAccepting default options won’t stop your server from delivering mail, but it will stop it from being as efficient as it can be

The router task uses threads to access the mailboxes, the more efficiently it can do this the faster your mail will be delivered

If your server does nothing other than route mail you can manually increase the number of transfer threads and concurrent transfer threads to improve throughput

Being pre-warned of bottlenecks allow you to deal with a problem or re-route the traffic before it becomes a backlog of too much mail

Having a defined mail topology, even in a small organisation makes it easier to trace mail and customise routing

If you have a cluster you can route the mail via a cluster mate without it being specifically defined in a connection document. Doing this for all hops can give you an entire alternate mail routing topology.

Don’t Go Too Far

Beyond what’s needed, more mailboxes is not always better. Giving the router additional mailboxes to process can be detrimental to performance

More transfer threads don’t always do what you think if you haven’t configured multiple concurrent threads to the same destination

Forcing the server to have more delivery and transfer threads than it thinks it needs, takes resources from other server activity

Ugly Emails

Your users complain of writing beautifully formatted mail and yet to people outside the company it looks completely wrong.

Internal messages forwarded between users sometimes lose a lot of formatting

What Do We Want?

Mail created in Notes or iNotes by your users, with custom formatting and layout shouldn’t be noticeably altered in transit

Internal mail should retain formatting as it is forwarded and replied to

You need to explain to your users the behaviour and limitations of mail routing

What Could Be Going Wrong?The default configuration for routing mail externally is to convert to MIME as plain text

User Mail Files contain mixed message formats

The default setting for internal mail is to be generated and delivered as Notes Rich Text

but if read via iNotes/Browser it converts to MIME on the fly

The default setting for inbound mail is that it remains and is stored in MIME format in the user’s mail file.

but if read via Notes it converts to Rich Text on the fly

The default setting for a Notes user is for an internet addressed message to be converted to MIME by the client as it sends

Different versions of Notes and Domino will convert differently so a mixed environment will return mixed results

How Do We Fix It?In the server configuration document

Set outbound MIME conversion to HTML or HTML and Plain Text (if you send to very old mail systems who can’t read HTML messages)

If your users are on different client versions then set their location documents to send internet mail as rich text

The server will then handle the mail conversion to MIME for everyone

Make users aware of limitations / behaviour of mail rendering

If they read a Notes mail message via iNotes and reply to it, they are replying with a converted MIME version which will look similar but not identical. Notes features such as sections and tabbed tables are most noticeable

Similarly if a Notes user receives a MIME message and opens it, the quality of the rendering is dependent upon their Notes client version. Two users with differing versions may see different results.

Some features are not compatible with sending internet mail, these include ‘letterhead’ and ‘mood stamp’

If a user complains that a message they sent ‘looks terrible’ get the message headers before doing anything else

Never accept a forwarded copy of the message as proof

Why This Works

Understanding how mail is stored, formatted and rendered for reading explains why you will see different effects from different messages and clients

You can only control your own mail, so making sure you send out as MIME and HTML is the best you can do to preserve fidelity

User dissatisfaction stems often from unrealistic or misconceived expectations.

Don’t Go Too Far

Standardise mail if you have radically different client and server versions and some of them are pre v7

Configure clients to send as Rich Text so the server does the conversion

Don’t let all servers do MIME conversion, just the newer ones

most sites have outbound routing hubs they can use

Make sure you allow MIME routing of messages within your domain so mail isn’t converted to Rich Text as it moves between servers