Admin Tech Clash: Discussing Best (and Worst) Administration Practices from hundreds of customers
-
Upload
christoph-adler -
Category
Software
-
view
95 -
download
2
Transcript of Admin Tech Clash: Discussing Best (and Worst) Administration Practices from hundreds of customers
Admin Tech Clash: Discussing Best (and Worst) Administration Practices from
hundreds of customers
Presented by: Christoph Adler
Ben Menesi
Speakers
Christoph Adler
Senior Consultant at panagenda 15 years of IBM (Lotus) solutions
experience Since 2007:
IBM Notes Client Management ICS Infrastructure Analysis and
Optimization Good to know:
Lives in Germany, hence the funny accent
Frequent traveler and speaker at international conferences
No energy drinks – squirrel alert! Beer or wine? Beer!
Head of Product at Ytria IBM Notes Administrator and Developer
wearing both hats Working with Ytria since 2011
C API Development Server side Administration Enterprise IT Security
Good to know: Born Hungarian, raised in Hungary &
Switzerland, living in Canada for the past 5 years
Frequent traveler and speaker at international conferences
Whiskey or Vodka? Beer with Whiskey!
Ben Menesi
Introduction
Your collaboration infrastructure contains a TON of valuable data
In order to extract this data you need to knowWHERE to look
Visualization helps to see trends and patterns andunderstand relationships
IBM can help with IDD (IBM Domino Doublecheck)
We (Ytria / panagenda) can help with our softwaresolutions, services and expertise BUT
Scout‘s honor: We won‘t talk about ourproducts in this session!
Agenda
User Activity
Server-side Logging
Client-side Logs & Performance
Usage Activity
Usage Activity
Why should we care?
Understanding usage data allows us to find unused / heavily used applications
In the light of this data we can calculate resourcerequirements and optimize performance
We can profile our high impact users andprocesses and optimize licensing
Usage Activity
What data source should we use?
Activity Trends
DB User Activity
Log & Catalog
3rd Party Solutions
Valuable high level reports
Not very customizable
Lack of customization leads to inaccuracies
Very detailed logs Really crucial info
buried inside one text field
Unless through 3rd
party tools: NONO
Great amount of detail
Only available per database
To be discussed later
Amazing products out there
Some installed on server side
Some client side To be discussed later
Activity Trends
How does it work??
Log.nsf Catalog.nsf Names.nsf Statistics
Activity Trend Collector
Activity.nsf
Activity Trends
Activity data – advantages DB Usage (active / inactive), space growth, etc… User details (active / inactive), connection times, dbs and
network used etc...) Server tasks (HTTP, IMAP, LDAP, Replication etc...) Some visual reports
Activity data – disadvantages What you get is what you get – no digging deeper Raw data isn’t always used properly
Think about a DB marked “Active” with only servers and agents using it
Doesn’t properly handle space used percentages
Log & Catalog
Activity data – advantages It’s only two databases to analyze NOT hundreds or thousands You’ll get the full picture instead of only application usage
Think about usage of system databases or client overhead
Activity data – disadvantages Only contains data of the last 7 days
Actually: There is an undocumented parameter for the Log= notes.ini line:
LOG=logfilename, log_option, cant_touch_this, days, size, [activitydays]
Data structure (Later)
Log & Catalog
Understanding Usage log documents in log.nsf Important: Logging works on a per User, per Session bases! These documents contain a TON of raw data
Log & Catalog
What is an activity event?
Log & Catalog
What is an activity event?
Transactions SentFrmServer SentToServer Body
Database Reads Writes Transactions
Bytes Read Bytes Written DB Open Time
UserName Pathname Reads Writes
Log & Catalog
How to get valuable data out of log.nsf
• Create additional views and select the Activity form• Rearrange columns to make more sense (e.g. User \ db)
Log & Catalog
Details: Database usage document
• Note there is a 1400 activity entry maximum per database (FIFO)• There is also a 64K size limit for the user activity• More details in IBM Technote #1086245
Log & Catalog
Database activity: catalog.nsf
• related information, but different focus
• Full text index details• Replication information• ACL overview
Note:Domino does not distinguishbetween user, server or maintenance tasks activity at this level
Log & Catalog
Example visualizations: Domino Environment Overview
Log & Catalog
Example visualizations: Network Bandwith Demand
Log & Catalog
Example visualizations: User Impact
Log & Catalog
Example visualizations: Database Demand Characteristics
Log & Catalog
Example visualizations: Network demand
Log & Catalog
Real customer example: 15 European Locations 8000 Users 1200 Domino Apps Logistics
„We want to mobilize the 20 most heavily used applications that sales users throughout Europe access in read only mode. How can we find them?“
Log & Catalog
Domino Data Collect session logs from all servers log.nsf Collect nsf inventory from all servers catalog.nsf Collect person information from all users names.nsf
Non Domino Data Merge organizational info to person info
Analyze and Visualize
Log & Catalog
Applications used from Sales in read only mode
Log & Catalog
Applications used from Sales in read only mode
Log & Catalog
Applications used from Sales in read only mode
Log & Catalog
Applications used from Sales in read only mode
DB User Activity
Recorded User Activity User Activity can be a *VERY* rich source of data (incl. HTTP!) Tip: No_Force_Activity_Logging=0 (Default value) allows on ALL
dbs Tip: Set confidential to prevent users with lower than Designer
access
DB User Activity
Prior to ODS 48 Entry size allocated 44 bytes, object size 61600 bytes (max 1400)
ODS 48+ (undocumented improvement) Entry size 92 bytes. Object size 128800 bytes (same max 1400)
Curious: only two additional counters added, yet ODS48+ entry size is over 200% the size of ODS48 – why?
Date & Time
Reads Writes User
Date & Time
Reads Writes Updates Deletes User
DB User Activity
Does Notes only log what it displays? Nope
Read more: http://techlab.ytria.com/5448/lotus-notes-articles/interpret-user-activity-traces-portrait-global-usage
Reads Adds Updates Deletes
Data
NonData
Σ
Data
NonData
Σ
Data
NonData
Σ
Data
NonData
Σ
DB User Activity
Limitations Only available per database
Free tools available to facilitate working with this data across multiple databases: http://www.agecom.com.au/useractivity
Or implement via W32_NSFDbGetUserActivity Makes no difference between servers & users
Filter user names using free or paid tools Only maintains 1400 entries
Isn’t this the same for the log file activity documents? Wait, this doesn’t need to be an issue
DB User Activity
Customer Example: calculating usage patterns from usage activity
Entry Date
Entry Date#1400
#1
5 days
Entry Date
Entry Date
120 days
#1400
#1
Database A Database B
3rd Party Solutions
panagenda iDNA, ytria databaseEZ…
3rd Party products for usage analysis IDD (IBM Domino Doublecheck) free offer
http://goo.gl/maL0fo panagenda iDNA
https://www.panagenda.com/products/idna Ytria databaseEZ & consoleEZ
http://ytria.com/databaseezhttp://ytria.com/consoleEZ
Tableau (for visualization) http://www.tableau.com
3rd Party Solutions
Server-side Logs
Mining logs
Domino Server
The Admin
Event lifecycle
1.• Events take place
2. • Events are logged
3.• Some events captured
4• Few events addressed
5• Tons of events missed, angry users on the phone
Dissecting Log.nsf
Server tasks can log events to multiple types of documents! But no duplicate logs: different information! Example: Server 1 replicator opens session to Server 2 as per
Connection Document settings
Misc.
Replication
Dissecting Log.nsf
Miscellaneous events: what do we see? Some high level info (Server, Start & Finish time) and UI shows one text type item: Date + Time & Description
What is an event?
Time Status Code Severity Type
Addin Name Target Server Target DB Target User
Dissecting Log.nsf
Miscellaneous events – event attributes logged in multi-value Text items
EventListTime &
Description
EventSeverity
EventStatus
EventType
Event 1
Severity
HEX Status
Type
Time & Description
Event 2
Severity
HEX Status
Type
Time & Description
Event 3
Severity
HEX Status
Type
Dissecting Log.nsf
Can we get more data? We can use formulas and the list of Severities, Events & Status
Codes to get a better idea of each Misc. log document
Total # of events / document [Eventlist item]
– @Elements(Eventlist)
Total # of unique events [EventStatus item]
– @Elements(@Unique(EventStatus))
Dissecting Log.nsf
Highest severity / document [EventSeverity item]
– _sev:=@If(@Sort(@Unique(@Text(EventSeverity));[Ascending])[1]="0";@Sort(@Unique(@Text(EventSeverity));[Ascending])[2];@Sort(@Unique(@Text(EventSeverity));[Ascending])[1]);
– With the _sev variable, use icons to display (from DDM resources)
– @If(_sev="0";"iconNormal";_sev="1";"iconFatal";_sev="2";"iconFailure";_sev="3";"iconHigh";_sev="4";"iconLow";_sev="5";"iconNormal";"")+".gif“
– What you’ll get:
Dissecting Log.nsf
A view displaying the highest severity events encountered, # of events, unique events and all levels of severities per document:
More info here: http://www.slideshare.net/BenedekMenesi/ibm-connected-2015
Client-side Logs & Performance
ClientClocking
NOTE: All of this should only be done on “ONE” client – slows down client, creates huge log files!
Use the following to see and measure client performance CLIENT_CLOCK=1 CONSOLE_LOG_ENABLED=1 Debug_Outfile=c:\debug.txt Debug_Console=1 DEBUG_TCP_ALL=1 (use only if needed)
Fileserver or local disk Performance debugging is virtually impossible If on >= 8.5 try adding DEBUGGINGWCTENABLED=16386 in Notes
ClientClocking
Client_Clock=1 leads to a debug file of the following format:
ClientClocking
OpenNTF – NRPC Parser
Link: http://www.openntf.org/internal/home.nsf/project.xsp?action=openDocument&name=Notes%20RPC%20Parser
Developed by: Jamie Magee, MartinScott Consulting LLC, US Andrew Magerman, Magerman Collaborative Software GmbH,
Switzerland
ClientClocking
OpenNTF – NRPC Parser
Cache.ndk
We've all been “advised to”/guilty of deleting the cache.ndk in an effort to cure Notes performance problems/symptoms
• Some companies even delete cache.ndk on a scheduled basis
Cache.ndk
HOWEVER, let us explain the impact of that on Notes performance ... Increases traffic between client and server by 4,000% (forty times
more traffic, yes)(10,000+% if the data directory is on a network drive)
The more databases end users use, the worse it gets
There is always a root cause – deleting cache.ndk only fixes symptoms
Let's look at some detailed traffic analysis for just ONE application (the mail file) of ONE user ...
Cache.ndk
100 users =
150 MB vs. 3,7 MB
1000 users =
1.5 GB vs. 37 MB
Excluding Fileservers!
After deleting cache.ndk Keeping / NOT deleting cache.ndk
ODS
On Disk Structure
ODS = On Disk Structure ODS 16 = Notes 2 ODS 17 = Notes 3 ODS 20 = Notes 4 (or Templates) ODS 41 = Notes 5 ODS 43 = Notes 6 & 7 ODS 48 = Notes 8 ODS 51 = Notes 8.5 & 9 ODS 52 = Notes 9.0.1
The difference between ODS 43 and 52 = up to 80% LESS FILE I/OThis helps with ANY client AND server hardware
On Disk Structure
CAUTION:The ODS is not automatically updated when upgrading clients
Except for names.nsf, bookmark.nsf, and desktop*.dsk/ndk – IF –you upgrade properly (think ini:Setup/TemplateSetup=)
ODS 51 requires Notes >= 8.5 and CREATE_R85_DATABASES=1 in notes.ini for NEW db's
ODS 52 requires Notes >= 9.0.1 and CREATE_R9_DATABASES=1 in notes.ini for NEW db's
On Disk Structure
Good News for Notes > 8.5 Use NSF_UpdateODS=1 with CREATE_R85/9_DATABASES=1 for a
one-time upgrade of most local databases NOTE: USE WITH EXTREME CARE IF YOUR DATA DIRECTORIES
ARE ON A NETWORK DRIVE! NOTE: End users can't access databases during compact
For databases on server: no more downtime required, use: Compact [dbpath] –Replica to create replica in the
background (new in domino 9) More info: https://www-
01.ibm.com/support/docview.wss?uid=swg27039379&aid=1
On Disk Structure
Impact examples Startup time of a Notes 8.5.3 client with 3 ODS 20 apps in Notes
data = 10 seconds After ODS upgrade: 2 seconds Reduced File I/O of your disks/SAN/NAS after ODS 41 to 52 upgrade
by 60% Removing 70% of all old files in Data directories on SAN/NAS = 45%
less managed storage (backup)
The bad news: There is no way to tell what is “out there” 3rd party tools or random inspections are highly recommended News for Notes > 8.5
On Disk Structure
IBM helps!
Encryption/Compression
Slow clients
Port settings are included in the notes.ini Type / Active / Encryption / Compression / Timeout
Bad idea Remove LAN0 / COMx / DisabledPorts
Good ideaUse TCPIP according to the client type PORTS = TCPIP TCPIP = TCP,0,15,0,,12320, Desktops TCPIP = TCP,0,15,0,,45088, Laptops TCPIP = TCP,0,15,0,,12288, Citrix / VDI
Riverbed !?Don‘t forget your Servers!
Q & A
Thank you