NSD ‘Crash Course’
Gregory Engels, KompurityILUG 2010
Some Rights Reserved
Copyright 2010 by Gregory Engels, Kompurity. Except where otherwise noted, this work is licensed under the Creative Commons Attribution Share Alike 3.0 License
About meGregory EngelsBorn in Moscow, live in Frankfurt, Germany since 1989Been around Lotus Notes since 1998Advanced Certified Domino Developer and Admin and Security R5, R6, R7, R8Co-Author "Migrating Microsoft Exchange 2000/2003 to IBM Lotus Notes 7"Founder Kompurity http://www.kompurity.deSpecialize in Hosting, Infrastructure management
Blog: http://inotes.de
Twitter: dichter
Agenda
• NSD
• ADC
• Upgrading NSD
• LND
• Notes Systems Diagnostic
• binary on W32 and iSeries
• shell script nsd.sh on Unix
What is NSD
Running the NSD
• Manually
• Automatically as part of Fault Recovery
• Enabled by default
• Enable it in the Server document
• Go enable it now.
Fault Recovery
Help is on the way
• nsd -help
Automated Data Collection (ADC)
Enable in the Config Doc
Enable in Desktop Policy
Fault Reports DB
Put this in the nsd.ini
• canceldisable=1
• (BOFH tip of the day)
Real Life
• ADC Enabled 9 Month ago
• User Population c.a. 6000 Users
• 1700 Users have created at least 1 NSD
• c.a. 13000 Client NSD
• 13 Users (0.05%) created 1700 NSD (14%)
How often does Notes Crashes?
0
10
20
30
40
50
60
70
80
90
100
1 5 9 14 21 30 52 75 163 1258
Number of Users that had N CrashesOccurency
Number of NSDs per User
0
375
750
1125
1500
Occurences
NSD’s per Day
0
18
36
54
72
90
22.08.2009 25.08.2009 28.08.2009 31.08.2009 03.09.2009 06.09.2009 09.09.2009 12.09.2009 15.09.2009 18.09.2009 21.09.2009 24.09.2009
Upgrading NSD
• See TN #4013182 for ftp links
• See TN #1233676 for list of SPR’s fixed
• Version of NSD is <NotesVersion><Fixpack>.<YNNN>
• NNN = Days since year begin
• example: NSD 8.0.10.8057 (Release 8.0.1) = Feb 27 2008 (57 Days since Jan 01)
Reading the NSD log
Finding the log
• Logs are stored in the IBM_TECHNICAL_SUPPORT subdirectory
• in the Lotus Notes Domino Fault Reports Database lndfr.nsf
• ND6 - nsd_all_<Platform>_<Host>_MM_DD@HH_MM.log • ND7 -
nsd_<Platform>_<ServerName>_YYYY_MM_DD@HH_MM_SS.log
Contents of the NSD
• Rather verbose
• Process Info (Call Stack)
• Memcheck (Domino Memory Objects)
• System Information
• Environment Information
• Process Information (Call Stacks) – Process Information is composed of the list of all running processes system wide, followed by a list of all Domino specific processes.
• the code path involved in a particular problem.
• Memcheck (Domino Memory Objects) – The Memcheck section dumps information about Domino-specific structures.
• memory pools (both private and shared)
• list of open resources such as open database, open view, open documents, connected users, and open files.
• System Information – This section provides information regarding version of OS,
• kernel configurations, patch information, disk information, network connections, and memory usage.
• Environment Information:
• user environment,
• notes.ini,
• file listing
Finding the fault
• Look for the words FATAL, PANIC, ETC### FATAL THREAD 67/135 [ nSERVER:0908: 2692] ### FP=0x0ae5ec70, PC=0x6018eb23, SP=0x0ae5e2f8 ### stkbase=0ae60000, total stksize=262144, used stksize=2424 ### EAX=0x010d088c, EBX=0x00000000, ECX=0x00ba0000, EDX=0x00ba0000 ### ESI=0x0ae5e904, EDI=0x00001d34, CS=0x0000001b, SS=0x00000023 ### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010202 Exception code: c0000005 (ACCESS_VIOLATION) Process ID: Thread ID
Reading the Callstack############################################################
### FATAL THREAD 1/4 [ncompact: 3044: 2620]### FP=0x0012e644, PC=0x60193297, SP=0x0012dacc, stksize=2936### EAX=0x00e3088c, EBX=0x00000000, ECX=0x00900000, EDX=0x00900000### ESI=0x0012e1d8, EDI=0x00d495d8, CS=0x0000001b, SS=0x00000023### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010202Exception code: c0000005 (ACCESS_VIOLATION)############################################################@[ 1] 0x60193297 nnotes._Panic@4+631 (60a30013)@[ 2] 0x600027a6 nnotes._LockHandle@12+294 (3963e26,12e674,12e680)@[ 3] 0x600025e7 nnotes._OSLockObject@4+23 (60c7df40)@[ 4] 0x606c09cc nnotes.CArchSetPolicy::GetPrivateArchiveSettings+3404 (1)@[ 5] 0x606bf1a8 nnotes.CArchSetPolicy::GetArchiveSettings+1032 (0)@[ 6] 0x606be9f9 nnotes.CArchSetPolicy::Init+553 (33a0a99)@[ 7] 0x606c7554 nnotes._PolicyCheckServerArchiveEnabled@4+52 (a9)@[ 8] 0x606e2fc1 nnotes._CheckServerArchiveEnabled+113 (10900a9)@[ 9] 0x00402111 nCompact._CompactFile@16+529 (0,a9,81000000,0)@[10] 0x00401b61 nCompact._AddInMain@12+2913 (3,3,398644)@[11] 0x004028cf nCompact._NotesMain@8+47 (3,400000)@[12] 0x004029d6 nCompact._notes_main+214 (0,0)@[13] 0x004028f6 nCompact._main+22 (3,3b2da0)@[14] 0x00402cc1 nCompact._mainCRTStartup+197 (0,0)@[15] 0x7c816ff7 KERNEL32.RegisterWaitForInputIdle+105 (402bfc,0)
Pid : Tid
Finding the Resources
• Search for “Open Databases”
• Search for “Open Documents”
DBH NOTEID HANDLE CLASS FLAGS IsProf #Pools #Items Size Database
531 7330 0x24ff 0x0001 0x0200 Yes 1 4 2984 d:\notedata\drmail\jsmith.nsf
Open By: CN=John Smith/O=ACME/C=US
Flags2 = 0x0404
Flags3 = 0x0000
OrigHDB = 531
First Item = [ 9471: 836]
Last Item = [ 9471: 1228]
Non-pool size : 0
Note Class Description
• 0x0001 Data Note (document)
• 0x0004 Form Note
• 0x0008 View note
• 0x0040 ACL Note
• 0x0200 Agent Note
• 0x0800 Replication Formula Note
Search for VTread info
• Search on KEYWORDs “Resource Usage” or “Resource Usage Summary”
• Search on KEYWORD “Process” until you located the necessary process ID
• Search on the PHYSICAL THREAD ID of the thread you are interested in (cosection after you have looked at call stacks)
Troubleshooting NSD
• Try NSD -info
• If this runs to completion, perhaps there is a problem with the pid.nbf file
• Try upgrading NSD
• Check if symbols are installed
• lotusnotes.sym
LND Tool
• See TN # 4019151 for download link
• ftp://ftp.software.ibm.com/software/lotus/tools/LND_2.8_Setup.zip
Auto KB Seatch
See the Process Resources
Works on Right-Click in Explorer
Questions
Thank You!
• Please fill out the evaluation (so I can get back next year)
• Contact: Gregory.Engels ※ kompurity.deBlog http://inotes.detwitter: dichter
Backup
NSD parameters
• nsd –stack (collects only call stacks, speeds execution)
• nsd –info (collects only system info)
• nsd –noinfo (collects all but info)
• nsd –memcheck (collects only memcheck info)
• nsd –nomemcheck (collects all but memcheck)
• nsd –perf (collects process memory usage)
• nsd –noperf (collects all but performance data)
• nsd –handles (collects OS level handle info)
• nsd –nohandles (collects all but handle info)
• nsd –kill (kills all notes process and associated memory)
• nsd –monitor (attaches and waits for exceptions)
• nsd –p (runs against a specific process – call stacks only)
Top Related