Writing Wrongs: Expressive Writing to Facilitate Forgiveness
Soccnx11 Two wrongs don't make a right - Troubleshooting Connections
-
Upload
nico-meisenzahl -
Category
Technology
-
view
1.024 -
download
0
Transcript of Soccnx11 Two wrongs don't make a right - Troubleshooting Connections
![Page 1: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/1.jpg)
Social Connections 11 Chicago, June 1-2 2017
Two wrongs don’t make a right –Troubleshooting Connections
Terri Warren, panagenda
@TerriLLBean
Nico Meisenzahl, panagenda
@nmeisenzahl
![Page 2: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/2.jpg)
PLATINUM SPONSORS
GOLD SPONSORS
SILVER SPONSORS
![Page 3: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/3.jpg)
Social Connections 11 Chicago, June 1-2 2017
Nico Meisenzahl
• Consultant at panagenda
• IBM Connections since version 3.0 / 2010
• IBM Notes / Domino since 2008
• Focusing in ICS
• Deployment & consulting
• Optimization and migration
• “panagendian” since 2016
• IBM Champion
@nmeisenzahl
linkedin.com/in/nicomeisenzahl
meisenzahl.org
nico.meisenzahl
+49 170 7355081
![Page 4: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/4.jpg)
Social Connections 11 Chicago, June 1-2 2017
Terri Warren
@TerriLLBean
linkedin.com/in/terrillbean
live:TerriLLBean
• Consultant at panagenda (since 2016)
• In my recent past I was a developer on IBM
Connections, IBM Domino
(Directory/Security) and an SRE on Watson
• Frequent presenter at IBM conferences
focusing in ICS
• Deployment & consulting
• Optimization and migration
![Page 5: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/5.jpg)
Social Connections 11 Chicago, June 1-2 2017
Agenda
• Howto: Troubleshooting
• Troubleshooting…• Connections itself
• Backend like Spectrum CfC, DB2, TDI, WebSphere, Directory, SSO
• Optional add-ons like Orient Me, Docs, FEB, Cognos, CCM
• Tools
![Page 6: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/6.jpg)
Social Connections 11 Chicago, June 1-2 2017
Howto: Troubleshooting
![Page 7: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/7.jpg)
Social Connections 11 Chicago, June 1-2 2017
Reproduce the error
• Reproducible and/or periodically?
• A sequence error?
• Client-side or server-side problem?
• Analyze the root cause
![Page 8: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/8.jpg)
Social Connections 11 Chicago, June 1-2 2017
Be aware of the big picture
• Client-side problems• Debug in different Browsers (IE, FF, Chrome)
• Do not use a server IE
• Server-side: IBM Connections is based on many components• debug on “high level” first
• get an overview which backend service is causing the error
![Page 9: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/9.jpg)
Social Connections 11 Chicago, June 1-2 2017
Configuration changes
• Changes in…
• Connections configuration
• Backend (WebSphere, Database, HTTP, CfC)
• Firewall or network
• OS, hardware or VM
Tip: Even the smallest configuration change can have big consequences!
![Page 10: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/10.jpg)
Social Connections 11 Chicago, June 1-2 2017
Analyze log files and browser
• Analyse log files• Atom.io, Notepad++ or less/tail
• Baretail or tail –f
• kubectl logs
• ELK stack
• Tools for client-side problems• Firebug or Developer-Tools
• BurpSuite or Fiddler
![Page 11: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/11.jpg)
Social Connections 11 Chicago, June 1-2 2017
Analyze root cause
• Find a hint inside the log
• Network timeout or DNS
• SQL errors
• LDAP errors
• Syntax errors in configuration files „xxx-config.xml“
• Error stack
Tip: In a clustered environment, start and analyze only one
Node (if possible)
![Page 12: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/12.jpg)
Social Connections 11 Chicago, June 1-2 2017
Find support
• Knowledge Center https://goo.gl/up6cxG
Troubleshooting Tips https://goo.gl/IaVinx
• IBM Connections Forum http://goo.gl/CVvQCU
• Community Blogs and/or Chats
• Fix Central
• PMR
![Page 13: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/13.jpg)
Social Connections 11 Chicago, June 1-2 2017
Troubleshooting
![Page 14: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/14.jpg)
Social Connections 11 Chicago, June 1-2 2017
WebSphere Application Server logs
• SystemOut.log
• SystemErr.log
• trace.log (if tracing is enabled)
• Log path:• <wasroot>/profiles/<profilename>/logs/<servername>/
![Page 15: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/15.jpg)
Social Connections 11 Chicago, June 1-2 2017
Analyze WAS log files
• Time stap: 24h time stamp with milli-seconds
• Thread id: eight character hexadecimal value
• Short name: typically java class name
• Event type: one character only (E, W, I,…)
• Message identifier: String based on component
• Message: Some information
![Page 16: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/16.jpg)
Social Connections 11 Chicago, June 1-2 2017
WAS Event types
• F - Fatal message
• E - Error message
• W - Warning message
• A - Audit message
• I - Informational message
• C - Configuration message
• D - Detail message
• O - Messages that are written directly to System.out by an application
• R - Messages that are written directly to System.err by an application
• Z - Place holder to indicate type was not recognized
![Page 17: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/17.jpg)
Social Connections 11 Chicago, June 1-2 2017
WAS Message identifier
• Prefix by Application or Server (CLFRW)
• Specific application code (0042)
• Event Type (I)
![Page 18: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/18.jpg)
Social Connections 11 Chicago, June 1-2 2017
Read trace stack
• First line displays key information
• “Caused by” displays root cause
![Page 19: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/19.jpg)
Social Connections 11 Chicago, June 1-2 2017
Environment information
• First log lines on server startup
• WebSphere version
• OS version, Process id
• Installation path
• Java version
![Page 20: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/20.jpg)
Social Connections 11 Chicago, June 1-2 2017
Enable tracing
• Enable tracing using ISC
• Runtime or configuration
only
• Define tracing based on
• App prefix / error stack
• Must gather (PMR)
![Page 21: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/21.jpg)
Social Connections 11 Chicago, June 1-2 2017
Search issues
• http(s)://<fqdn>/search/serverStatus
• Display index, seedlists, log information
• Data is displayed for one node only• Using node fqdn to access different nodes
Tip: This will create persistent files on your hard disk!
![Page 22: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/22.jpg)
Social Connections 11 Chicago, June 1-2 2017
Debug Search
• Search queries (runtime)
• com.ibm.connections.search.index.searching.*=all
• Crawling & seedlists
• com.ibm.connections.search.index.indexing.*=all:
com.ibm.connections.search.seedlist.*=all:
com.ibm.connections.httpClient.*=all
![Page 23: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/23.jpg)
Social Connections 11 Chicago, June 1-2 2017
Recreate Search Index
• SearchService.startBackgroundIndex()
• Crawls seedlists
• Extracts the file content
• Create index
Tip: Use „all_configured“ to index all apps
![Page 24: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/24.jpg)
Social Connections 11 Chicago, June 1-2 2017
CLFRW0394E: Search indexing of services
...
• Search index not ready
• interruption at index creation• CLFRW0283E: Search has encountered a problem
while crawling
• CLFRW0027E: Error Indexing component <app> for search
• INDEX.READY file not present
• Recreate and enable tracing
![Page 25: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/25.jpg)
Social Connections 11 Chicago, June 1-2 2017
Database connections
• Check datasouces• ISC – Resources – JDBC – Data sources
• Check logs for more information
• DB2 server log• <instanceroot>/sqllib/db2dump/
• db2diag.log
• db2diag.xxx.log (log rotation, you should enable this!)
Tip: Oracle users have password expiration enabled by default!
![Page 26: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/26.jpg)
Social Connections 11 Chicago, June 1-2 2017
HTTP Server (IHS & Plugins) logs
• IBM HTTP Server
• <installroot>/logs/
• error_log
• access_log
• based on configuration
• WebSphere AppServer Plugins
• <installroot>/logs/<webserver>/http_plugin.log
![Page 27: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/27.jpg)
Social Connections 11 Chicago, June 1-2 2017
HTTP 404 not found
• Outdated Plugin configuration
• Restart IHS
• WAS Plugin configuration issue
• http_plugin.log
![Page 28: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/28.jpg)
Social Connections 11 Chicago, June 1-2 2017
HTTP 404 not found
• AppServer or App down
• Network issue
• http_plugin.log
![Page 29: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/29.jpg)
Social Connections 11 Chicago, June 1-2 2017
HTTP 500 Internal Server Error
• Unexpected error
• http_plugin.log
• Configuration issue
• WAS Root certificate not trusted or missing
• SSL certificate expired
![Page 30: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/30.jpg)
Social Connections 11 Chicago, June 1-2 2017
Debug wsadmin
• Enable trace within wsadmin session
• AdminControl.trace(‘com.ibm.*=all’)• <wasroot>/profiles/<profilename>/logs/wsadmin.traceout
![Page 31: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/31.jpg)
Social Connections 11 Chicago, June 1-2 2017
Directory/Troubleshooting/ Tuning
• Waltz and Sonata overview
• Enabling Log parameters…
• Cache Tuning
• SSO Config Glitches
• CCM Troubleshooting
![Page 32: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/32.jpg)
Social Connections 11 Chicago, June 1-2 2017
Connections Troubleshooting
• Waltz: "common directory services"
• Directory Service eXtension (DSX) REST API.
• Sonata: common HttpClient
services through a RESTful service
• configurable authenticators that support
• SSO security tokens and cookies for
secure server-to-server communication.
![Page 33: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/33.jpg)
Social Connections 11 Chicago, June 1-2 2017
Troubleshooting:
DSOutOfServiceExceptions
• Symptoms:• Access Connections not possible
• DSX not working
![Page 34: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/34.jpg)
Social Connections 11 Chicago, June 1-2 2017
Troubleshooting: Enable Logging
• Track all basic Waltz configuration settings and transactions.• com.ibm.connections.directory.services.*=all
• Track all LDAP transactions in between Waltz & LDAP server(s).
• WaltzLDAPUsage=all
• Track all LDAP entry to be cached and hit from cache upon DN of LDAP entries.
• WaltzDNEntryCache=all
• Track all LDAP entry to be cached and hit from cache upon ID of LDAP entries.
• WaltzExactIDMatchCache=all• Track all group membership (a list of groups) for a given user upon ID.
• WaltzGroupMembershipCache=all
• Track all members (a list of users) for a given group upon ID.
• WaltzMemberExpansionCache=all
TIP: Save runtime changes to make changes persistent
![Page 35: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/35.jpg)
Social Connections 11 Chicago, June 1-2 2017
Troubleshooting: Enable Logging
• WAS VMM/WIM:
• com.ibm.websphere.wim.*=all:com.ibm.ws.wim.*=all
• WAS SPNEGO:
• com.ibm.ws.security.spnego.*=all
• Apache commons HttpClient:
• org.apache.commons.httpclient.*=all
• WAS security:
• com.imb.ws.security.*=all:com.ibm.ws.security.policy.*=off;
![Page 36: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/36.jpg)
Social Connections 11 Chicago, June 1-2 2017
Troubleshooting: Enable Logging
• Basic Sonata configuration settings and transactions:• com.ibm.connections.httpClient.*=all"
• Track the amount of HTTP(S) transactions.• SonataHttpUsage=all
• Track headers received for all HTTP(S) transactions.• SonataHttpHeader=all
• Track all bodies received for all HTTP(S) transactions.• SonataHttpBody=all
![Page 37: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/37.jpg)
Social Connections 11 Chicago, June 1-2 2017
EJPVJ9284E: Unable to get the groups
from the directory for the user…
• User was not able to access Connections
anymore
• WAS LDAP bind user had no read access
to one of the groups the user was member
![Page 38: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/38.jpg)
Social Connections 11 Chicago, June 1-2 2017
SSO Configuration Glitches:
• Warning indicating SSO config is missing or
incomplete enabling application security:
• Ensure you filled
in your domain
Tip: Connections CRs sometimes
resets SSO domain
![Page 39: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/39.jpg)
Social Connections 11 Chicago, June 1-2 2017
SSO Configuration Glitches:
• LtpaToken Cookies: • More secure- recommended to use version 2 LTPA cookies
In other words- interoperability mode is left as disabled by default.
Only enable interoperability mode ifsome app servers are still relying on LTPA version 1 and need to interoperate.
Connections Security supports custom LTPA cookie names since 4.5 CR1
Sets com.ibm.websphere.security.customSSOCookieName custom properties (verify 3rd party products Portal/Domino can handle it)
![Page 40: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/40.jpg)
Social Connections 11 Chicago, June 1-2 2017
Waltz Cache: Changing Group
Membership/permissions remains in Waltz Cache
• Issue: Administrator changes access or Group membership for a user. That user does not get immediate access.
• Cache: Connection's Directory Services keeps a “list” of directory objects it has already searched for performance reasons
● Improves performance of connections services
● Reduces load on remote directory servers
● Cache is based on a timing mechanism
● Cache is "flushed and renewed" on a 12 hour time schedule
![Page 41: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/41.jpg)
Social Connections 11 Chicago, June 1-2 2017
Tuning the Cache: Continued
• How does this affect Group Membership?● Group membership exists in the cache for 12 hours
● Operations such as renaming, deleting, or updating groups remain in the cache for that time
• Configuration parameter for cache timing mechanism
● Enables Connections administrators to flush the cache in configurable time increments
● Set cache parameter via JVM arguments to flush them (we'll get to that!)
● Default remains 12 hours
![Page 42: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/42.jpg)
Social Connections 11 Chicago, June 1-2 2017
Cache Configuration: Setting the cache
flush parameters• Setting the cache parameters via JVM configured in
WebSphere● Google Setting generic JVM arguments in WebSphere Application
Server
• Time Values must be the same for all cache settings
• Adjust with care! Flushing the cache has implications!
![Page 43: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/43.jpg)
Social Connections 11 Chicago, June 1-2 2017
IBM Connections Content Manager
(CCM): Troubleshooting• During library creation, retrieving or modifying content:
• Calls are made between Communities, Profiles, and FileNet Issues with Filenet / Connections Profile or Community configuration can
adversely affect access to Libraries and CCM
Logging from Connections components (previous slides) and FileNet essential
• SystemOut.log & trace.log – these are your friend
• FileNet core logging: • <wasprofile>/<servername>/p8_server_error.log
• <wasprofile>/<servername>/p8_server_trace.log
• <wasprofile>/<servername>/pesvr_system.log
• <wasprofile>/<servername>/pesvr_trace.log
![Page 44: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/44.jpg)
Social Connections 11 Chicago, June 1-2 2017
CCM Troubleshooting: Filenet URLS
• Status about FileNet Collaboration Services:
• check the status page at http://<fqdn>/dm
Navigator version and configuration
• Status about FileNet Content Engine
• check the status page at http://<fqdn>/FileNet/Engine
Server status, version, sonata/waltz version
• Status about FileNet Content Engine Domain and Object Store
• check the status page at http://<fqdn>/P8CE/Health
Health checks for authentication, stores and database
![Page 45: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/45.jpg)
Social Connections 11 Chicago, June 1-2 2017
Debug CCM Widget
• Widget issues
• com.ibm.quickr.communitylibrary.*=all:
com.ibm.lconn.widgets.service.*=all:
com.ibm.lconn.widgets.actions.*=all
• Authentication issues (Remember those cool tips a few
slides back? )
• com.ibm.connections.directory.services.*=all:
com.ibm.connections.httpClient.*=all
![Page 46: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/46.jpg)
Social Connections 11 Chicago, June 1-2 2017
Debug FileNet using ACCE
• ACCE tool: check if FileNet Content Engine can
authenticate using Connections and inspect other data
for errors:
http(s)://<fqdn>/acce
![Page 47: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/47.jpg)
Social Connections 11 Chicago, June 1-2 2017
CCM: Enable logging for Connections and FileNet
using JVM Properties
• Download and copy the sample waltz.sonata.log4j. as log4j.xml:• Linux: /<absolute_path_for_log4j_file>/log4j.xml
• Windows:\<absolute_path_for_log4j_file>\log4j.xml
• Add generic JVM properties
• -Dlog4j.configuration=file:<path>/log4j.xml -DskipTLC=true
• Tells the Content Engine (CE) server to skip it’s tracing configuration and writes to waltz.sonata.trace.log Windows: c:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01Linux: /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
![Page 48: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/48.jpg)
Social Connections 11 Chicago, June 1-2 2017
CCM: Enable logging for Connections and FileNet
using JVM Properties
• Copy & customize sample log4j.xml
• <ContentEngineRoot>/config/samples
• 20 subsystems (db, engine, security, search,…)
![Page 49: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/49.jpg)
Social Connections 11 Chicago, June 1-2 2017
TDI logs
• <tdisol>/log/ibmdi.log• TDI log file
• <tdisol>/employee.*• Files include all changed users (adds, update, delete, error, skip)
• <tdisol>/syncupdates/*• Temporary files within the sync
• Including database dump and ldiff
• sync_updates_clean_temp_files=false (default: true)• profiles_tdi.properties
Tip: Check the lock file
![Page 50: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/50.jpg)
Social Connections 11 Chicago, June 1-2 2017
Analyze TDI logs
• Error code prefix• CLFRN: Profile & User synchronization
• CTGDIS: TDI itself
• Error code suffix
• I, E, W, …
![Page 51: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/51.jpg)
Social Connections 11 Chicago, June 1-2 2017
Debug TDI
• Profile & User synchronization
(<tdisol>/etc/profiles_tdi.properties)
• source_ldap_debug=true
• debug_update_profile=true
• debug_collect=true
• TDI issues (<tdisol>/etc/log4j.properties)
• log4j.rootCategory=DEBUG, Default
![Page 52: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/52.jpg)
Social Connections 11 Chicago, June 1-2 2017
Cognos BI logs
• Cognos BI• SystemOut.log & trace.log
• <installroot>/logs/cogserver.log
• Cognos Transformer• <installroot>/logs/cogserver.log
• PowerCube build• <installroot>/metricsmodel/trxschelog.log
• <userhome>/Transformer/Logs/*.log (win only)
![Page 53: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/53.jpg)
Social Connections 11 Chicago, June 1-2 2017
BMT-MD-6003 No connection to the
data source …
• PowerCubes not created yet
• Check cronjobs or scheduled jobs
• trxschelog.log
![Page 54: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/54.jpg)
Social Connections 11 Chicago, June 1-2 2017
Debug Cognos BI & Metrics
• Communication between Cognos BI and Connections Metrics
• SonataHttpUsage=all:SonataHttpHeader=all:SonataHttpBody=all: com.ibm.connections.httpClient.*=all: com.ibm.connections.metrics.*=all
• Connections Metrics Servlet
• com.ibm.connections.metrics.cognos.servlet.*=all
![Page 55: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/55.jpg)
Social Connections 11 Chicago, June 1-2 2017
Docs/Viewer logs & urls
• SystemOut.log & trace.log
• http(s):<fqdn>/vsanity/check
• http(s):<fqdn>/sanity/check?app=all&querytype=report
• http(s):<fqdn>/*/version
![Page 56: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/56.jpg)
Social Connections 11 Chicago, June 1-2 2017
Debug LTPA between Domino & WAS
• Debug on Domino side (notes.ini)
• Debug_SSO_Trace_Level=2
• Webauth_verbose_trace=1
• WebSess_verbose_trace=1
• Debug_outfile=<logfilepath>
• Debug on WebSphere
• com.ibm.ws.security.ltpa.*=all
![Page 57: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/57.jpg)
Social Connections 11 Chicago, June 1-2 2017
Debug Kerberos
• Configuration• com.ibm.ws.security.spnego.*=all:
com.ibm.ws.security.*=all: com.ibm.issw.spnegoTAI.*=all: com.ibm.security.krb5.*=all
• Runtime• com.ibm.connections.httpClient.*=all:
com.ibm.connections.directory.services.*=all:com.ibm.websphere.wim.*=all:com.ibm.ws.wim.*=all
• Fiddler & BurpSuite
![Page 58: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/58.jpg)
Social Connections 11 Chicago, June 1-2 2017
IBM Spectrum CfC stats
• https://fqdn:8443
![Page 59: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/59.jpg)
Social Connections 11 Chicago, June 1-2 2017
Docker & Kubernetes status
• kubectl get pods (--all-namespaces)
• kubectl get services
• kubectl get deployment
• kubectl describe pods xxx
• docker ps
• docker info
![Page 60: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/60.jpg)
Social Connections 11 Chicago, June 1-2 2017
Docker & Kubernetes status
• Persistent Volume status
• kubectl get pv,pvc
![Page 61: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/61.jpg)
Social Connections 11 Chicago, June 1-2 2017
Orient Me logs
• Display application logs
• kubectl logs xxx• -f
• --tail=20
• --since=1h
• Special tasks for MongoDB and Solr
![Page 62: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/62.jpg)
Social Connections 11 Chicago, June 1-2 2017
MongoDB Troubleshooting
• Check persistent volumes
• Access logs
• kubectl logs xxx [mongo|mongo-sidecar]
• Get replica status
• kubectl exec -it mongo-0 -- mongo mongo-0.mongo:27017 --eval
'rs.status()’
• Check “health” status for all 3
members
![Page 63: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/63.jpg)
Social Connections 11 Chicago, June 1-2 2017
Solr logs
• kubectl exec -it xxx -- cat /var/solr/logs/solr.log
• kubectl logs xxx
• only display deploment logs
![Page 64: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/64.jpg)
Social Connections 11 Chicago, June 1-2 2017
Pods startup issues
• kubectl describe pod xxx
![Page 65: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/65.jpg)
Social Connections 11 Chicago, June 1-2 2017
CfC logs
• docker logs -f kubelet
• docker logs -f calico
![Page 66: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/66.jpg)
Social Connections 11 Chicago, June 1-2 2017
Docker tracing
• service docker stop
• docker daemon –debug
• journalctl -u docker.service
![Page 67: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/67.jpg)
Social Connections 11 Chicago, June 1-2 2017
What to get more insights on Orient Me?
![Page 68: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/68.jpg)
Social Connections 11 Chicago, June 1-2 2017
Tools
![Page 69: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/69.jpg)
Social Connections 11 Chicago, June 1-2 2017
Analyze logs
• Analyze logs live• Baretail
• tail –f
• View logs • Atom.io, Notepad++
• less, tail
• ELK Stack• Elasticsearch, Logstash, Kibana
• Small Docker deployment
• Kubectl (local client)
![Page 70: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/70.jpg)
Social Connections 11 Chicago, June 1-2 2017
Analyze logs
• Kubectl (local client)
![Page 71: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/71.jpg)
Social Connections 11 Chicago, June 1-2 2017
Analyze Client-side
• Browser
• Firebug / Developer Tools
• Intercepting proxies
• Fiddler
• Burp Suite
• VMs with different IE versions
• Without GPO
• https://www.modern.ie/en-us/virtualization-tools
![Page 72: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/72.jpg)
Social Connections 11 Chicago, June 1-2 2017
Database Clients
• db2 command
• Dbeaver or IBM Datastudio
• Robomongo (MongoDB)
Tip: Database write access is not supported!
![Page 73: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/73.jpg)
Social Connections 11 Chicago, June 1-2 2017
LDAP & Search
• LDAP
• ldapsearch command
• Softerra LDAP Browser
• Apache Directory Studio
• Search Index
• Luke (Lucence Index Toolbox)
![Page 74: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/74.jpg)
Social Connections 11 Chicago, June 1-2 2017
Analyze Network
• Wireshark
• tcpdump
![Page 75: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/75.jpg)
Social Connections 11 Chicago, June 1-2 2017
Q&A
![Page 76: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/76.jpg)
![Page 77: Soccnx11 Two wrongs don't make a right - Troubleshooting Connections](https://reader033.fdocuments.us/reader033/viewer/2022052117/5a6d98177f8b9abd418b7177/html5/thumbnails/77.jpg)
PLATINUM SPONSORS
GOLD SPONSORS
SILVER SPONSORS