Troubleshooting

There are logs available for troubleshooting the DCC as well as the operational tools.

• Troubleshooting the Device Command and Control UI, page 1

• Operational Tools Troubleshooting, page 2

• Using the Logs for Troubleshooting, page 3

• Cleaning Up PMG, Upload Server, and DCC UI Logs, page 5

• Configuring or Changing Log Threshold, page 7

Troubleshooting the Device Command and Control UIThe DCCUI log files are generated and stored in the logs directory,/rms/log/dcc_ui/, by default. The Auditlogs capture the entire tool access process. You can check the logs for checking the tool access details.

A sample Audit log of the application is shown below:[03-13-2012 00:00:02]Timer-1 - Writing perf log header after file rotation[03-13-2012...05:11:33]http-80-3 - [000001-000000001000|FPGe89e1e45-dba9-46d9-a688-72f4aba3de73]Attr: xmlns = http://www.cisco.com/ca/sse/PMGMessages-v1_0_0[03-13-2012 05:11:33]http-80-3- [000001-000000001000|FPGe89e1e45-dba9-46d9-a688-72f4aba3de73] Attr: xmlns:xsd= http://www.w3.org/2001/XMLSchema[03-13-2012 05:11:33]http-80-3 -[000001-000000001000|FPGe89e1e45-dba9-46d9-a688-72f4aba3de73] Attr: xmlns:xsi =http://www.w3.org/2001/XMLSchema-instance[03-13-2012 05:11:33]http-80-3 -[000001-000000001000|FPGe89e1e45-dba9-46d9-a688-72f4aba3de73] Attr:xsi:schemaLocation = http://www.cisco.com/ca/sse/PMGMessages-v1_0_0pmg-messages.xsd[03-13-2012 05:11:33]http-80-3 -...The Debug logs are used for debugging the errors that occur during programming. A sampleDebug log ofthe application is shown below[04-20-2012 00:00:14-668]Timer-1 Servlet INFO [SystemStatusMonitor$FPGHourlyStatusTask.run]- Start running hourly timer task[04-20-2012 00:00:14-670]Timer-1 Servlet DEBUG[SingletonInstanceManager.getInstance] - Using existing SingletonInstanceManagerobject...[04-20-2012 00:00:14-671]Timer-1 Servlet DEBUG [SingletonInstanceManager.getInstance]- Reusing existing singleton instance:...

Cisco RAN Management System Administration Guide, Release 5.1 July 6, 2015 1

14:14:09-332]main Servlet DEBUG [SetupServlet.destroy] - Destroying SetupServlet... Shuttingdown SystemMonitor timers...[04-20-2012 14:14:09-333]main Servlet DEBUG[SetupServlet.destroy] - Destroying SetupServlet... Shutting down SystemMonitor timers..DONE.[04-20-2012 14:14:09-335]main Servlet INFO [SetupServlet.destroy] - DestroyingSetupServlet... DONE.[04-20-2012 14:14:10-525]ol_Thread-1 Core DEBUG [DebuggableThread.run]- Exiting SCICARE_UMTThreadPool_Thread-1

Operational Tools TroubleshootingThe following logs are available for troubleshooting the Operational tools:

• Audit logs

• Debug logs

• Error logs

For all operational tools, the logs are written into the Central Log file as defined in $RMS_OPS. The logfiles are generated and stored under $RMS_OPS/<time-stamped directory>/logs. However the outputdirectory location can be overridden by using -outdir option. Please see tools -outdir option for details.

Note

Audit Logs

The path to the log file is printed to the console after the tool execution is complete. The Audit logs capturethe entire tool execution process. The administrator can check the logs to ensure that the tool execution is asuccess.

A sample Audit log of the massReboot.sh tool execution is shown below:

$ cat /home/krajain/massReboot.sh/MassReboot-20120906-072717/logs/audit.log[09-06-2012 07:27:19-376]main Audit INFO [ScriptUtils.initLog4j:428] - Audit log configuredand ready.[09-06-2012 07:27:19-912]main Audit INFO [BacConnection.initialize:125] - RDU connectionestablished (host=localhost, port=49187)....Summary Report *****************************************[09-06-2012 07:27:20-489]main Audit INFO [Category$info.call:?] - Number of successfulReboots :: 10[09-06-2012 07:27:20-489]main Audit INFO [Category$info.call:?] - Number of failed Reboots:: 0[09-06-2012 07:27:20-489]main Audit INFO [Category$info.call:?] -********************************************************************************[09-06-2012 07:27:20-490]main Audit INFO [Category$info.call:?] -

Debug Logs

The Debug logs are used for debugging the errors that occur during programming. A sample Debug log ofmassReboot.sh tool execution is shown below:

$ cat /home/krajain/massReboot.sh/MassReboot-20120906-090645/logs/debug.log[09-06-2012 09:06:47-411]main Core INFO [DCCProperties.loadProperties:93] - Property file/rms/app/rms/conf/dcc.properties loaded....

Cisco RAN Management System Administration Guide, Release 5.12 July 6, 2015

TroubleshootingOperational Tools Troubleshooting

[09-06-2012 09:06:48-306]PoolWorker3 Threadpool WARN [QueueClosedException.<init>:31] -Queue closed exception[09-06-2012 09:06:48-362]Thread-2 Core INFO [ApplicationInstanceManager$1.run:102] - Scriptterminated. Releasing lock.

Error Logs

The Error logs are generated and stored in the $RMS_OPS/<time-stamped directory>/logs by default. Ifanother output directory is specified in the arguments, then the logs are stored in the specified directory. TheError logs capture the errors and exceptions causing the tool to terminate. If the tool terminates, the administratorcan check the Error logs for the error details. This facilitates troubleshooting decisions.

Using the Logs for TroubleshootingThe following log files are available in the RMS:

Description / UsageLog

Logs all DCCUI Operation Logs. Location:/rms/log/dcc_ui/ui-debug.log/rms/log/dcc_ui/ui-audit.log

DCC UI Audit and Debuglogs

Logs all PMG Processing logs. Location:rms/log/pmg/pmg-audit.log/rms/log/pmg/pmg-debug.log

PMG logs

Records all RDU events according to the configured logging severitylevel.Location:/rms/data/CSCObac/rdu/logs/rdu.logTo enable this log, use this command on the Central node:

/rms/app/CSCObac/rdu/bin/setLogLevel.sh 6

/rms/app/CSCObac/rdu/bin/setLogLevel.sh 6Please enter RDU username: bacadminPlease enter RDU password:<RMS_APP_Password>

RDU log

Records all high-level changes to the Cisco BAC configuration or functionalityincluding the user who made the change.

Audit log

Records detailed device information for troubleshooting a specific device.Troubleshooting logs are enabled from the BACUI (admin UI) as follows:

1 Login to the Admin UI.

2 Search for devices.

3 Click the Identifier.

4 Click on Enable Troubleshooting.

5 Click Submit.

Troubleshooting log

Cisco RAN Management System Administration Guide, Release 5.1 July 6, 2015 3

TroubleshootingUsing the Logs for Troubleshooting

Description / UsageLog

Records the performance of message statistics related to the system on RDU.Location:/rms/data/CSCObac/rdu/logs/statisticsRecords the TR-069 messages received and processed by the DPE. Location:/rms/data/CSCObac/dpe/logs/statistics

Performance statistics log

LogsDPE activities such as CWMPmessages, file downloadmessages, extensionflows, PG re-directs, Lease Query, Proxy operations, and so on. To enable thislog, do the following:

1 Log on to the Serving node.

2 telnet localhost 2323

3 show config

4 log level 6-info

[root@blr-rms12-serving_41N ~]# telnet localhost 2323Trying 127.0.0.1...Connected to localhost.Escape character is '^]'.blr-rms12-serving_41N BAC Device Provisioning EngineUser Access VerificationPassword:blr-rms12-serving> enPassword:blr-rms12-serving# show configblr-rms12-serving# log level 6-info% OK* Password will be RMS_APP_Password

DPE logs

Location:rms/data/nwreg2/local/logs/name_dhcp_1_log

NR logs

Location:/rms/app/CSCOar/logs/name_radius_1_traceTo enable this log, use the following command:

/rms/app/CSCOar/usrbin/aregcmd -N admin

/rms/app/CSCOar/usrbin/aregcmd -N adminCisco Prime Access Registrar 6.0.1 Configuration UtilityCopyright (C) 1995-2013 by Cisco Systems, Inc. All rights reserved.Cluster:Password:[ //localhost ]LicenseInfo = PAR-NG-TPS 6.0(100TPS:expires on 11-Sep-2014)Radius/Administrators/Server 'Radius' is Running, its health is 10 out of 10--> trace 5Trace level is set to 5

*Password =<RMS_APP_PASSWORD>

AR logs

Cisco RAN Management System Administration Guide, Release 5.14 July 6, 2015

TroubleshootingUsing the Logs for Troubleshooting

Cleaning Up PMG, Upload Server, and DCC UI LogsThe cleanup cron script run at configured interval and performs rollover and clean-up of configured log filesbased on the size thresholds. The following sections describe how the script works and how you can changethe configuration, for example, disk size thresholds.

• Console Log Rollover and Clean-up of PMG Logs, on page 5

• Clean-up Configured Logs for PMG and DCC UI, on page 5

• Console Log Rollover and Clean-up of Upload Server Logs, on page 6

• Clean-up Configured Logs for Upload Server, on page 6

Console Log Rollover and Clean-up of PMG LogsThe cron script for PMG console logs is /etc/cron.hourly/pmgCron

This cron script is configured to run every hour and execute the following configuration script:/etc/pmgConsoleLogRotate.conf

This cron script is responsible for rollover of PMG console log file located at /rms/logs/pmg/PMGServer.console.log and also the purging of the archived log file.

The default configuration of the PMG console log file is to roll over when it reaches the maximum limit of100 MB and maintain the maximum of 24 archived files (older files will be deleted).

To change the configuration, edit these scripts. It uses the standard Linux cron utility and log rotation module.

Clean-up Configured Logs for PMG and DCC UIThe cron script for PMG and DCC UI configured logs is /etc/cron.d/rmsLogsPurgeCron.conf

This cron script is configured to run every 30 minutes and execute the following configuration script:/etc/ rmsLogsPurgeCron.sh

This cron script is responsible for cleanup of logs listed in the following table. For PMG it includes thefollowing log patterns and the location of the log files is /rms/log/pmg/ .

Size Threshold (GB)Log File Pattern

39pmg-debug*.gz

30pmg-audit*.gz

0.8pmg-msg-perf*.gz

1pmg-outbound-msg*.gz

1pmg-events*.gz

0.2pmg-inbound-msg*.gz

Cisco RAN Management System Administration Guide, Release 5.1 July 6, 2015 5

TroubleshootingCleaning Up PMG, Upload Server, and DCC UI Logs

Size Threshold (GB)Log File Pattern

0.01pmg-alarms*.gz

For DCC UI it includes the following log patterns

Size threshold (GB)Log File pattern

0.08ui-debug*.gz

0.08ui-audit*.gz

Console Log Rollover and Clean-up of Upload Server LogsThe cron script for the Upload server logs is /etc/cron.hourly/ulsCron

This cron script is configured to run every hour and execute the following configuration script:/etc/ulsConsoleLogRotate.conf

This cron script is responsible for rollover of the Upload server console log file located at /opt/CSCOuls/logs/UploadServer.console.log and also the purging of the archived log file.

The default configuration of the Upload server console log file is to roll over when it reaches the maximumlimit of 100 MB and maintain the maximum of 24 archived files (older files will be deleted).

Clean-up Configured Logs for Upload ServerThe cron script for Upload server configured logs is /etc/cron.d/ulsLogsPurgeCron.conf

This cron script is configured to run every 30 minutes and execute the following configuration script:/etc/ulsLogsPurgeCron.sh

This cron script is responsible for cleanup of logs defining the log threshold and maximum size thresholds inGB.

Size Threshold (GB)Log File Pattern

5nb-events*.gz

16sb-events*.gz

5uls*.gz

This cron script also compresses the .tmp files created and deletes the compressed file if it crosses thethresholds (combined) and if the log pattern is configured.

Note

Cisco RAN Management System Administration Guide, Release 5.16 July 6, 2015

TroubleshootingConsole Log Rollover and Clean-up of Upload Server Logs

Configuring or Changing Log ThresholdTo change the threshold values specified for PMG, DCC UI, and Upload Server, see the table in this sectionfor the log files, configuration file, and the field name where the log threshold values is configured.

The PMG log settings are provided in the following table:Log Configuration FilenameField NameLog File Pattern

/etc/rmsLogsPurgeCron.shPMG-DEBUG-MAXGBpmg-debug*.gz

PMG-AUDIT-MAXGBpmg-audit*.gz

PMG-MSG-PERF-MAXGBpmg-msg-perf*.gz

PMG-OUTBOUND-MSG-MAXGBpmg-outbound-msg*.gz

PMG-EVENTS-MAXGBpmg-events*.gz

PMG-INBOUND-MAXGBpmg-inbound-msg*.gz

PMG-ALARMS-MAXGBpmg-alarms*.gz

The DCC UI log settings are provided in the following table:Log Configuration FilenameField NameLog File Pattern

/etc/rmsLogsPurgeCron.shDCCUI-DEBUG-MAXGBui-debug*.gz

DCCUI-AUDIT-MAXGBui-audit*.gz

The Upload server log settings are provided in the following table:Log Configuration FilenameField NameLog File Pattern

server-logback.xmlULS_SB_LOGS_MAXGBsb-event*.gz

ULS_NB_LOGS_MAXGBnb-event*.gz

post-config-logback.xmlULS_DEBUG_LOGS_MAXGBuls*.gz

The field name is the key and the value refers to the disk threshold values in GB.

Cisco RAN Management System Administration Guide, Release 5.1 July 6, 2015 7

TroubleshootingConfiguring or Changing Log Threshold

Cisco RAN Management System Administration Guide, Release 5.18 July 6, 2015

TroubleshootingConfiguring or Changing Log Threshold