Troubleshooting TMSP
description
Transcript of Troubleshooting TMSP
Copyright 2009 Trend Micro Inc.Classification 04/22/23 1
Troubleshooting TMSPMarks Shen • Senior Engineer – QAEvan Wang • Engineer - QA
Copyright 2009 Trend Micro Inc.Classification 04/22/23 2
Agenda
Frequent Case
Debug log and information
Troubleshooting
Q&A
Copyright 2009 Trend Micro Inc.
No report was generated• Logs need to collect on Daemon Server
– /opt/TrendMicro/tdss/tdes/log/iae_log.txt– /opt/TrendMicro/tdss/tdes/reports/tdes.log(Since
2.5R3)– /var/log/cron
• Information – Customer expiration date– Device register to TMSP
Copyright 2009 Trend Micro Inc.
No report was generated Cont.
• Normal debug log for log correlation and report generation
• Crontab taskcron_iae.sh will be executed at 2:15 am every day
Copyright 2009 Trend Micro Inc.
No report was generated Cont.
Report will not be generated if• Customer service get expired
• Customer without device registered
Copyright 2009 Trend Micro Inc.
No incident in report• Logs need to collect on Daemon Server
– /opt/TrendMicro/tdss/tdes/log/iae_log.txt
• Information– Check if TDA log has been uploaded
Latest log time
Copyright 2009 Trend Micro Inc.
Cannot access Admin console (err 404)
• Logs that need to collect on Daemon Server– /var/log/httpd/access_log– /var/log/httpd/error_log
• Information – ps –ef | grep httpd– netstat –anp | grep httpd
Copyright 2009 Trend Micro Inc.
No Rsync log uploaded
• Logs need to collect on Access Server– Log receiver
• /var/log/messages• /home/tdalog/log/pre-post-exec.log • /home/tdalog/log/db_import_tda.log• /home/tdalog/log/db_import_tdm.log
– Authentication (describe in next sides)
• Information – ps –ef | grep tmsshd– netstat –anp | grep tmsshd
• Listen on port 22
Copyright 2009 Trend Micro Inc.
No Rsync log uploaded Cont.• normal log of tmsshd and rsync
Classification 04/22/23 9
Copyright 2009 Trend Micro Inc.
No Rsync log uploaded Cont.• Normal debug log of TDA log processing
– /home/tdalog/log/pre-post-exec.log
Copyright 2009 Trend Micro Inc.
CAS server caseProblems caused by CAS failure:
– Device register to TMSP fail– Customer portal login fail (only before R3)– Log uploading fail through RSYNC
Log on Access Server:– /var/log/messages– /var/log/cas_8000.log– /var/log/cas_8001.log– /var/log/cas_8002.log
Information:• ps –ef | grep pound• ps –ef | grep rubcasd
Normal log of CAS authentication– /var/log/cas_8000.log
Copyright 2009 Trend Micro Inc.
CAS server case – Service down• TDA register fail
• Check /var/log/messages
• Recover– If pound or cas service is down
• /etc/init.d/pound start• /etc/init.d/rubcasd start
Copyright 2009 Trend Micro Inc.
Data Gateway CaseProblems caused by Data Gateway failure:
– OCS Heartbeat / OCS log cannot be handled– T2 / T3 mitigation request cannot be delivered to TMTM– SIC sample cannot be handled
Logs on Access Server:– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/webapps/dg/WEB-
INF/logs/dg.log– $APACHE_HOME/logs/ssl_request_log_dg– $APACHE_HOME/logs/error_log– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/logs/catalina.out
Information:• ps –ef | grep httpd• ps –ef | grep tomcat• netstat –anp | grep 443• netstat –anp | grep 8009• netstat –anp | grep 8080
Copyright 2009 Trend Micro Inc.
Data Gateway Case – DB disconnect• TDA register fail
• Check apache error log: /usr/apache/logs/error_log
Copyright 2009 Trend Micro Inc.
Customer portal cannot login• Logs need to collect on Access Server:
– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/webapps/tms2/WEB-INF/logs/tms.log
– $APACHE_HOME/logs/ssl_request_log_portal– $APACHE_HOME/logs/error_log– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/logs/
catalina.out
• Information• ps –ef | grep httpd• ps –ef | grep tomcat• netstat –anp | grep 443• netstat –anp | grep 8009• netstat –anp | grep 8080
Copyright 2009 Trend Micro Inc.
Cannot get eMail notification• Exclude Mail server problem, collect debug
logs:– /root/infomation.log
– /var/log/cron
No Subscription
DB connection fail
Copyright 2009 Trend Micro Inc.
23/4/22 17Classification
FAQWhy no daily report can be found from web UI?
1. Check TDES log from #tail -n 100 /opt/TrendMicro/tdss/tdes/log/iae_log.txt
If content like "Daily report: customer_ID, 2009 02 16 JP" cannot be found, that mean the scheduled job has not started so far.
2. Daily report is auto generated at 7:15 am every day, so check the system time of TDES:
3. Report generation need take some time, please check if the report is generating:#ps -ef | grep php
if some php process is running, it means the reports are under generating.4. Check if customer has expired for TMSP service, find the profile “expire time”
from admin console
Copyright 2009 Trend Micro Inc.
FAQWhy there is no data in report?
1. Check if the log has been uploaded to TMS and imported into DatabaseLogin log receiver machine and check the file last modification time#ll /home/tdalog/userdata/USERID/DEVICE_GUID/*.db
If not latest data, that means TDA did not upload logs.2. Log in database and query yesterday's log
Some times, TDA did not detect any events, if so, there will be no yesterday's data in DB
3. Check iae_log.txt, check if “Running TDES 2.1 for XXXX (device=50)(customer_id=30) on date: 2009 2 16” existing, this means TMS run IAE for this customer. if NO this content exist, means there is something wrong when process IAE.
Copyright 2009 Trend Micro Inc.
FAQHow to re-generate report manually?
Login TDES machine, change dir to "/opt/TrendMicro/tdss/tdes/";– 1. Daily Report
#php gendailydata.php user_id YEAR MONTH DAYexample: "php gendailydata.php trend 2009 01 04" generate daily report of 2009.01.04 for customer "trend“
– 2. Executive Report (Weekly / Monthly)#php genexecdata.php user_id START_DATE START_DATE yes m/w START_DATE: report start data with format “YYYY-MM-DD” START_DATE: report end data with format “YYYY-MM-DD” yes: if this report will be imported into DB m/w: monthly or weeklyexample: "php genexecdata.php trend 2009-01-01 2009-01-31 yes m" to generate monthly report for customer "trend" of 2009-01
– 3. Upsell Report# php genupselldata.php user_id START_DATE END_DATE noexample: " php genupselldata.php trend 2009-01-01 2009-01-31 no" to generate upsell report for customer "trend" from 2009-01-01 to 2009-01-31Note: Upsell report will not be imported into DB and cannot download from admin console
After execute these command, reports will be re-generated and imported into database for downloading
Copyright 2009 Trend Micro Inc.
FAQWhat’s the steps to deploy new report php file?
Sometimes, reports generation related php need to be modified and deploy to TDES, here are the steps to do this:1. back up old php file2. replace with new php file3. remove cache_*.php under TDES installation folder4. Re-generate report to verify new php files if necessary
Copyright 2009 Trend Micro Inc.
Q&A
Copyright 2009 Trend Micro Inc.Classification 04/22/23 22
THANK YOU!