LHCb Logging System
description
Transcript of LHCb Logging System
LHCb Logging System
Nikolaidis Fotis ( fotis.nikolaidis @ cern.ch )University Of Crete, Greece
A computer log is a diary or archive of events, in this case generated by a computer system or systems. In the late- 23rd century, Federation starships were equipped with a "black box" that stored computer logs. The logs could be used in criminal investigations or to determine the cause of a lost ship. Computer logs were for official purposes only and were available to authorities only under specific legal circumstances or court-order.
Sources Web Servers Gateways Network Components Farm Nodes PVSS FMC
Storage SchemaFARM
HOSTS
PARTITIONS
PVSS
SERVICES
hlt[a-e][1-11] Messages, crond, maild, dnsd,secure, secureNagios
hostName Messages, crond, maild, dnsd,secure, secureNagios{Other files either from FMC or web sites}
hostName Project Name PVSS_II.logPVSS00ctrl50.logAnd other ...
LHCbTFCFESTECAL........
DAQ
TELL1 Messages, crond, maild, dnsd,secure, secureNagios
$partition.log
Dataremove, Dimrpc, Writerd, Xmlrpc
Needs
Forensic / Troubleshooting Splunk ( http://admin01/splunk )
Real Time Alert Ossec
Splunk
Is a High performance, scalable software server written in C/C++ and Python.
Index and Normalize logs (disk fail , disk error are the same)
Can be combined with with Ossec, Snort and other IDS via plugins
Does not need an external Database.
Splunk - Features
Advanced search Regular Expressions / Time Windows
Runtime statistical analysis Extensible
Modules, Patterns Dashboards
Splunk - More Features
Can correlate events of different hosts/formats Supports many log formats out of the box
(For non standard logs such as FMC configuration is needed)
If run on CLI , can be integrated to scripts
Have a closer look here ...
The first line is excludedThe second line is now the first
Who is keeping ssh busy ? ;p
New Patterns can be generated almost automatically
Internal Information
OSSEC
OSSEC
Open Source Host-based Intrusion Detection System.
Log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response.
OSSEC
Analyzes incoming logs runtime and reacts if needed
Every event can be ranked with a value [1-14] If event > mailRank , send a mail If event > scriptRank , execute a script Rules are defined in XML files
Message, frequency, priority, etc
Fault Tolerance
Normal
Logsrv01 failure
Log analysis failure
Logsrv02 failure