System Montoring
-
Upload
pheletso-rigdway -
Category
Documents
-
view
14 -
download
3
description
Transcript of System Montoring
Linux log files explained – /var/log/by RAHUL SHAH on SEPTEMBER 14, 2013
1
Linux has an extensive set of log files under the /var/log directory. This directory is the central place where all applications and programs put their log files. Most log files are text files that can be viewed using a standard text editor like vi or nano.
To check which log files you have, just do an ls on the /var/log directory:
ls -la /var/log
This outputs:
total 59672
drwxr-xr-x 4 root root 4096 Sep 14 11:58 .
drwxr-xr-x 21 root root 4096 Jun 30 2012 ..
-rw------- 1 root root 0 Jun 25 2012 boot.log
-rw------- 1 root utmp 1852032 Sep 14 04:14 btmp
-rw------- 1 root utmp 8346624 Sep 1 02:15 btmp-20130901
-rw------- 1 root root 95407 Sep 14 11:01 cron
-rw------- 1 root root 102324 Aug 18 03:08 cron-20130818
-rw------- 1 root root 102572 Aug 25 03:29 cron-20130825
-rw------- 1 root root 101870 Sep 1 03:10 cron-20130901
-rw------- 1 root root 103864 Sep 8 03:13 cron-20130908
-rw-r--r-- 1 root root 0 May 1 19:25 dmesg
-rw-r--r-- 1 root root 0 Jan 17 2013 dmesg.old
drwx------ 2 root root 4096 Sep 8 03:14 httpd
-rw-r--r-- 1 root root 146876 Sep 14 11:54 lastlog
-rw------- 1 root root 0 Sep 8 03:14 maillog
-rw------- 1 root root 0 Aug 11 03:44 maillog-20130818
-rw------- 1 root root 0 Aug 18 03:08 maillog-20130825
-rw------- 1 root root 0 Aug 25 03:29 maillog-20130901
-rw------- 1 root root 0 Sep 1 03:11 maillog-20130908
-rw------- 1 root root 1360474 Sep 14 09:45 messages
-rw------- 1 root root 1114942 Aug 18 03:08 messages-20130818
-rw------- 1 root root 3384773 Aug 25 03:29 messages-20130825
-rw------- 1 root root 7643984 Sep 1 03:11 messages-20130901
-rw------- 1 root root 5960143 Sep 8 03:14 messages-20130908
-rw-r----- 1 mysql mysql 580120 Sep 14 11:06 mysqld.log
-rw------- 1 root root 0 Jul 25 03:11 pureftpd.log
-rw------- 1 root root 223 Apr 25 21:03 pureftpd.log-20130428
-rw------- 1 root root 111 May 2 17:33 pureftpd.log-20130505
-rw------- 1 root root 222 Jun 10 15:26 pureftpd.log-20130611
-rw------- 1 root root 115 Jul 24 14:56 pureftpd.log-20130725
drwx------ 3 root root 4096 Apr 30 2012 samba
-rw------- 1 root root 3091237 Sep 14 11:54 secure
-rw------- 1 root root 2429153 Aug 18 01:50 secure-20130818
-rw------- 1 root root 4695728 Aug 25 03:29 secure-20130825
-rw------- 1 root root 12348973 Sep 1 02:24 secure-20130901
-rw------- 1 root root 7211819 Sep 8 01:22 secure-20130908
-rw------- 1 root root 0 Sep 8 03:14 spooler
-rw------- 1 root root 0 Aug 11 03:44 spooler-20130818
-rw------- 1 root root 0 Aug 18 03:08 spooler-20130825
-rw------- 1 root root 0 Aug 25 03:29 spooler-20130901
-rw------- 1 root root 0 Sep 1 03:11 spooler-20130908
-rw------- 1 root root 0 Aug 29 2011 tallylog
-rw-rw-r-- 1 root utmp 139392 Sep 14 11:54 wtmp
-rw------- 1 root root 56859 Jun 28 2012 xferlog
-rw------- 1 root root 0 Jan 1 2013 yum.log
-rw------- 1 root root 13215 Jul 14 2012 yum.log-20130101
I will explain most of these files in this article.
/var/log/boot.log – This file contains the boot and reboot information of this Linux box/var/log/btmp – This file records failed login attempts/var/log/utmp – This file has the current login state of each user/var/log/wtmp – This file contains all logins and logouts history/var/log/cron – This file contains all cron related messages like when the cron daemon started a job, failure messages etc./var/log/dmesg – This file contains messages related to device drivers. There is also a command dmesg which can be used to view the messages in this file./var/log/lastlog – This file contains information about last logins of all users. It is a binary file which can be read by the command lastlog./var/log/maillog – This log files has the messages from mail server running on the system e.g. sendmail/var/log/messages – This is the general system activity log. Everything is logged to this file including logins, authentication failed, anonymous logins, network connections, ftp sessions etc. If you need to debug a process running on your system, this is the log file to go to./var/log/mysqld.log – This is the MySQL log file. MySQL logs all debug, failure and success messages to this file. It also has information about the starting, stopping and restarting of the MySQL daemon mysqld./var/log/pureftp.log – This log file is of the pureftp process that listens for FTP connections. All connections, FTP logins and authentication failures are
logged to this file./var/log/secure – This file contains all security related messages on the system. This includes authentication failures, possible break-in attempts, SSH logins, failed passwords, sshd logouts, invalid user accounts etc./var/log/spooler – This file is rarely used and is empty on my server. This log files used to contain messages from USENET/var/log/xferlog – This file conatains all FTP file transfer sessions including the file name and the user who initiated the FTP transfer/var/log/yum.log – This file contains the activity of the yum package installer. All packages installed, updated and deleted are logged to this file. Any errors or warnings are also logged./var/log/httpd/ – This directory contains the error_log and the access_log files of the Apache httpd daemon. As the names suggest, the error_log contains all errors encountered by httpd including memory issues, and other system related errors. The access_log contains a log of all requests received over HTTP.How to limit the size of the log filesLinux log files can grow to huge sizes in a very short while. For the Apache logs, since all requests are logged in the access_log file, it can grow to an unmanageable size in a very short period of time. Read this article on how to setup log file rotation of Apache logs through WHM. There is also a utility called logrotate that can be used to rotate the log files automatically using a cron job.How to view logsThere are several ways in which a log file can be viewed – using vi, tail, nano etc. If you want to monitor activity on a log file in real time, use the following command:
tail -f /var/log/secure
This will output something like the following:
[root@server ~]# tail -f /var/log/secure
Sep 14 10:56:05 server sshd[3114]: pam_unix(sshd:auth): authentication
failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=221.226.56.22
Sep 14 10:56:07 server sshd[3114]: Failed password for invalid user admin
from 221.226.56.22 port 50118 ssh2
Sep 14 10:56:07 server sshd[3115]: Received disconnect from 221.226.56.22:
11: Bye Bye
Sep 14 11:30:38 server sshd[3133]: Accepted password for root from
122.169.27.71 port 12031 ssh2
Sep 14 11:30:38 server sshd[3133]: pam_unix(sshd:session): session opened for
user root by (uid=0)
Sep 14 11:35:28 server sshd[3133]: pam_unix(sshd:session): session closed for
user root
Sep 14 12:56:06 server sshd[3246]: Accepted password for root from
122.169.27.71 port 14113 ssh2
Sep 14 12:56:06 server sshd[3246]: pam_unix(sshd:session): session opened for
user root by (uid=0)
The -f option to tail will instruct it to poll the file and output any updates to the file immediately. It can be used to poll the file in real time./etc/rsyslog.confThe file /etc/rsyslog.conf contains information on which processes log to which files.
Here is a sample rsyslog.conf file:
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none -/var/log/messages
# The authpriv file has restricted access.
authpriv.* -/var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* -/var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit -/var/log/spooler
# Save boot messages also to boot.log
local7.* -/var/log/boot.log
So there you are! I mentioned about 15 different log files and the messages that are logged to the files on a Linux box. These files are common between all flavors of Linux like CentOS, Debian, Ubuntu etc. I find that tail is the best way to poll a log file in real time, if you suspect suspicious activity on your server, like too many failed login attempts and so on. Post your suggestions and questions in the comments section!
Incoming search terms: var log spooler (2) linux log files explained (1)
Linux Log Files Location And How Do I View Logs Files on Linux?by VIVEK GITE on JULY 17, 2006 LAST UPDATED DECEMBER 6, 2014
in COMMANDS , FILE SYSTEM , LINUX
Iam a new Linux user. I would like to know where are the log files located under
Debian/Ubuntu or CentOS/RHEL/Fedora Linux server? How do I open or view log files
on Linux operating systems?
Almost all logfiles are located under /var/log directory and its sub-directories on Linux.
You can change to this directory using the cd command. You need be the root user to
view or access log files on Linux or Unix like operating systems. You can use the
following commands to see the log files:
1. less command
2. more command
3. cat command
4. grep command
5. tail command
6. zcat command
7. zgrep command
8. zmore command
How do I view log files on Linux?
Open the Terminal or login as root user using ssh command. Go to /var/log directory
using the following cd command:
# cd /var/log
To list files use the following ls command:
# ls
Sample outputs from RHEL 6.x server:
anaconda.ifcfg.log boot.log-20111225 cron-20131110.gz maillog-
20111218 messages-20131103.gz secure-20131027.gz spooler-20131117.gz
up2date-20131117.gz
anaconda.log btmp cron-20131117.gz maillog-
20111225 messages-20131110.gz secure-20131103.gz squid
uptrack.log
anaconda.program.log btmp-20120101 cups maillog-
20120101 messages-20131117.gz secure-20131110.gz swinstall.d
uptrack.log.1
anaconda.storage.log btmp-20131101.gz dkms_autoinstaller maillog-
20131027.gz mysqld.log secure-20131117.gz tallylog
uptrack.log.2
anaconda.syslog collectl dmesg maillog-
20131103.gz ntpstats setroubleshoot UcliEvt.log
varnish
anaconda.yum.log ConsoleKit dmesg.old maillog-
20131110.gz prelink spooler up2date
wtmp
arcconfig.xml cron dracut.log maillog-
20131117.gz rhsm spooler-20111211 up2date-20111211
yum.log
atop cron-20111211 dracut.log-20120101 messages
sa spooler-20111218 up2date-20111218 yum.log-
20120101
audit cron-20111218 dracut.log-20130101.gz messages-
20111211 secure spooler-20111225 up2date-20111225
yum.log-20130101.gz
boot.log cron-20111225 httpd messages-
20111218 secure-20111211 spooler-20120101 up2date-20120101
boot.log-20111204 cron-20120101 lastlog messages-
20111225 secure-20111218 spooler-20131027.gz up2date-20131027.gz
boot.log-20111211 cron-20131027.gz maillog messages-
20120101 secure-20111225 spooler-20131103.gz up2date-20131103.gz
boot.log-20111218 cron-20131103.gz maillog-20111211 messages-
20131027.gz secure-20120101 spooler-20131110.gz up2date-20131110.gz
To view a common log file called /var/log/messages use any one of the following
command:
# less /var/log/messages
# more -f /var/log/messages
# cat /var/log/messages
# tail -f /var/log/messages
# grep -i error /var/log/messages
Sample outputs:
Jul 17 22:04:25 router dnsprobe[276]: dns query failed
Jul 17 22:04:29 router last message repeated 2 times
Jul 17 22:04:29 router dnsprobe[276]: Primary DNS server Is Down...
Switching To Secondary DNS server
Jul 17 22:05:08 router dnsprobe[276]: Switching Back To Primary DNS server
Jul 17 22:26:11 debian -- MARK --
Jul 17 22:46:11 debian -- MARK --
Jul 17 22:47:36 router -- MARK --
Jul 17 22:47:36 router dnsprobe[276]: dns query failed
Jul 17 22:47:38 debian kernel: rtc: lost some interrupts at 1024Hz.
Jun 17 22:47:39 debian kernel: IN=eth0 OUT=
MAC=00:0f:ea:91:04:07:00:08:5c:00:00:01:08:00 SRC=61.4.218.24
DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=46 ID=21599 DF PROTO=TCP
SPT=59297 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0
Common Linux log files names and usage
/var/log/messages : General message and system related stuff
/var/log/auth.log : Authenication logs
/var/log/kern.log : Kernel logs
/var/log/cron.log : Crond logs (cron job)
/var/log/maillog : Mail server logs
/var/log/qmail/ : Qmail log directory (more files inside this directory)
/var/log/httpd/ : Apache access and error logs directory
/var/log/lighttpd/ : Lighttpd access and error logs directory
/var/log/boot.log : System boot log
/var/log/mysqld.log : MySQL database server log file
/var/log/secure or /var/log/auth.log : Authentication log
/var/log/utmp or /var/log/wtmp : Login records file
/var/log/yum.log : Yum command log file.
GUI tool to view log files on Linux
System Log Viewer is a graphical, menu-driven viewer that you can use to view and
monitor your system logs. This tool is only useful on your Linux powered laptop or
desktop system. Most server do not have X Window system installed. You can start
System Log Viewer in the following ways:
Click on System menu > Choose Administration > System Log:
Sample outputs:
Fig.01 Gnome log file viewer
A note about rsyslogd
All of the above logs are generated using rsyslogd service. It is a system utility providing
support for message logging. Support of both internet and unix domain sockets enables
this utility to support both local and remote logging. You can view its config file by tying
the following command:
# vi /etc/rsyslog.conf
# ls /etc/rsyslog.d/
In short /var/log is the location where you should find all Linux logs file. However, some
applications such as httpd have a directory within /var/log/ for their own log files. You
can rotate log file using logrotate software and monitor logs files
using logwatch software.
How To View and Configure Linux Logs on Ubuntu and CentosPosted Dec 17, 2013
8
130.1kviews Logging Linux Basics CentOS Ubuntu Debian
Introduction
Linux system administrators often need to look at log files for troubleshooting purposes.
In fact, this is the first thing any sysadmin would do.
Linux and the applications that run on it can generate all different types of messages,
which are recorded in various log files. Linux uses a set of configuration files,
directories, programs, commands and daemons to create, store and recycle these log
messages. Knowing where the system keeps its log files and how to make use of
related commands can therefore help save valuable time during troubleshooting.
In this tutorial, we will have a look at different parts of the Linux logging mechanism.
Disclaimer
The commands in this tutorial were tested in plain vanilla installations of CentOS 6.4,
Ubuntu 12 and Debian 7.
Default Log File LocationThe default location for log files in Linux is /var/log.
You can view the list of log files in this directory with a simple ls -l /var/log command.
This is what I see in my CentOS system:
[root@TestLinux ~]# ls -l /var/log total 1472 -rw-------. 1 root root 4524 Nov 15 16:04 anaconda.ifcfg.log -rw-------. 1 root root 59041 Nov 15 16:04 anaconda.log -rw-------. 1 root root 42763 Nov 15 16:04 anaconda.program.log -rw-------. 1 root root 299910 Nov 15 16:04 anaconda.storage.log -rw-------. 1 root root 40669 Nov 15 16:04 anaconda.syslog -rw-------. 1 root root 57061 Nov 15 16:04 anaconda.xlog -rw-------. 1 root root 1829 Nov 15 16:04 anaconda.yum.log drwxr-x---. 2 root root 4096 Nov 15 16:11 audit -rw-r--r-- 1 root root 2252 Dec 9 10:27 boot.log -rw------- 1 root utmp 384 Dec 9 10:31 btmp -rw-------. 1 root utmp 1920 Nov 28 09:28 btmp-20131202 drwxr-xr-x 2 root root 4096 Nov 29 15:47 ConsoleKit-rw------- 1 root root 2288 Dec 9 11:01 cron-rw-------. 1 root root 8809 Dec 2 17:09 cron-20131202 -rw-r--r-- 1 root root 21510 Dec 9 10:27 dmesg-rw-r--r-- 1 root root 21351 Dec 6 16:37 dmesg.old -rw-r--r--. 1 root root 165665 Nov 15 16:04 dracut.log
-rw-r--r--. 1 root root 146876 Dec 9 10:44 lastlog -rw------- 1 root root 950 Dec 9 10:27 maillog -rw-------. 1 root root 4609 Dec 2 17:00 maillog-20131202 -rw------- 1 root root 123174 Dec 9 10:27 messages-rw-------. 1 root root 458481 Dec 2 17:00 messages-20131202 -rw------- 1 root root 2644 Dec 9 10:44 secure -rw-------. 1 root root 15984 Dec 2 17:00 secure-20131202 -rw------- 1 root root 0 Dec 2 17:09 spooler -rw-------. 1 root root 0 Nov 15 16:02 spooler-20131202 -rw-------. 1 root root 0 Nov 15 16:02 tallylog-rw-rw-r--. 1 root utmp 89856 Dec 9 10:44 wtmp-rw------- 1 root root 3778 Dec 6 16:48 yum.log
Viewing Log File ContentsHere are some common log files you will find under /var/log:
wtmp
utmp
dmesg
messages
maillog or mail.log
spooler
auth.log or secure
The wtmp and utmp files keep track of users logging in and out of the system. You
cannot directly read the contents of these files using cat– there are specific commands
for that.
We will now use some of these commands.
To see who is currently logged in to the Linux server, simply use the who command.
This command gets its values from the /var/run/utmp file (for CentOS and Debian) or
/run/utmp (for Ubuntu).
Here is an example from CentOS:
[root@TestLinux ~]# who root tty1 2013-12-09 10:44 root pts/0 2013-12-09 10:29 (10.0.2.2) sysadmin pts/1 2013-12-09 10:31 (10.0.2.2) joeblog pts/2 2013-12-09 10:39 (10.0.2.2)
In this particular case, I am the sole user of the system. I was running the server from
an Oracle VirtualBox and accessing it as root from both the console and an SSH
session. Two other user accounts (sysadmin and joebolg) were also accessing the
system.
The last command tells us the login history of users:
[root@TestLinux ~]# last | grep sysadmin sysadmin pts/1 10.0.2.2 Mon Dec 9 10:31 still logged insysadmin pts/0 10.0.2.2 Fri Nov 29 15:42 - crash (00:01) sysadmin pts/0 10.0.2.2 Thu Nov 28 17:06 - 17:13 (00:06) sysadmin pts/0 10.0.2.2 Thu Nov 28 16:17 - 17:05 (00:48) sysadmin pts/0 10.0.2.2 Thu Nov 28 09:29 - crash (06:04) sysadmin pts/0 10.0.2.2 Wed Nov 27 16:37 - down (00:29) sysadmin tty1 Wed Nov 27 14:05 - down (00:36) sysadmin tty1 Wed Nov 27 13:49 - 14:04 (00:15)
In this example, I am trying to find the login history of the user sysadmin. As you can
see, there were couple of instances where he managed to crash the system.
To find out when was the system last rebooted, we can run the following command:
[root@TestLinux ~]# last reboot
The result may look like this
reboot system boot 2.6.32-358.el6.x Mon Dec 9 10:27 - 10:47 (00:19)reboot system boot 2.6.32-358.el6.x Fri Dec 6 16:37 - 10:47 (2+18:10)reboot system boot 2.6.32-358.el6.x Fri Dec 6 16:28 - 16:36 (00:08) reboot system boot 2.6.32-358.el6.x Fri Dec 6 11:06 - 16:36 (05:29)reboot system boot 2.6.32-358.el6.x Mon Dec 2 17:00 - 16:36 (3+23:36)reboot system boot 2.6.32-358.el6.x Fri Nov 29 16:01 - 16:36 (7+00:34)reboot system boot 2.6.32-358.el6.x Fri Nov 29 15:43 - 16:36 (7+00:53)... ... wtmp begins Fri Nov 15 16:11:54 2013
To see when did someone last log in to the system, use lastlog:
[root@TestLinux ~]# lastlog
In my system, the output looked like this:
Username Port From Latest root tty1 Mon Dec 9 10:44:30 +1100 2013bin **Never logged in**daemon **Never logged in** adm **Never logged in** lp **Never logged in** sync **Never logged in** shutdown **Never logged in** halt **Never logged in** mail **Never logged in** uucp **Never logged in** operator **Never logged in** games **Never logged in** gopher **Never logged in** ftp **Never logged in** nobody **Never logged in** vcsa **Never logged in** saslauth **Never logged in** postfix **Never logged in** sshd **Never logged in** sysadmin pts/1 10.0.2.2 Mon Dec 9 10:31:50 +1100 2013 dbus **Never logged in** joeblog pts/2 10.0.2.2 Mon Dec 9 10:39:24 +1100 2013
For other text-based log files, you can use cat, head or tail commands to read the
contents.
In the example below, I am trying to look at the last ten lines of /var/log/messages file in
a Debian box:
debian@debian:~$ sudo tail /var/log/messages
Output:
Dec 16 01:21:08 debian kernel: [ 9.584074] Bluetooth: BNEP (Ethernet Emulation) ver 1.3 Dec 16 01:21:08 debian kernel: [ 9.584074] Bluetooth: BNEP filters: protocol multicast Dec 16 01:21:08 debian kernel: [ 9.648220] Bridge firewalling registeredDec 16 01:21:08 debian kernel: [ 9.696728] Bluetooth: SCO (Voice Link) ver 0.6 Dec 16 01:21:08 debian kernel: [ 9.696728] Bluetooth: SCO socket layer initialized Dec 16 01:21:08 debian kernel: [ 9.832215] lp: driver loaded but no devices found Dec 16 01:21:08 debian kernel: [ 9.868897] ppdev: user-space parallel port driver Dec 16 01:21:11 debian kernel: [ 12.748833] [drm] Initialized drm 1.1.0 20060810Dec 16 01:21:11 debian kernel: [ 12.754412] pci 0000:00:02.0: PCI INT A -> Link[LNKB] -> GSI 11 (level, low) -> IRQ 11 Dec 16 01:21:11 debian kernel: [ 12.754412] [drm] Initialized vboxvideo 1.0.0 20090303 for 0000:00:02.0 on minor 0
The rsyslog DaemonAt the heart of the logging mechanism is the rsyslog daemon. This service is
responsible for listening to log messages from different parts of a Linux system and
routing the message to an appropriate log file in the /var/log directory. It can also
forward log messages to another Linux server.
The rsyslog Configuration FileThe rsyslog daemon gets its configuration information from the rsyslog.conf file. The
file is located under the /etc directory.
Basically, the rsyslog.conf file tells the rsyslog daemon where to save its log messages.
This instruction comes from a series of two-part lines within the file.
This file can be found at rsyslog.d/50-default.conf on ubuntu.
The two part instruction is made up of a selector and an action. The two parts are
separated by white space.
The selector part specifies what's the source and importance of the log message and
the action part says what to do with the message.
The selector itself is again divided into two parts separated by a dot (.). The first part
before the dot is called *acility (the origin of the message) and the second part after the
dot is called priority (the severity of the message).
Together, the facility/priority and the action pair tell rsyslog what to do when a log
message matching the criteria is generated.
Here is excerpt from a CentOS rsyslog.conf file:
# rsyslog v5 configuration file... ... # Include all config files in /etc/rsyslog.d/ IncludeConfig /etc/rsyslog.d/*.conf
#### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console
# Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access. authpriv.* /var/log/secure
# Log all the mail messages in one place. mail.* -/var/log/maillog
# Log cron stuff cron.* /var/log/cron
# Everybody gets emergency messages *.emerg *
# Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log local7.* /var/log/boot.log ... ...
To understand what this all means, let's consider the different types of facilities
recognized by Linux. Here is a list:
auth or authpriv: Messages coming from authorization and security related events
kern: Any message coming from the Linux kernel
mail: Messages generated by the mail subsystem
cron: Cron daemon related messages
daemon: Messages coming from daemons
news: Messages coming from network news subsystem
lpr: Printing related log messages
user: Log messages coming from user programs
local0 to local7: Reserved for local use
And here is a list of priorities in ascending order:
debug: Debug information from programs
info: Simple informational message - no intervention is required
notice: Condition that may require attention
warn: Warning
err: Error
crit: Critical condition
alert: Condition that needs immediate intervention
emerg: Emergency condition
So now let's consider the following line from the file:
cron.* /var/log/cron
This just tells the rsyslog daemon to save all messages coming from the cron daemon
in a file called /var/log/cron. The asterix (*) after the dot (.) means messages of all
priorities will be logged. Similarly, if the facility was specified as an asterix, it would
mean all sources.
Facilities and priorities can be related in a number of ways.
In its default form, when there is only one priority specified after the dot, it means all
events equal to or greater than that priority will be trapped. So the following directive
causes any messages coming from the mail subsystem with a priority of warning or
higher to be logged in a specific file under /var/log:
mail.warn /var/log/mail.warn
This will log every message equal to or greater than the warn priority, but leave
everything below it. So messages with err, crit, alert or emerg will also be recorded in
this file.
Using an equal sign (=) after the dot (.) will cause only the specified priority to be
logged. So if we wanted to trap only the info messages coming from the mail
subsystem, the specification would be something like the following:
mail.=info /var/log/mail.info
Again, if we wanted to trap everything from mail subsystem except info messages, the
specification would be something like the following
mail.!info /var/log/mail.info
or
mail.!=info /var/log/mail.info
In the first case, the mail.info file will contain everything with a priority lower than info. In
the second case, the file will contain all messages with a priority above info.
Multiple facilities in the same line can be separated by commas.
Multiple sources (facility.priority) in the same line is separated by semicolon.
When an action is marked as an asterix (*), it means all users. This entry in my CentOS
rsyslog.conf file is saying exactly that:
# Everybody gets emergency messages *.emerg *
Try to see what's the rsyslog.conf is saying in your Linux system. Here is an excerpt
from the Debian server I am running:
# /etc/rsyslog.conf Configuration file for rsyslog. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html ... ... auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log daemon.* -/var/log/daemon.log kern.* -/var/log/kern.log lpr.* -/var/log/lpr.log mail.* -/var/log/mail.log user.* -/var/log/user.log
## Logging for the mail system. Split it up so that # it is easy to write scripts to parse these files. # mail.info -/var/log/mail.info mail.warn -/var/log/mail.warn mail.err /var/log/mail.err # # Logging for INN news system. # news.crit /var/log/news/news.crit news.err /var/log/news/news.err news.notice -/var/log/news/news.notice
As you can see, Debian saves all security/authorization level messages
in /var/log/auth.log whereas CentOS saves it under /var/log/secure.
The configurations for rsyslog can come from other custom files as well. These custom
configuration files are usually located in different directories under /etc/rsyslog.d. The
rsyslog.conf file includes these directories using $IncludeConfig directive.
Here is what it looks like in Ubuntu:
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf .... .... $IncludeConfig /etc/rsyslog.d/*.conf
The contents under the /etc/rsyslog.d directory looks like the following:
-rw-r--r-- 1 root root 311 Mar 17 2012 20-ufw.conf -rw-r--r-- 1 root root 252 Apr 11 2012 21-cloudinit.conf
-rw-r--r-- 1 root root 1655 Mar 30 2012 50-default.conf
Now the destination for a log message does not necessarily have to be a log file; the
message can be sent to a user's console. In this case, the action field will contain the
username. If more than one user needs to receive the message, their usernames are
separated by commas. If the message needs to be broadcast to every user, it's
specified by an asterix (*) in the action field.
Because of being part of a network operating system, rsyslog daemon can not only
save log messages locally, it can also forward them to another Linux server in the
network or act as a repository for other systems. The daemon listens for log messages
in UDP port 514. The example below will forward kernel critical messages to a server
called "texas".
kern.crit @texas
Creating and Testing Your Own Log MessagesSo now it's time for us to create our own log files.
To test this, we will do the following
Add a log file specification in /etc/rsyslog.conf file
Restart the rsyslog daemon
Test the configuration using the logger utility
In the following example, I am adding two new lines in my CentOS Linux system's
rsyslog.conf file. As you can see, each of them are coming from a facility called local4
and they have different priorities.
[root@TestLinux ~]# vi /etc/rsyslog.conf .... ....
# New lines added for testing log message generation
local4.crit /var/log/local4crit.log local4.=info /var/log/local4info.log
Next, the service is restarted so the config file data is reloaded:
[root@TestLinux ~]# /etc/init.d/rsyslog restart Shutting down system logger: [ OK ] Starting system logger: [ OK ] [root@TestLinux ~]#
To generate the log message now, the logger application is called:
[root@TestLinux ~]# logger -p local4.info " This is a info message from local 4"
Looking under the /var/log directory now shows two new files:
...
... -rw------- 1 root root 0 Dec 9 11:21 local4crit.log -rw------- 1 root root 72 Dec 9 11:22 local4info.log
The size of the local4info.log is non-zero. So when it's opened, I see the message has
been recorded:
[root@TestLinux ~]# cat /var/log/local4info.log Dec 9 11:22:32 TestLinux root: This is a info message from local 4
Rotating Log FilesAs more and more information is written to log files, they get bigger and bigger. This
obviously poses a potential performance problem. Also, the management of the files
become cumbersome.
Linux uses the concept of "rotating" log files instead of purging or deleting them. When
a log is rotated, a new log file is created and the old log file is renamed and optionally
compressed. A log file can thus have multiple old versions remaining online. These files
will go back over a period of time and will represent the backlog. Once a certain number
of backlogs have been generated, a new log rotation will cause the oldest log file to be
deleted.
The rotation is initiated through the logrotate utility.
The logrotate Configuration FileLike rsyslog, logrotate also depends on a configuration file and the name of this file is
logrotate.conf. It's located under /etc.
Here is what I see in the logrotate.conf file of my Debian server:
debian@debian:~$ cat /etc/logrotate.conf # see "man logrotate" for details # rotate log files weekly weekly
# keep 4 weeks worth of backlogs rotate 4
# create new (empty) log files after rotating old ones create
# uncomment this if you want your log files compressed #compress
# packages drop log rotation information into this directory include /etc/logrotate.d
# no packages own wtmp, or btmp -- we'll rotate them here /var/log/wtmp { missingok monthly create 0664 root utmp rotate 1
}
/var/log/btmp { missingok monthly create 0660 root utmp rotate 1 }
# system-specific logs may be configured here
The lines are fairly self-explanatory. By default, log files are to be rotated weekly with
four backlogs remaining online at any one time. When the program runs, a new, empty
log file will be generated and optionally the old ones will be compressed.
The only exception is for wtmp and btmp files. wtmp keeps track of system logins and
btmp keeps track of bad login attempts. Both these log files are to be rotated every
month and no error is returned if any previous wtmp or btmp file can be found.
Custom log rotation configurations are kept under etc/logrotate.d directory. These are
also inluded in the logrotate.conf with the include directive. The Debian installation
shows me the content of this directory:
debian@debian:~$ ls -l /etc/logrotate.d total 44 -rw-r--r-- 1 root root 173 Apr 15 2011 apt -rw-r--r-- 1 root root 79 Aug 12 2011 aptitude -rw-r--r-- 1 root root 135 Feb 24 2010 consolekit -rw-r--r-- 1 root root 248 Nov 28 2011 cups -rw-r--r-- 1 root root 232 Sep 19 2012 dpkg -rw-r--r-- 1 root root 146 May 12 2011 exim4-base -rw-r--r-- 1 root root 126 May 12 2011 exim4-paniclog -rw-r--r-- 1 root root 157 Nov 16 2010 pm-utils -rw-r--r-- 1 root root 94 Aug 8 2010 ppp -rw-r--r-- 1 root root 515 Nov 30 2010 rsyslog -rw-r--r-- 1 root root 114 Nov 26 2008 unattended-upgrades
The contents of the rsyslog shows how to recycle a number of log files:
debian@debian:~$ cat /etc/logrotate.d/rsyslog/var/log/syslog { rotate 7 daily missingok notifempty delaycompress compress postrotate invoke-rc.d rsyslog reload > /dev/null endscript }
/var/log/mail.info /var/log/mail.warn /var/log/mail.err /var/log/mail.log /var/log/daemon.log /var/log/kern.log /var/log/auth.log /var/log/user.log /var/log/lpr.log /var/log/cron.log /var/log/debug /var/log/messages { rotate 4 weekly missingok notifempty compress delaycompress sharedscripts postrotate invoke-rc.d rsyslog reload > /dev/null endscript }
As you can see, the syslog file will be reinitialized every day with seven days' worth of
logs being kept online. Other log files are rotated every week.
Also worth noting is the postrotate directive. This specifies the action that happens after
the whole log rotation has completed.
Testing the RotationLogrotate can be manually run to recycle one or more files. And to do that, we simply
specify the relevant configuration file as an argument to the command.
To see how this works, here is a partial list of log files under /var/log directory in my test
CentOS server:
[root@TestLinux ~]# ls -l /var/log total 800 ... -rw------- 1 root root 359 Dec 17 18:25 maillog -rw-------. 1 root root 1830 Dec 16 16:35 maillog-20131216 -rw------- 1 root root 30554 Dec 17 18:25 messages -rw-------. 1 root root 180429 Dec 16 16:35 messages-20131216 -rw------- 1 root root 591 Dec 17 18:28 secure -rw-------. 1 root root 4187 Dec 16 16:41 secure-20131216 ... ...
The partial contents of the logrotate.conf file looks like this:
[root@TestLinux ~]# cat /etc/logrotate.conf# see "man logrotate" for details# rotate log files weeklyweekly
# keep 4 weeks worth of backlogs rotate 4
# create new (empty) log files after rotating old ones
create ... ...
Next we run the logrotate command:
[root@TestLinux ~]# logrotate -fv /etc/logrotate.conf
Messages scroll over as new files are generated, errors are encountered etc. When the
dust settles, we try to check for new mail, secure or messages files:
[root@TestLinux ~]# ls -l /var/log/mail* -rw------- 1 root root 0 Dec 17 18:34 /var/log/maillog -rw-------. 1 root root 1830 Dec 16 16:35 /var/log/maillog-20131216 -rw------- 1 root root 359 Dec 17 18:25 /var/log/maillog-20131217
[root@TestLinux ~]# ls -l /var/log/messages* -rw------- 1 root root 148 Dec 17 18:34 /var/log/messages-rw-------. 1 root root 180429 Dec 16 16:35 /var/log/messages-20131216 -rw------- 1 root root 30554 Dec 17 18:25 /var/log/messages-20131217
[root@TestLinux ~]# ls -l /var/log/secure* -rw------- 1 root root 0 Dec 17 18:34 /var/log/secure-rw-------. 1 root root 4187 Dec 16 16:41 /var/log/secure-20131216 -rw------- 1 root root 591 Dec 17 18:28 /var/log/secure-20131217 [root@TestLinux ~]#
As we can see, all three new log files have been created. The maillog and secure files
are still empty, but the new messages file already has some data in it.