Tame your logs with (an) ELK - suse.com · Tame your logs with (an) ELK State-of-the-art monitoring...
Transcript of Tame your logs with (an) ELK - suse.com · Tame your logs with (an) ELK State-of-the-art monitoring...
Tame your logs with (an) ELKState-of-the-art monitoring and log analysis
Klaus Kämpf
Product Owner SUSE Manager
SUSE Linux
2
Preface
This is a research project and technology preview.
It may or may not result in a maintained product offering in the future.
3
4
5
Problem Statement
6
127.0.0.1 - - [05/Oct/2016:15:30:18 +0200] "GET /cgi-bin/translate_key.cgi?scout_shared_key=7a6a63848194 HTTP/1.1" 200 1 "-" "SatIDXL8r/1.0 libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:30:18 +0200] "POST /satconfig/cgi-mod-perl/accept_status_log.cgi HTTP/1.1" 200 -127.0.0.1 - - [05/Oct/2016:15:30:39 +0200] "GET /cgi-bin/translate_key.cgi?scout_shared_key=7a6a63848194 HTTP/1.1" 200 1 "-" "SatIDXL8r/1.0 libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:30:39 +0200] "POST /tsdb HTTP/1.1" 200 82 "-" "libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:30:39 +0200] "POST /cgi-bin/eventHandler.cgi HTTP/1.1" 200 82127.0.0.1 - - [05/Oct/2016:15:30:40 +0200] "GET /cgi-bin/translate_key.cgi?scout_shared_key=7a6a63848194 HTTP/1.1" 200 1 "-" "SatIDXL8r/1.0 libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:30:40 +0200] "GET /satconfig/cgi-mod-perl/fetch_commands.cgi?cluster_id=7a6a63848194&node_id=2&role=lead&version=1.0 HTTP/1.1" 200 2310.160.4.230 - - [05/Oct/2016:15:30:41 +0200] "GET /satconfig/cgi-mod-perl/fetch_commands.cgi?cluster_id=7a6a63848194&node_id=2&role=lead&version=1.0 HTTP/1.1" 200 2310.160.4.230 - - [05/Oct/2016:15:30:50 +0200] "POST /tsdb HTTP/1.1" 200 164 "-" "libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:30:50 +0200] "POST /cgi-bin/eventHandler.cgi HTTP/1.1" 200 16410.162.166.1 - - [05/Oct/2016:15:30:58 +0200] "POST /XMLRPC HTTP/1.1" 200 16310.162.166.1 - - [05/Oct/2016:15:30:58 +0200] "POST /XMLRPC HTTP/1.1" 200 73110.160.4.230 - - [05/Oct/2016:15:31:00 +0200] "POST /cobbler_api HTTP/1.1" 200 144 "-" "Java/1.7.0"10.160.4.230 - - [05/Oct/2016:15:31:00 +0200] "POST /cobbler_api HTTP/1.1" 200 129 "-" "Java/1.7.0"10.160.4.230 - - [05/Oct/2016:15:31:00 +0200] "POST /tsdb HTTP/1.1" 200 111 "-" "libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:31:00 +0200] "POST /cgi-bin/eventHandler.cgi HTTP/1.1" 200 11110.160.4.230 - - [05/Oct/2016:15:31:05 +0200] "POST /tsdb HTTP/1.1" 200 87 "-" "libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:31:05 +0200] "POST /cgi-bin/eventHandler.cgi HTTP/1.1" 200 87127.0.0.1 - - [05/Oct/2016:15:31:18 +0200] "GET /cgi-bin/translate_key.cgi?scout_shared_key=7a6a63848194 HTTP/1.1" 200 1 "-" "SatIDXL8r/1.0 libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:31:18 +0200] "POST /satconfig/cgi-mod-perl/accept_status_log.cgi HTTP/1.1" 200 -10.160.4.230 - - [05/Oct/2016:15:31:41 +0200] "GET /satconfig/cgi-mod-perl/fetch_commands.cgi?cluster_id=7a6a63848194&node_id=2&role=lead&version=1.0 HTTP/1.1" 200 2310.160.4.230 - - [05/Oct/2016:15:32:00 +0200] "POST /cobbler_api HTTP/1.1" 200 144 "-" "Java/1.7.0"10.160.4.230 - - [05/Oct/2016:15:32:00 +0200] "POST /cobbler_api HTTP/1.1" 200 129 "-" "Java/1.7.0"127.0.0.1 - - [05/Oct/2016:15:32:18 +0200] "GET /cgi-bin/translate_key.cgi?scout_shared_key=7a6a63848194 HTTP/1.1" 200 1 "-" "SatIDXL8r/1.0 libwww-perl/5.816"10.160.4.230 - - [05/Oct/2016:15:32:18 +0200] "POST /satconfig/cgi-mod-perl/accept_status_log.cgi HTTP/1.1" 200 -10.162.166.1 - - [05/Oct/2016:15:32:26 +0200] "POST /XMLRPC HTTP/1.1" 200 16310.162.166.1 - - [05/Oct/2016:15:32:27 +0200] "POST /XMLRPC HTTP/1.1" 200 731
7
8
Elasticsearch
Kibana
Logstash
9Apache Logos: The Apache Software Foundation - http://svn.apache.org/viewvc/jakarta/site/xdocs/images/logos/tomcat.eps, Apache License 2.0, https://commons.wikimedia.org/w/index.php?curid=11302180
...
Elasticsearch
Logstash Kibana
10
Elasticsearch
11
Elasticsearch
Full text database
Scalable
Terminology● Index: Database● Mapping: Schema● Document: Record● Field: key-value pair
12
Elasticsearch – raw data
13
Elasticsearch – Kibana fields
14
Elasticsearch – internal fields
15
Logstash
16
Logstash - Overview
Log server
Scalable
Focus on time-based events
JRuby
Main components● Input - typically text or json● Filter - parse and manipulate● Output - usually Elasticsearch
17
Logstash - input.conf
input {
stdin {}
}
18
Logstash - input.conf
input {
tcp {
port => 9000
type => "access_log"
}
tcp {
port => 9001
type => "error_log"
tags => ["tag1", "tag2"]
...
19
Logstash - filter.conf
filter {
if ([type] == "osa-dispatcher") {
grok {
match => {
"message" => "\d\d\d\d\/\d\d\/\d\d\s\d\d:\d\d:\d\d\s[+-]\d\d:\d\d:timestamp ...”
}
}
}
}
20
Logstash - filter.pattern
# osa-dispatcher
# 2015/06/12 11:39:04 +02:00 14117 0.0.0.0: osad/jabber_lib.main('ERROR',...)
TIMESTAMP \d\d\d\d\/\d\d\/\d\d\s\d\d:\d\d:\d\d\s[+-]\d\d:\d\d
PID [\d]+
FUNCTION [\w_\.]+
ARGS \([^\)]+\)
21
Logstash - filter.conf
filter {
if ([type] == "osa-dispatcher") {
grok {
match => {
"message" => "%{TIMESTAMP:timestamp} %{PID:pid:int} %{IPV4:clientip}: ...”
}
}
}
}
22
Logstash - output.conf
output {
stdout { codec => rubydebug }
}
23
Logstash - output.conf
output {
elasticsearch {
hosts => [“localhost:9200”]
}
}
24
Logstash - Start
# ls
filter.conf input.conf osa-dispatcher.pattern output.conf rhn_web_api.pattern
# logstash -f ./\*.conf --auto-reload
25
Kibana
26
Kibana
Visualization frontend to Elasticsearch
Web frontend
Focus on time-based events
Comfortable query interface
Dashboard management
● Settings● Discover● Visualize● Dashboard
27
Kibana - Settings
● Select index pattern– wildcards possbile
● Time based ?– Time-field name
Loads mapping● field names● field types● analyzed ?
28
Kibana - Discover
● No results found ?– Expand your time range
● Explore fields● Include/Exclude
● Create query● Save search● Visualize !
29
Kibana - Visualize
● Create new● Select visualization type
● New/Saved search● Graph-specific parameters
30
Kibana - Dashboard
Collection of visualization tiles
Can be saved/shared
31
SUSE Manager
32
SUSE Manager - Components
Apache Web Server
Tomcat Application Server
PostgreSQL database
Java Application Stack
Python API
Salt
33
SUSE Manager - Apache
# grep ErrorLog /etc/apache2/httpd.conf
#ErrorLog /var/log/apache2/error_log
# cat /etc/apache2/sysconfig.d/logstash.conf
CustomLog "|/usr/bin/nc logstash.mgr.suse.de 9000" combined
ErrorLog "|$/usr/bin/tee -a /var/log/apache2/error_log
| /usr/bin/nc logstash.mgr.suse.de 9001"
34
SUSE Manager - Tomcat
# /etc/tomcat/log4j.properties
log4j.rootLogger=debug, R, LogstashAppender
...
log4j.appender.LogstashAppender=org.apache.log4j.net.SocketAppender
log4j.appender.LogstashAppender.port=9006
log4j.appender.LogstashAppender.remoteHost=logstash.mgr.suse.de
35
SUSE Manager - Java
# /usr/share/rhn/classes/log4j.properties
log4j.rootLogger=WARN,RootAppender,LogstashAppender
...
log4j.appender.LogstashAppender=org.apache.log4j.net.SocketAppender
log4j.appender.LogstashAppender.port=9007
log4j.appender.LogstashAppender.remoteHost=logstash.mgr.suse.de
36
Elastic Beats
37
Beats
Formerly ‘logstash-forwarder’
Unobtrusive (log) file forwarder
Written in Go, fast
Simple configuration
38
SUSE Manager - Filebeat
# /usr/filebeat/filebeat.yml
filebeat:
prospectors:
-
paths:
- /var/log/apache2/access_log
encoding: utf-8
input_type: log
document_type: access_log
...
output:
logstash:
hosts: ["logstash.mgr.suse.de:5045"]
39
Packages
40
Packages
Watch this space for packages
https://build.opensuse.org/package/show/security:logging/elasticsearch
41
Outlook
42
Outlook
● Complete packaging● Package queries ?● Package visualizations ?
● Document best practices
● Use to observe testing
● Automated analysis of supportconfigs
● Productize ?