Open Source Logging and Metrics Tools
-
Upload
phase2 -
Category
Technology
-
view
393 -
download
0
description
Transcript of Open Source Logging and Metrics Tools
![Page 1: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/1.jpg)
D E V O P S T R A C K , D R U P A L C O N A M S T E R D A M
OPEN-SOURCE LOGG ING AND MONITORING TOOLS
S T E V E N W . M E R R I L L
![Page 2: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/2.jpg)
INTRODUCTION
![Page 3: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/3.jpg)
Director of Engineering, Phase2 Volunteer, drupal.org infrastructure
Steven Merrill
Twitter: @stevenmerrill
![Page 4: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/4.jpg)
ABOUT THIS TALK
• Help you visualize your data with OSS tools
• Information on customizing logs from common daemons
• Strong focus on log aggregation, parsing, and search
• Information about drupal.org's logging setup
• Some information on performance metrics tools and monitoring
• Two-machine demo of Drupal and logging tools
![Page 5: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/5.jpg)
DEMO SETUP
• 2 Google Cloud Engine g1.small instances
• Both instances run collectd to grab system metrics
• 1 'drupal' instance with rsyslog shipping logs, Apache, Varnish, MySQL, PHP
• 1 'utility' instance with rsyslog host, Jenkins, Graphite, Grafana, ElasticSearch, Logstash, Kibana, bucky (statsd)
![Page 6: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/6.jpg)
INITIAL DEMO:ELK STACK IN ACTION
![Page 7: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/7.jpg)
KIBANA
![Page 8: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/8.jpg)
KIBANA
![Page 9: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/9.jpg)
LOGS
![Page 10: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/10.jpg)
CECI N'EST PAS UNE LOG
https://www.flickr.com/photos/data_op/2575797856/
![Page 11: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/11.jpg)
"LOGS ARE TIME + DATA."- Jordan Sissel, creator of Logstash
![Page 12: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/12.jpg)
WHAT ARE LOGS
• Ultimately, logs are about keeping track of events
• Logs are very different; some use custom formats, while some may be in pure XML or JSON
• Some are one line, some are many, like Java stacktraces or MySQL slow query logs
![Page 13: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/13.jpg)
WHO PRODUCES LOGS
• Drupal
• nginx
• Apache
• Varnish
• Jenkins
• ElasticSearch or SOLR
• MySQL
• cron
• sudo
• ...
![Page 14: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/14.jpg)
TYPES OF LOGS
• Error Logs
• Transaction Logs
• Trace and Debug Logs
![Page 15: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/15.jpg)
ISSUES WITH LOGS
• Legal retention requirements
• Require shell access to view
• Not often human-parseable
• Cyborg-friendly tooling
![Page 16: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/16.jpg)
SOLVING PROBLEMS WITH LOG DATA
• Find slow pages or queries
• Sort through Drupal logs to trace user action on a site
• Get an average idea of traffic to a particular area
• Track new PHP error types
![Page 17: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/17.jpg)
LOG FORMATSADDING MORE SMARTS TO YOUR LOGS
![Page 18: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/18.jpg)
SYSLOG
Apr 11 18:35:53 shiftiest dnsmasq-dhcp[23185]: DHCPACK(br100) 192.168.32.4 fa:16:3e:c4:2f:fd varnish4 !Mar 11 15:38:14 drupal drupal: http://192.168.32.3|1394566694|system|192.168.32.1|http://192.168.32.3/admin/modules/list/confirm|http://192.168.32.3/admin/modules|1||php module installed.
![Page 19: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/19.jpg)
APACHE
127.0.0.1 - - [08/Mar/2014:00:36:44 -0500] "GET /dashboard HTTP/1.0" 302 20 "https://68.232.187.42/dashboard/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.146 Safari/537.36"
![Page 20: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/20.jpg)
NGINX
192.168.32.1 - - [11/Apr/2014:10:44:36 -0400] "GET /kibana/font/fontawesome-webfont.woff?v=3.2.1 HTTP/1.1" 200 43572 "http://192.168.32.6/kibana/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36"
![Page 21: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/21.jpg)
VARNISH
192.168.32.1 - - [11/Apr/2014:10:47:52 -0400] "GET http://192.168.32.3/themes/seven/images/list-item.png HTTP/1.1" 200 195 "http://192.168.32.3/admin/config" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36"
![Page 22: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/22.jpg)
ADDITIONAL FEATURES
• Apache, nginx, and Varnish all support additional output
• Time spent processing the request
• Varnish can log cache hit/miss
• Any header
• With Logstash we can normalize these
![Page 23: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/23.jpg)
APACHE
• Configurable log formats are available – http://httpd.apache.org/docs/2.2/mod/mod_log_config.html
• A single LogFormat directive in any Apache configuration file will override all log formats
• The default NCSA combined log format is as follows
• LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
![Page 24: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/24.jpg)
APACHE
• Additional useful information:
• %D Time taken to serve request in microseconds
•%{Host}i Value of the Host HTTP header
•%p Port
• New LogFormat line:
• LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D %{Host}i %p" combined
![Page 25: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/25.jpg)
APACHE
127.0.0.1 - - [29/Jul/2014:22:03:07 +0000] "GET /admin/config/development/performance HTTP/1.0" 200 3500 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36" !127.0.0.1 - - [29/Jul/2014:22:03:07 +0000] "GET /admin/config/development/performance HTTP/1.0" 200 3500 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36" 45304 23.251.149.143 80
![Page 26: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/26.jpg)
NGINX
• Log formats are defined with the log_format directive – http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format
• You may not override the default NCSA combined format
• log_format combined '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"';
![Page 27: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/27.jpg)
NGINX
• Additional useful information:
• $request_time Time taken to serve request in seconds with
millisecond resolution (e.g. 0.073)
•$http_host Value of the Host HTTP header
•$server_post Port
![Page 28: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/28.jpg)
NGINX
• New log_format line and example config for a vhost:
• log_format logstash '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '$request_time $http_host $server_port';
• access_log /var/log/nginx/access.log logstash;
![Page 29: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/29.jpg)
NGINX
70.42.157.6 - - [22/Jul/2014:22:03:30 +0000] "POST /logstash-2014.07.22/_search HTTP/1.0" 200 281190 "http://146.148.34.62/kibana/index.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36" !70.42.157.6 - - [22/Jul/2014:22:03:30 +0000] "POST /logstash-2014.07.22/_search HTTP/1.0" 200 281190 "http://146.148.34.62/kibana/index.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36" 0.523 146.148.34.62 80
![Page 30: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/30.jpg)
VARNISH
• The varnishncsa daemon outputs NCSA-format logs
• You may pass a different log format to the varnishncsa daemon
• Many of the configuration options share the same format as Apache
![Page 31: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/31.jpg)
VARNISH
• Additional useful information:
• %D Time taken to serve request in seconds with microsecond precision (e.g. 0.000884)
• %{Varnish:hitmiss}x The text "hit" or "miss"
• varnishncsa daemon argument:
• -F '%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %D %{Varnish:hitmiss}x'
![Page 32: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/32.jpg)
VARNISH
70.42.157.6 - - [29/Jul/2014:22:03:07 +0000] "GET http://23.251.149.143/admin/config/development/performance HTTP/1.0" 200 3500 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36" !70.42.157.6 - - [29/Jul/2014:22:03:07 +0000] "GET http://23.251.149.143/admin/config/development/performance HTTP/1.0" 200 3500 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36" 0.045969 miss
![Page 33: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/33.jpg)
SHIPPING LOGS
![Page 34: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/34.jpg)
SHIP THOSE LOGS!
• syslog-ng
• rsyslogd
• Ship syslog
• Ship other log files
• Logstash
• logstash-forwarder (formerly Lumberjack)
• Beaver
![Page 35: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/35.jpg)
SHIPPING CONCERNS
• Queueing
• Behavior when shipping to remote servers
• Max spool disk usage
• Retries?
• Security
• Encrypted channel
• Encrypted at rest
• Access to sensitive data
![Page 36: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/36.jpg)
SHIP LOGS WITH RSYSLOGD
![Page 37: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/37.jpg)
CONFIGURING RSYSLOGD CLIENTS
• Ship logs to another rsyslog server over TCP
• *.* @@utility:514
• This defaults to shipping anything that it would normally log to /var/log/syslog or /var/log/messages
![Page 38: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/38.jpg)
CONFIGURING RSYSLOGD SERVERS
• Prevent remote logs from showing up in /var/log/messages
• if $source != 'utility' then ~
• Store logs coming in based on hostname and date
• $template DailyPerHostLogs,"/var/log/rsyslog/%HOSTNAME%/%HOSTNAME%.%$YEAR%-%$MONTH%-%$DAY%.log"*.* -?DailyPerHostLogs;RSYSLOG_TraditionalFileFormat
![Page 39: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/39.jpg)
CONFIGURING RSYSLOGD SHIPPING
• Read lines from a particular file and ship over syslog
• $ModLoad imfile$InputFileName /var/log/httpd/access_log$InputFileTag apache_access:$InputFileStateFile state-apache_access$InputFileSeverity info$InputFileFacility local0$InputFilePollInterval 10$InputRunFileMonitor
![Page 40: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/40.jpg)
CONFIGURING RSYSLOGD SPOOLING
• Configure spooling and queueing behavior
• $WorkDirectory /var/lib/rsyslog # Where to place spool files$ActionQueueFileName fwdRule1 # Unique name prefix for spool files$ActionQueueMaxDiskSpace 1g # 1 GB space limit$ActionQueueSaveOnShutdown on # Save messages to disk on shutdown$ActionQueueType LinkedList # Run asynchronously$ActionResumeRetryCount -1 # Infinite retries if host is down
![Page 41: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/41.jpg)
SYSLOG-SHIPPED LOG FILES
Mar 11 15:38:14 drupal drupal: http://192.168.32.3|1394566694|system|192.168.32.1|http://192.168.32.3/admin/modules/list /confirm|http://192.168.32.3/admin/modules|1||php module installed. !Jul 30 15:04:14 drupal varnish_access: 156.40.118.178 - - [30/Jul/2014:15:04:09 +0000] "GET http://23.251.149.143/misc/tableheader.js?n9j5uu HTTP/1.1" 200 1848 "http://23.251.149.143/admin/modules" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36" 0.000757 miss
![Page 42: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/42.jpg)
AUTOMATED LOG AGGREGATION TOOLS
![Page 43: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/43.jpg)
PROPRIETARY LOG AGGREGATION TOOLS
• Third-party SaaS systems are plentiful in this area
• Splunk
• SumoLogic
• Loggly
• LogEntries
![Page 44: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/44.jpg)
LOGSTASH
• http://logstash.net/
• Great tool to work with logs of ALL sorts
• Has input, filter, and output pipelines
• Inputs can be parsed with different codecs (JSON, netflow)
• http://logstash.net/docs/1.4.2/ describes many options
![Page 45: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/45.jpg)
ELASTICSEARCH
• http://www.elasticsearch.com/
• A Java search engine based on Lucene, similar to SOLR
• Offers a far nicer REST API; easy discovery for clustering
![Page 46: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/46.jpg)
KIBANA
• Great viewer for Logstash logs
• Needs direct HTTP access to ElasticSearch
• You may need to protect this with nginx or the like
• Uses ElasticSearch features to show statistical information
• Can show any ElasticSearch data, not just Logstash
• Logstash JARs/packages can also run Kibana from an embedded server
• ./bin/logstash web -a ADDRESS -p PORT
![Page 47: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/47.jpg)
GROK
• Tool for pulling semantic data from logs; logstash filter
• A regex engine with built-in named patterns
• Online tools to parse sample logs
• http://grokdebug.herokuapp.com/
• http://grokconstructor.appspot.com/
![Page 48: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/48.jpg)
EXAMPLE:GROKKING NGINX LOGS
192.168.32.1 - - [11/Apr/2014:10:44:36 -0400] "GET /kibana/font/fontawesome-webfont.woff?v=3.2.1 HTTP/1.1" 200 43572 "http://192.168.32.6/kibana/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36"
![Page 49: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/49.jpg)
CONFIGURING LOGSTASH
![Page 50: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/50.jpg)
LOGSTASH CONFIG
• By default Logstash looks in /etc/logstash/conf.d/*.conf
• You many include multiple files
• Each must have at least an input, filter, or output stanza
• Example Logstash configuration follows to parse Drupal and nginx logs from syslog and normalize duration to milliseconds
• The full example is posted at http://git.io/e6TvAg
![Page 51: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/51.jpg)
LOGSTASH CONFIG
input { file { path => "/var/log/rsyslog/*/*.log" exclude => "*.bz2" type => syslog sincedb_path => "/var/run/logstash/sincedb" sincedb_write_interval => 10 } } !output { elasticsearch_http { host => "localhost" flush_size => 5000 } }
![Page 52: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/52.jpg)
LOGSTASH CONFIG
filter { if [type] == "syslog" { mutate { add_field => [ "syslog_message", "%{message}" ] remove_field => "message" } grok { match => [ "syslog_message", "%{SYSLOGLINE}" ] } date { match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] }
![Page 53: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/53.jpg)
LOGSTASH CONFIG
# Parse Drupal logs that are logged to syslog. if [program] == "drupal" { grok { match => [ "message", "https?://%{HOSTNAME:vhost}?\|%{NUMBER:d_timestamp}\|(?<d_type>[^\|]*)\|%{IP:d_ip}\|(?<d_request_uri>[^\|]*)\|(?<d_referer>[^\|]*)\|(?<d_uid>[^\|]*)\|(?<d_link>[^\|]*)\|(?<d_message>.*)" ] } }
![Page 54: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/54.jpg)
LOGSTASH CONFIG
# Parse nginx logs that have been shipped over syslog. if [program] == "nginx_access" { mutate { remove_field => [ "path", "pid", "syslog_message", "timestamp" ] } grok { match => [ "message", "%{COMBINEDAPACHELOG} %{BASE16FLOAT:duration} %{IPORHOST:hostname} %{POSINT:port}" ] } # Put the hostname into the request, a la Varnish. if [port] == "443" { mutate { replace => [ "request", "https://%{hostname}%{request}" ] } }
![Page 55: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/55.jpg)
LOGSTASH CONFIG
else { mutate { replace => [ "request", "http://%{hostname}%{request}" ] } } ruby { code => "event['duration'] = event['duration'].to_f * 1000.0" } date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] } useragent { source => "agent" target => "browser" } }
![Page 56: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/56.jpg)
MONITORING AND PERFORMANCE METRICS
![Page 57: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/57.jpg)
LOGS VS PERFORMANCE COUNTERS
• Generally, logs capture data at a particular time
• You may also want to keep information about how your servers are running and performing
• A separate set of tools are often used to help monitoring and manage systems performance
• This data can then be trended to chart resource usage and capacity
![Page 58: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/58.jpg)
PROPRIETARY METRICS TOOLS
• Third-party SaaS systems are also plentiful in this area
• DataDog
• Librato Metrics
• Circonus
• New Relic / AppNeta
![Page 59: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/59.jpg)
TIME-SERIES DATA
• Generally, performance counters are taken with regular sampling at an interval, known as time-series data
• Several OSS tools exist to store and query time-series data:
• RRDTool
• Whisper
• InfluxDB
![Page 60: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/60.jpg)
FIRST WAVE: RRD-BASED TOOLS
• Many tools can graph metrics and make and plot RRD files
• Munin
• Cacti
• Ganglia
• collectd
![Page 61: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/61.jpg)
SECOND WAVE: GRAPHITE
• Graphite is a more general tool; it does not collect metrics
• It uses an advanced storage engine called Whisper
• It can buffer data and cache it under heavy load
• It does not require data to be inserted all the time
• It's fully designed to take time-series data and graph it
![Page 62: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/62.jpg)
STATSD
• Statsd is a tool created by Etsy as a simple stats aggregator
• It aggregates counters and sends them to Graphite
• It accepts data over TCP or UDP
• Logstash can increment statsd counters based on log data
• The canonical implementation is node.js, but many ports exist
• statsite (C), bucky (Python)
• For counters, sends raw stats each flush (stats.*) and also event counts per second (stats_counts.*)
![Page 63: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/63.jpg)
LOGSTASH CONFIG
output { if [program] == "varnish_access" { statsd { sender => "%{logsource}" # Count each event based on the HTTP response code. increment => "varnish.response.%{response}" # Also count Varnish hits and misses. increment => "varnish.cache.%{cache}" # Use the 'bytes' field from the varnish log as the count value. count => [ "varnish.bytes", "%{bytes}" ] } } }
![Page 64: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/64.jpg)
GRAFANA
• Grafana is to Graphite as Kibana is to ElasticSearch
• HTML / JavaScript app
• The user's browser needs direct HTTP access to Graphite
• You may need to protect this with nginx or the like
![Page 65: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/65.jpg)
COLLECTD
• http://collectd.org/
• Collectd is a tool that makes it easy to capture many system-level statistics
• It can write to RRD databases or to Graphite
• Collectd is written in C and is efficient; it can remain resident in memory and report on a regular interval
![Page 66: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/66.jpg)
DEMO: GRAPHITE / COLLECTD / GRAFANA
![Page 67: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/67.jpg)
MONITORING & METRICS
![Page 68: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/68.jpg)
MONITORING TOOLS
• Monitoring tools
• Nagios
• Icinga
• Sensu
• Zenoss
• Zabbix
• Monitoring check tools
• Monitoring Plugins (née Nagios Plugins)
• https://www.monitoring-plugins.org/
• #monitoringsucks
• #monitoringlove
![Page 69: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/69.jpg)
MONITORING CHECKS
• Monitoring is generally used to get server state.
• Are the web servers online - can I ping them? check_ping
• Is the disk more than 80% full on any of my servers? check_disk
• Is Redis running and responding to a PING command? check_tcp
• Is the load average on any machine over 4? check_load
• Have any Drush processes run for more than 10m? check_procs
![Page 70: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/70.jpg)
METRICS EXAMPLES
• Metrics have far deeper insight into application state.
• How many failed login attempts have happened in the last 10m?
• How many users have logged into the system this week?
• What is the average Varnish cache hit rate over the last 5m?
• How many 404s have we gotten in the last day?
• What is the average backend response time in the last hour?
![Page 71: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/71.jpg)
BRIDGING METRICS AND MONITORING
• Specific metrics values can be very useful to turn into monitoring alerts.
• The https://github.com/heroku/umpire project watches Graphite or Librato Metrics and responds with a HTTP 200 or 500.
• Point check_http checks or external monitoring like Pingdom at it.
• To avoid alert fatigue, try to establish conditions that you know are truly anomalous.
![Page 72: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/72.jpg)
BRIDGING METRICS AND MONITORING
• Example metrics/monitoring bridges with Umpire
• Did the average Varnish cache hit rate fall under 60 over the last 5m?
• /check?metric=statsd.varnish-hit-rate&min=60&range=300
• Did uid 1 have more than 5 failed login attempts in the past 1h?
• /check?metric=statsd.uid-1-login-failure&max=60&range=3600
![Page 73: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/73.jpg)
DEMO: UMPIRE / STATSD
![Page 74: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/74.jpg)
THE DRUPAL.ORGLOGGING SETUP
![Page 75: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/75.jpg)
SINGLE LOG HOST MACHINE
• CentOS 5
• Dual quad-core Gulftown Xeons (8 cores, 16 threads)
• 16 GB RAM
• 600 GB of HDD storage dedicated to Logstash
![Page 76: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/76.jpg)
SOFTWARE (CURRENT)
• ElasticSearch 0.90
• Logstash 1.2
• Kibana 3.0.0m3
• Curator 0.6.2
![Page 77: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/77.jpg)
SOFTWARE (THIS WEEK)
• ElasticSearch 1.3.2
• Logstash 1.4.2
• Kibana 3.1
• Curator 2.0.0
![Page 78: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/78.jpg)
STATS
• Consolidating logs from ≈ 10 web servers
• Incoming syslog (Drupal), Apache, and Varnish logs
• Looking to add EdgeCast (CDN) logs if we have capacity
• Non-syslog logs are updated every hour with rsync
• > 2 billion logs processed per month
![Page 79: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/79.jpg)
FINDING A PROBLEM USING LOGHOST
• https://www.drupal.org/node/2186105
• PHP error on saving issues after the Drupal 7 upgrade
• Did a Lucene query to get xjm's PHP errors
• program:drupal AND d_uid:65776 AND d_type:php
• Found PDOExceptions and an issue with a SQL field size
![Page 80: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/80.jpg)
QUESTIONS?
![Page 81: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/81.jpg)
RESOURCES
![Page 82: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/82.jpg)
LINKS
• http://httpd.apache.org/docs/2.2/mod/mod_log_config.html
• http://logstash.net/
• http://elasticsearch.com/
• https://github.com/elasticsearch/kibana/
• http://graphite.wikidot.com/
• http://grafana.org/
• http://www.phase2technology.com/blog/adding-mysql-slow-query-logs-to-logstash/
![Page 83: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/83.jpg)
LINKS
• https://github.com/elasticsearch/logstash-forwarder
• https://collectd.org/
• https://www.drupal.org/documentation/modules/syslog
• https://www.monitoring-plugins.org/
• https://github.com/etsy/statsd/
• https://github.com/heroku/umpire
• http://git.io/e6TvAg - sample Logstash config for Drupal/nginx
![Page 84: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/84.jpg)
SPRINTS!
![Page 85: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/85.jpg)
![Page 86: Open Source Logging and Metrics Tools](https://reader034.fdocuments.us/reader034/viewer/2022042521/553a449f4a7959f66c8b4a60/html5/thumbnails/86.jpg)
D E V O P S T R A C K , D R U P A L C O N A M S T E R D A M
WHAT D ID YOU TH INK?E V A L U A T E T H I S S E S S I O N - AMSTERDAM2014.DRUPAL.ORG/SCHEDULE