1. Rails exception tracking with Graylog2 Manage your logs in the dark and have lasers going and make it look like you're from space Lennart Koopmann RubyAndRails EU Amsterdam, 2010

2. About me Lennart Koopmann 22 years old Living in Hamburg, working at Jimdo lennartkoopmann.net / @_lennart 3. Log server with web interface. 4. phpLogCon let's not talk about that Graylog1: August 2009 Graylog2: August 2010 5. 6. Syslog daemon (TCP/UDP) 7. GELF (UDP) 8. Graylog extended log format UDP - Max (GZIP) 8192 byte per message (Chunking is supported) Libraries for Ruby and PHP available 9. { } 10. { ' message ':'Exception: Something went wrong.' } 11. { ' message ':'Exception: Something went wrong.', ' full_message ':'Stacktrace. Some env vars' } 12. { ' message ':'Exception: Something went wrong.', ' full_message ':'Stacktrace. Some env vars', ' host ':'www19' } 13. { ' message ':'Exception: Something went wrong.', ' full_message ':'Stacktrace. Some env vars', ' host ':'www19', ' file ':'/var/www/index.php' } 14. { ' message ':'Exception: Something went wrong.', ' full_message ':'Stacktrace. Some env vars', ' host ':'www19', ' file ':'/var/www/index.php', ' line ':2638 } 15. { ' message ':'Exception: Something went wrong.', ' full_message ':'Stacktrace. Some env vars', ' host ':'www19', ' file ':'/var/www/index.php', ' line ':2638, ' level ':1 } 16. { ' message ':'Exception: Something went wrong.', ' full_message ':'Stacktrace. Some env vars', ' host ':'www19', ' file ':'/var/www/index.php', ' line ':2638, ' level ':1, ' something ':'foo' } 17. { ' message ':'Exception: Something went wrong.', ' full_message ':'Stacktrace. Some env vars', ' host ':'www19', ' file ':'/var/www/index.php', ' line ':2638, ' level ':1, ' something ':'foo', ' something_else ':'bar' } 18. Chunking 19. Use Cases 20. Plain syslog Collect everything of your /var/log from all servers, aggregate, analyze and get a warning if something goes wrong 21. GELF Embed it into the logging class of your application and trigger with every error. Include stacktrace and env vars in full message. Later filter by file:line to get statistics. 22. Aggregate, analyze and get a warning if something goes wrong 23. Notifications Currently only Nagios hook (rake task) Triggered at too many messages in last X minutes 24. Average page generation time Graylog1 / MySQL: ~12000ms 25. Average page generation time Graylog1 / MySQL: ~12000ms Graylog2 / MongoDB: ~250ms 26. Exception tracking with Rails Rack middleware available as gem 27. lennart ~$sudo gem install graylog2_exceptions Successfully installed gelf-0.9.1 Successfully installed graylog2_exceptions-0.5.3 28. # config/environment.rb config.middleware.use " Graylog2Exceptions ", { :host=> ' 127.0.0.1 ', :port=> ' 12201 ', :local_app_name=> ' mama ' } 29. Roadmap 30. Incident management Define chained rules that explain an incident. Trigger several actions when incident occurs. 31. Fine grained notifications Be informed via Email, XMPP, SMS by Graylog2 directly without Nagios hook 32. More analyzing 33. More search features 34. Live demo. www.graylog2.org / @Graylog2 www.lennartkoopmann.net / @_lennart (Slides are on Slideshare)