Logging Application Behavior to MongoDB
Robert Stewart @wombatnation
1/1/2012
TOC
• Why Log to MongoDB?
• Logging Library Basics
• Logging by Language
• Other Ways to Distribute Your Logging
Why Log to MongoDB?
• Centralized application logging
• Easy remote access, compared to files
• Log events good fit for document model
• Flexible schema
• Indexes for frequent queries
• Capped collections are very efficient
• Analyze data in-place with JavaScript MapReduce
Event Storage
• Create a database for your app
• Create a capped collection to store log events
• Time-based eviction more natural, but less efficient
• Capped collection reduces chance of filling up disk
Log Analysis
• Compared to files, easier to analyze app behavior across servers and days
• Log values to custom keys where possible
• JavaScript MapReduce
Miscellaneous Free Advice
• Take advantage of separating storage from presentation
• Be aware of which classes your Mongo driver knows how to BSONify
• If logging from multiple languages, sticking to Log4Lang family can help
Logging Library Basics
• Some provide API with pluggable implementation
• Configuration
• Programmatic
• Declarative
• Dynamic
Logging Concepts
• Named, Hierarchical Logger Objects
• Identify a class or an area of an app
• Log Level, e.g., error, warn, info, debug
• Thresholds determine if event is logged
• Logging Event
• The message
• Metadata, e.g., time, level, class, method, line
Logging Concepts
• Handler - determines event destination
• File, console, socket, email, syslog, MongoDB, etc.
• Handler level threshold
• Async vs. sync
• Filter - for logger and/or handler
• Formatter - event and message
• Pattern with placeholder tokens
App calls logger.info(“freak out”)
Common Flow for Logging
Is logger info
enabled?
Passes filter, if any?
For each handler:
No
No
Yes
Yes
Is handler info
enabled?
Passes filter, if any?
No
Yes
Yes
Format and store event
No
Polyglotism
• Good news - lots of client language bindings and logging libraries for MongoDB
• Mostly good news - Log4J very influential
• Sort of bad news - default log event formats can be very different
Logging by Language
• Java
• Python
• Ruby
• PHP
• C#
Java
• APIs
• Apache Commons Logging
• SLF4J
• java.util.logging, a.k.a., JUL
• Limited feature set
• Destinations - console, file, socket, memory
• JVM-level config
Log4J • Apache Log4J is dominant logging library
• Logback is potential successor
• Logger - usually, represents class logging the event
• LoggingEvent - message and metadata
• Appender - sends LoggingEvent to destination
• Layout - formats the event
• Converter - convert placeholder to value
Log4mongo-java
• Set of Log4J Appenders for MongoDB
• log4mongo.org
• Open source project on GitHub
• I’m a committer with Peter Monks and Jozef Ševčík
• I used it at Voxify with speech reco applications and infrastructure services
• Peter has used it at Alfresco
MongoDbAppender
• Stores a Log4J LoggingEvent object as a document in a collection
• Format is very similar to structure of LoggingEvent Java class
• Where possible, data is stored as appropriate BSON objects, rather than just strings
MongoDbPatternLayoutAppender
• Stores log event based on user-specified pattern
• Standard Log4J pattern layout, parser and converter functionality
• Very flexible formatting at cost of additional config
• Values stored as strings or arrays
• Custom values determined at time event is logged
BsonAppender
• Custom log format by extending BsonAppender
• Can use it to log BSON objects to other data stores
• Includes code for BSONifying Java exceptions
Enabling log4mongo-java • Associate a name with an appender implementation log4j.appender.MongoDB = org.log4mongo.MongoDbAppender!
• Specify host (optional port, username and password). For replica set, specify all active hosts.
log4j.appender.MongoDB.hostname = localhost
• Specify database and collection log4j.appender.MongoDB.databaseName = app_name
log4j.appender.MongoDB.collectionName = log
• Add named appender to rootLogger list log4j.rootLogger = INFO, file, MongoDB
Using Pattern Layout Appender • Specify pattern layout appender
log4j.appender.MongoDB = org.log4mongo.MongoDbPatternLayoutAppender
• Specify your custom Pa2ernLayout class log4j.appender.MongoDB.layout = com.voxify.log.log4j.MongoDBPatternLayout
• Specify a conversion pattern
• Must be a valid JSON document
• Can have sub-documents
• Values can be strings or arrays
log4j.appender.MongoDB.layout.ConversionPattern = {"ts":"%d{yyyy-MM-dd HH:mm:ss,SSS}", "level":"%p","class":"%c{1}","message":"%m"}
Shell and Tail
In the Shell
> use app_name switched to db app_name > db.log.findOne() {
"_id" : ObjectId("4c200c4d28cf037460e82363"), "ts" : "2010-10-21 18:05:17,237", "level" : "INFO", "class" : "NodeFlagGenerator", "message" : "Generating me some node flags”
}
Recent Log Events If using mongo 1.9.1+, put useful functions like this in .mongorc.js
> function last(count) {
if (count==null) count = 1;
return db.log.find({}, {_id:0}).sort({ts:-1}).limit(count);
}
> last(2)
{ "ts" : "2010-10-28 23:49:14,567", "level" : "INFO", "session" : "122", "class" : "Site", "message" : "uninformative message" }
{ "ts" : "2010-10-28 23:49:14,566", "level" : "INFO", "session" : "122", "class" : "Site", "message" : "some info kind of message" }
Tailing Logs
• You’ll really miss ability to tail logfiles
• Or, ... will you?
• MongoDB offers tailable cursors
• Specify cursor, e.g., from find, as tailable
• While cursor alive, print new docs then sleep
Python Tailing Example from pymongo import Connection import time db = Connection().my_db coll = db.my_collection cursor = coll.find(tailable=True) while cursor.alive: try: doc = cursor.next() print doc except StopIteration: time.sleep(1)
Python
• Feature-rich logging module in Python 2.3
• Influenced by Log4J
Components • Logger - represents area of library or project
• LogRecord
• Handler
• Filter
• Formatter
• LoggerAdapter (2.6) - add contextual info
mongodb-log
• Python logging handler to log to MongoDB
• Stores dictionary directly as BSON doc
• https://github.com/andreisavu/mongodb-log
log4mongo-python
• Similar design to log4mongo PHP and .NET appenders
• Example programmatic handler config and usage: import logging from log4mongo.handlers import MongoHandler
logger = logging.getLogger('test') logger.addHandler(MongoHandler(host='localhost')) logger.warning('test')
• http://log4mongo.org/display/PUB/Log4mongo+for+Python
• https://github.com/log4mongo/log4mongo-python
Ruby
• Built-in Logger library
• Limited feature set
• Console and file outputs
• Message format is fixed
• Datetime format is configurable
Log4r • Influenced by Log4J
• Logger
• Outputter
• Formatter
• Configurator
• http://log4r.rubyforge.org
• Code for using Log4r with MongoMapper
• http://gist.github.com/207347
Logging
• Similar to Log4r with a few enhancements
• Supports JSON as output format
• http://logging.rubyforge.org/
mongo_db_logger • Rails plugin which uses Rails Logger, which can use
Ruby Logger, Log4r, etc.
• Logs a single document per request
• IP address, time, etc.
• All messages logged during request
• Custom request data
• Configure via database.yml
• http://github.com/peburrows/mongo_db_logger
Central Logger
• Forked from mongo_db_logger
• Added Rails 3 support and gemified
• Web UI for searching, filtering and analyzing logs
• Configure via database.yml (or mongoid.yml or central_logger.yml)
• http://www.slideshare.net/leopard_me/logging-rails-application-behavior-to-mongodb
• http://files.meetup.com/1742411/Logging%20With%20MongoDB.pdf
• https://github.com/customink/central_logger
MongodbLogger
• Based on central_logger
• Different web UI for searching, filtering and analyzing logs
• Configure via database.yml (or mongoid.yml or mongodb_logger.yml)
• http://mongodb-logger.catware.org/
PHP
• error_log() logs to file, email address or system logger (e.g., syslog or NT EventLog)
• Log4PHP - heavily influenced by Log4J
• PEAR Log - PHP-specific logging library
• Zend_Log - Logging for Zend Framework
Log4mongo-php
• Log4PHP appender
• Programmatic & declarative config
• Timestamp stored as Mongo Date
• Exception stored as array - msg, code, stacktrace
• Inner exception stored if available
• https://github.com/log4mongo/log4mongo-php
PEAR Log
• PEAR Log class - lots of handlers (Firebug!), but none yet for MongoDB
• Programmatic config
• Specify handler when creating logger object
• Composite handler for multiple destinations
• http://pear.php.net/package/Log/
Log Writer for Zend���
• Project -> Recordshelf_Log_Writer_MongoDb
• Described as a “prototypesque implementation” • Standard Zend tools to config and init
• Zend log event is associative array
• Project has code for accessing & filtering events • http://raphaelstolt.blogspot.com/2009/09/logging-to-mongodb-and-accessing-log.html
C#/.NET
• Log4net
• Heavily influenced by Log4J
• http://logging.apache.org/log4net/
Log4mongo-net
• Uses community developed C# driver, but may move to official driver
• Tested with .NET 3.5+ and Mono 2.8
• Standard App.config/Web.config or programmatic
• http://log4mongo.org/display/PUB/Log4mongo+for+.NET
Other Options • Destinations include MongoDB
• Flume, logstash, Fluentd
• Primary Destination is HDFS
• Chukwa and Scribe
• Commercial Hosted
• Loggly
• Commercial On-Premise
• Splunk
Flume • If you can’t change the source of logs ...
• Can aggregate files, syslog, etc.
• Agent --> Collector --> Storage
• Storage plugins for HDFS, MongoDB, etc.
• Decorators transform data & extract metadata
• https://github.com/cloudera/flume
• https://github.com/mongodb/mongo-hadoop
• Chukwa, Honu and Scribe more HDFS-centric
logstash
• Log event storage, search and graphing
• Read from file, syslog, AMQP, tcp, etc.
• Store to ElasticSearch, AMQP queue, websockets, MongoDB, etc.
• http://logstash.net/
• http://code.google.com/p/logstash
Fluentd • Ruby-based open source log collector
• Input, output and buffer plug-ins
• For example, plug-ins for tailing Apache log files and storing log events to MongoDB
• http://blog.treasure-data.com/post/13766262632/real-time-log-collection-with-fluentd-and-mongodb
• http://fluentd.org/doc/
• https://github.com/fluent/fluentd
Questions?
• Photo/Image Credits • Log analysis - ww.flickr.com/photos/linuxtuxguy/2579295702
• Don’t Stop Believing - www.flickr.com/photos/loozrboy/3908830690/
• Homeland Security Advisory System - www.theonion.com/articles/iraq-adopts-terror-alert-system,1258/
• Advice - www.flickr.com/photos/wurzle/659315/
• Shell - http://www.flickr.com/photos/geishaboy500/305555652/
• Tail - http://www.flickr.com/photos/autumnsonata/2632194442/
• Log flume - www.flickr.com/photos/halloweenjack/1330806722/
• Cufflinks - www.flickr.com/photos/oberazzi/318947873/
Top Related