Fluentd meetup dive into fluent plugin

Dive into Fluent Plugin

2012年2月4日土曜日

Site:repeatedly.github.com

Company:Preferred Infrastructure, Inc.

Love plugins:input: tailbuffer: memoryoutput: mongo


developed by

The missing log collector

What's Fluentd?

See keynote


Fluentd is abufferroutercollectorconverteraggregatoretc...


... but,Fluentd doesn’t have such features as a built-in.


Instead,Fluentd has flexible plugin architecturewhich consists of Input, Output and Buffer.


We can customize Fluentd using plugins :)


Agenda

Yes, I talk about - an example of Fluentd plugins - Fluentd and libraries - how to develop a Fluentd plugins

No, I don’t talk about - the details of each plugin - the experience of production


Examplebased on bit.ly/fluentd-with-mongo


Install

Plugin name is ,and fluent-gem is included in Fluentd gem.

fluent-plugin-xxx


Let’s type!

$ fluent-gem install fluent-plugin-mongo


Me!

Many Many

ManyPlugins!


<source> type tail format apache path /path/to/log tag mongo.apache</source>

<match mongo.**> type mongo database apache collection access host otherhost</match>

Input Output

fluentd.conf


Start!

$ fluentd -c fluentd.conf2012-02-04 00:00:14 +0900: starting fluentd-0.10.82012-02-04 00:00:14 +0900: reading config file path="fluentd.conf"2012-02-04 00:00:14 +0900: adding source type="tail"2012-02-04 00:00:14 +0900: adding match pattern="mongo.**" type="mongo"█


Attack!

$ ab -n 100 -c 10 http://localhost/


$ mongo --host otherhost> use apache> db.access.find() { "type": "127.0.0.1", "method": "GET", "path": "/", "code": "200", "size": "44", "time": ISODate("2011-11-27T07:56:27Z") ... }has more...


Mongo

Apache

Fluentd

write

tail

insert

I’m a log!

eventbuffering


Warming up


Ruby

Fluentd Stack

OS

Cool.io

MessagePack

BufferInput

Output


Rubyruby-lang.org


Fluentd and plugins are written in Ruby.


... but note thatFluentd works on Ruby 1.9, goodbye 1.8!


MessagePackmsgpack.org


Serialization:JSON like fast and compact format.

RPC:Async and parallelism for high performance.

IDL:Easy to integrate and maintain the service.


Binary format,Header + Body,andVariable length.


Note thatRuby version can’t handle a Time object.


So,we use an Integer object instead of a Time.


Source:github.com/msgpack

Wiki:wiki.msgpack.org/display/MSGPACK

Mailing List:groups.google.com/group/msgpack


Cool.iocoolio.github.com


Event driven framework built on top of libev.


Cool.io has Loop and Watchers withTransport wrappers.


Fluentd has a default event loop.We can use @default_loop in the plugin.


Configuration


Fluentd loads plugins from $LOAD_PATH.


Input:$LOAD_PATH/fluent/plugin/in_<type>.rb

Buffer:$LOAD_PATH/fluent/plugin/buf_<type>.rb

Output:$LOAD_PATH/fluent/plugin/out_<type>.rb


We use ‘register_input’, ‘register_buffer’ and ‘register_output’ to register a plugin.


We can load the plugin configuration usingconfig_param and configure method.config_param set config value to@<config name> automatically.


class TailInput < Input Plugin.register_input(’tail’, self) config_param :path, :string ...end

<source> type tail path /path/to/log ...</source> fluentd.conf

in_tail.rb2012年2月4日土曜日

One trick is here:

Fluentd’s configuration module does notverify a default value. So,we can use the nil like Tribool :)

config_param :tag, :string, :default => nil

Fluentd does not check the type


Fluentd provides some useful mixins forinput and output plugins.


SetTagKeyMixin:Provide ‘tag_key’ and ‘include_tag_key’.

SetTimeKeyMixin:Provide ‘time_key’ and ‘include_time_key’.

DetachMultiProcessMixin:Provide ‘detach_process’ andexecute an action in the multi-process.


Code Flow

Mixin usage

class MongoOutput < BufferedOutput ... include SetTagKeyMixin config_set_default :include_tag_key, false ...end MongoOutput

SetTagKeyMixin

BufferedOutput

super

super

super


Input


Default 3rd party

Available plugins

execforwardhttpstreamsyslogtailetc...

mongo_tailscribemsgpackdstatzmqamqpetc...


class NewInput < Input ... def configure(conf) # parse a configuration manually end

def start # invoke action end

def shutdown # cleanup resources endend


In action method,we use Engine.emit to input data.

tag = "app.tag"time = Engine.nowrecord = {"key" => "value", ...}Engine.emit(tag, time, record)

Sample:


How to read an input in an efficient way?We use a thread and an event loop.


class ForwardInput < Fluent::Input ... def start ... @thread = Thread.new(&method(:run)) end

def run ... endend

Thread


class ForwardInput < Fluent::Input ... def start @loop = Coolio::Loop.new @lsock = listen @loop.attach(@lsock) ... end ...end

Event loop


Note thatWe must use Engine.now instead of Time.now


Buffer


Default 3rd party

Available plugins

memoryfilezfile (?)


In most cases,Memory and File are enough.


Memory type is default.It’s fast but can’t resume data.


File type is persistent type.It can resume data from file.


Output


Default 3rd party

Available plugins

copyexecfileforwardnullstdoutetc...

mongos3scribecouchhoopsplunketc...


class NewOutput < BufferedOutput # configure, start and shutdown # are same as input plugin

def format(tag, time, record) # convert event to raw string end

def write(chunk) # write chunk to target # chunk has multiple formatted data endend


Output has 3 buffering modes.NoneBufferedTime sliced


Buffered Time sliced

Buffering type

chunk

go out

chunk

chunk

from in

chunklimit

queuelimit

Buffer has an internal map to manage a chunk. A key is tag in Buffered, but a key is time slice in TimeSliced buffer.

def write(chunk) # chunk.key is time sliceend


How to write an output in an efficient way?We can use multi-process (input too).

See: DetachMultiProcessMixin with detach_multi_process


Test


Input:Fluent::Test::InputTestDriver

Buffer:Fluent::Test::BufferedOutputTestDriver

Output:Fluent::Test::OutputTestDriver


class MongoOutputTest < Test::Unit::TestCase def setup Fluent::Test.setup require 'fluent/plugin/out_mongo' end

def create_driver(conf = CONFIG) Fluent::Test::BufferedOutputTestDriver.new (Fluent::MongoOutput) { def start # prevent external access super end ... }.configure(conf) end


...

def test_format # test format using emit and expect_format

end

def test_write d = create_driver t = emit_documents(d)

# return a result of write method collection_name, documents = d.run assert_equal([{...}, {...}, ...], documents) assert_equal('test', collection_name) end ...

end


It’s a weak point in Fluentd... right?


Release


Bundle with git

$ edit lib/fluent/plugin/out_<name>.rb

$ git add / commit

$ cat VERSION0.1.0

$ bunlde exec rake release

See: rubygems.org/gems/fluent-plugin-<name>


See released pluginsfor more details about each file.


Lastly...


Help!2012年2月4日土曜日

Question?


Fluentd meetup dive into fluent plugin

Technology

Transcript of Fluentd meetup dive into fluent plugin