Download - Fluentd v11 at tokuben

Jan 23, 2014

www.treasuredata.com/

Fluentd v11

Masahiro NakagawaTreasure Data, Inc

1

Sunday, January 26, 14

http://www.treasuredata.com/


Who are you?

• Masahiro Nakagawa

• @repeatedly

• Treasure Data, Inc.

• Senior Software Engineer

• Fluentd, td-agent, etc...

• Dlang, MessagePack, ...

2


Structured logging

Reliable forwarding

Pluggable architecture

http://fluentd.org/


http://fluentd.org/

http://fluentd.org/

M x N → M + N

4

Nagios

MongoDB

Hadoop

Alerting

Amazon S3

Analysis

Archiving

MySQL

Apache

Frontend

Access logs

syslogd

App logs

System logs

Backend

Databasesbuffer / buffer / routing


v10

• Mainly for log forwarding

• with good performance

• working in many productions reliably

• Various plugins are released

• There are 200+ plugins!

• Mainly for CRuby

5


Why v11?

• We want more useful features, but...

• Changing the internal architecture is hard

• Keeping the external API is even harder

• Originally, v10 was started as a prototype

• Need drastic changes for the future :)

6


What’s new?

• ServerEngine based reliable architecture

• Zero-downtime restarting

• Filter / Label for flexible record handling

• Improved configuration

• Improved plugin development

7


ServerEngine based

• Robust signal handling

• Put a signal into Queue first

• Built-in supervisor

• Multiprocess support

• No need for in_multiprocess plugin

8


Multi Process

9

Worker

Supervisor

Worker Worker

Separate stream pipelines in one instance!

<worker> input tail output mongo</worker>

<worker> input forward output webhdfs</worker>

<worker> input foo output bar</worker>


• SocketManager shares resources with workers

Zero downtime restart

10

SupervisorTCP

1. Listen to TCP socket




11

Worker

Supervisor

heartbeat

TCP

TCP


2. Pass its socket to worker




12

Worker

Supervisor

Worker

TCP

TCP


2. Pass its socket to worker

3. Do same actionat worker restartingwith keeping TCP socket

heartbeat


Filter / Label support

• No more tag-related tricks!

• add_tag_xxx, remove_tag_xxx, etc...

• Redirect events to another group

• Much easier to group and share plugins

13


Filter

• <match> can have nested <match>

• Configuration format is not fixed!

14

<match access.**> type flowcounter add_tag_prefix counted</match>

<match counted.**> typo growthforecast</match>

<match access.** copy> type flowcounter

<match **> typo growthforecast </match></match>

v10: v11:


Label

• <label> can contain multiple <match>

• out_redirect can forward events to <label>

15

<match access.**> type rewrite_tag_filter ... <match bang.**> type redirect to_label blackhole </match> ...</match>

<label blackhole> <match **> type null </match></label>

bang’s record go away!


Improved configuration

• Can use Hash, Array and others

• No need for “,” or similar tricks

• You can write Ruby directly

• Worker pragma

• Separate plugins in each process

• Support the v10 compatible mode

16


• Can write complex values without DSL!

• Can use Ruby code for configuration

New parameter types

17

<source> type my_tail keys ["k1", "k2", "k3"]</source>

<match **> typo my_filter add_keys {"k1" : "v1"}</match>

<match ** copy> type my_filter env "#{ENV['KEY']}"</match>

Hash, Array, etc: Embedded Ruby code:

• Socket.gethostname

• `command`

• etc...


Improved plugin

• Actor

• Abstraction layer for several actions

• Error stream

• with @ERROR label

• Log level per plugin

• No more global API, Engine.emit, $log, etc...

18


Actor

• Easy to write popular routines

19

class TimerWatcher < Coolio::TimerWatcher ...end

def start @loop = Coolio::Loop.new @timer = ... @loop.attach(@timer) @thread = ...end

actor.every(@interval) { event_router.emit(...)}

v10: v11:


ERROR!

Error stream

• Can handle an error at each record level

20

{"event":1, ...}

{"event":2, ...}

{"event":3, ...}

chunk1

{"event":4, ...}

{"event":5, ...}

{"event":6, ...}

chunk2

…

Input

OK

ERROR!

OK

OK

OK

Output

<label @ERROR> <match **> type file ... </match></label>

Error stream

Built-in @ERROR is usedwhen error occurred in “emit”


Lastly...

• Super alpha status!

• Adding / Removing features may occur

• There are several “to be implemented” features

• The release date is not fixed yet

• The initial release should be useful from day one

• Need feedbacks!

21


Jan 23, 2014

www.treasuredata.com/22

We’re Hiring!


