Fluentd - road to v1 -

29
Masahiro Nakagawa Jan 27, 2015 Data Transfer Middleware Meetup #1 Fluentd - road to v1 - #dtm_meetup

Transcript of Fluentd - road to v1 -

Masahiro NakagawaJan 27, 2015

Data Transfer Middleware Meetup #1

Fluentd - road to v1 -

#dtm_meetup

Who are you?

> Masahiro Nakagawa > github/twitter: @repeatedly

> Treasure Data, Inc. > Senior Software Engineer > Fluentd / td-agent developer

> I love OSS :) > D language - Phobos committer > Fluentd - Main maintainer > MessagePack / RPC - D and Python (only RPC) > The organizer of Presto Source Code Reading / meetup > etc…

Structured logging !

Reliable forwarding !

Pluggable architecture

http://fluentd.org/

What’s Fluentd?

> Data collector for unified logging layer > Streaming data transfer based on JSON > Written in Ruby

> Gem based various plugins > http://www.fluentd.org/plugins

> Working in production > http://www.fluentd.org/testimonials

Divide & Conquer & Retry

error retry

error retry retry

retryBatch

Stream

Other stream

Core Plugins

> Divide & Conquer

> Buffering & Retrying

> Error handling

> Message routing

> Parallelism

> read / receive data > from API, database,

command, etc… > write / send data

> to API, database, alert, graph, etc…

Architecture (v0.12 or later)

EngineInput

Filter Output

Buffer

> grep > record_transfomer > …

> Forward > File tail > ...

> Forward > File > ...

Output

> File > Memory

not pluggable

FormatterParser

Before

After

or Embulk

Nagios

MongoDB

Hadoop

Alerting

Amazon S3

Analysis

Archiving

MySQL

Apache

Frontend

Access logs

syslogd

App logs

System logs

Backend

Databasesbuffering / processing / routing

M x N → M + N

v0.10 (old stable)

> Mainly for log forwarding > with good performance > working in production

> http://www.fluentd.org/testimonials > Various plugins

> http://www.fluentd.org/plugins

v0.11 (dead version)

http://repeatedly.github.io/ja/2014/03/about-fluentd-v11/

v0.12 (current stable)

> Event handling improvement > Filter > Label

> At-least-once semantics in forwarding > require_ack_response parameter > http://ogibayashi.github.io/blog/2014/12/16/try-

fluentd-v0-dot-12-at-least-once/

> Apply filtering routine to event stream > No more tag tricks!

Filter

<match access.**> type record_reformer tag reformed.${tag} </match> !<match reformed.**> type growthforecast </match>

<filter access.**> type record_transformer … </filter>

v0.10: v0.12:

<match access.**> type growthforecast </match>

> Internal event routing > Redirect events to another group

> much easier to group and share plugins

Label

<source> type forward </source> !<match app1.**> type record_reformer </match> !…

<source> type forward @label @APP1 </source><label @APP1> <match access.**> type s3 </match> </label>

v0.10: v0.12:

Error stream with Label

> Can handle an error at each record level > It is still prototype ERROR!

{"event":1, ...}

{"event":2, ...}

{"event":3, ...}

chunk1

{"event":4, ...}

{"event":5, ...}

{"event":6, ...}

chunk2

Input

OK

ERROR!

OK

OK

OK

Output

<label @ERROR> <match **> type file ... </match> </label>

Error stream

Built-in @ERROR is used when error occurred in “emit”

v0.14 (next stable)

> New plugin APIs > Actor > New base classes (#309)

> ServerEngine based engine > Robust supervisor

> Sub-second time support (#461) > Zero downtime restart

Actor> Easy to write popular routines > Hide implementation details

class TimerWatcher < Coolio::TimerWatcher ... end !def start @loop = Coolio::Loop.new @timer = ... @loop.attach(@timer) @thread = ... end

def configure(conf) actor.every(@interval) { router.emit(...) } end !def start actor.start end

v10: v0.14:

> Socket manager shared resources with workers

19

SupervisorTCP

1. Listen to TCP socket

Zero downtime restart

20

Worker

Supervisor

heartbeat

TCP

TCP

1. Listen to TCP socket

2. Pass its socket to worker

Zero downtime restart

> Socket manager shared resources with workers

21

Worker

Supervisor

Worker

TCP

TCP

1. Listen to TCP socket

2. Pass its socket to worker

3. Do same actionat worker restartingwith keeping TCP socket

heartbeat

Zero downtime restart

> Socket manager shared resources with workers

TODO: How to implement on JRuby?

v1 (future stable)

> Fix new features / APIs > Plugin APIs > Default configurations

> Clear versioning and stability > No breaking API compatibility!

> Breaking compatibility by Fluentd v2 ?

Roadmap summary> v0.10 (old stable) > v0.12 (current stable)

> Filter / Label / At-least-once > v0.14 (spring, 2015)

> New plugin APIs, ServerEngine, Time… > v1 (early summer, 2015)

> Fix new features / APIs

https://github.com/fluent/fluentd/wiki/V1-Roadmap

Other TODO

> Windows support > Need feedback! > https://github.com/fluent/fluentd/tree/windows

> Also check: http://qiita.com/okahashi117

> JRuby support > msgpack / cool.io now work on JRuby > https://github.com/fluent/fluentd/issues/317

Ecosystem

Treasure Agent (td-agent)

> Treasure Data distribution of Fluentd > Treasure Agent 2 is current stable

> Update core components > We recommend to use v2, not v1

> Next version, 2.1.4, uses fluentd v0.12 > In this week or next week

fluentd-forwarder> Forwarding agent written in Go

> Focusing log forwarding to Fluentd > Work on Windows

> Bundle TCP input/output and TD output > No flexible plugin mechanizm > We have a plan to add some input/output

> Need feedback!

fluentd-ui

> Manage Fluentd instance via Web UI > https://github.com/fluent/fluentd-ui

Check: treasuredata.comCloud service for the entire data pipeline