Sadayuki Furuhashi!uentd.org

What’s new in v11

Non-stop restart

supervisor process

worker process

v10 (old):

listen TCP socket heartbeat

Non-stop restart

supervisor process

worker process

v10 (old):

Downtime!

restart

initializing...

heartbeat

Non-stop restart

supervisor process

worker process

v11:

listen TCP socket heartbeat

Non-stop restart

supervisor process

worker process

v11:

listen TCP socket

restart

Zero downtime!

initializing...

heartbeat

Multiprocess

supervisor process

worker processheartbeat

<worker> ...</worker>

<worker> ...</worker>

<worker> ...</worker>

Multiprocess

worker process

･ in_tail･ out_groupcounter･ out_webhdfs

<worker> ... </worker>

<worker>

</worker>

･ in_forward･ out_VERY_UNSTABLE

<worker>

</worker>

Separate importantlogging fromexperimental plugins

Error stream

{“event”: 1, ...}

{“event”: 2, ...}

{“event”: 3, ...}

{“event”: 4, ...}

{“event”: 5, ...}

{“event”: 6, ...}

v10 (old):

chunk

chunk

<match **>

OK

ERROR!

?

ERROR!

?

?

event

<source>

Error handlingdepends on plugin(throw away! etc.)

Error stream

{“event”: 1, ...}

{“event”: 2, ...}

{“event”: 3, ...}

{“event”: 4, ...}

{“event”: 5, ...}

{“event”: 6, ...}

v10 (old):

chunk

chunk

<match **>

OK

ERROR!

OK

ERROR!

OK

OK

event

<source>

Fluentd corehandles errorevents

<label @ERROR> <match **> ...

特殊ラベル

OK

OK

Plugin version managementv11:

$ cat /etc/td-agent/Gemfile

gem “fluent-plugin-s3”, “~> 0.11.3”

gem “fluent-plugin-webhdfs”, “~> 0.2.2”

gem “fluentd”, “~> 0.11.2”

$

Log level per pluginv11:

<source>

type forward

log_level error

</source>

<match **>

type mongo

log_level debug

</match>

Variables in a config filev11:

<source>

type forward

port “#{ENV[‘FLUENTD_LISTEN_PORT’]}”

</source>

<server> @set port 9800 @include “/etc/fluentd/server.conf”</server>

<server> @set port 9801 @include “/etc/fluentd/server.conf”</server>

Variables in a config filev11?：

<match raw.**> type flowcounter add_tag_prefix counted</match>

<match counted.**> type growthforecast</match>

Streaming processing without tag hackingv10 (old):

<match **> type flowcounter

<match **> type growthforecast </match></match>

v11:

v11

Reliability &Performance

✓ Non-stop restart✓ Multiprocess✓ Error stream

✓ Variables in a config file✓ Streaming processing without tag hacking

✓ Plugin version management✓ Log level per plugins

Management

Configuration