Fluentd v1.0 in a nutshell

23
Fluentd v1.0 in a nutshell March 30, 2017 Masahiro Nakagawa

Transcript of Fluentd v1.0 in a nutshell

Page 1: Fluentd v1.0 in a nutshell

Fluentd v1.0 in a nutshell

March 30, 2017 Masahiro Nakagawa

Page 2: Fluentd v1.0 in a nutshell

Fluentd v0.12• Current stable and widely used on production

• Input, Parser, Filter, Formatter, Buffer, Output plugins

• Known issues • Event time is second unit • No multi core support • No Windows support • Need to improve plugin API to support more various

use cases

Page 3: Fluentd v1.0 in a nutshell

• Development version of v1

• Latest version is v0.14.14: March 24, 2017

• Implemented New features • New Plugin APIs • Time with Nanosecond resolution • ServerEngine based Supervisor • Windows support • Plugin Helpers & Plugin Storage

Fluentd v0.14

Page 4: Fluentd v1.0 in a nutshell

Fluentd v1• Stable announcement for APIs / features

• No breaking API changes in v1.x

• Compatible with v0.12 and v0.14 • exclude v0 config syntax and detach_process

• Release plan • Q2, 2017 • Need v0.14 feedback from developers and users

https://hub.docker.com/r/fluent/fluentd/

Page 5: Fluentd v1.0 in a nutshell

New Plugin APIs• Input/Output plugin APIs w/ well-controlled lifecycle

• stop, shutdown, close, terminate • Integrate all output plugin into Fluent::Plugin::Output

• New Buffer API for delayed commit and flexible chunking with metadata • parallel/async "commit" operation for chunks

• For high latency case: forward’s at-least-once, issuing job, etc… • Users can choose chunk keys by configuration for dynamic parameters

• Compatible w/ v0.12 plugins • compatibility layer for traditional APIs • it will be supported between v1.x versions

Page 6: Fluentd v1.0 in a nutshell

Router

buffer_chunk_limit

enqueue: exceed flush_intervalor buffer_chunk_limit

Key pattern:

- BufferedOutputempty string or specified key-ObjectBufferedOutput tag-TimeSlicedOutput time slice

emit emit

Buffer

Queue

buffer_queue_limit

Output

OutputInput / Filter

Tag Time

Record Chunk

Chunk

Chunk Chunk

Chunk

key:foo

key:bar

key:baz

v0.12 buffer design

Page 7: Fluentd v1.0 in a nutshell

v0.14 buffer design

Page 8: Fluentd v1.0 in a nutshell

Buffer keys and placeholders• Dynamic parameters for table name, object path and more

• We can embed time, tag and any field with placeholder

<match s3.**> @type s3 aws_key_id "#{ENV['AWS_ACCESS_KEY']}" aws_sec_key "#{ENV['AWS_SECRETA_KEY']}" s3_bucket fluent-plugin-s3 path test/%Y/%m/${tag}/${key}/

<buffer time,tag,key> timekey 3600 </buffer> </match>

http://docs.fluentd.org/v0.14/articles/buffer-section for more details

time: 2017-03-30 12:00:00 +0200 tag: “test” record: {“key”:”hello”}

- Event sample

test/2017/3/test/hello/

- Generated “path”

Page 9: Fluentd v1.0 in a nutshell

Time with nanosecond• For sub-second systems: Elasticsearch, InfluxData, etc…

• Fluent::EventTime • behaves as Integer for v0.12’s second unit compatibility • has methods to get sub-second resolution • be serialized into msgpack using Ext type • Fluent::Engine.now now returns EventTime, not Integer

• Fluentd core can handle both of Integer and EventTime as time • compatible with older versions and software in eco-system

(e.g., fluent-logger, Docker logging driver)

Page 10: Fluentd v1.0 in a nutshell

ServerEngine based Supervisor• ServerEngine is a framework for building robust server

• https://github.com/treasure-data/serverengine

• Replacing supervisor process with ServerEngine • it has SocketManager to share listening sockets

between 2 or more worker processes

• Replacing Fluentd's processing model from fork to spawn

• to support Windows environment

Page 11: Fluentd v1.0 in a nutshell

Windows support

• Fluentd and core plugins work on Windows • Windows service registration is also supported • http://docs.fluentd.org/v0.14/articles/install-by-msi

• Use HTTP RPC instead of signals

• https://github.com/fluent/fluent-plugin-windows-eventlog • We can collect windows eventlog :)

Page 12: Fluentd v1.0 in a nutshell

Symmetric multi core processing

• 2 or more workers share a configuration file • and share listening sockets via PluginHelper • under a supervisor process (ServerEngine)

• Multi core scalability for huge traffic • one input plugin for a tcp port, some filters and one

(or some) output plugin • buffer paths are managed automatically by

Fluentd. Need root_dir and @id parameters

Page 13: Fluentd v1.0 in a nutshell

Worker0

Supervisor

v0.14’s multi process feature

grep

forward

tdlog

Worker1 Worker2

grep

forward

tdlog

grep

forward

tdlog

socket

Page 14: Fluentd v1.0 in a nutshell

Configuration example

<system> workers 2 root_dir /var/log/fluentd</system>

<source> @type forward</source>

<filter pattern> @type grep</filter>

<match pattern> @type tdlog @id out_td</match>

/var/log/fluentd/worker0/out_td/buffer/buffer.xxx.log

/var/log/fluentd/worker0/out_td/buffer/buffer.xxx.log.meta

- buf_file’s path is automatically generated

worker id root_dir

plugin’s @id

Page 15: Fluentd v1.0 in a nutshell

TLS/Authn/Authz support for forward plugin

• secure-forward is merged into built-in forward • TLS w/ at-least-one semantics • Simple authentication/authorization w/o SSL

• Different points • secure-forward uses keep-alive, but forward doesn’t • secure-forward uses thread per connection, but

forward uses libev based IO.

http://www.fluentd.org/blog/fluentd-v0.14.12-has-been-released

Page 16: Fluentd v1.0 in a nutshell

Plugin Storage & Helpers• Plugin Storage: new plugin type for plugins

• provides key-value storage to persistent intermediate status • built-in plugins: in-memory, local file • pluggable: 3rd party plugin to store data into storage

• storage-redis

• Plugin Helpers: • collections of utility methods for plugins • making threads, sockets, network servers, ... • fully integrated with test drivers to run test code after setup

phase of helpers (e.g. test started after created threads)

Page 17: Fluentd v1.0 in a nutshell

server helper: beforedef start @loop = Coolio::Loop.new @handler = Coolio::TCPServer.new(@bind, @port, SocketUtil::TcpHandler, log, @delimiter, method(:on_message)) @loop.attach(@handler) @thread = Thread.new(&method(:run)) end

def shutdown @loop.watchers.each { |w| w.detach } @loop.stop @handler.close @thread.join end

def run @loop.run(@blocking_timeout) rescue => e log.error "unexpected error", error: e log.error_backtrace end

def on_message(msg, addr) # bodyend

Page 18: Fluentd v1.0 in a nutshell

server helper: after

def start server_create(:foo_server, @port, bind: @bind) do |data, conn| # body end end

Page 19: Fluentd v1.0 in a nutshell

v0.12 plugins

ParserInput Buffer Output FormatterFilter

“output-ish”“input-ish”

Page 20: Fluentd v1.0 in a nutshell

v0.14 plugins

ParserInput Buffer Output FormatterFilter

“output-ish”“input-ish”

Storage

Helper

Page 21: Fluentd v1.0 in a nutshell

TODO list for v1• <worker N> directive to separate plugins

• https://github.com/fluent/fluentd/pull/1507

• Counter API to store metrics between processes • https://github.com/fluent/fluentd/tree/counter-api

• Migrate more plugin into v0.14 API

• Fix regressions and bugs

• Write docuemnt: design, plugin, operation, etc…

Page 22: Fluentd v1.0 in a nutshell

<worker N> directive• To execute plugins under one process

• Good for non-multiprocess supported plugins like in_tail

<system> workers 4</system>

<source> @type forward </source>

<match pattern> @type mongo</match>

<worker 0> <source> @type in_tail </source>

<match pattern> @type s3 </match></worker>

in_tail/out_s3 works under worker 0

in_forward/out_mongo worksunder multiprocess environment withworker 1, worker 2, and worker 3

Page 23: Fluentd v1.0 in a nutshell

Treasure Agent 3.0 (td-agent 3)

• fluentd v0.14, Ruby 2.4, and latest core components

• Environments • Add msi Windows package • Remove CentOS 5, Ubuntu 10.04 support

• Beta packages have beed released • http://docs.fluentd.org/v0.14/categories/installation • beta will be removed after v0.14 becomes stable :)