Masahiro NakagawaNov 6, 2014

Cloudera World Tokyo

FluentdUnified Logging Layer

Who are you?

> Masahiro Nakagawa > github/twitter: @repeatedly

> Treasure Data, Inc. > Senior Software Engineer > Fluentd / td-agent developer

> I love OSS :) > D language - Phobos committer > Fluentd - Main maintainer > MessagePack / RPC- D and Python (only RPC) > The organizer of Presto Source Code Reading > etc…

Structured logging !

Reliable forwarding !

Pluggable architecture

http://fluentd.org/

Collect Store Process Visualize

Data source

Reporting

Monitoring

Data Processing

Related Products

Store Process

Cloudera

Horton Works

Treasure Data

Collect Visualize

Tableau

Excel

R

easier & shorter time

???

What’s Fluentd?

• Open sourced log collector written in Ruby

• Customization is essential small core + many plugins

6

It’s like syslogd, but uses JSON for log messages

Divide & Conquer & Retry

7

error retry

error retry retry

retry

Core Plugins!

• Divide & Conquer

• Buffring & Retrying

• Error handling

• Message routing

• Parallelism

!

!

• read / receive data

• write / send data

8

Pluggable Architecture

9

Buffer Output

Input

> Forward > HTTP > File tail > dstat > ...

> Forward > File > MongoDB > ...

> File > Memory

Engine

Output

> rewrite > ...

Pluggable Pluggable

Before

10

After

11

M x N → M + N

12

Nagios

MongoDB

Hadoop

Alerting

Amazon S3

Analysis

Archiving

MySQL

Apache

Frontend

Access logs

syslogd

App logs

System logs

Backend

Databasesbuffer / buffer / routing

Ecosystem

Treasure Agent

• Treasure Data distribution of Fluentd

• including Ruby, core libraries andthird party plugins

• Current stable is 2.1.1

• rpm / deb / dmg

• 2.1.2 will be released soon with fluentd-ui

14

fluentd-forwarder

• Forwarding agent written in Go

• mainly for Windows support

• immature than Fluentd

• Bundle TCP input/output and TD output

• No plugin mechanizm

• Similar product: ik, fluent-agent-hydra, etc

15

fluentd-ui

• Manage Fluentd instance using Web GUI

16

Use-case

Users

• Companies

• Web service, Ad Tech, Game, Media, EC, IoT, PoS, etc..

• Products

• Kubernetes, Cloudn, Harvester, Bloomberg Clustered Private Cloud, etc..

18

Simple forwarding

19

HDFS HBaseMongo

S3

Separate log stream

20

CEP for stream processing

21

Roadmap

• v0.10 (current)

• v0.12 (Nov, 2014)

• Filter / Label / at-least-once

• v0.14 (TBD)

• New Plugin APIs, ServerEngine

• v1 (TBD. In the first half of 2015)

22

Check: treasuredata.com

Cloud service for the entire data pipeline