FLUENTD: UNIFIED LOGGING LAYER

Kiyoto TamuraJuly 23, 2015

Silicon Valley Data Engineering Meetup

WHOAMI

Kiyoto TamuraGitHub/Twitter: kiyoto/kiyototamura

Treasure Data, Inc.Vice President, MarketingTRUST ME, I used to code daily =)

Tweet NOW!“At #svde learning how to collect more event data using #Fluentd”

WHAT’S FLUENTD?

An extensible & reliable data collection tool

simple core + plugins

buffering, HA (failover),load balancing, etc.

like syslogd

data collection tool

✓ duplicated code for error handling...✓ messy code for retrying mechnism...

Blueflood

MongoDB

Hadoop

Metrics

Amazon S3

Analysis

Archiving

MySQL

Apache

Frontend

Access logs

syslogd

App logs

System logs

Backend

Your system

bash scripts ruby scripts

rsync

log file

bash

python scripts

customloggger

cron

other customscripts...

(this is painful!!!)

Blueflood

MongoDB

Hadoop

Metrics

Amazon S3

Analysis

Archiving

MySQL

Apache

Frontend

Access logs

syslogd

App logs

System logs

Backend

Your systemfilter / buffer / route

extensible

CORE PLUGINS• Divide & Conquer• Buffering & Retries• Error Handling• Message Routing• Parallelism

• Read Data• Parse Data• Buffer Data• Write Data• Format Data

CommonConcerns

Use CaseSpecific

architecture

INTERNAL ARCHITECTURE

“input-ish” “output-ish”

Input Parser Buffer Output FormatterFilter

reliable

reliable data transfer

DIVIDE & CONQUER & RETRY

error retry

error retry retry

retry

reliable process

THIS?

OR THIS?

M X N → M + N

Nagios

MongoDB

Hadoop

Alerting

Amazon S3

Analysis

Archiving

MySQL

Apache

Frontend

Access logs

syslogd

App logs

System logs

Backend

Databasesbuffer/filter/route

use cases

SIMPLE FORWARDING

# logs from a file<source> type tail path /var/log/httpd.log format apache2 tag backend.apache</source>

# logs from client libraries<source> type forward port 24224</source>

# store logs to ES and HDFS<match backend.*> type mongo database fluent collection test</match>

LESS SIMPLE FORWARDING

LAMBDA ARCHITECTURE

# logs from a file<source> type tail path /var/log/httpd.log format apache2 tag web.access</source>

# logs from client libraries<source> type forward port 24224</source>

# store logs to ES and HDFS<match backend.*> type copy

<store> type elasticsearch logstash_format true </store>

<store> type webhdfs host namenode port 50070 path /path/on/hdfs/ </store></match>

CEP FOR STREAM PROCESSING

FLUENTD ON KUBERNETES (NOV 2015)

FLUENTD LOGGING DRIVER (APR 2015)

Tweet Again!“Happy v1 #k8s and congrats #Fluentd for becoming a #docker logging driver”

DEMO: FLUENTD + DOCKER

THANK YOU!

AND TREASURE DATA IS HIRING!WWW.TREASUREDATA.COMC/CAREERS