Fluentd:Towards Unified Logging Layer

Kiyoto Tamura@kiyototamura

Treasure Data, Inc.

• Not an ops engineer myself

• “marketing guy”

• My laptop isn’t very hipster

Why you shouldn’t listen to me

• Fluentd maintainer and

community manager

• Former programmer

• data nerd

Why you should listen to me

why Fluentd?

I was a data scientist before it was cool to be one.

http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow-overview-and-challenges/fulltext

Acquire Data (or so you think)

WUT!? Invalid UTF8?

Fix the encoding issue…

Yell at the engineers

Some columns are missing!?

Run the script…DIVISION BY

ZERO!!!

Hmm…

Logging.priority=> :not_super_high

analytics.priority=> :very_high

analytics.needs? :logs=> true

outage.priority=> :super_high

outage.needs? :logs=> [“no”, “shit”]

Unified Logging Layer

overview

127.0.0.1 - - [05/Feb/2012:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"

{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…"}

Parse as JSON!

?

[“05/Feb/2012:17:11:55”,“web.access”,{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…"}]

?

web.mongodb

web.file

web.hdfs

web.s3

web.mysql

Fluentd Design Patterns(credit: @tagomoris’s talk at RedDotRubyConf)

Real World Use Cases

• Automate monitoring!

• App and System metrics

• JSON everywhere

• 2000+ node• ~1B events/day• Forwarder-

Aggregator

Demo

Fluentd

Treasure Data

Twitter API#NED #ARG

Postgres + d3.js

Collect Raw Tweets

Tokenize + Count

serve + visualize

{ “install”: “gem install fluentd”, “website”: “www.fluentd.org”, “github” : “fluent/fluentd”, “twitter”: “@fluentd”}