Fluentd: Towards Unified Logging Layer Kiyoto Tamura @kiyototamura Treasure Data, Inc.

Not an ops engineer myself marketing guy My laptop isnt very hipster Why you shouldnt listen to me

Fluentd maintainer and community manager Former programmer data nerd Why you should listen to me

why Fluentd?

I was a data scientist before it was cool to be one.

http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow- overview-and-challenges/fulltext

Acquire Data (or so you think) WUT!? Invalid UTF8? Fix the encoding issue Yell at the engineers Some columns are missing!? Run the scriptDIVISION BY ZERO!!!

Hmm

Logging.priority => :not_super_high

analytics.priority => :very_high

analytics.needs? :logs => true

outage.priority => :super_high

outage.needs? :logs => [no, shit]

Unified Logging Layer

overview

127.0.0.1 - - [05/Feb/2012:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"

{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": Mozilla/5.0 (Windows" }

Parse as JSON!

?

[05/Feb/2012:17:11:55,web.access,{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": Mozilla/5.0 (Windows" }]

? web.mongodb web.file web.hdfs web.s3 web.mysql

Fluentd Design Patterns (credit: @tagomoriss talk at RedDotRubyConf)

Real World Use Cases

Automate monitoring! App and System metrics JSON everywhere

2000+ node ~1B events/day Forwarder- Aggregator

Demo

Fluentd Treasure Data Twitter API #NED #ARG Postgres + d3.js Collect Raw Tweets Tokenize + Count serve + visualize

{ install: gem install fluentd, website: www.fluentd.org, github : fluent/fluentd, twitter: @fluentd }