Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
-
Upload
kiyoto-tamura -
Category
Software
-
view
194 -
download
1
Embed Size (px)
description
Transcript of Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

Fluentd:Towards Unified Logging Layer
Kiyoto Tamura@kiyototamura
Treasure Data, Inc.


• Not an ops engineer myself
• “marketing guy”
• My laptop isn’t very hipster
Why you shouldn’t listen to me

• Fluentd maintainer and
community manager
• Former programmer
• data nerd
Why you should listen to me

why Fluentd?

I was a data scientist before it was cool to be one.

http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow-overview-and-challenges/fulltext

Acquire Data (or so you think)
WUT!? Invalid UTF8?
Fix the encoding issue…
Yell at the engineers
Some columns are missing!?
Run the script…DIVISION BY
ZERO!!!

Hmm…

Logging.priority=> :not_super_high

analytics.priority=> :very_high

analytics.needs? :logs=> true

outage.priority=> :super_high

outage.needs? :logs=> [“no”, “shit”]




Unified Logging Layer

overview


127.0.0.1 - - [05/Feb/2012:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"

{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…"}


Parse as JSON!

?

[“05/Feb/2012:17:11:55”,“web.access”,{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…"}]

?
web.mongodb
web.file
web.hdfs
web.s3
web.mysql

Fluentd Design Patterns(credit: @tagomoris’s talk at RedDotRubyConf)

Real World Use Cases


• Automate monitoring!
• App and System metrics
• JSON everywhere


• 2000+ node• ~1B events/day• Forwarder-
Aggregator

Demo

Fluentd
Treasure Data
Twitter API#NED #ARG
Postgres + d3.js
Collect Raw Tweets
Tokenize + Count
serve + visualize

{ “install”: “gem install fluentd”, “website”: “www.fluentd.org”, “github” : “fluent/fluentd”, “twitter”: “@fluentd”}