Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

36
Fluentd: Towards Unified Logging Layer Kiyoto Tamura @kiyototamura Treasure Data, Inc.

description

Fluentd is an open source data collector to simplify and scale log management. In this talk, I gave a brief overview of Fluentd, focusing on the why: Why should you care about logging? Why do you want a unified logging layer? Why Fluentd?

Transcript of Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

Page 1: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

Fluentd:Towards Unified Logging Layer

Kiyoto Tamura@kiyototamura

Treasure Data, Inc.

Page 2: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
Page 3: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

• Not an ops engineer myself

• “marketing guy”

• My laptop isn’t very hipster

Why you shouldn’t listen to me

Page 4: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

• Fluentd maintainer and

community manager

• Former programmer

• data nerd

Why you should listen to me

Page 5: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

why Fluentd?

Page 6: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

I was a data scientist before it was cool to be one.

Page 7: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow-overview-and-challenges/fulltext

Page 8: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

Acquire Data (or so you think)

WUT!? Invalid UTF8?

Fix the encoding issue…

Yell at the engineers

Some columns are missing!?

Run the script…DIVISION BY

ZERO!!!

Page 9: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

Hmm…

Page 10: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

Logging.priority=> :not_super_high

Page 11: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

analytics.priority=> :very_high

Page 12: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

analytics.needs? :logs=> true

Page 13: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

outage.priority=> :super_high

Page 14: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

outage.needs? :logs=> [“no”, “shit”]

Page 15: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
Page 16: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
Page 17: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
Page 18: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

Unified Logging Layer

Page 19: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

overview

Page 20: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
Page 21: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

127.0.0.1 - - [05/Feb/2012:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"

Page 22: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…"}

Page 23: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
Page 24: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

Parse as JSON!

Page 25: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

?

Page 26: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

[“05/Feb/2012:17:11:55”,“web.access”,{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…"}]

Page 27: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

?

web.mongodb

web.file

web.hdfs

web.s3

web.mysql

Page 28: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

Fluentd Design Patterns(credit: @tagomoris’s talk at RedDotRubyConf)

Page 29: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

Real World Use Cases

Page 30: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
Page 31: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

• Automate monitoring!

• App and System metrics

• JSON everywhere

Page 32: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
Page 33: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

• 2000+ node• ~1B events/day• Forwarder-

Aggregator

Page 34: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

Demo

Page 35: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

Fluentd

Treasure Data

Twitter API#NED #ARG

Postgres + d3.js

Collect Raw Tweets

Tokenize + Count

serve + visualize

Page 36: Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

{ “install”: “gem install fluentd”, “website”: “www.fluentd.org”, “github” : “fluent/fluentd”, “twitter”: “@fluentd”}