Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
date post
27-Aug-2014Category
Software
view
187download
1
Embed Size (px)
description
Fluentd is an open source data collector to simplify and scale log management. In this talk, I gave a brief overview of Fluentd, focusing on the why: Why should you care about logging? Why do you want a unified logging layer? Why Fluentd?
Transcript of Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
- Fluentd: Towards Unified Logging Layer Kiyoto Tamura @kiyototamura Treasure Data, Inc.
- Not an ops engineer myself marketing guy My laptop isnt very hipster Why you shouldnt listen to me
- Fluentd maintainer and community manager Former programmer data nerd Why you should listen to me
- why Fluentd?
- I was a data scientist before it was cool to be one.
- http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow- overview-and-challenges/fulltext
- Acquire Data (or so you think) WUT!? Invalid UTF8? Fix the encoding issue Yell at the engineers Some columns are missing!? Run the scriptDIVISION BY ZERO!!!
- Hmm
- Logging.priority => :not_super_high
- analytics.priority => :very_high
- analytics.needs? :logs => true
- outage.priority => :super_high
- outage.needs? :logs => [no, shit]
- Unified Logging Layer
- overview
- 127.0.0.1 - - [05/Feb/2012:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"
- { "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": Mozilla/5.0 (Windows" }
- Parse as JSON!
- ?
- [05/Feb/2012:17:11:55,web.access,{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": Mozilla/5.0 (Windows" }]
- ? web.mongodb web.file web.hdfs web.s3 web.mysql
- Fluentd Design Patterns (credit: @tagomoriss talk at RedDotRubyConf)
- Real World Use Cases
- Automate monitoring! App and System metrics JSON everywhere
- 2000+ node ~1B events/day Forwarder- Aggregator
- Demo
- Fluentd Treasure Data Twitter API #NED #ARG Postgres + d3.js Collect Raw Tweets Tokenize + Count serve + visualize
- { install: gem install fluentd, website: www.fluentd.org, github : fluent/fluentd, twitter: @fluentd }