Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

download Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

of 36

  • date post

    27-Aug-2014
  • Category

    Software

  • view

    177
  • download

    1

Embed Size (px)

description

Fluentd is an open source data collector to simplify and scale log management. In this talk, I gave a brief overview of Fluentd, focusing on the why: Why should you care about logging? Why do you want a unified logging layer? Why Fluentd?

Transcript of Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

  • Fluentd: Towards Unified Logging Layer Kiyoto Tamura @kiyototamura Treasure Data, Inc.
  • Not an ops engineer myself marketing guy My laptop isnt very hipster Why you shouldnt listen to me
  • Fluentd maintainer and community manager Former programmer data nerd Why you should listen to me
  • why Fluentd?
  • I was a data scientist before it was cool to be one.
  • http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow- overview-and-challenges/fulltext
  • Acquire Data (or so you think) WUT!? Invalid UTF8? Fix the encoding issue Yell at the engineers Some columns are missing!? Run the scriptDIVISION BY ZERO!!!
  • Hmm
  • Logging.priority => :not_super_high
  • analytics.priority => :very_high
  • analytics.needs? :logs => true
  • outage.priority => :super_high
  • outage.needs? :logs => [no, shit]
  • Unified Logging Layer
  • overview
  • 127.0.0.1 - - [05/Feb/2012:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"
  • { "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": Mozilla/5.0 (Windows" }
  • Parse as JSON!
  • ?
  • [05/Feb/2012:17:11:55,web.access,{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": Mozilla/5.0 (Windows" }]
  • ? web.mongodb web.file web.hdfs web.s3 web.mysql
  • Fluentd Design Patterns (credit: @tagomoriss talk at RedDotRubyConf)
  • Real World Use Cases
  • Automate monitoring! App and System metrics JSON everywhere
  • 2000+ node ~1B events/day Forwarder- Aggregator
  • Demo
  • Fluentd Treasure Data Twitter API #NED #ARG Postgres + d3.js Collect Raw Tweets Tokenize + Count serve + visualize
  • { install: gem install fluentd, website: www.fluentd.org, github : fluent/fluentd, twitter: @fluentd }