Fluentd meetup

download Fluentd meetup

of 21

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of Fluentd meetup

  • 1.FluentdStructured loggingPluggable architectureReliable forwardingThe Event Collector ServiceSadayuki FuruhashiTreasure Data, Inc.@frsyuki

2. Fluentd in briefIts like syslogd, but uses JSON for log messages 3. Fluentd :: format of logsApplication2012-02-04 01:33:51myapp.buylog { Fluentduser: me,path: /buyItem,price: 150,referer: /landing Storage} 4. Fluentd :: format of logstimeApplication tag2012-02-04 01:33:51myapp.buylog { Fluentduser: me,path: /buyItem,price: 150,referer: /landing Storage} record 5. Fluentd :: plugins ApplicationFluentdfilter / buffer / routingSaaSStorageFluentd Plug-in Plug-in Plug-in 6. Fluentd :: plugins syslogd Scribe ApplicationFile Plug-intailPlug-in Plug-in Fluentd filter / buffer / routing SaaSStorageFluentd Plug-inPlug-in Plug-in 7. Fluentd :: client libraries Client libraries> Ruby> Perl Application> PHP> Python> Java Fluentd> ...Fluent.open(myapp)Fluent.event(login, {user=>38})#=> 2012-02-04 04:56:01 myapp.login {user:38} 8. Typical architecture before Fluentd App serverApp serverApp server Application ApplicationApplication File File File ...File File File ... File File File ...FileHigh latencymust wait for a dayLog serverHard to analyzecomplex text parsers 9. Architecture after Fluentd App serverApp server App server Application ApplicationApplicationFluentdFluentd FluentdRealtime! FluentdFluentd 10. Architecture after FluentdFluentdFluentd FluentdRealtime! FluentdFluentdHadoop Amazon Ready toMongoDB / HiveS3 / EMR Analyze! 11. Case studyRuby on RailsRuby on RailsRuby on Rails FluentdFluentd Fluentd 127 RoR servers 70,000 msgs/secFluentd Fluentd routing 120Mbps at peak 650GB/dayHadoop User behavior PV logsMongoDBlogs / Hive 12. # read logs from a file # forward other logs to servers# (load-balancing + fail-over)type tail path /var/log/httpd.log type forwardformat apache tag apache.access host weight 20# save access logs to MongoDB host mongoweight 60host 13. Scribe Scribe: log collector by Facebook Frontend serversAggregator nodes scribescribe scribe HadoopHDFS scribescribe scribe 14. Scribes Pros & Cons Pros.> Fast (C++) Cons.> VERY hard to install> Deals with unstructured logsyou must parse logs before analyzing them> Hard to extendyou must re-compile C++ programs> No longer maintained? 15. Fluentd vs Scribe Easy to install> gem install uentd> stable RPM and DEB packageshttp://packages.treasure-data.com/ Easy to write plugins> you can use Ruby Easy to distribute plugins> gem search -rd uent-plugin 16. FlumeFlume: distributed log collector by Cloudera PhisicalFlume MasterTopology FlumeFlume Flume LogicalTopology HadoopHDFS 17. Flumes Pros & Cons Pros.> Central master server manages all nodes Cons.> Difcult to understandlogical topologies, phisical servers and a conguration ofthe logical/phisical mapping> Dicult to congurereplicated master servers, log servers and agents> Big footprint50,000 lines of Java codes 18. Fluentd vs Flume Easy to understand> syslogd that understands JSON Easy to setup> sudo uentd --setup && uentd Very small footprint> small engine (3,000 lines) + plugins Easy to congure 19. Fluentd vs Scribe/Flume Fluentd Scribe FlumeInstallationgem/rpm/debmake rpm/deb 3000 lines of8000 lines of 50,000 lines ofFootprint Ruby C++JavaPluginRubyN/A JavaPlugin distribution RubyGems.orgN/AN/AMaster ServerNo No YesLicense Apache License Apache License Apache License 20. Fluentd Documents> http://uentd.org Source code> http://github.com/uent> 14 committers acrossmany organizations Mailing list> Google groups 21. Sadayuki Furuhashi> twitter: @frsyuki Treasure Data, Inc.> Software Engineer; founder Author of MessagePack Author of Fluentd