Fluentd meetup #2

download Fluentd meetup #2

If you can't read please download the document

  • date post

    08-May-2015
  • Category

    Technology

  • view

    9.435
  • download

    5

Embed Size (px)

Transcript of Fluentd meetup #2

  • 1.Log everyting in JSON. Treasuare Data, Inc. Sadayuki Furuhashi

2. Self-introduction> Sadayuki Furuhashitwitter: @frsyuki> Original author of Fluentd> Treasure Data, Inc.Software Architect; Founder> open-sourceMessagePack - efficient serialization format 3. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd 4. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd 5. 0. Why logging? >Error notifications >Performance monitoring >User segment analysis >Funnel analysis >Heatmap analysis >Market prediction etc... 6. 0. Why logging? - Error notificationsError! 7. 0. Why logging? - Performance monitor 8. 0. Why logging? - User segment analysis 9. 0. Why logging? - Funnel analysis-28%! -27%! 10. 0. Why logging? - Heatmap analysis 11. 0. Why logging? - Market prediction 12. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd 13. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd 14. AlertingNagiosAnalysis MongoDB MySQL Hadooplog utilizationArchivingAmazon S3 15. Access logsAlertingApache NagiosApp logs Analysis Frontend log sources MongoDB BackendMySQLSystem logs Hadoopsyslogdlog utilization ArchivingDatabasesAmazon S3 16. Access logs AlertingApacheNagiosApp logsAnalysis FrontendMongoDB Backend MySQLSystem logsHadoopsyslogdArchivingDatabases Amazon S3 17. Access logs rsync serversAlertingApacheNagiosApp logsAnalysis FrontendMongoDB Backend MySQLbash scriptsSystem logsHadoopsyslogdArchivingDatabases Amazon S3perl scripts 18. Problems...No unified method to collect logs> Too many bash/perl scriptsFragile for changesLess reliable> Mixed log formatsOld-fashioned Human-readable text logsNot ready to analyze> High latencymust wait a day for log rotation 19. Access logs AlertingApacheNagiosApp logsAnalysis FrontendMongoDB Backend MySQLSystem logsHadoopsyslogdArchivingDatabases Amazon S3 20. Access logs AlertingApacheNagiosApp logsAnalysis FrontendMongoDB Backend MySQLSystem logsHadoopsyslogdArchivingfilter / buffer / routingDatabases Amazon S3 21. Input PluginsOutput PluginsBuffer PluginsFilter Plugins 22. Input Plugins Output Plugins 2012-02-04 01:33:51 JSON format myapp.buylog { user: me, path: /buyItem, price: 150, referer: /landing } 23. Input Plugins Output Plugins time tag 2012-02-04 01:33:51 JSON format myapp.buylog { user: me, path: /buyItem, price: 150, referer: /landing }record 24. Why Fluentd?> Extensibility - Plugin architecturecollect logs from various systemsforward logs to various systems> Unified log format - JSON formatmodern Machine-readable log formatimmediately ready to analyze> Reliable - HA configuration> Easy to install - RPM/deb packagesdeploy instantly to everywhere 25. Comparision with other log collectors:> ScribeLess extensibleNo unified log formatNo longer developped?> FlumeLess simpleNo unified log formatLittle information about Flume-NG 26. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd 27. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd 28. NHN JapanCOOKPADNAVERCrocoshttp://www.quora.com/Who-uses-Fluentd-in-production 29. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd 30. 0. Why logging?1. Why Fluentd? - Design of Fluentd > Extensibility > Unified log format > Simplicity2. Who uses Fluentd?3. Future of Fluentd 31. Future of Fluentd> > in > > MessagePack for Ruby v5> td-agent-lite> Pub/Sub & Monitoring API> New process model & Live restart> Backward compatibility 32. type tailtype forwardpath /var/log/httpd.loghost log.serverformat apachetag not_filtered.apache Mysterious tagtype rewriteremove_prefix not_filtered tag operationskey statuspattern ^500$ignore true Before 33. type tailtype forwardpath /var/log/httpd.loghost log.serverformat apachetag apacheFilter plugins!type rewritekey statuspattern ^500$ignore true After (v11) 34. type tail type forwardpath /var/log/httpd.log host log.serverformat apache tag apachetype rewritekey statuspattern ^500$/ inignore true After (v11) 35. type tailtype forwardpath /var/log/httpd.loghost log.servertag apache I want to add flowcounter here... Before 36. type tail type forwardpath /var/log/httpd.log host log.servertag apachetype forwardhost traffic.serverNested!type copytype flowcountertag flow.traffic Before 37. type tail type forwardpath /var/log/httpd.log host log.servertag apachetype copytype flowcountertag flow.traffictype forwardhost traffic.serverFiltering pipelineAfter (v11) 38. # copy & label & forwardtype forward type copy type forwardtype copy label alert host alerting.servertype file path /mnt/local_archive # copy & label & forward type copy... type forwardlabel analysishost analysis.server...After (v11) 39. MessagePack for Ruby v5(tweets/sec)400003000020000100000SerializeDeserialize msgpack v5 msgpack v4 yajljson 40. td-agent-lite> in_tail + out_forward in single binarystatically linked ruby binary + scripts tied with the binary 41. New process model & Live restartOld multiprocess model detachedprocess fork()SupervisorEngine detached all data pass throughprocess the central process 42. New process model & Live restartNew multiprocess model detachedprocess Process SupervisorEngine Manager detachedprocessdirect communication 43. New process model & Live restartNew multiprocess modeldetached processProcess SupervisorEngineManagerdetached Live restartprocessProcess EngineManager 44. Backward compatibilityFluentd v11 includes 2 namespaces:> Fluentd:: new code base> Fluent:: old code base + wrapper classesCheckout the repository for details: > http://github.com/frsyuki/fluentd-v11 45. ConculutionFluentd makes logging better> Plugin architecture> JSON format> HA configuration> RPM/deb packageFluentd is under active developmentFluentd is suppored by many committers 46. contact: sales@treasure-data.com 47. / 48. 49. Fluentd