The basics of fluentd

download The basics of fluentd

of 37

  • date post

    08-May-2015
  • Category

    Technology

  • view

    9.463
  • download

    1

Embed Size (px)

description

Fluentd introduction slide

Transcript of The basics of fluentd

  • 1.Structured logging! Reliable forwarding! Pluggable architecturehttp://uentd.org/

2. Agenda > Background > Overview > Product Comparison > Use cases 3. Background 4. Data Processing Collect Store Process Visualize Data source Reporting Monitoring 5. Related Products Store Process Cloudera Horton Works Treasure Data Collect Visualize Tableau Excel R easier & shorter time ??? 6. Before Fluentd Application Server2 Application Server3 Application Server1 FluentLog High Latency! must wait for a day... 7. After Fluentd Application Server2 Application Server3 Application Server1 In streaming! Fluentd Fluentd Fluentd Fluentd Fluentd 8. Overview 9. > Open sourced log collector written in Ruby > Reliable, scalable and easy to extend > Using rubygems ecosystem for plugins ! ! In short Its like syslogd, but uses JSON for log messages 10. tail insert event buffering 127.0.0.1 - - [11/Dec/2012:07:26:27] "GET / ... 127.0.0.1 - - [11/Dec/2012:07:26:30] "GET / ... 127.0.0.1 - - [11/Dec/2012:07:26:32] "GET / ... 127.0.0.1 - - [11/Dec/2012:07:26:40] "GET / ... 127.0.0.1 - - [11/Dec/2012:07:27:01] "GET / ... ... Fluentd Web Server 2012-02-04 01:33:51apache.log{"host": "127.0.0.1","method": "GET",...} Example (apache to mongo) 11. > default second unit > from data source or adding parsed time Event structure(log message) Time > for message routing Tag > JSON format > MessagePack internally > non-unstructured Record 12. Pluggable Architecture Buffer Output Input > Forward > HTTP > File tail > dstat > ... > Forward > File > MongoDB > ... > File > Memory Engine Output > rewrite > ... Pluggable Pluggable 13. Fluentd # Ruby! Fluent.open(myapp)! Fluent.event(login, {user => 38})! #=> 2012-12-11 07:56:01 myapp.login {user:38} > Ruby> Java> Perl> PHP> Python> D> Scala> ... Application Time:Tag:Record Client libraries 14. Conguration and operation > No central / master node > HTTP include helps configuration sharing > Operation depends on your environment > Use your deamon management > Use Chef in Treasure Data > Apache like syntax and Ruby DSL 15. # receive events via HTTP type http port 8888 ! # read logs from a le type tail path /var/log/httpd.log format apache tag apache.access ! # save access logs to MongoDB type mongo database apache collection log # save alerts to a letype lepath /var/log/uent/alerts! # forward other logs to serverstype forwardhost 192.168.0.11weight 20host 192.168.0.12weight 60! include http://example.com/conf 16. Reliability (core + plugin) > Buffering > Use file buffer for persistent data > buffer chunk has ID for idempotent > Retrying > Error handling > transaction, failover, etc on forward plugin > secondary for backup 17. Plugins - use rubygems $ fluent-gem search -rd fluent-plugin! ! $ fluent-gem search -rd fluent-mixin! ! $ fluent-gem install fluent-plugin-mongo 18. http://www.fluentd.org/plugins 19. in_tail read a log le! read log les in directory! custom regexp! custom parser in Ruby FluentdApache access.log > apache > apache2 > syslog > nginx > json > csv > tsv > ltsv Supported format: > none > multiline 20. Fluentd out_mongo Apache bufferaccess.log retry automatically! exponential retry wait! persistent on a le 21. Fluentd out_webhdfs buffer retry automatically! exponential retry wait! persistent on a le slice les based on time 2013-01-01/01/access.log.gz! 2013-01-01/02/access.log.gz! 2013-01-01/03/access.log.gz! ... HDFS custom text formatter Apache access.log 22. out_copy + other plugins routing based on tags! copy to multiple storages Amazon S3 Hadoop Fluentd buffer Apache access.log 23. out_forward apache automatic fail-over! load balancing FluentdApache bufferaccess.log retry automatically! exponential retry wait! persistent on a le Fluentd Fluentd Fluentd 24. Forward topology send/ack Fluentd Fluentd Fluentd Fluentd Fluentd Fluentd Fluentd send/ack 25. Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing 26. Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing 27. Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases filter / buffer / routing 28. td-agent > Open sourced distribution package of fluentd > ETL part of Treasure Data > deb, rpm, dmg (since td-agent 2.0) > Including useful components > ruby, jemalloc, fluentd > 3rd party gems: td, mongo, webhdfs, etc > http://packages.treasure-data.com/ 29. v1 > New features without breaking compatibility > Filter, Label and better error handling > Serverengine based: multi-process, signal, etc. > New configuration and DSL format > JRuby and Windows support > github issue: Plan for v1 release #251 30. Use cases 31. Treasure Data Frontend Job Queue Worker Hadoop Hadoop Fluentd Applications push metrics to Fluentd (via local Fluentd) Librato Metrics for realtime analysis Treasure Data for historical analysis Fluentd sums up data minutes (partial aggregation) 32. hundreds of app servers sends event logs sends event logs sends event logs Rails app td-agent td-agent td-agent Google Spreadsheet Treasure Data MySQL Logs are available after several mins. Daily/Hourly Batch KPI visualizationFeedback rankings Rails app Rails app Unlimited scalability Flexible schema Realtime Less performance impact Cookpad Over 100 RoR servers (2012/2/4) 33. http://www.slideshare.net/tagomoris/log-analysis-with-hadoop-in-livedoor-2013 NHN Japan by @tagomoris 16 nodes! 120,000+ lines/sec! 400Mbps at peak! 1.5+ TB/day (raw) Web Servers FluentdCluster ArchiveStorage(scribed) FluentdWatchers GraphTools Notications(IRC) Hadoop ClusterCDH4(HDFS, YARN) webhdfs HuahinManager hiveserver STREAM Shib ShibUI BATCH SCHEDULED BATCH 34. Other usecases > Collect censor logs > Embedded devise, Rapsberry Pi, etc > Integrated with Elasticsearch and Kibana > Integrated with Norikra CEP engine http://www.uentd.org/guides 35. Other companies http://www.uentd.org/testimonials 36. > Fluentd is a widely-used log collector > There are many use cases > Many contributors and plugins > Keep it simple > Easy to integrate your environment Conclusion