Fluentd - Unified logging layer

download Fluentd -  Unified logging layer

of 31

Embed Size (px)

Transcript of Fluentd - Unified logging layer

  1. 1. FLUENTD: UNIFIED LOGGING LAYER Kiyoto Tamura July 23, 2015 Silicon Valley Data Engineering Meetup
  2. 2. WHOAMI Kiyoto Tamura GitHub/Twitter: kiyoto/kiyototamura Treasure Data, Inc. Vice President, Marketing TRUST ME, I used to code daily =)
  3. 3. Tweet NOW! At #svde learning how to collect more event data using #Fluentd
  4. 4. WHATS FLUENTD? An extensible & reliable data collection tool simple core + plugins buffering, HA (failover), load balancing, etc. like syslogd
  5. 5. data collection tool
  6. 6. duplicated code for error handling... messy code for retrying mechnism... Blueood MongoDB Hadoop Metrics Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Your system bash scripts ruby scripts rsync log le bash python scripts custom loggger cron other custom scripts...
  7. 7. (this is painful!!!)
  8. 8. Blueood MongoDB Hadoop Metrics Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Your system filter / buffer / route
  9. 9. extensible
  10. 10. CORE PLUGINS Divide & Conquer Buffering & Retries Error Handling Message Routing Parallelism Read Data Parse Data Buffer Data Write Data Format Data Common Concerns Use Case Specic
  11. 11. architecture
  12. 12. INTERNAL ARCHITECTURE input-ish output-ish Input Parser Buffer Output FormatterFilter
  13. 13. reliable
  14. 14. reliable data transfer
  15. 15. DIVIDE & CONQUER & RETRY error retry error retry retry retry
  16. 16. reliable process
  17. 17. THIS?
  18. 18. OR THIS?
  19. 19. M X N M + N Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases buffer/filter/route
  20. 20. use cases
  21. 21. SIMPLE FORWARDING
  22. 22. # logs from a file type tail path /var/log/httpd.log format apache2 tag backend.apache # logs from client libraries type forward port 24224 # store logs to ES and HDFS type mongo database fluent collection test
  23. 23. LESS SIMPLE FORWARDING
  24. 24. LAMBDA ARCHITECTURE
  25. 25. # logs from a file type tail path /var/log/httpd.log format apache2 tag web.access # logs from client libraries type forward port 24224 # store logs to ES and HDFS type copy type elasticsearch logstash_format true type webhdfs host namenode port 50070 path /path/on/hdfs/
  26. 26. CEP FOR STREAM PROCESSING
  27. 27. FLUENTD ON KUBERNETES (NOV 2015)
  28. 28. FLUENTD LOGGING DRIVER (APR 2015)
  29. 29. Tweet Again! Happy v1 #k8s and congrats #Fluentd for becoming a #docker logging driver
  30. 30. DEMO: FLUENTD + DOCKER
  31. 31. THANK YOU! AND TREASURE DATA IS HIRING! WWW.TREASUREDATA.COMC/CAREERS