Fluentd v11 at tokuben

download Fluentd v11 at tokuben

of 22

  • date post

  • Category


  • view

  • download


Embed Size (px)


Introducing Fluentd v11 features.

Transcript of Fluentd v11 at tokuben

  • 1.Fluentd v11 Jan 23, 2014 Masahiro Nakagawa Treasure Data, Incwww.treasuredata.com/ Sunday, January 26, 141

2. Who are you? Masahiro Nakagawa Treasure Data, Inc. @repeatedlySenior Software Engineer Fluentd, td-agent, etc...Dlang, MessagePack, ...2 Sunday, January 26, 14 3. Structured logging http://uentd.org/Reliable forwarding Pluggable architectureSunday, January 26, 14 4. MxNM+N Access logsAlertingApacheNagiosApp logsAnalysisFrontendMongoDBBackendMySQL HadoopSystem logs syslogd buffer / buffer / routingDatabasesArchiving Amazon S3 4Sunday, January 26, 14 5. v10 Mainly for log forwarding with good performance working in many productions reliably Various plugins are released There are 200+ plugins! Mainly for CRuby 5 Sunday, January 26, 14 6. Why v11? We want more useful features, but... Changing the internal architecture is hard Keeping the external API is even harder Originally, v10 was started as a prototype Need drastic changes for the future :) 6 Sunday, January 26, 14 7. Whats new? ServerEngine based reliable architecture Zero-downtime restarting Filter / Label for exible record handling Improved conguration Improved plugin development 7 Sunday, January 26, 14 8. ServerEngine based Robust signal handling Put a signal into Queue rst Built-in supervisor Multiprocess support No need for in_multiprocess plugin 8 Sunday, January 26, 14 9. Multi Process SupervisorWorkerWorkerWorker input tail output mongo input forward output webhdfs input foo output bar Separate stream pipelines in one instance! 9 Sunday, January 26, 14 10. Zero downtime restart SocketManager shares resources with workersSupervisor1. Listen to TCP socketTCP10 Sunday, January 26, 14 11. Zero downtime restart SocketManager shares resources with workersSupervisor TCP1. Listen to TCP socket 2. Pass its socket to workerheartbeatWorker TCP11 Sunday, January 26, 14 12. Zero downtime restart SocketManager shares resources with workersSupervisor TCP heartbeatWorker1. Listen to TCP socket 2. Pass its socket to worker 3. Do same action at worker restarting with keeping TCP socketWorker TCP12 Sunday, January 26, 14 13. Filter / Label support No more tag-related tricks! add_tag_xxx, remove_tag_xxx, etc... Redirect events to another group Much easier to group and share plugins 13 Sunday, January 26, 14 14. Filter can have nested Conguration format is not xed! v10: type owcounter add_tag_prex counted typo growthforecast v11: type owcounter typo growthforecast 14Sunday, January 26, 14 15. Label can contain multiple out_redirect can forward events to type rewrite_tag_lter ... type redirect to_label blackhole ... type null bangs record go away! 15Sunday, January 26, 14 16. Improved conguration Can use Hash, Array and others No need for , or similar tricks You can write Ruby directly Worker pragma Separate plugins in each process Support the v10 compatible mode 16 Sunday, January 26, 14 17. New parameter types Can write complex values without DSL! Can use Ruby code for conguration Hash, Array, etc:Embedded Ruby code: type my_tail keys ["k1", "k2", "k3"] typo my_lter add_keys {"k1" : "v1"} Sunday, January 26, 14 type my_lter env "#{ENV['KEY']}" Socket.gethostname `command` etc... 17 18. Improved plugin Actor Abstraction layer for several actions Error stream with @ERROR label Log level per plugin No more global API, Engine.emit, $log, etc... 18 Sunday, January 26, 14 19. Actor Easy to write popular routines v10:v11:class TimerWatcher < Coolio::TimerWatcher ... end def start @loop = Coolio::Loop.new @timer = ... @loop.attach(@timer) @thread = ... end Sunday, January 26, 14actor.every(@interval) { event_router.emit(...) }19 20. Error stream Can handle an error at each record level Input {"event":1, ...}chunk1{"event":2, ...} {"event":3, ...}{"event":4, ...}Output OK ERROR! OKOK{"event":5, ...} {"event":6, ...}chunk2ERROR! OKSunday, January 26, 14Error stream type le ... Built-in @ERROR is used when error occurred in emit 20 21. Lastly... Super alpha status! Adding / Removing features may occur There are several to be implemented features The release date is not xed yet The initial release should be useful from day one Need feedbacks! 21 Sunday, January 26, 14 22. Were Hiring! Jan 23, 2014www.treasuredata.com/ Sunday, January 26, 1422