Fluentd v0.12 master guide

download Fluentd v0.12 master guide

of 31

  • date post

    02-Aug-2015
  • Category

    Technology

  • view

    3.405
  • download

    9

Embed Size (px)

Transcript of Fluentd v0.12 master guide

1. Masahiro Nakagawa June 1, 2015 Fluentd meetup 2015 Summer Fluentd- v0.12 master guide - #uentdmeetup 2. Who are you? > Masahiro Nakagawa > github/twitter: @repeatedly > Treasure Data, Inc. > Senior Software Engineer > Fluentd / td-agent developer > I love OSS :) > D language - Phobos committer > Fluentd - Main maintainer > MessagePack / RPC - D and Python (only RPC) > The organizer of Presto Source Code Reading / meetup > etc 3. Structured logging Reliable forwarding Pluggable architecture http://uentd.org/ 4. Whats Fluentd? > Data collector for unied logging layer > Streaming data transfer based on JSON > Written in Ruby > Gem based various plugins > http://www.uentd.org/plugins > Working in production > http://www.uentd.org/testimonials 5. v0.10 (old stable) > Mainly for log forwarding > with good performance > working in production > with td-agent 1 and td-agent 2.0 / 2.1 > Robust but not good for log processing 6. Architecture (v0.10) Buffer Output Input > Forward > HTTP > File tail > dstat > ... > Forward > File > MongoDB > ... > File > Memory Engine Output > rewrite > ... Pluggable Pluggable 7. v0.12 (current stable) > v1 conguration by default > Event handling improvement > Filter, Label, Error Stream > At-least-once semantics in forwarding > Add require_ack_response parameter > HTTP RPC based management > Latest release is v0.12.11 8. Architecture (v0.12 or later) EngineInput Filter Output Buffer > grep > record_transfomer > > Forward > File tail > ... > Forward > File > ... Output > File > Memory not pluggable FormatterParser 9. v1 conguration > hash, array and enum types are added > hash and array are json > Embed Ruby code using "#{}", > easy to set variable values: "#{ENV['KEY']}" > Add :secret option to mask parameters > @ prex for built-in parameters > @type, @id and @log_level 10. New v1 formats > Easy to write complex values > No trick or additional work for common cases @type my_tail keys ["k1", "k2", "k3"] @typo my_lter add_keys {"k1" : "v1"} @type my_lter env "#{ENV['KEY']}" lter> Hash,Array, etc: Embedded Ruby code: Socket.gethostname `command` etc... 11. :secret option > For masking sensitive parameters > In uentd logs and in_monitor_agent 2015-05-29 19:50:10 +0900 [info]: using conguration le: @type forward @type test sensitive_param xxxxxx cong_param :sensitive_param, :string, :secret => true 12. > Apply ltering routine to event stream > No more tag tricks and cant modify tag type record_reformer tag reformed.${tag} type growthforecast @type record_transformer lter> v0.10: v0.12: @type growthforecast Filter 13. Processing pipeline comparison Output Engine Filter Output Output 1 transaction 1 transaction 1 transaction v0.10 v0.12 14. > Mutate events > http://docs.uentd.org/articles/ lter_record_transformer @type record_transformer hostname "#{Socket.gethostname}" lter> @type mongodb Filter: record_transformer 15. > Grep event streams > http://docs.uentd.org/articles/lter_grep @type grep regexp1 1 message cool regexp2 hostname ^webd+.example.com$ exclude1 message uncool lter> @type mongodb Filter: grep 16. > Print events to stdout > No need copy and stdout plugins combo! > http://docs.uentd.org/articles/lter_stdout @type stdout lter> @type mongodb Filter: stdout 17. > Override lter method module Fluent::AddTagFilter < Filter # Same as other plugins, initialize, congure, start, shudown # Dene congurations by cong_param utilities def lter(tag, time, record) # Process record record["tag"] = tag # Return processed record, # If return nil, that records are ignored record end end Filter: Plugin development 1 18. > Override lter_stream method module Fluent::AddTagFilter < Filter def lter_stream(tag, es) new_es = MultiEventStream.new es.each { |time, record| begin record["tag"] = tag new_es.add(time, record) rescue => e router.emit_error_event(tag, time, record, e) end } new_es end end Filter: Plugin development 2 19. > Internal event routing > Redirect events to another group > much easier to group and share plugins @type forward @type s3 @type forward @label @APP1 @type s3 v0.10: v0.12: Label 20. > Use router.emit instead of Engine.emit > Engine#emit API is deprecated tag = "" time = Engine.now record = {} Engine.emit(tag, time, record) v0.10: v0.12: tag = "" time = Engine.now record = {} router.emit(tag, time, record) Label : Need to update plugin 21. > Redirect events to another label @type forward @label @RAW Label: relabel output @type copy @type owcounter @type relabel @label @MAIN @type s3 22. Error stream with Label > Can handle an error at each record level > router.emit_error_event(tag, time, record, error) ERROR! {"event":1, ...} {"event":2, ...} {"event":3, ...} chunk1 {"event":4, ...} {"event":5, ...} {"event":6, ...} chunk2 Input OK ERROR! OK OK OK Output type le ... Error stream Built-in @ERROR is used when error occurred in emit 23. Support at-least-once semantics > Delivery guarantees in failure scenarios > At-most-once: messages may be lost > At-least-once: messages may be duplicated > Exactly-once: No lost and duplication > Fluentd supports at-most-once in v0.10 > Fluentd supports at-least-once since v.12! > set require_ack_response parameter 24. At-most-once and At-least-once @type forward require_ack_response may be duplicated Error! @type forward may be lost Error! 25. HTTP RPC based management > Use HTTP/JSON API instead of signals > For Windows and JRuby support > RPC is based on HTTP RPC style, not REST > See https://api.slack.com/web#basics > Enabled by rpc_endpoint in > Have a plan to add more APIs > stop input plugins, check plugins and etc 26. Supported RPCs > /api/processes.interruptWorkers > /api/processes.killWorkers > Same as SIGINT and SIGTERM > /api/plugins.ushBuffers > Same as SIGUSR1 > /api/cong.reload > Same as SIGHUP 27. RPC example > Conguration > Curl rpc_endpoint 127.0.0.1:24444 $ curl http://127.0.0.1:24444/api/plugins.ushBuffers {"ok":true} 28. Ecosystem 29. Almost ecosystems are v0.12 based > Treasure Agent > v2.2 is shipped with v0.12 > docs.uentd.org are now v0.12 > You can see v0.10 document via v0.10 prex > http://docs.uentd.org/v0.10/articles/quickstart > If your used plugins dont use v0.12 feature, please contribute it! 30. Roadmap > v0.10 (old stable) > v0.12 (current stable) Filter / Label / At-least-once / HTTP RPC > v0.14 (summer, 2015) > New plugin APIs, ServerEngine, Time > v1 (fall/winter, 2015) > Fix new features / APIs https://github.com/uent/uentd/wiki/V1-Roadmap 31. https://jobs.lever.co/treasure-data Cloud service for the entire data pipeline. Were hiring!