Docker and Fluentd (revised)
Embed Size (px)
Transcript of Docker and Fluentd (revised)
1. Docker and Fluentd 2015/06/04 Golang & Docker users meetup Satoshi Tagomori (@tagomoris) 2. @tagomoris Treasure Data, Inc. Fluentd, Norikra, Hadoop, ... 3. Which more cute than another? 4. Logging of container architecture Storage: > should be outside of container / hosts Transferring: > should be over network Aggregation: > should be done per container / per service 5. Container-based world needs unied logging layer! 6. Structured logging Reliable forwarding Pluggable architecture http://uentd.org/ 7. Whats Fluentd? > Data collector for unied logging layer > Streaming data transfer based on JSON > Written in Ruby > Gem based various plugins > http://www.uentd.org/plugins > Working in production > http://www.uentd.org/testimonials 8. Architecture EngineInput Filter Output Buffer > grep > record_transfomer > > Forward > File tail > ... > Forward > File > ... Output > File > Memory not pluggable FormatterParser 9. Aggregation Patterns (1) 1-level aggregation (2) 2-level aggregation 10. Aggregation Patterns (1) 1-level aggregation (2) 2-level aggregation pros: simple conguration cons: xed aggr. address many connections high load for aggregator pros: less connections lower load for aggr. cons: a bit more container 11. Aggregation Patterns (1) 1-level aggregation (2) 2-level aggregation Apps sends logs over uent-logger there're some patterns... or ... 12. Pattern of 2-level aggr. w/ Fluentd 1. Network transferring 2. Container log & tail 13. (1) Network transferring Apps sends logs: > using uent-logger-* > over inter-container network Pros: > no performance penalty Cons: > logs of docker is out of scope > a bit complex apps for logging over TCP app 14. (2) Container logger & tail Apps write logs to STDOUT: > docker writes it to log le > uentd container reads le Pros: > simple conf for apps & docker > logs include container logs Cons: > in_tail performance penalty log le app http://www.uentd.org/guides/recipes/docker-logging 15. Logging drivers New from docker v1.6! 'Add new Logging driver "uentd"' > --log-driver=fluentd > https://github.com/docker/docker/pull/12876 > New for docker v1.7.0? 16. over TCP (3) Container logging driver "uentd" Apps write logs to STDOUT: > docker send it to uentd directly! Pros: > simple conf for apps & docker > logs include container logs Cons: > none? app Stay tune! 17. Fluentd docker image Ofcial image by uent organization https://registry.hub.docker.com/u/uent/uentd/ Use it as it is, or build your own container! https://github.com/uent/uentd-docker-image 18.