Docker and Fluentd
2015/06/04 Golang & Docker users meetup Satoshi Tagomori (@tagomoris)
@tagomorisTreasure Data, Inc.
Fluentd, Norikra, Hadoop, ...
Which more cute than another?
Logging of container architecture
Storage:> should be outside of container / hosts
Transferring:> should be over network
Aggregation:> should be done per container / per service
Container-based worldneeds unified logging layer!
Structured logging
Reliable forwarding
Pluggable architecture
http://fluentd.org/
What’s Fluentd?
> Data collector for unified logging layer> Streaming data transfer based on JSON> Written in Ruby
> Gem based various plugins> http://www.fluentd.org/plugins
> Working in production> http://www.fluentd.org/testimonials
Architecture
EngineInput
Filter Output
Buffer
> grep > record_transfomer> …
> Forward> File tail> ...
> Forward> File> ...
Output
> File> Memory
not pluggable
FormatterParser
Aggregation Patterns
(1) 1-level aggregation (2) 2-level aggregation
Aggregation Patterns
(1) 1-level aggregation (2) 2-level aggregation
pros:simple configuration
cons:fixed aggr. addressmany connectionshigh load for aggregator
pros:less connectionslower load for aggr.
cons:a bit more container
Aggregation Patterns
(1) 1-level aggregation (2) 2-level aggregation
Apps sends logs over fluent-logger
there're some patterns...
or ...
Pattern of 2-level aggr. w/ Fluentd
1. Network transferring
2. Container log & tail
(1) Network transferring
Apps sends logs:> using fluent-logger-*> over inter-container network
Pros:> no performance penalty
Cons:> logs of docker is out of scope> a bit complex apps for logging
over TCP
app
(2) Container logger & tail
Apps write logs to STDOUT:> docker writes it to log file> fluentd container reads file
Pros:> simple conf for apps & docker> logs include container logs
Cons:> in_tail performance penalty
log file
app
http://www.fluentd.org/guides/recipes/docker-logging
Logging drivers
New from docker v1.6!
'Add new Logging driver "fluentd"'> --log-driver=fluentd
> https://github.com/docker/docker/pull/12876
> New for docker v1.7.0?
over TCP
(3) Container logging driver "fluentd"
Apps write logs to STDOUT:> docker send it to fluentd
directly!
Pros:> simple conf for apps & docker> logs include container logs
Cons:> none?
app
Stay tune!
Fluentd docker image
Official image by fluent organizationhttps://registry.hub.docker.com/u/fluent/fluentd/
Use it as it is, or build your own container!https://github.com/fluent/fluentd-docker-image
❤
Top Related