Docker and Fluentd

2015/06/01 Fluentd meetup 2015 Summer

Satoshi Tagomori (@tagomoris)

@tagomorisTreasure Data, Inc.

Fluentd, Norikra, Hadoop, ...

Announce: fluent-plugin-secure-forward

Security issue: v0.2.x or earlierhttp://tagomoris.tumblr.com/post/120090873793/vulnerability-and-update-of-secure-forward-plugin

http://tagomoris.hatenablog.com/entry/2015/05/28/182245

Please update to v0.3.2!

v0.3.2:

Updated w/ new config param set / code

Incompatible protocol & configuration :(

Which more cute than another?

Docker and Fluentd

Docker:> container platform what you love :D

Fluentd:> logging platform what you love :D

Logging of container architecture

Storage:> should be outside of container / hosts

Transferring:> should be over network

Aggregation:> should be done per container / per service

Aggregation Patterns

(1) 1-level aggregation (2) 2-level aggregation

Aggregation Patterns

(1) 1-level aggregation (2) 2-level aggregation

pros:simple configuration

cons:fixed aggr. addressmany connectionshigh load for aggregator

pros:less connectionslower load for aggr.

cons:a bit more container

Aggregation Patterns

(1) 1-level aggregation (2) 2-level aggregation

Apps sends logs over fluent-logger

there're some patterns...

or ...

Pattern of 2-level aggr. w/ Fluentd

1. Network transferring

2. Container log & tail

(1) Network transferring

Apps sends logs:> using fluent-logger-*> over inter-container network

Pros:> no performance penalty

Cons:> logs of docker is out of scope> a bit complex apps for logging

over TCP

app

(2) Container logger & tail

Apps write logs to STDOUT:> docker writes it to log file> fluentd container read file

Pros:> simple conf for apps & docker> logs include container logs

Cons:> in_tail performance penalty

log file

app

http://www.fluentd.org/guides/recipes/docker-logging

Logging drivers

New from docker v1.6!

'Add new Logging driver "fluentd"'> --log-driver=fluentd

> https://github.com/docker/docker/pull/12876

> New for docker v1.7.0?

over TCP

(3) Container logging driver "fluentd"

Apps write logs to STDOUT:> docker send it to fluentd

directly!

Pros:> simple conf for apps & docker> logs include container logs

Cons:> none?

app

Stay tune!

Fluentd docker image

Official image by fluent organizationhttps://registry.hub.docker.com/u/tagomori/fluentd/

(it should be "fluent/fluentd" ...)

Use it as it is, or build your own container!https://github.com/fluent/fluentd-docker-image

❤