FluentD and DockerVinay Krishna

SolutionsIQ

Monitoring

Is it simple?

• Monitoring

• Applications

• Servers

Logging

What?

• Status of application

• Keeps information about errors/failure

• Status of Network

Why?

• Developers• Get help in Debugging

• IT admin / support • Get help in Trouble-shooting• Apps running smoothly• Security

• Business• Input data – analytics• User interaction / behaviors• Improvements

Assumptions

• I have enough disk space

• I/O operations will not block

• Log messages are human readable

• My logging mechanism scale

• Basically, yeah.. it should work.

Concerns

• Logs increase = data increase

• Message format get more complex

• Did the Kernel flush the buffers ? (sync(2))

• Multi-thread application ?, locking ?

• Multiple Applications = Multiple Logs

• If Multiple Applications = Multiple logs

• Multiple Hosts x Multiple Applications = ???

How to parse/store multiple data sources ?

Fluentd is an open source data collector for unified logging layer.

It allows you to unify data collection and consumption for a better use and understanding of data.

• Structured logging

• Reliable forwarding

• Pluggable architecture

Fluentd

• Data collection for unified logging layer• Streaming data transfer based on JSON

• Written in Ruby

• Gem based various plugins• http://www.fluentd.org/plugins

• Working on lots of productions• http://www/fluentd.org/testimonials

Before

After

Highlights

• Unified Logging Layer• Fluentd tries to structure data as JSON as much as possible

• Simple and yet flexible• 300+ plugins

• Open Source

• Proven Reliability and Performance• 2000+ data-driven companies rely on FluentD• Minimum resources required - vanilla instance runs on 30-40MB of

memory and can process 13,000 events/second/core• Data loss should never happen. • Fluentd supports memory- and file-based buffering to prevent inter-node

data loss. • Fluentd also supports robust failover and can be set up for high availability

• Community

Docker Monitoring

Monitor

• Resource utilization• How much RAM and CPU is each container using?

• Health of docker environments

• As the Docker ecosystem continues to evolve, we have to ask ourselves the following questions:

• How can we log and monitor Docker effectively? • This includes logging the Docker runtime infrastructure, the container itself and

what goes on inside of it, and how to ensure to collect log data from ephemeral containers.

• How can we use feedback from containers to manage and improve the quality of our services?

• Can we build off of decades of experience logging monolithic applications, or do we have to start from scratch?

• If we have to start from scratch, how can we build a solution that helps us make better decisions?

FluentD + Docker

Logging of container architecture

• Storage: • should be outside of container / hosts

• Transferring: • should be over network

• Aggregation: • should be done per container / per service

FluentD Architecture

Logging Driver

• Docker v1.6 released the concept of logging drivers

• Route container output

• Add new logging driver – fluentd• --log-driver=fluentd

• https://github.com/docker/docker/pull/12876

• New for docker v1.7.0?

Container logging driver “fluentd”

• Apps write logs to STDOUT:• docker sends it to fluentd

directly!

• Pros:• simple conf for apps and

docker• logs include container logs

• Cons:• ?

Fluentd docker image

• Official image by fluentd organization

https://registry.hub.docker.com/u/fluent/fluentd/

• Use it as it is, or build your own container!

https://github.com/fluent/fluentd-docker-image

Demo

Install fluentd

• Install fluentd via td-agent

curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh

• Start td-agent

sudo /etc/init.d/td-agent start

Verify installation

• Check the logs to make sure it was installed successfully• tail /var/log/td-agent/td-agent.log

Build fluentd image

• Create a new directory for your Fluentd Docker resources, and move into it• mkdir ~/fluentd-docker && cd ~/fluentd-docker

• Create the following Dockerfile• sudo nano Dockerfile

• Add the following content:

FROM ruby:2.2.0

MAINTAINER kiyoto@treausuredata.com

RUN apt-get update

RUN gem install fluentd -v "~>0.12.3"

RUN mkdir /etc/fluent

RUN apt-get install -y libcurl4-gnutls-dev make

RUN /usr/local/bin/gem install fluent-plugin-elasticsearch

ADD fluent.conf /etc/fluent/

ENTRYPOINT ["/usr/local/bundle/bin/fluentd", "-c", "/etc/fluent/fluent.conf"]

Build fluentd image

• Create a fluent.conf file in the same directory• sudo nano fluent.conf

<source>

type tail

read_from_head true

path /var/lib/docker/containers/*/*-json.log

pos_file /var/log/fluentd-docker.pos

time_format %Y-%m-%dT%H:%M:%S

tag docker.*

format json

</source>

# Using filter to add container IDs to each event

<filter docker.var.lib.docker.containers.*.*.log>

type record_transformer

<record>

container_id ${tag_parts[5]}

</record>

</filter>

<match docker.var.lib.docker.containers.*.*.log>

type elasticsearch

logstash_format true

host "#{ENV['ES_PORT_9200_TCP_ADDR']}" # dynamically configured to use Docker's link feature

port 9200

flush_interval 5s

</match>

• Build docker image• docker build -t fluentd-es .

• Check successfully built the images• docker images

ElasticSearch Container

• Move to home directory• Cd ~

• Download and start the Elasticsearch container• docker run -d -p 9200:9200 -p 9300:9300 --name es

elasticsearch

• Check elasticsearch container is running• docker ps

Start the Fluentd-to-ElasticsearchContainer• Start the container that runs Fluentd, collects the

logs, and sends them to Elastcisearch• docker run -d --link es:es -v

/var/lib/docker/containers:/var/lib/docker/containers fluentd-es

• Check that container is running• docker ps

Confirm ElasticSearch receives events• curl -XGET 'http://localhost:9200/_all/_search?q=*‘

What’s next

• Setup Dashboard

References

• http://www.fluentd.org/guides/recipes/docker-logging

• http://www.slideshare.net/repeatedly/docker-and-fluentd-51821582

• http://www.slideshare.net/tagomoris/docker-and-fluentd-revised

• https://www.socallinuxexpo.org/sites/default/files/presentations/fluentd.pdf

Questions?