Fluentd and docker monitoring
-
Upload
vinay-krishna -
Category
Technology
-
view
3.733 -
download
2
Transcript of Fluentd and docker monitoring
FluentD and DockerVinay Krishna
SolutionsIQ
Monitoring
Is it simple?
• Monitoring
• Applications
• Servers
Logging
What?
• Status of application
• Keeps information about errors/failure
• Status of Network
Why?
• Developers• Get help in Debugging
• IT admin / support • Get help in Trouble-shooting• Apps running smoothly• Security
• Business• Input data – analytics• User interaction / behaviors• Improvements
Assumptions
• I have enough disk space
• I/O operations will not block
• Log messages are human readable
• My logging mechanism scale
• Basically, yeah.. it should work.
Concerns
• Logs increase = data increase
• Message format get more complex
• Did the Kernel flush the buffers ? (sync(2))
• Multi-thread application ?, locking ?
• Multiple Applications = Multiple Logs
• If Multiple Applications = Multiple logs
• Multiple Hosts x Multiple Applications = ???
How to parse/store multiple data sources ?
Fluentd is an open source data collector for unified logging layer.
It allows you to unify data collection and consumption for a better use and understanding of data.
• Structured logging
• Reliable forwarding
• Pluggable architecture
Fluentd
• Data collection for unified logging layer• Streaming data transfer based on JSON
• Written in Ruby
• Gem based various plugins• http://www.fluentd.org/plugins
• Working on lots of productions• http://www/fluentd.org/testimonials
Before
After
Highlights
• Unified Logging Layer• Fluentd tries to structure data as JSON as much as possible
• Simple and yet flexible• 300+ plugins
• Open Source
• Proven Reliability and Performance• 2000+ data-driven companies rely on FluentD• Minimum resources required - vanilla instance runs on 30-40MB of
memory and can process 13,000 events/second/core• Data loss should never happen. • Fluentd supports memory- and file-based buffering to prevent inter-node
data loss. • Fluentd also supports robust failover and can be set up for high availability
• Community
Docker Monitoring
Monitor
• Resource utilization• How much RAM and CPU is each container using?
• Health of docker environments
• As the Docker ecosystem continues to evolve, we have to ask ourselves the following questions:
• How can we log and monitor Docker effectively? • This includes logging the Docker runtime infrastructure, the container itself and
what goes on inside of it, and how to ensure to collect log data from ephemeral containers.
• How can we use feedback from containers to manage and improve the quality of our services?
• Can we build off of decades of experience logging monolithic applications, or do we have to start from scratch?
• If we have to start from scratch, how can we build a solution that helps us make better decisions?
FluentD + Docker
Logging of container architecture
• Storage: • should be outside of container / hosts
• Transferring: • should be over network
• Aggregation: • should be done per container / per service
FluentD Architecture
Logging Driver
• Docker v1.6 released the concept of logging drivers
• Route container output
• Add new logging driver – fluentd• --log-driver=fluentd
• https://github.com/docker/docker/pull/12876
• New for docker v1.7.0?
Container logging driver “fluentd”
• Apps write logs to STDOUT:• docker sends it to fluentd
directly!
• Pros:• simple conf for apps and
docker• logs include container logs
• Cons:• ?
Fluentd docker image
• Official image by fluentd organization
https://registry.hub.docker.com/u/fluent/fluentd/
• Use it as it is, or build your own container!
https://github.com/fluent/fluentd-docker-image
Demo
Install fluentd
• Install fluentd via td-agent
curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
• Start td-agent
sudo /etc/init.d/td-agent start
Verify installation
• Check the logs to make sure it was installed successfully• tail /var/log/td-agent/td-agent.log
Build fluentd image
• Create a new directory for your Fluentd Docker resources, and move into it• mkdir ~/fluentd-docker && cd ~/fluentd-docker
• Create the following Dockerfile• sudo nano Dockerfile
• Add the following content:
FROM ruby:2.2.0
MAINTAINER [email protected]
RUN apt-get update
RUN gem install fluentd -v "~>0.12.3"
RUN mkdir /etc/fluent
RUN apt-get install -y libcurl4-gnutls-dev make
RUN /usr/local/bin/gem install fluent-plugin-elasticsearch
ADD fluent.conf /etc/fluent/
ENTRYPOINT ["/usr/local/bundle/bin/fluentd", "-c", "/etc/fluent/fluent.conf"]
Build fluentd image
• Create a fluent.conf file in the same directory• sudo nano fluent.conf
<source>
type tail
read_from_head true
path /var/lib/docker/containers/*/*-json.log
pos_file /var/log/fluentd-docker.pos
time_format %Y-%m-%dT%H:%M:%S
tag docker.*
format json
</source>
# Using filter to add container IDs to each event
<filter docker.var.lib.docker.containers.*.*.log>
type record_transformer
<record>
container_id ${tag_parts[5]}
</record>
</filter>
<match docker.var.lib.docker.containers.*.*.log>
type elasticsearch
logstash_format true
host "#{ENV['ES_PORT_9200_TCP_ADDR']}" # dynamically configured to use Docker's link feature
port 9200
flush_interval 5s
</match>
• Build docker image• docker build -t fluentd-es .
• Check successfully built the images• docker images
ElasticSearch Container
• Move to home directory• Cd ~
• Download and start the Elasticsearch container• docker run -d -p 9200:9200 -p 9300:9300 --name es
elasticsearch
• Check elasticsearch container is running• docker ps
Start the Fluentd-to-ElasticsearchContainer• Start the container that runs Fluentd, collects the
logs, and sends them to Elastcisearch• docker run -d --link es:es -v
/var/lib/docker/containers:/var/lib/docker/containers fluentd-es
• Check that container is running• docker ps
Confirm ElasticSearch receives events• curl -XGET 'http://localhost:9200/_all/_search?q=*‘
What’s next
• Setup Dashboard
References
• http://www.fluentd.org/guides/recipes/docker-logging
• http://www.slideshare.net/repeatedly/docker-and-fluentd-51821582
• http://www.slideshare.net/tagomoris/docker-and-fluentd-revised
• https://www.socallinuxexpo.org/sites/default/files/presentations/fluentd.pdf
Questions?