Download - Docker and Fluentd

Transcript
Page 1: Docker and Fluentd

andDocker meetup Tokyo #5 Aug 19, 2015

Page 2: Docker and Fluentd

Who are you?

• Masahiro Nakagawa • github: @repeatedly

• Treasure Data Inc. • Fluentd / td-agent developer • https://jobs.lever.co/treasure-data

• I love OSS :) • D Language, MessagePack, The organizer of several meetups, etc…

Page 3: Docker and Fluentd

Logging for containers

• Storage: • should be outside of containers / hosts

• Transferring: • should be over network

• Aggregation: • should be done per container / per service

Page 4: Docker and Fluentd

Real world…

Page 5: Docker and Fluentd

Container-based world needs unified logging layer

Page 6: Docker and Fluentd

Structured logging

Reliable forwarding

Pluggable architecturehttp://fluentd.org/

Page 7: Docker and Fluentd

What’s Fluentd?• Data collector for unified logging layer

• Streaming data transfer based on JSON • Written in Ruby

• Gem based various plugins • http://www.fluentd.org/plugins

• Working on lots of productions • http://www.fluentd.org/testimonials

Page 8: Docker and Fluentd

Before…

Page 9: Docker and Fluentd

After!

Page 10: Docker and Fluentd

M x N → M + NNagios

MongoDB

Hadoop

Alerting

Amazon S3

Analysis

Archiving

MySQL

Apache

Frontend

Access logs

syslogd

App logs

System logs

Backend

Databasesbuffering / processing /

Page 11: Docker and Fluentd

Core Plugins• Divide & Conquer

• Buffering & Retrying

• Error handling

• Message routing

• Parallelism

• Read data

• Parse data

• Filter data

• Format data

• Write data

Page 12: Docker and Fluentd

Architecture

EngineInput

Filter Output

Buffer

> grep > record_transfomer > …

> Forward > File tail > ...

> Forward > File > ...

Output> File > Memory

not pluggable

FormatterParser

Page 13: Docker and Fluentd

Aggregation patterns1) 1-level aggregation 2) 2-level aggregation

Pros:

• simple configuration

Cons:

• fixed aggr, address

• many connections

• high load for aggr

Pros:

• less connections

• lower load for aggr

Cons:

• more containers

Page 14: Docker and Fluentd

Aggregation patterns1) 1-level aggregation 2) 2-level aggregation

Apps sends logs over fluent-logger

or …

There are some patterns - via udp / tcp - tail file - etc…

Page 15: Docker and Fluentd

2-level aggr w/ Fluentd

1. Transfer logs over network

2. Container logs & tail generated log files

Page 16: Docker and Fluentd

1) Network transferring• Apps sends logs:

• using fluent-logger-* • over inter-container network

• Pros: • no performance penalty

• Cons: • docker logs is out of scope • a bit complex for logging

over TCP

app

Page 17: Docker and Fluentd

2) Tail container logs• Apps write logs to

STDOUT • docker writes it to log file • fluentd container reads its file

• Pros: • simple conf for apps & docker • logs include container logs

• Cons: • in_tail performance penalty &

a bit complex configuration

log file

app

http://www.fluentd.org/guides/recipes/docker-logging

Page 18: Docker and Fluentd

Seems good? But we now have smarter way!

Page 19: Docker and Fluentd

Logging drivers• New from docker v1.6

• We can get docker logs directly via drivers

• “fluentd" driver is coming from v1.8 • enabled by “—log-driver=fluentd” • contributed by @tagomoris

https://github.com/docker/docker/pull/12876

• https://docs.docker.com/reference/logging/fluentd/

Page 20: Docker and Fluentd

3) Logging driver• Apps write logs to STDOUT

• Docker sends its logs to fluentd directly

• Pros: • simple conf for apps & docker • Logs includes container logs

• Cons: • Do you see any concerns?

over TCP

app

Page 21: Docker and Fluentd
Page 22: Docker and Fluentd

Demo?

Page 23: Docker and Fluentd

Fluentd docker image

• Official image by fluent organization • Main maintainer is @tagomoris • https://registry.hub.docker.com/u/fluent/fluentd/

• Or build your container! • https://github.com/fluent/fluentd-docker-image

Page 24: Docker and Fluentd

With Kubernetes

• Kubernetes provides fluentd + Elasticsearch + Kibana image officially • https://hub.docker.com/r/kubernetes/fluentd-

elasticsearch/

• http://www.dasblinkenlichten.com/logging-in-kubernetes-with-fluentd-and-elasticsearch/

Page 25: Docker and Fluentd

Cloud service for entire data pipeline!