Docker and Fluentd

download Docker and Fluentd

of 25

  • date post

    16-Apr-2017
  • Category

    Technology

  • view

    9.670
  • download

    0

Embed Size (px)

Transcript of Docker and Fluentd

  • andDocker meetup Tokyo #5 Aug 19, 2015

  • Who are you?

    Masahiro Nakagawa github: @repeatedly

    Treasure Data Inc. Fluentd / td-agent developer https://jobs.lever.co/treasure-data

    I love OSS :) D Language, MessagePack, The organizer of several meetups, etc

    https://jobs.lever.co/treasure-data

  • Logging for containers

    Storage: should be outside of containers / hosts

    Transferring: should be over network

    Aggregation: should be done per container / per service

  • Real world

  • Container-based world needs unified logging layer

  • Structured logging

    Reliable forwarding

    Pluggable architecturehttp://fluentd.org/

    http://fluentd.org/

  • Whats Fluentd? Data collector for unified logging layer

    Streaming data transfer based on JSON Written in Ruby

    Gem based various plugins http://www.fluentd.org/plugins

    Working on lots of productions http://www.fluentd.org/testimonials

    http://www.fluentd.org/pluginshttp://www.fluentd.org/testimonials

  • Before

  • After!

  • M x N M + NNagios

    MongoDB

    Hadoop

    Alerting

    Amazon S3

    Analysis

    Archiving

    MySQL

    Apache

    Frontend

    Access logs

    syslogd

    App logs

    System logs

    Backend

    Databasesbuffering / processing /

  • Core Plugins Divide & Conquer

    Buffering & Retrying

    Error handling

    Message routing

    Parallelism

    Read data

    Parse data

    Filter data

    Format data

    Write data

  • Architecture

    EngineInput

    Filter Output

    Buffer

    > grep > record_transfomer >

    > Forward > File tail > ...

    > Forward > File > ...

    Output> File > Memory

    not pluggable

    FormatterParser

  • Aggregation patterns1) 1-level aggregation 2) 2-level aggregation

    Pros:

    simple configuration

    Cons:

    fixed aggr, address

    many connections

    high load for aggr

    Pros:

    less connections

    lower load for aggr

    Cons:

    more containers

  • Aggregation patterns1) 1-level aggregation 2) 2-level aggregation

    Apps sends logs over fluent-logger

    or

    There are some patterns - via udp / tcp - tail file - etc

  • 2-level aggr w/ Fluentd

    1. Transfer logs over network

    2. Container logs & tail generated log files

  • 1) Network transferring Apps sends logs:

    using fluent-logger-* over inter-container network

    Pros: no performance penalty

    Cons: docker logs is out of scope a bit complex for logging

    over TCP

    app

  • 2) Tail container logs Apps write logs to

    STDOUT docker writes it to log file fluentd container reads its file

    Pros: simple conf for apps & docker logs include container logs

    Cons: in_tail performance penalty &

    a bit complex configuration

    log file

    app

    http://www.fluentd.org/guides/recipes/docker-logging

    http://www.fluentd.org/guides/recipes/docker-logging

  • Seems good? But we now have smarter way!

  • Logging drivers New from docker v1.6

    We can get docker logs directly via drivers

    fluentd" driver is coming from v1.8 enabled by log-driver=fluentd contributed by @tagomoris

    https://github.com/docker/docker/pull/12876

    https://docs.docker.com/reference/logging/fluentd/

    https://github.com/docker/docker/pull/12876https://docs.docker.com/reference/logging/fluentd/

  • 3) Logging driver Apps write logs to STDOUT

    Docker sends its logs to fluentd directly

    Pros: simple conf for apps & docker Logs includes container logs

    Cons: Do you see any concerns?

    over TCP

    app

  • Demo?

  • Fluentd docker image

    Official image by fluent organization Main maintainer is @tagomoris https://registry.hub.docker.com/u/fluent/fluentd/

    Or build your container! https://github.com/fluent/fluentd-docker-image

    https://registry.hub.docker.com/u/fluent/fluentd/https://github.com/fluent/fluentd-docker-image

  • With Kubernetes

    Kubernetes provides fluentd + Elasticsearch + Kibana image officially https://hub.docker.com/r/kubernetes/fluentd-

    elasticsearch/

    http://www.dasblinkenlichten.com/logging-in-kubernetes-with-fluentd-and-elasticsearch/

    https://hub.docker.com/r/kubernetes/fluentd-elasticsearch/http://www.dasblinkenlichten.com/logging-in-kubernetes-with-fluentd-and-elasticsearch/

  • Cloud service for entire data pipeline!