Fluentd - Flexible, Stable, Scalable

FluentdFlexible, Stable, Scalable

Suiting @Taipei.py

ho am I

Suiting (@suitingtseng)

Gogolook Inc.

Data Team

Before

What is Fluentd?

• Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.

• Treasure Data: td-agent

What is a log?

Log definition

Time + Tag + Content

How?

• Lightweight: C + Ruby + MessagePack

• Pluggable architecture

• Built-in Reliability

Input plugins

• forward

• tail

• AWS Simple Queue Service

• AWS CloudWatch

input: tail$ cat /etc/td-‐agent/conf.d

<source>

type tail

path /var/log/nginx/access.log

pos_file /var/log/td-‐agent/httpd-‐access.log.pos

tag nginx.access

</source>

<match nginx.access>

blah blah

</match>

input: forward$ cat /etc/td-‐agent/conf.d

<source>

type forward

port 24224

</source>

<match flask.index>

blah blah

</match>

input: forward$ cat ~/example.py

from fluent import sender

from fluent import event

sender.setup('flask', host='localhost', port=24224)

event.Event("index", {

"user": "foo",

"token": "bar",

"action": "POST"

})

Output plugins

• forward

• copy

• Elasticsearch / MongoDB

• statsd / influxDB / graphite

• S3 / GCS / BigQuery

output: elasticsearch$ cat /etc/td-‐agent/conf.d

<source>

foo bar

tag nginx.access

</source>


type elasticsearch

hosts es-‐host1,es-‐host2

index_name nginx

type_name access

flush_interval 60s

</match>

output: splunk$ cat /etc/td-‐agent/conf.d

<source>

foo bar

tag nginx.access

</source>


type splunk

hosts splunk-‐host1

</match>

Filter plugins

• grok

• grep

• record-modifier / record-reformer

• geoip

Buffer types

• Memory

• File

Buffer example$ cat /etc/td-‐agent/conf.d

<source>

foo bar

tag nginx.access

</source>


type splunk

hosts splunk-‐host1

buffer_chunk_limit 10m

buffer_queue_limit 1000

flush_interval 5m

</match>

Scalability

• Scale up: multi-process plugin

• Scale out: out-forward plugin

App + Fluentd

Fluentd

Elastic search

Elastic search

Elastic search

Elastic search

App + Fluentd

App + Fluentd

Fluentd

Elastic search

Elastic search

Elastic search

Elastic search

Fluentd

App + Fluentd

App + Fluentd

App + Fluentd

Fluentd

Elastic search

Elastic search

Elastic search

Elastic search

Fluentd

FluentdLoad

balance

App + Fluentd

App + Fluentd

App + Fluentd

Auto scaling group

Stability

• Auto retry

• Persistent file buffer

• At-most-once delivery

Message Delivery

• At-most-once: data may be lost

• At-least-once: data may be duplicated

• Exactly-once: perfect

Idempotent

• HTTP PUT

• Maintain a unique id in application level or

• Concatenate (instance-id, time, ….) as id

Gogolook use cases

• MongoDB, nginx log

• API, worker log

• Monitor

• Benchmark

Active users by day

System monitor

Queue monitor

Benchmark?

FluentdApp + Fluentd DB

Benchmark?

FluentdApp + Fluentd DB

Local files

Fluentd - Flexible, Stable, Scalable

Data & Analytics

Transcript of Fluentd - Flexible, Stable, Scalable