Shipping & Visualize Your Data With ELK

Adam Chen

HELLO Everyone!I am AdamI am just an engineer like to share some experience with others.Thanks Gentoo let me familiar with Linux.

You can find me at @adaam

Let’s Start This Session

ELK? What !?

Develop by

Elastic Family

ELK ?DevOps ?

Log Still Play An Important Role

Today’s characters

Logstash/Beat seriesShipping all of your log to where it should go, like ES, AWS, or just text.

ElasticsearchThe main part to store your data with High Availability.

KibanaVisualize will power your data. To know more about its value.

Traditional Way to Collect Log

When error happened, administrator or RD/QA will need to login or write/use tool to grab log from each machines then analysis.

Hey Bob, Please collect the error log to analyze.

OK, Boss.

BOSS

Hey Bob, Please collect the error log to analyze.

Traditional Way to Collect Log

Old Way to Collect Log

How ELK Help ?

Centralize Log To One Place

Collect Log using ELK

Introduce The E, The L and The K

Logstash

An agent install at where log need to be collect.

Have much filter to process your log.Also Input/Output module.

Logstash Module

Logstash Filters

Date, geoip, json, kv ...etc

GROK !!

Grok online tester

Logstash DEMO

Input apache/nginx log then output to stdout, using local logstash.

Elasticsearch

SImply a data store with near real time search

Store data in index, can by hours, day, week, month. Setting at Logstash.

Machine spec depends on data size.

Elasticsearch Modules

They are improve ES function if you need.

Watcher, Shield, Marvel, Cloud-AWS

Elasticsearch DEMO

API Webpage

Kibana

Show your data to you understand.But you need to know your data fields first.

Kibana Demo

Create some template to show (data pregen at ES? )

Real time insert data to ES and see from template pregen. (apache/nginx?)

What else ?

Push Metrics to elsewhere

THANKS!Any questions?You can find me at:@adaam