Log analysis OpenSource con Logstash, Elasticsearch e Kibana
-
Upload
valentino-gagliardi -
Category
Technology
-
view
5.165 -
download
6
description
Transcript of Log analysis OpenSource con Logstash, Elasticsearch e Kibana
![Page 1: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/1.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Log AnalysisLog Analysis Open Source Open Sourcecon con
LogstashLogstashElasticsearch &Elasticsearch &
KibanaKibana
![Page 2: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/2.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Slides a cura di Slides a cura di Valentino GagliardiValentino Gagliardi
Technical Manager at ServerManaged.itTechnical Manager at ServerManaged.it
Devop & Sysadmin vecchia scuola, consulente informatico Devop & Sysadmin vecchia scuola, consulente informatico per small/medium business, cloud, hosting operations.per small/medium business, cloud, hosting operations.
![Page 3: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/3.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
SommarioSommario●Log analysis, a cosa serve?Log analysis, a cosa serve?●In principio era...In principio era...●Cos'e' LogstashCos'e' Logstash●Cos'e' ElasticsearchCos'e' Elasticsearch●Cos'e' KibanaCos'e' Kibana●The big picture, un setup tipicoThe big picture, un setup tipico●Ma Splunk? E Loggly?Ma Splunk? E Loggly?●Logstash, vantaggiLogstash, vantaggi
![Page 4: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/4.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Log analysisLog analysis::““is an art and science is an art and science
seeking to make sense seeking to make sense out of computer-out of computer-
generated records”generated records”
![Page 5: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/5.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Tradotto:Tradotto:dare un senso ad una dare un senso ad una
montagna di log montagna di log provenienti da server, provenienti da server,
routers, eccrouters, ecc
![Page 6: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/6.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
LogLog::I registri di un sistema.I registri di un sistema.Se c'e' un problema sul Se c'e' un problema sul server Y c'e' anche una server Y c'e' anche una
traccia nei log.traccia nei log.
![Page 7: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/7.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Log analysisLog analysis per: per:- tracciare i problemi- tracciare i problemi
- prevenire incidenti di - prevenire incidenti di sicurezzasicurezza
- ricostruirli se avvengono- ricostruirli se avvengono
![Page 8: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/8.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
In principio In principio era era tailftailf
......
![Page 9: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/9.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
# tailf /var/log/secure
![Page 10: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/10.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
# tailf /var/log/messages # tailf /var/log/secure
![Page 11: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/11.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
# tailf /var/log/messages # tailf /var/log/secure
# multitail /var/log/httpd/error.log /var/log/httpd/access.log
![Page 12: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/12.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Oggi e'Oggi e'data data
visualizationvisualization
![Page 13: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/13.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
![Page 14: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/14.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Cos'e' Cos'e' LogstashLogstash??““Logstash helps you take Logstash helps you take logs and other event data logs and other event data
from your systems and from your systems and store them in a central store them in a central
place. “place. “
![Page 15: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/15.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
LogstashLogstash::trasforma qualsiasi fonte trasforma qualsiasi fonte di di eventieventi e e loglog in qualcosa in qualcosa
di digeribile e di digeribile e processabileprocessabile
![Page 16: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/16.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
LogstashLogstash::36 inputs (and growing)36 inputs (and growing)14 codecs (and growing)14 codecs (and growing)40 filtri (and growing)40 filtri (and growing)
50 outputs (and growing)50 outputs (and growing)
![Page 17: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/17.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
## Una configurazione minimale di Logstashinput { file { type => "linux-syslog" path => ["/var/log/*.log"] exclude => [ "*.gz" ] }}output { redis { host => "127.0.0.1" data_type => "list" key => "syslog" }}##
![Page 18: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/18.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Cos'e' Cos'e' RedisRedis??““Redis is an open source, Redis is an open source, BSD licensed, advanced BSD licensed, advanced
key-value store.“key-value store.“
![Page 19: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/19.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
RedisRedis::in un sistema di in un sistema di logging logging
centralizzatocentralizzato puo' essere puo' essere usato come usato come bufferbuffer per i per i
loglog
![Page 20: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/20.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Cos'e' Cos'e' ElasticsearchElasticsearch??““flexible and powerful flexible and powerful
open source, distributed open source, distributed real-timereal-time
search and analytics engine search and analytics engine for the cloud“for the cloud“
![Page 21: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/21.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
ElasticsearchElasticsearch::in un sistema di logging in un sistema di logging
centralizzato puo' essere centralizzato puo' essere usato come output per usato come output per
indicizzare i logindicizzare i log
![Page 22: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/22.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Cos'e' Cos'e' KibanaKibana??““Make Sense of your Make Sense of your
Data“Data“
![Page 23: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/23.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
KibanaKibana::una dashboard per una dashboard per
estrarre i dati da estrarre i dati da ElasticsearchElasticsearch
![Page 24: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/24.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
![Page 25: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/25.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Big picture: Big picture: logging logging
centralizzatocentralizzato
![Page 26: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/26.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Un setup tipico per il Un setup tipico per il logging centralizzato logging centralizzato
con Rsyslog, Logstash, con Rsyslog, Logstash, Redis, Elasticsearch e Redis, Elasticsearch e
Kibana.Kibana.
![Page 27: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/27.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
server
server
server
server
Server di Logging Centralizzato
rsyslog
![Page 28: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/28.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
server
server
server
server
server
server
Server di Logging Centralizzato
rsyslog
![Page 29: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/29.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
server
server
server
server
server
server
Server di Logging Centralizzato
rsyslog
![Page 30: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/30.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
server
server
server
server
server
server
Server di Logging Centralizzato
rsyslog
![Page 31: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/31.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
server
server
server
server
server
server
Server di Logging Centralizzato
rsyslog
![Page 32: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/32.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
server
server
server
server
server
server
Server di Logging Centralizzato
rsyslog
![Page 33: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/33.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Ok tutto bello. “Ma Ok tutto bello. “Ma cosa te ne fai di questi cosa te ne fai di questi grafici?” (tratto da una grafici?” (tratto da una
domanda vera)domanda vera)
![Page 34: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/34.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Centralizzare i logCentralizzare i log::- visibilita' dei trend- visibilita' dei trend
- visibilita' dei problemi- visibilita' dei problemi- analisi di sicurezza- analisi di sicurezza
![Page 35: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/35.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Caso di studioCaso di studio..Mitigazione di una Mitigazione di una
serie di potenti serie di potenti attacchi bruteforceattacchi bruteforce
![Page 36: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/36.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
In rosso: attacco bruteforce massivo su siti web JoomlaIn verde: mitigazione dell'attacco. Le richieste anomale vengono respinte
Visualizzare le conseguenze di un attacco, anticipare il trend e mitigare la minaccia.
![Page 37: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/37.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Non solo Logstash:Non solo Logstash:Le alternative costose Le alternative costose
al logging al logging centralizzato open centralizzato open
source.source.
![Page 38: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/38.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Splunk>Splunk>Grab a 20GB license for Grab a 20GB license for 12187631461319$/month12187631461319$/month
(gratuito fino a (gratuito fino a 500MB/giorno)500MB/giorno)
![Page 39: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/39.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Splunk>Splunk>
![Page 40: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/40.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
LogglyLoggly10GB di log al giorno 10GB di log al giorno con una ritenzione di con una ritenzione di
90 giorni costano circa 90 giorni costano circa 1482 dollari al mese.1482 dollari al mese.
![Page 41: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/41.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Ogni prodotto ha pro e Ogni prodotto ha pro e contro. Splunk e contro. Splunk e
Loggly: compliance e Loggly: compliance e immediatezza. immediatezza.
Logstash: per i nerd.Logstash: per i nerd.
![Page 42: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/42.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Logstash: Logstash: vantaggivantaggi- open source- open source
- gratuito- gratuito- una folta community- una folta community- in continuo sviluppo- in continuo sviluppo
![Page 43: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/43.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
KEEP CALMKEEP CALMANDAND
LOGSTASHLOGSTASH
![Page 44: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/44.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
http://www.logstash.nethttp://www.logstash.nethttp://www.redis.iohttp://www.redis.io
http://www.elasticsearch.orghttp://www.elasticsearch.org
![Page 45: Log analysis OpenSource con Logstash, Elasticsearch e Kibana](https://reader031.fdocuments.us/reader031/viewer/2022013111/554f7a52b4c905435d8b46ab/html5/thumbnails/45.jpg)
#servertraining
twitter.com/servermanageditwww.servermanaged.it
Slides a cura di Slides a cura di Valentino GagliardiValentino Gagliardi
Technical Manager at ServerManaged.itTechnical Manager at ServerManaged.it
Devop & Sysadmin vecchia scuola, consulente informatico Devop & Sysadmin vecchia scuola, consulente informatico per small/medium business, cloud, hosting operations.per small/medium business, cloud, hosting operations.
(Vieni a trovarmi su Google+, LinkedIn e Twitter)(Vieni a trovarmi su Google+, LinkedIn e Twitter)
Immagine di sfondo: http://medialoot.com/item/free-dark-noise-backgroundsImmagine di sfondo: http://medialoot.com/item/free-dark-noise-backgrounds