Elasticsearch logstash kibana meetup
-
Upload
bharvi-dixit -
Category
Data & Analytics
-
view
293 -
download
8
Transcript of Elasticsearch logstash kibana meetup
![Page 1: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/1.jpg)
Who Am I
![Page 2: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/2.jpg)
••••
![Page 3: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/3.jpg)
![Page 4: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/4.jpg)
![Page 5: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/5.jpg)
•
•
•
•
•
••••
![Page 6: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/6.jpg)
••••••••
•••••
![Page 7: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/7.jpg)
•••
•••••••
RAM, CPU type/cores, DISK, Networks matter a lot, But the cluster design, data structures of documents and queries has a huge impact on the Elasticsearch Clusters and your search experiences.
I have experienced it a lot. You can check out some of the scenarios here : https://www.found.no/foundation/crash-elasticsearch/
![Page 8: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/8.jpg)
![Page 10: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/10.jpg)
![Page 11: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/11.jpg)
•
•
••
![Page 12: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/12.jpg)
![Page 13: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/13.jpg)
•
•
•
•
•
![Page 14: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/14.jpg)
••
•••••
![Page 15: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/15.jpg)
•••
••••
•••
•••
![Page 16: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/16.jpg)
••
••••
•••
•••
![Page 17: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/17.jpg)
![Page 18: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/18.jpg)
![Page 19: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/19.jpg)
Centralize
Analyze Monitor
Share
![Page 20: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/20.jpg)
• •
•
![Page 21: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/21.jpg)
•••
•••
![Page 22: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/22.jpg)
#!/bin/bash
sudo apt-get purge openjdk-\*
wget --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz
tar -xvf jdk-7u79-linux-x64.tar.gz
sudo mkdir -p /usr/lib/jvm
sudo mv ./jdk1.7.0_79 /usr/lib/jvm/
sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.7.0_79/bin/java" 1
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.7.0_79/bin/javac" 1
sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/lib/jvm/jdk1.7.0_79/bin/javaws" 1
sudo chmod a+x /usr/bin/java
sudo chmod a+x /usr/bin/javac
sudo chmod a+x /usr/bin/javaws
sudo chown -R root:root /usr/lib/jvm/jdk1.7.0_79
sudo update-alternatives --config java
##########Skip above if you have java already available.
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.4.deb
sudo dpkg -i elasticsearch-1.4.4.deb
sudo update-rc.d elasticsearch defaults 95 10
![Page 23: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/23.jpg)
script.disable_dynamic: false
cluster.name: give_your_cluster_name
node.name:"es-master-3"
node.master: true
node.data: false
node.max_local_storage_nodes: 1
index.number_of_shards: 4
index.number_of_replicas: 1
bootstrap.mlockall: true
transport.tcp.port: 9300
http.enabled:false
discovery.zen.minimum_master_nodes:2
discovery.zen.ping.timeout:10s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts:["es-master-1:9300","es-master-2:9300", "es-master-3:9300"]
action.disable_delete_all_indices: false
action.destructive_requires_name: true
marvel.agent.exporter.es.hosts: ['es-monitor-1:6200']
/etc/elasticsearch/elasticsearch.yml
Configuration for Master Node.
Give half of total available RAM to ES:vim /etc/init.d/elasticsearchES_HEAP_SIZE=2g ( on a 4 GB RAM, the more heap size the better performance, But need to keep sweet spot of 32 GB in mind)
![Page 24: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/24.jpg)
…
![Page 25: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/25.jpg)
![Page 26: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/26.jpg)
![Page 27: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/27.jpg)
![Page 28: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/28.jpg)
![Page 29: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/29.jpg)
ShipperLogstash
ShipperLogstash
ShipperLogstash
BrokerRedis/RabbitMQ
IndexerLogstash Elasticsearch
Using Logstash Shippers and Redis or RabiitMQ as a broker
![Page 30: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/30.jpg)
img source: https://deviantony.wordpress.com/2014/05/19/centralized-logging-with-an-elk-stack-elasticsearch-logback-kibana/
Using Logstash Forwarder : Lumberjack Protocol
![Page 32: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/32.jpg)
Don’t want to use any agent/shipper for system events logging??
Use Rsyslog:
1. Send logs directly to Elasticsearch: omelasticsearch http://www.rsyslog.com/doc/v8-stable/configuration/modules/omelasticsearch.html
2. Use base logs sending : Create and edit the file /etc/rsyslog.d/logstash.conf with
*.* @logserver.example.com:5544 and parse the logs on centralized logstash server.
![Page 33: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/33.jpg)
Rivers are finally removed:https://github.com/elastic/elasticsearch/pull/11568#event-332821650
Want to get a deep dive into Elasticsearch??
Join us @ 3rd Delhi Elasticsearch Meetup
http://www.meetup.com/Delhi-Elasticsearch-Meetup/events/223470631/
![Page 34: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/34.jpg)
![Page 35: Elasticsearch logstash kibana meetup](https://reader031.fdocuments.us/reader031/viewer/2022013118/55c2bfc3bb61ebca2d8b47f5/html5/thumbnails/35.jpg)