Elk with Openstack
-
Upload
arun-prasath -
Category
Engineering
-
view
852 -
download
0
Transcript of Elk with Openstack
![Page 1: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/1.jpg)
ELK with Openstack
Arun prasath SJune 16, 2016
![Page 2: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/2.jpg)
Challenges in log analysis
• Multiple services • Multiple servers behind load balancers• Searching the logs (cat, tail, sed, grep, awk)• Finding logs in particular time in multiple servers• Finding fields (Instance ID, name, IP address) in multiple servers and
correlating them• Log analysis , summary, visualization
![Page 3: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/3.jpg)
ELK user operation demo
• Performing a normal search• Filtering based on time, fields• Viewing document data• Viewing field data statistics• Visualize data• Dashboards
![Page 4: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/4.jpg)
ELK Architecture
Logstash forwarder
Servers
Logstash(Indexer)
Storage and search
Dashboard
![Page 5: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/5.jpg)
Broker
• Temporary buffer between logstash agents and central server• Enhance performance by providing caching buffer for log events• Adds resiliency
• Incase the indexing fails, the events are held in queue instead of getting lost
![Page 6: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/6.jpg)
Logstash
• Runs on JVM• Multiple Input / Multiple output• Centralize logs• Collect• Parse• Store / Forward• 3 components – Input, Filter and Output
![Page 7: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/7.jpg)
Logstash conf - /etc/logstash/conf.d
input {}
filter{}
output{}
![Page 8: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/8.jpg)
Logstash - Input
• Input plugin enables a specific source of events to be read by Logstash.
• Some examples of input• Beats• File• Stdin• Eventlog• More here
![Page 9: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/9.jpg)
Logstash - Filter
• A filter plugin performs intermediary processing on an event. Filters are often applied conditionally depending on the characteristics of the event.
• Some examples are• Csv• Date• Grok• Json• More here
![Page 10: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/10.jpg)
Logstash - Output
• An output plugin sends event data to a particular destination.• Some examples are
• Csv• redis• elasticsearch• File• Jira, Nagios, pagerduty• stdout• More here
![Page 11: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/11.jpg)
Logstash - codec
• A codec plugin changes the data representation of an event• Some examples are
• Collectd - Reads events from the collectd binary protocol using UDP• Graphite - Reads graphite formatted lines• Json - Reads JSON formatted content, creating one event per element in a JSON array• Plain - Reads plaintext with no delimiting between events• rubydebug - Applies the Ruby Awesome Print library to Logstash events
• More here
![Page 12: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/12.jpg)
Example - 1 (stdin input / rubydebug output)
input {stdin {}
}filter{}output{stdout { codec => rubydebug }}
![Page 13: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/13.jpg)
Example – 2 (Basic filter on syslog with stdin input and rubydebug output)
input {stdin {}
}filter{grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]add_field => [ "received_from", "%{host}" ]
}syslog_pri { }date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]}
}output{stdout { codec => rubydebug }}
![Page 14: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/14.jpg)
Example – 3 (Openstack filters with file input and multiple output)
input {file {
path => ['/var/log/nova/nova-api-metadata.log']tags => ['nova', 'oslofmt', 'novametaapi']type => "nova"
}…}filter{…else if "novaapi" in [tags] {
if [module] == "nova.osapi_compute.wsgi.server" {mutate {
gsub => ['logmessage',"\"",""]}grok {
match => { "logmessage" => "\[req\-%{NOTSPACE:requestid} %{NOTSPACE:user_id} %{NOTSPACE:tenant}\] %{NOTSPACE:requesterip} %{NOTSPACE:method} %{NOTSPACE:url_path} %{NOTSPACE:http_ver} status\: %{NUMBER:response} len\: %{NUMBER:bytes} time\: %{NUMBER:seconds}" }
add_field => ["api", "nova"]add_tag => ["apimetrics"]
}…}output {
elasticsearch { hosts => ["localhost:9200"] }stdout { codec => rubydebug }
}
Ref: https://github.com/bingoarunprasath/logstash-openstack-filters
![Page 15: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/15.jpg)
Elasticsearch
• Searching and storing of logs• Built on Apache lucene (https://lucene.apache.org/core/)• Massively distributed• High availability• Developer friendly , RESTful API
![Page 16: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/16.jpg)
Kibana
• Dashboard• Provides various options to search data• Creates bar charts, pie charts and various other data visualizations. • Can create custom dashboard and add the saved visualizations. • Simple data export
![Page 17: Elk with Openstack](https://reader034.fdocuments.us/reader034/viewer/2022051320/587706631a28ab890e8b572d/html5/thumbnails/17.jpg)
Installation notes
• Install Java• Install elasticsearch and Kibana
• Install nginx for reverse proxy and basic AUTH• Install logstash, generate SSH certificates
• Configure one output to elasticsearch• Load Kibana dashboard• Setup filebeat / logstash in agent machines and output to logstash• Ansible role - https://galaxy.ansible.com/bingoarunprasath/elk/• Filters - https://github.com/bingoarunprasath/logstash-openstack-
filters