CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd...

25
Matthias Runge CentOS Opstools SIG - Update February 1, 2019 1

Transcript of CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd...

Page 1: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

Matthias Runge

CentOS Opstools SIG - Update

February 1, 2019

1

Page 2: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update

● Intro● What is included● A closer look at performance monitoring● Taking the next step

Agenda

2

Page 3: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

INTRODUCTION

3

Page 4: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update4

● Intended to support operators in second day tasks● a piece in the puzzle for RHOSP

Opstools - tools for operators

Page 5: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update5

Community and products

**

Page 6: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update6

Relationship with community and OSP repos

collectdOpenStack

CentOS opstools

CentOS MessagingRDO

QPID

Cloud SIG

Red Hat AMQ

oVirt

OSP OSP opstools

RHV

EPEL

Page 7: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CENTRALIZED LOGGING

7

Page 8: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update8

Centralized logging

fluentd fluentd

fluentd

fluentd

elasticsearch

Page 9: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

AVAILABILITY MONITORING

9

Page 10: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update10

Availability monitoring - Sensu

Page 11: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

PERFORMANCE MONITORING

11

Page 12: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update

● collectd● Graphite/CarbonDB● Grafana● Gnocchi

12

Performance monitoring (past)

collectd

collectd

collectd

graphite

GrafanaMonitoring node

Page 13: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update

● collectd 5.8.0, released Nov 17, 2017● release 5.8.1 in preparation● small resource usage● around since 2005● about 80 plugins, reporting mostly low level system information

○ bindings for Python, Java, exec, Perl● generates both metrics and events

13

collectd

Page 14: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update14

Performance monitoring (past) - HA

collectd

collectd

collectd

graphite

GrafanaMonitoring node

Page 15: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update

Kubernetes is a portable, extensible open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation.

15

Kubernetes

Page 16: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update16

Prometheus

Page 17: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update

● Decouple Prometheus instances from configuration

17

Prometheus operator

Page 18: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update

● QPID proton● dispatch router● qpid cpp broker

18

Apache QPID

qdr qdr qdrqdr

producer consumer

broker broker

Page 19: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update

infra node

19

Service assurance framework

collectd

qdr

qdr

qdr

qdr

Prometheus

grafana

telemetry consumer

kubernetes cluster

broker broker

Page 20: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update20

Grafana

Page 21: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update21

● Alarming rules defined in prometheus● alarm actions executed by alertmanager● - name: openstack-api slow

rules: - alert: openstack-api expr: avg_over_time(collectd_curl_response_time[30s]) > 0.2 for: 3m labels: severity: pager annotations: summary: openstack api slow

Alarming

Page 22: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

OUTLOOK

22

Page 23: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

CentOS Opstools SIG Update

● integration of “self healing” parts?● adding tools like vitrage, or doctor?● deprecating sensu in favor of alertmanager

23

Outlook

Page 24: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

Questions?

[email protected]

24

Page 25: CentOS Opstools SIG - Update Update FOSDEM 2019.pdf · Centralized logging fluentd fluentd fluentd fluentd elasticsearch. AVAILABILITY MONITORING 9. 10 CentOS Opstools SIG Update

THANK YOU!

plus.google.com/+RedHat

linkedin.com/company/red-hat

youtube.com/user/RedHatVideos

facebook.com/redhatinc

twitter.com/RedHatNews

25