Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total...
Transcript of Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total...
![Page 1: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/1.jpg)
Gabriel Ciciliani - Fernando Ipar
Taking PMM to the next level
![Page 2: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/2.jpg)
AGENDA
2© The Pythian Group Inc., 2017
PMM in 2 minutes
Production notes
Beyond what's in the box
QA
![Page 3: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/3.jpg)
PMM in 2 minutes
![Page 4: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/4.jpg)
4© The Pythian Group Inc., 2017https://www.percona.com/doc/percona-monitoring-and-management/architecture.html
![Page 5: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/5.jpg)
© The Pythian Group Inc., 2017 5
PMM Server
Distributed as:● A Docker image
● An Open Virtual Appliance (OVA) package
● An Amazon Machine Image (AMI)
![Page 6: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/6.jpg)
© The Pythian Group Inc., 2017 6
PMM Client
Distributed as:● An RPM, deb package
● Generic Linux binaries
![Page 7: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/7.jpg)
© The Pythian Group Inc., 2017 7
Quick Start Reference
1. Create pmm-data container (lives "forever")
2. Create pmm-server container (destroyed on every upgrade)
3. Install at least one agent
4. Point agent to pmm-server
5. Deploy at least one service to monitor
https://www.percona.com/doc/percona-monitoring-and-management/deploy/server/docker.setting-up.html
https://www.percona.com/doc/percona-monitoring-and-management/deploy/index.html#installing-clients
![Page 8: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/8.jpg)
© The Pythian Group Inc., 2017 8
# docker pull percona/pmm-server:latest
# docker create \
-v /opt/prometheus/data \
-v /opt/consul-data \
-v /var/lib/mysql \
-v /var/lib/grafana \
--name pmm-data \
percona/pmm-server:latest /bin/true
Quick Start Reference: Step 1
![Page 9: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/9.jpg)
© The Pythian Group Inc., 2017 9
# docker run -d \
-p 80:80 \
--volumes-from pmm-data \
--name pmm-server \
--restart always \
percona/pmm-server:latest
Quick Start Reference: Step 2
![Page 10: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/10.jpg)
© The Pythian Group Inc., 2017 10
Quick start reference : Steps 3 through 5
# yum install pmm-client
# pmm-admin config --server <pmm-server-host>:<pmm-server-port>
# pmm-admin add mysql
![Page 11: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/11.jpg)
Production notes
Running PMM in production
![Page 12: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/12.jpg)
© The Pythian Group Inc., 2017 12
Production notes
● Store the pmm-data volumes on an dedicated volume
● Adjust total and Prometheus (if possible) memory on shared hosts
● Adjust metrics and queries retention
● Adjust metrics resolution if needed
● Enable authentication
● Backups
● Capacity planning
![Page 13: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/13.jpg)
© The Pythian Group Inc., 2017 13
Production notes: pmm-data
# docker create \
-v /pmmdata/prometheus/data:/opt/prometheus/data \
-v /pmmdata/consul-data:/opt/consul-data \
-v /pmmdata/mysql:/var/lib/mysql \
-v /pmmdata/grafana:/var/lib/grafana \
--name pmm-data \
percona/pmm-server:latest /bin/true
There are gotchas:
● When using docker.io (ie. Centos < 7) bind mounts will hide any image existing files in the mount point.
● On Centos, file privileges and owners set from within the container are visible from the host OS and may overlap with existing
![Page 14: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/14.jpg)
© The Pythian Group Inc., 2017 14
Production notes: adjusting memory
# docker run --memory=4G -e METRICS_MEMORY=209152 ...
● METRICS_MEMORY only works on Prometheus v1 (PMM < 1.13)
● Set it to 2/3 of the total memory assigned
![Page 15: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/15.jpg)
© The Pythian Group Inc., 2017 15
Production notes: adjusting retention
# docker run -e METRICS_RETENTION=4400h -e QUERIES_RETENTION=4400h ...
● METRICS_RETENTION sets storage.local.retention for Prometheus 1.x (PMM < 1.13) or storage.tsdb.retention for Prometheus 2.x.
● QUERIES_RETENTION is used as argument for the purge-qan-data script.
![Page 16: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/16.jpg)
© The Pythian Group Inc., 2017 16
Production notes: metrics resolution
# docker run -e METRICS_RESOLUTION=5 ...
● Can be used in cases where latency is above 1 second
![Page 17: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/17.jpg)
© The Pythian Group Inc., 2017 17
Production notes: enabling authentication
# docker run -e SERVER_USER=<username> -e SERVER_PASSWORD=<password> …
# pmm-admin config --server 127.0.0.1 --server-user <username>
--server-password <password>
OK, PMM server is alive.
PMM Server | 127.0.0.1 (password-protected)
Client Name | coltrane
Client Address | 172.17.0.1
![Page 18: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/18.jpg)
© The Pythian Group Inc., 2017 18
Production notes: backups
● pmm-server container is ephemeral by design
● Cold backup is the official method
● pmm-data should be backed up while pmm-server is not running
https://www.percona.com/doc/percona-monitoring-and-management/deploy/server/docker.backing-up.html
![Page 19: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/19.jpg)
© The Pythian Group Inc., 2017 19
As of v1.13:
● At least 2Gb of memory for a production environment (up 5 monitored target
systems)
● From there, consider 8 target systems per Gb of memory available for
pmm-server
● CPU: ~50 monitored systems per core (or 100K metrics/sec per CPU core)
● ~1 GB of storage for each monitored database node (data retention=1w)
https://www.percona.com/blog/2018/09/28/scaling-percona-monitoring-management-pmm/
Production notes: capacity planning
![Page 20: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/20.jpg)
© The Pythian Group Inc., 2017 20
Monitoring PMM resources
● System Overview dashboard, choosing container host or container name
● If container name chosen, memory limits not reflected (full host memory
will be displayed)
● Visible with docker stats pmm-server
● Keep an eye on:
● CPU Saturation and Max Core Usage
● Memory Utilization
● Disk IO Loadhttps://www.percona.com/blog/2018/09/28/scaling-percona-monitoring-management-pmm/
Production notes: capacity planning
![Page 21: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/21.jpg)
Beyond what's in the box
![Page 22: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/22.jpg)
22© The Pythian Group Inc., 2017
Beyond what's in the box: Application Instrumentation 1
![Page 23: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/23.jpg)
© The Pythian Group Inc., 2017 23
Beyond what's in the box: Application Instrumentation 1
$ cat /usr/local/percona/pmm-client/queries-mysqld.yml
---
mysql_orders_placed_last_minute:
query: "select count(id) as orders from pleu18.orders where created_at between now() -
interval 1 minute and now()"
metrics:
- orders:
usage: "GAUGE"
description: "orders placed per minute"
Metric name
Legend
![Page 24: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/24.jpg)
24© The Pythian Group Inc., 2017
Beyond what's in the box: Application Instrumentation 2
![Page 25: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/25.jpg)
© The Pythian Group Inc., 2017 25
Beyond what's in the box: Application Instrumentation 2
● Using Grok exporter https://github.com/fstab/grok_exporter
● Config: global:
config_version: 2
input:
type: file
path: ./magento_exception.log
readall: true
![Page 26: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/26.jpg)
© The Pythian Group Inc., 2017 26
Config (continued): grok:
patterns_dir: ./patterns
additional_patterns:
- 'MAGENTO_MESSAGE .*'
- 'NUMBER [0-9]*'
Beyond what's in the box: Application Instrumentation 2
![Page 27: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/27.jpg)
© The Pythian Group Inc., 2017 27
Config (continued): metrics:
- type: counter
name: magento_critical_errors
help: Total number of main.CRITICAL entries in the magento log
match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'
labels:
error_message: '{{.message}}'
server:
host: localhost
port: 9144
Beyond what's in the box: Application Instrumentation 2
Tweak MAGENTO_MESSAGE pattern to get more info
![Page 28: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/28.jpg)
© The Pythian Group Inc., 2017 28
Start and add it: # ./grok_exporter --config ./config.yml
# pmm-admin add external:service magento --service-port=9144 --path /metrics
Add the graph:
Beyond what's in the box: Application Instrumentation 2
name from config.yml
labels from config.yml
![Page 29: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/29.jpg)
© The Pythian Group Inc., 2017 29
Beyond what's in the box: Cassandra
![Page 30: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/30.jpg)
© The Pythian Group Inc., 2017 30
● Using https://github.com/nabto/cassandra-prometheus
● Add jar to cassandra-env.sh JVM_OPTS:
● Start Cassandra, then add target to PMM: # pmm-admin add external:service cassandra --service-port=7400
● Import dashboard:
https://grafana.com/dashboards/371
https://gist.github.com/fipar/87fe346de88aa71b9f7656dc5938c23d
Beyond what's in the box: Cassandra
![Page 31: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/31.jpg)
© The Pythian Group Inc., 2017 31
● Metrics from X must be available to PMM's Grafana.
● Available data sources:
● Prometheus (exporter for X needed)
● QAN-API (if you want to create custom graphs from Query Analytics Data)
● pmm-admin add external:service.
● Create graphs or create/import dashboard
Beyond what's in the box: Monitoring X
![Page 32: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/32.jpg)
Q&A
![Page 33: Taking PMM to the next level - Percona · 2018-11-16 · name: magento_critical_errors help: Total number of main.CRITICAL entries in the magento log match: '.* main.CRITICAL: %{MAGENTO_MESSAGE:message}'](https://reader034.fdocuments.us/reader034/viewer/2022042219/5ec5596713b08355f20aa4da/html5/thumbnails/33.jpg)
33© The Pythian Group Inc., 2017
Thank you!