An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to...

56
An Elephant’s Habitat Alexander Sosna <[email protected]> pgconf.eu 2019

Transcript of An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to...

Page 1: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

An Elephant’s Habitat

Alexander Sosna <[email protected]>

pgconf.eu 2019

Page 2: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Speaker - Alexander Sosna

I PostgreSQL in production since 2006

I 5+ years at credativ

I Consultant Project Manager

I github.com/xxorde

Alexander Sosna <[email protected]> credativ GmbH 1 / 55

Page 3: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

PostgreSQL - Overview

I The World’s Most Advanced Open Source Relational Database

Alexander Sosna <[email protected]> credativ GmbH 2 / 55

Page 4: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

PostgreSQL - Features

PostgreSQL has a lot of great features,

but you already heard about them in the previous talks!

Alexander Sosna <[email protected]> credativ GmbH 3 / 55

Page 5: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

PostgreSQL - In The Real World

Today we talk about:

I Running PostgreSQL in production

I Building critical infrastructure

I Not relying on proprietary components or vendors while doing so!

Alexander Sosna <[email protected]> credativ GmbH 4 / 55

Page 6: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

PostgreSQL - Operation

Let’s focus on:

1. Reliability

2. Operating environment

Alexander Sosna <[email protected]> credativ GmbH 5 / 55

Page 7: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

PostgreSQL - Operation

Let’s focus on:

1. Reliability (already proven over many years, battle hardened since 1996)

2. Operating environment

Alexander Sosna <[email protected]> credativ GmbH 6 / 55

Page 8: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Reliability -> Anecdote 1/2

I ’‘I manage thousands of databases (PostgreSQL, SQL Server, and MySQL), and thispast weekend we had a massive power surge that knocked out two APC cabinets.[. . . ] Long story short, every single PostgreSQL machine survived the failurewith zero data corruption. I had a few issues with SQL Server machines, andvirtually every MySQL machine has required data cleanup and table scans and tweaksto get it back to ”production” status.”

Joshua Drake,http://archives.postgresql.org/pgsql-advocacy/2011-04/msg00085.php

Alexander Sosna <[email protected]> credativ GmbH 7 / 55

Page 9: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Reliability -> Anecdote 2/2

I ’‘I had exactly the same experience 3 years ago. Complete power failure (the stand-bygenerator took fire) in one small datacenter (around 500 machines). We had Oracle,SQL Server, DB2, MySQL, Progress, and of course PostgreSQL. The only databaseengine that restarted with no operation required was PostgreSQL. There werevery minimal problems with Oracle (typing recover on some instances), but we hadquite a few problems with the other engines.”

Marc Cousin,http://archives.postgresql.org/pgsql-advocacy/2011-04/msg00086.php

Alexander Sosna <[email protected]> credativ GmbH 8 / 55

Page 10: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

PostgreSQL - Drawbacks?

Unix-Philosophy

I Write programs that do one thing and do it well.

I Write programs to work together.

I Write programs to handle text streams, because that is a universal interface.

Summarized by Peter H. Salus in A Quarter-Century of Unix (1994)https://en.wikipedia.org/wiki/Unix_philosophy

I What is missing?

Alexander Sosna <[email protected]> credativ GmbH 9 / 55

Page 11: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Infrastructure Pillars

Alexander Sosna <[email protected]> credativ GmbH 10 / 55

Page 12: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Infrastructure Pillars -> Platform

Alexander Sosna <[email protected]> credativ GmbH 11 / 55

Page 13: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Platform

I PostgreSQL runs virtually anywhere

I Choose any common (Linux) OSI PostgreSQL loves LinuxTM

I DebianI UbuntuI CentOS / RHELI . . .

I BSDI $UNIXI Windows*

Alexander Sosna <[email protected]> credativ GmbH 12 / 55

Page 14: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Platform -> Linux

1. Enterprise distribution packagesI might will be outdatedI Version => OS : PostgreSQL (1 : 1)

2. Community repositories to the rescueI (https://apt.postgresql.org) Debian and UbuntuI (https://yum.postgresql.org) CentOS, RHEL, FedoraI (https://zypp.postgresql.org) SLES

3. Build your own PostgreSQLI very easyI not recommendedI the road less traveled might lead you to unknown places bugs

Alexander Sosna <[email protected]> credativ GmbH 13 / 55

Page 15: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Platform -> BSDs

I FreeBSD has a very good reputation

I OpenBSD for specialized environments (security)

I Binary packages available from the package collections.

Alexander Sosna <[email protected]> credativ GmbH 14 / 55

Page 16: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Platform -> Windows

I Binary installer (https://www.postgresql.org/download/windows/)

I Not an ideal platformI No fork() system callI General lower performance than on Linux / UnixI Not the whole ecosystem / tooling runs on Windows

Alexander Sosna <[email protected]> credativ GmbH 15 / 55

Page 17: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Platform -> Cloud

Software as a Service (SaaS) / Database as a Service (DBaaS)

I DBaaS is old news (˜20 years)

I I used PHP + MySQL (as DBaaS) in the early 2000s

I Quit using that long ago ;)

Alexander Sosna <[email protected]> credativ GmbH 16 / 55

Page 18: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Platform -> Cloud

PostgreSQL as a Service (PGaaS)

I Private Cloud InfrastructureI Kubernetes (Openshift, . . . )I Nomad

I Public Cloud (often proprietary / vendor lock possible!)

Alexander Sosna <[email protected]> credativ GmbH 17 / 55

Page 19: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Platform -> Cloud Provider (PGaaS)

Alexander Sosna <[email protected]> credativ GmbH 18 / 55

Page 20: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Platform -> Characteristics

I Good maintainabilityI Updates and upgradesI Migrations

I Good support in case you ran out of knowledgeI OperationI Architecture / migrationI Emergency / disaster recovery

Alexander Sosna <[email protected]> credativ GmbH 19 / 55

Page 21: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Platform -> Performance

I Bare metal vs. VM vs. cloud

I Benchmarks are good, but hard to make rightTM

I Cloud environments are sometimes tricky (IOPs, CPU, bursts vs. shaping, black box)

I Spectre/Meltdown and friends don’t make it easier(https://make-linux-fast-again.com/)

I Scalability?!

Alexander Sosna <[email protected]> credativ GmbH 20 / 55

Page 22: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Infrastructure Pillars -> Monitoring

Alexander Sosna <[email protected]> credativ GmbH 21 / 55

Page 23: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Monitoring

Monitoring should be considered key infrastructure

I Makes sure everything works as expected

I Reports failures

I Trend reporting (will the disk be full by Saturday?)

I Flexible for new services and metrics

I Easy to maintain and automate

Alexander Sosna <[email protected]> credativ GmbH 22 / 55

Page 24: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Monitoring -> Software

I Prometheus + Grafana

I Icinga2

I Log analyze alone is not enough!

Alexander Sosna <[email protected]> credativ GmbH 23 / 55

Page 25: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Infrastructure Pillars -> Availability

Alexander Sosna <[email protected]> credativ GmbH 24 / 55

Page 26: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Availability

“High Availability” is a buzzword, every infrastructure requires its own definition for it

I Nobody wants “low” availability

I Usually not only the database is critical, there are many SPOFs

I Know your:I (full) stackI (real) requirementsI (false) assumptions

Alexander Sosna <[email protected]> credativ GmbH 25 / 55

Page 27: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Availability -> State of the State

I Stateless systems are easyI Add more serversI Add load balancer

I Stateful is more complicatedI We need to propagate the state

Alexander Sosna <[email protected]> credativ GmbH 26 / 55

Page 28: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Availability -> Techniques

I Physical Streaming Replication

I Logical Streaming ReplicationI Usually not a good option for common availability considerationsI Good for migrations / special scenarios

I Cold Standby (shared storage)

I Maybe (fast) restore is enough for you?!

Alexander Sosna <[email protected]> credativ GmbH 27 / 55

Page 29: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Availability -> Automatic Failover

Patroni

I Cluster manager for PostgreSQL

I One primary / pultiple standby instances

I Etcd, Consul, ZooKeeper, . . . as external quorum sources

Alexander Sosna <[email protected]> credativ GmbH 28 / 55

Page 30: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Availability -> Automatic Failover

Pacemaker + Corosync

I Cluster manager for whole software stacks

I Building blocks for experienced architects, not an easy solutionI Very flexibleI Very complicated (watch your timeouts!)I Requires resource agents for specific services

(https://clusterlabs.github.io/PAF/configuration.html)

Alexander Sosna <[email protected]> credativ GmbH 29 / 55

Page 31: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Availability -> Setup Review

I Question your setup carefullyI Is a spilt brain theoretical possible? (disqualification!)I Are there still any SPOFs left?I What are likely failure scenarios, how will the setup behave?I Have you really increased the global availability?I Is the added complexity worth it?

I Traditional 2 node setups should be considered deprecated!

Alexander Sosna <[email protected]> credativ GmbH 30 / 55

Page 32: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Infrastructure Pillars -> Backup

Alexander Sosna <[email protected]> credativ GmbH 31 / 55

Page 33: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Backup

A good infrastructure needs has a backup “plan”:

I What to backup?

I When to backup?

I Where to backup?

I How many backups to keep?

I How to recover? (Important)

Alexander Sosna <[email protected]> credativ GmbH 32 / 55

Page 34: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Backup - Test or do not trust!

Alexander Sosna <[email protected]> credativ GmbH 33 / 55

Page 35: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Backup

I Create a recovery infrastructure

I Train your recovery

I Test your recovery, and automate it

no backup => no compassion mercy!

Alexander Sosna <[email protected]> credativ GmbH 34 / 55

Page 36: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Backup -> Tools

Good tools for PostgreSQL backup automation:

I pgBackRest (https://pgbackrest.org)

I Barman (https://pgbarman.org/)

I Many more. . .

Alexander Sosna <[email protected]> credativ GmbH 35 / 55

Page 37: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Infrastructure Pillars -> Integration

Alexander Sosna <[email protected]> credativ GmbH 36 / 55

Page 38: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Integration -> Extension Overview

Alexander Sosna <[email protected]> credativ GmbH 37 / 55

Page 39: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Integration -> Extensibility

I API and hooks for SQL, C, Python, . . . and many more

I Additional data types (e.g. postgresql-unit)

I Procedural Languages

I . . .

Alexander Sosna <[email protected]> credativ GmbH 38 / 55

Page 40: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Integration -> Foreign Data Wrapper (FDW)

I Other PostgreSQL instances (postgres fdw)

I Other DatabasesI OracleI InformixI CassandraI MySQL / MariaDBI . . .

I Arbitrary Data SourcesI CSVI XMLI TwitterI . . .

Alexander Sosna <[email protected]> credativ GmbH 39 / 55

Page 41: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Do You Need the Perfect, Tailor-Made Setup?

credativ supports customers since 1999 building infrastructure, custom environments andrunning PostgreSQL in production.

I From Single DB to PGaaS-Provider

I Architecture, Operation, Review, Training, . . .

I Migrations from other RDBMS

I Bugfixes and Code Level Support

I [LTS Support for EOL-OSS, e.g. PostgreSQL, Debian]

Alexander Sosna <[email protected]> credativ GmbH 40 / 55

Page 42: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Just in Need of a Good Examples?

I You want to see how other people run PostgreSQL?

I Need a full solution now, that can be customized later?I Not enough time to make all design decisions now?I Need to migrate from a commercial RDBMS fast? :)

Alexander Sosna <[email protected]> credativ GmbH 41 / 55

Page 43: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Goals

Have a look at https://elephant-shed.io/

I Ready to use BlueprintI Try it out on your workstationI Deploy to testingI Deploy to production

Code on github: https://github.com/credativ/elephant-shed

Alexander Sosna <[email protected]> credativ GmbH 42 / 55

Page 44: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Architecture

I Modular designI ExpandableI Components as construction kit for individual setups

I 100% Open Source, no proprietary components

I Long-term maintenance and commercial support

Alexander Sosna <[email protected]> credativ GmbH 43 / 55

Page 45: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Portal (Stable)

I Web based interface

I Every component on one page

I SSO via PAM Authentication

Alexander Sosna <[email protected]> credativ GmbH 44 / 55

Page 46: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Portal (Development)

Alexander Sosna <[email protected]> credativ GmbH 45 / 55

Page 47: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Monitoring Components

I Grafana - visualisation

I Prometheus - collecting metrics

I SQL Exporter - PostgreSQL metrics

I Node Exporter - system metrics

Alexander Sosna <[email protected]> credativ GmbH 46 / 55

Page 48: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Monitoring Visualisation

Metrics for:

I Current or old performance

I System

I Cluster

I Database

I Replication

Alexander Sosna <[email protected]> credativ GmbH 47 / 55

Page 49: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Monitoring Alerting

I Fast and simple via Grafana

I Complex alerting rules via Prometheus Alertmanager

Alexander Sosna <[email protected]> credativ GmbH 48 / 55

Page 50: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Database Administration

Alexander Sosna <[email protected]> credativ GmbH 49 / 55

Page 51: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Backup

pgBackRest

I WAL archiving and Point in Time Recovery (PITR)

I Compression (on disk / network)

I Encryption

I Full, differential and incremental backups

I Backup to disk, cifs, nfs or S3

Alexander Sosna <[email protected]> credativ GmbH 50 / 55

Page 52: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Backup

Alexander Sosna <[email protected]> credativ GmbH 51 / 55

Page 53: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Log Analyzing

Generate HTML Reports via pgBadger.

I Find in the Logs e.g.:I Error messagesI Long running actions (log min duration statement)I Connections and timingsI . . .

I Automatic Reports:I Daily via systemd timerI Ad-hoc via webinterfaceI For every cluster

Alexander Sosna <[email protected]> credativ GmbH 52 / 55

Page 54: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Platforms

Appliance and packages (Full)

I Debian

I Ubuntu

I CentOS / RHEL

DB-Server (PostgreSQL + Prometheus exporter)

I SLES

Porting to other platforms possible if requested

Alexander Sosna <[email protected]> credativ GmbH 53 / 55

Page 55: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Elephant Shed -> Try it out

VM via Vagrant + VirtualBox

vagrant init credativ/elephant-shed

vagrant up

I Default user: admin / admin

I Web-GUI: https://localhost:4433

I PG-Port: 55432 (psql -h localhost -p 55432 -U admin)

I pgAdmin4: admin@localhost / admin

https://elephant-shed.io/#download

Alexander Sosna <[email protected]> credativ GmbH 54 / 55

Page 56: An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to maintain and automate Alexander Sosna  credativ GmbH22

Q&A

Questions? Now or later!

I Alexander Sosna <[email protected]>

Slides

I https://share.credativ.com/~aso/2019.pgconf.eu.pdf

Additional Information

I https://elephant-shed.io

I Germany: https://credativ.de

I The world: https://credativ.com

Alexander Sosna <[email protected]> credativ GmbH 55 / 55