An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to...

An Elephant’s Habitat

Alexander Sosna <[email protected]>

pgconf.eu 2019

<[email protected]>

Speaker - Alexander Sosna

I PostgreSQL in production since 2006

I 5+ years at credativ

I Consultant Project Manager

I github.com/xxorde

Alexander Sosna <[email protected]> credativ GmbH 1 / 55

https://github.com/xxorde

<[email protected]>

PostgreSQL - Overview

I The World’s Most Advanced Open Source Relational Database


<[email protected]>

PostgreSQL - Features

PostgreSQL has a lot of great features,

but you already heard about them in the previous talks!


<[email protected]>

PostgreSQL - In The Real World

Today we talk about:

I Running PostgreSQL in production

I Building critical infrastructure

I Not relying on proprietary components or vendors while doing so!


<[email protected]>

PostgreSQL - Operation

Let’s focus on:

1. Reliability

2. Operating environment


<[email protected]>

PostgreSQL - Operation

Let’s focus on:

1. Reliability (already proven over many years, battle hardened since 1996)

2. Operating environment


<[email protected]>

Reliability -> Anecdote 1/2

I ’‘I manage thousands of databases (PostgreSQL, SQL Server, and MySQL), and thispast weekend we had a massive power surge that knocked out two APC cabinets.[. . . ] Long story short, every single PostgreSQL machine survived the failurewith zero data corruption. I had a few issues with SQL Server machines, andvirtually every MySQL machine has required data cleanup and table scans and tweaksto get it back to ”production” status.”

Joshua Drake,http://archives.postgresql.org/pgsql-advocacy/2011-04/msg00085.php


http://archives.postgresql.org/pgsql-advocacy/2011-04/msg00085.php

<[email protected]>

Reliability -> Anecdote 2/2

I ’‘I had exactly the same experience 3 years ago. Complete power failure (the stand-bygenerator took fire) in one small datacenter (around 500 machines). We had Oracle,SQL Server, DB2, MySQL, Progress, and of course PostgreSQL. The only databaseengine that restarted with no operation required was PostgreSQL. There werevery minimal problems with Oracle (typing recover on some instances), but we hadquite a few problems with the other engines.”

Marc Cousin,http://archives.postgresql.org/pgsql-advocacy/2011-04/msg00086.php


http://archives.postgresql.org/pgsql-advocacy/2011-04/msg00086.php

<[email protected]>

PostgreSQL - Drawbacks?

Unix-Philosophy

I Write programs that do one thing and do it well.

I Write programs to work together.

I Write programs to handle text streams, because that is a universal interface.

Summarized by Peter H. Salus in A Quarter-Century of Unix (1994)https://en.wikipedia.org/wiki/Unix_philosophy

I What is missing?


https://en.wikipedia.org/wiki/Unix_philosophy

<[email protected]>

Infrastructure Pillars


<[email protected]>

Infrastructure Pillars -> Platform


<[email protected]>

Platform

I PostgreSQL runs virtually anywhere

I Choose any common (Linux) OSI PostgreSQL loves LinuxTM

I DebianI UbuntuI CentOS / RHELI . . .

I BSDI $UNIXI Windows*


<[email protected]>

Platform -> Linux

1. Enterprise distribution packagesI might will be outdatedI Version => OS : PostgreSQL (1 : 1)

2. Community repositories to the rescueI (https://apt.postgresql.org) Debian and UbuntuI (https://yum.postgresql.org) CentOS, RHEL, FedoraI (https://zypp.postgresql.org) SLES

3. Build your own PostgreSQLI very easyI not recommendedI the road less traveled might lead you to unknown places bugs


<[email protected]>

Platform -> BSDs

I FreeBSD has a very good reputation

I OpenBSD for specialized environments (security)

I Binary packages available from the package collections.


<[email protected]>

Platform -> Windows

I Binary installer (https://www.postgresql.org/download/windows/)

I Not an ideal platformI No fork() system callI General lower performance than on Linux / UnixI Not the whole ecosystem / tooling runs on Windows


<[email protected]>

Platform -> Cloud

Software as a Service (SaaS) / Database as a Service (DBaaS)

I DBaaS is old news (˜20 years)

I I used PHP + MySQL (as DBaaS) in the early 2000s

I Quit using that long ago ;)


<[email protected]>

Platform -> Cloud

PostgreSQL as a Service (PGaaS)

I Private Cloud InfrastructureI Kubernetes (Openshift, . . . )I Nomad

I Public Cloud (often proprietary / vendor lock possible!)


<[email protected]>

Platform -> Cloud Provider (PGaaS)


<[email protected]>

Platform -> Characteristics

I Good maintainabilityI Updates and upgradesI Migrations

I Good support in case you ran out of knowledgeI OperationI Architecture / migrationI Emergency / disaster recovery


<[email protected]>

Platform -> Performance

I Bare metal vs. VM vs. cloud

I Benchmarks are good, but hard to make rightTM

I Cloud environments are sometimes tricky (IOPs, CPU, bursts vs. shaping, black box)

I Spectre/Meltdown and friends don’t make it easier(https://make-linux-fast-again.com/)

I Scalability?!


<[email protected]>

Infrastructure Pillars -> Monitoring


<[email protected]>

Monitoring

Monitoring should be considered key infrastructure

I Makes sure everything works as expected

I Reports failures

I Trend reporting (will the disk be full by Saturday?)

I Flexible for new services and metrics

I Easy to maintain and automate


<[email protected]>

Monitoring -> Software

I Prometheus + Grafana

I Icinga2

I Log analyze alone is not enough!


<[email protected]>

Infrastructure Pillars -> Availability


<[email protected]>

Availability

“High Availability” is a buzzword, every infrastructure requires its own definition for it

I Nobody wants “low” availability

I Usually not only the database is critical, there are many SPOFs

I Know your:I (full) stackI (real) requirementsI (false) assumptions


<[email protected]>

Availability -> State of the State

I Stateless systems are easyI Add more serversI Add load balancer

I Stateful is more complicatedI We need to propagate the state


<[email protected]>

Availability -> Techniques

I Physical Streaming Replication

I Logical Streaming ReplicationI Usually not a good option for common availability considerationsI Good for migrations / special scenarios

I Cold Standby (shared storage)

I Maybe (fast) restore is enough for you?!


<[email protected]>

Availability -> Automatic Failover

Patroni

I Cluster manager for PostgreSQL

I One primary / pultiple standby instances

I Etcd, Consul, ZooKeeper, . . . as external quorum sources


<[email protected]>

Availability -> Automatic Failover

Pacemaker + Corosync

I Cluster manager for whole software stacks

I Building blocks for experienced architects, not an easy solutionI Very flexibleI Very complicated (watch your timeouts!)I Requires resource agents for specific services

(https://clusterlabs.github.io/PAF/configuration.html)


<[email protected]>

Availability -> Setup Review

I Question your setup carefullyI Is a spilt brain theoretical possible? (disqualification!)I Are there still any SPOFs left?I What are likely failure scenarios, how will the setup behave?I Have you really increased the global availability?I Is the added complexity worth it?

I Traditional 2 node setups should be considered deprecated!


<[email protected]>

Infrastructure Pillars -> Backup


<[email protected]>

Backup

A good infrastructure needs has a backup “plan”:

I What to backup?

I When to backup?

I Where to backup?

I How many backups to keep?

I How to recover? (Important)


<[email protected]>

Backup - Test or do not trust!


<[email protected]>

Backup

I Create a recovery infrastructure

I Train your recovery

I Test your recovery, and automate it

no backup => no compassion mercy!


<[email protected]>

Backup -> Tools

Good tools for PostgreSQL backup automation:

I pgBackRest (https://pgbackrest.org)

I Barman (https://pgbarman.org/)

I Many more. . .


<[email protected]>

Infrastructure Pillars -> Integration


<[email protected]>

Integration -> Extension Overview


<[email protected]>

Integration -> Extensibility

I API and hooks for SQL, C, Python, . . . and many more

I Additional data types (e.g. postgresql-unit)

I Procedural Languages

I . . .


<[email protected]>

Integration -> Foreign Data Wrapper (FDW)

I Other PostgreSQL instances (postgres fdw)

I Other DatabasesI OracleI InformixI CassandraI MySQL / MariaDBI . . .

I Arbitrary Data SourcesI CSVI XMLI TwitterI . . .


<[email protected]>

Do You Need the Perfect, Tailor-Made Setup?

credativ supports customers since 1999 building infrastructure, custom environments andrunning PostgreSQL in production.

I From Single DB to PGaaS-Provider

I Architecture, Operation, Review, Training, . . .

I Migrations from other RDBMS

I Bugfixes and Code Level Support

I [LTS Support for EOL-OSS, e.g. PostgreSQL, Debian]


<[email protected]>

Just in Need of a Good Examples?

I You want to see how other people run PostgreSQL?

I Need a full solution now, that can be customized later?I Not enough time to make all design decisions now?I Need to migrate from a commercial RDBMS fast? :)


<[email protected]>

Elephant Shed -> Goals

Have a look at https://elephant-shed.io/

I Ready to use BlueprintI Try it out on your workstationI Deploy to testingI Deploy to production

Code on github: https://github.com/credativ/elephant-shed


https://elephant-shed.io/

https://github.com/credativ/elephant-shed

<[email protected]>

Elephant Shed -> Architecture

I Modular designI ExpandableI Components as construction kit for individual setups

I 100% Open Source, no proprietary components

I Long-term maintenance and commercial support


<[email protected]>

Elephant Shed -> Portal (Stable)

I Web based interface

I Every component on one page

I SSO via PAM Authentication


<[email protected]>

Elephant Shed -> Portal (Development)


<[email protected]>

Elephant Shed -> Monitoring Components

I Grafana - visualisation

I Prometheus - collecting metrics

I SQL Exporter - PostgreSQL metrics

I Node Exporter - system metrics


<[email protected]>

Elephant Shed -> Monitoring Visualisation

Metrics for:

I Current or old performance

I System

I Cluster

I Database

I Replication


<[email protected]>

Elephant Shed -> Monitoring Alerting

I Fast and simple via Grafana

I Complex alerting rules via Prometheus Alertmanager


<[email protected]>

Elephant Shed -> Database Administration


<[email protected]>

Elephant Shed -> Backup

pgBackRest

I WAL archiving and Point in Time Recovery (PITR)

I Compression (on disk / network)

I Encryption

I Full, differential and incremental backups

I Backup to disk, cifs, nfs or S3


<[email protected]>

Elephant Shed -> Backup


<[email protected]>

Elephant Shed -> Log Analyzing

Generate HTML Reports via pgBadger.

I Find in the Logs e.g.:I Error messagesI Long running actions (log min duration statement)I Connections and timingsI . . .

I Automatic Reports:I Daily via systemd timerI Ad-hoc via webinterfaceI For every cluster


<[email protected]>

Elephant Shed -> Platforms

Appliance and packages (Full)

I Debian

I Ubuntu

I CentOS / RHEL

DB-Server (PostgreSQL + Prometheus exporter)

I SLES

Porting to other platforms possible if requested


<[email protected]>

Elephant Shed -> Try it out

VM via Vagrant + VirtualBox

vagrant init credativ/elephant-shed

vagrant up

I Default user: admin / admin

I Web-GUI: https://localhost:4433

I PG-Port: 55432 (psql -h localhost -p 55432 -U admin)

I pgAdmin4: admin@localhost / admin

https://elephant-shed.io/#download


https://localhost:4433

https://elephant-shed.io/#download

<[email protected]>

Q&A

Questions? Now or later!

I Alexander Sosna <[email protected]>

Slides

I https://share.credativ.com/~aso/2019.pgconf.eu.pdf

Additional Information

I https://elephant-shed.io

I Germany: https://credativ.de

I The world: https://credativ.com


<[email protected]>

https://share.credativ.com/~aso/2019.pgconf.eu.pdf

https://elephant-shed.io/

https://credativ.de

https://credativ.com

<[email protected]>

An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to...

Documents

Transcript of An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to...