An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to...

An Elephant’s Habitat

Alexander Sosna <alexander.sosna@credativ.de>

pgconf.eu 2019

Speaker - Alexander Sosna

I PostgreSQL in production since 2006

I 5+ years at credativ

I Consultant Project Manager

I github.com/xxorde

Alexander Sosna <alexander.sosna@credativ.de> credativ GmbH 1 / 55

PostgreSQL - Overview

I The World’s Most Advanced Open Source Relational Database

PostgreSQL - Features

PostgreSQL has a lot of great features,

but you already heard about them in the previous talks!

PostgreSQL - In The Real World

Today we talk about:

I Running PostgreSQL in production

I Building critical infrastructure

I Not relying on proprietary components or vendors while doing so!

PostgreSQL - Operation

Let’s focus on:

1. Reliability

2. Operating environment

PostgreSQL - Operation

Let’s focus on:

1. Reliability (already proven over many years, battle hardened since 1996)

2. Operating environment

Reliability -> Anecdote 1/2

I ’‘I manage thousands of databases (PostgreSQL, SQL Server, and MySQL), and thispast weekend we had a massive power surge that knocked out two APC cabinets.[. . . ] Long story short, every single PostgreSQL machine survived the failurewith zero data corruption. I had a few issues with SQL Server machines, andvirtually every MySQL machine has required data cleanup and table scans and tweaksto get it back to ”production” status.”

Joshua Drake,http://archives.postgresql.org/pgsql-advocacy/2011-04/msg00085.php

Reliability -> Anecdote 2/2

I ’‘I had exactly the same experience 3 years ago. Complete power failure (the stand-bygenerator took fire) in one small datacenter (around 500 machines). We had Oracle,SQL Server, DB2, MySQL, Progress, and of course PostgreSQL. The only databaseengine that restarted with no operation required was PostgreSQL. There werevery minimal problems with Oracle (typing recover on some instances), but we hadquite a few problems with the other engines.”

Marc Cousin,http://archives.postgresql.org/pgsql-advocacy/2011-04/msg00086.php

PostgreSQL - Drawbacks?

Unix-Philosophy

I Write programs that do one thing and do it well.

I Write programs to work together.

I Write programs to handle text streams, because that is a universal interface.

Summarized by Peter H. Salus in A Quarter-Century of Unix (1994)https://en.wikipedia.org/wiki/Unix_philosophy

I What is missing?

Infrastructure Pillars

Infrastructure Pillars -> Platform

Platform

I PostgreSQL runs virtually anywhere

I Choose any common (Linux) OSI PostgreSQL loves LinuxTM

I DebianI UbuntuI CentOS / RHELI . . .

I BSDI $UNIXI Windows*

Platform -> Linux

1. Enterprise distribution packagesI might will be outdatedI Version => OS : PostgreSQL (1 : 1)

2. Community repositories to the rescueI (https://apt.postgresql.org) Debian and UbuntuI (https://yum.postgresql.org) CentOS, RHEL, FedoraI (https://zypp.postgresql.org) SLES

3. Build your own PostgreSQLI very easyI not recommendedI the road less traveled might lead you to unknown places bugs

Platform -> BSDs

I FreeBSD has a very good reputation

I OpenBSD for specialized environments (security)

I Binary packages available from the package collections.

Platform -> Windows

I Binary installer (https://www.postgresql.org/download/windows/)

I Not an ideal platformI No fork() system callI General lower performance than on Linux / UnixI Not the whole ecosystem / tooling runs on Windows

Platform -> Cloud

Software as a Service (SaaS) / Database as a Service (DBaaS)

I DBaaS is old news (˜20 years)

I I used PHP + MySQL (as DBaaS) in the early 2000s

I Quit using that long ago ;)

Platform -> Cloud

PostgreSQL as a Service (PGaaS)

I Private Cloud InfrastructureI Kubernetes (Openshift, . . . )I Nomad

I Public Cloud (often proprietary / vendor lock possible!)

Platform -> Cloud Provider (PGaaS)

Platform -> Characteristics

I Good maintainabilityI Updates and upgradesI Migrations

I Good support in case you ran out of knowledgeI OperationI Architecture / migrationI Emergency / disaster recovery

Platform -> Performance

I Bare metal vs. VM vs. cloud

I Benchmarks are good, but hard to make rightTM

I Cloud environments are sometimes tricky (IOPs, CPU, bursts vs. shaping, black box)

I Spectre/Meltdown and friends don’t make it easier(https://make-linux-fast-again.com/)

I Scalability?!

Infrastructure Pillars -> Monitoring

Monitoring

Monitoring should be considered key infrastructure

I Makes sure everything works as expected

I Reports failures

I Trend reporting (will the disk be full by Saturday?)

I Flexible for new services and metrics

I Easy to maintain and automate

Monitoring -> Software

I Prometheus + Grafana

I Icinga2

I Log analyze alone is not enough!

Infrastructure Pillars -> Availability

Availability

“High Availability” is a buzzword, every infrastructure requires its own definition for it

I Nobody wants “low” availability

I Usually not only the database is critical, there are many SPOFs

I Know your:I (full) stackI (real) requirementsI (false) assumptions

Availability -> State of the State

I Stateless systems are easyI Add more serversI Add load balancer

I Stateful is more complicatedI We need to propagate the state

Availability -> Techniques

I Physical Streaming Replication

I Logical Streaming ReplicationI Usually not a good option for common availability considerationsI Good for migrations / special scenarios

I Cold Standby (shared storage)

I Maybe (fast) restore is enough for you?!

Availability -> Automatic Failover

Patroni

I Cluster manager for PostgreSQL

I One primary / pultiple standby instances

I Etcd, Consul, ZooKeeper, . . . as external quorum sources

Availability -> Automatic Failover

Pacemaker + Corosync

I Cluster manager for whole software stacks

I Building blocks for experienced architects, not an easy solutionI Very flexibleI Very complicated (watch your timeouts!)I Requires resource agents for specific services

(https://clusterlabs.github.io/PAF/configuration.html)

Availability -> Setup Review

I Question your setup carefullyI Is a spilt brain theoretical possible? (disqualification!)I Are there still any SPOFs left?I What are likely failure scenarios, how will the setup behave?I Have you really increased the global availability?I Is the added complexity worth it?

I Traditional 2 node setups should be considered deprecated!

Infrastructure Pillars -> Backup

Backup

A good infrastructure needs has a backup “plan”:

I What to backup?

I When to backup?

I Where to backup?

I How many backups to keep?

I How to recover? (Important)

Backup - Test or do not trust!

Backup

I Create a recovery infrastructure

I Train your recovery

I Test your recovery, and automate it

no backup => no compassion mercy!

Backup -> Tools

Good tools for PostgreSQL backup automation:

I pgBackRest (https://pgbackrest.org)

I Barman (https://pgbarman.org/)

I Many more. . .

Infrastructure Pillars -> Integration

Integration -> Extension Overview

Integration -> Extensibility

I API and hooks for SQL, C, Python, . . . and many more

I Additional data types (e.g. postgresql-unit)

I Procedural Languages

I . . .

Integration -> Foreign Data Wrapper (FDW)

I Other PostgreSQL instances (postgres fdw)

I Other DatabasesI OracleI InformixI CassandraI MySQL / MariaDBI . . .

I Arbitrary Data SourcesI CSVI XMLI TwitterI . . .

Do You Need the Perfect, Tailor-Made Setup?

credativ supports customers since 1999 building infrastructure, custom environments andrunning PostgreSQL in production.

I From Single DB to PGaaS-Provider

I Architecture, Operation, Review, Training, . . .

I Migrations from other RDBMS

I Bugfixes and Code Level Support

I [LTS Support for EOL-OSS, e.g. PostgreSQL, Debian]

Just in Need of a Good Examples?

I You want to see how other people run PostgreSQL?

I Need a full solution now, that can be customized later?I Not enough time to make all design decisions now?I Need to migrate from a commercial RDBMS fast? :)

Elephant Shed -> Goals

Have a look at https://elephant-shed.io/

I Ready to use BlueprintI Try it out on your workstationI Deploy to testingI Deploy to production

Code on github: https://github.com/credativ/elephant-shed

Elephant Shed -> Architecture

I Modular designI ExpandableI Components as construction kit for individual setups

I 100% Open Source, no proprietary components

I Long-term maintenance and commercial support

Elephant Shed -> Portal (Stable)

I Web based interface

I Every component on one page

I SSO via PAM Authentication

Elephant Shed -> Portal (Development)

Elephant Shed -> Monitoring Components

I Grafana - visualisation

I Prometheus - collecting metrics

I SQL Exporter - PostgreSQL metrics

I Node Exporter - system metrics

Elephant Shed -> Monitoring Visualisation

Metrics for:

I Current or old performance

I System

I Cluster

I Database

I Replication

Elephant Shed -> Monitoring Alerting

I Fast and simple via Grafana

I Complex alerting rules via Prometheus Alertmanager

Elephant Shed -> Database Administration

Elephant Shed -> Backup

pgBackRest

I WAL archiving and Point in Time Recovery (PITR)

I Compression (on disk / network)

I Encryption

I Full, differential and incremental backups

I Backup to disk, cifs, nfs or S3

Elephant Shed -> Backup

Elephant Shed -> Log Analyzing

Generate HTML Reports via pgBadger.

I Find in the Logs e.g.:I Error messagesI Long running actions (log min duration statement)I Connections and timingsI . . .

I Automatic Reports:I Daily via systemd timerI Ad-hoc via webinterfaceI For every cluster

Elephant Shed -> Platforms

Appliance and packages (Full)

I Debian

I Ubuntu

I CentOS / RHEL

DB-Server (PostgreSQL + Prometheus exporter)

I SLES

Porting to other platforms possible if requested

Elephant Shed -> Try it out

VM via Vagrant + VirtualBox

vagrant init credativ/elephant-shed

vagrant up

I Default user: admin / admin

I Web-GUI: https://localhost:4433

I PG-Port: 55432 (psql -h localhost -p 55432 -U admin)

I pgAdmin4: admin@localhost / admin

https://elephant-shed.io/#download

Questions? Now or later!

I Alexander Sosna <alexander.sosna@credativ.de>

Slides

I https://share.credativ.com/~aso/2019.pgconf.eu.pdf

Additional Information

I https://elephant-shed.io

I Germany: https://credativ.de

I The world: https://credativ.com

An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to...

Documents

Transcript of An Elephant's Habitataso/2019.pgconf.eu.pdfI Flexible for new services and metrics I Easy to...

Connecting the Debian and PostgreSQL worlds · Connecting the Debian and PostgreSQL worlds Christoph Berg christoph.berg@credativ.de myon@debian.org PGconf.EU 2011, Amsterdam October

Making apt.postgresql.org a Realitywiki.postgresql.org/images/4/4f/Pgdg-apt-fosdem2013.pdf · Making apt.postgresql.org a Reality ChristophBerg Debian

April 3-4, 2003 Garden Grove Proven Programs for Community Supervision Settings 1 Proven Program for Community Supervision Settings Todd Sosna, Ph.D.

Porting Oracle Applications to PostgreSQL - PGCon 2018 · Porting Oracle Applications to PostgreSQL Peter Eisentraut, credativ GmbH / credativ Ltd. ... Porting is usually straightforward,

Christoph Berg ...Freezing I Transaktionsnummern sind 64 Bit I gespeichert werden nur 32 Bit I nach 231 Transaktionen mussen alte Tupel (genauer:

07162010 Greenberger Bay Letter SOSNA

Scaling Up Evidence-Based Practices - Blueprints …2012.blueprintsconference.com/presentations/F4-A.pdf · Scaling Up Evidence-Based Practices Lynne Marsenich, LCSW Todd Sosna, Ph.D.

A resampling approach to gender relations: the Rebesˇovice ... et al. 2008.pdf · A resampling approach to gender relations: the Rebesˇovice cemetery Daniel Sosna a,*, Patrik Galeta

5 t h S u n d a y o f L e n t · 4/7/2019 · Larry Tanking Whity Lehmkuhl April 14, SUN 8:00am Sharon ole Kathy Devlin-N Dot Sosna-S onnor Hammer* Victor Organista Alyssa erns Joan

How Common are Bone Islands on Abdominal and Pelvic CT? Tamar Sella MD Nurith Hiller MD Azraq Yusef MD Eugene Libson MD Jacob Sosna MD Dept. of Radiology.

Dumping the Mainframe: Migration Study from DB2 …mbanck/pgconfde-2015-mba.pdfDumping the Mainframe: Migration Study from DB2 UDB to PostgreSQL Michael Banck

SECURITY SPOTLIGHT - info.blackducksoftware.com · Bareos GmbH & Co. KG, Black Duck Software, Capital One, Chamilo, Chef, CloudFoundry Corp, Confer, Coolan, Couchbase, Credativ, DEIS/Engineyard,

Alexander Sosna, Adrian Vondendrieschcredativ - Open Source for Business I IT-Dienstleister seit 1999 I Grunder u. Mitarbeiter aus Debian- u. PostgreSQL-Community I Betrieb kritischer

AZURE REGIONS - download.microsoft.comdownload.microsoft.com/.../2016-01-Azure...webinar.pdf · Debian on Azure Microsoft partnered with Credativ, a member of the Linux Foundation,

Volume 60, No. 2 Week of July 24, 2017 THE ACORN · Mary E. Oksala, Ph.D ... The backpacks are made by inmates at the Todd ... 85-86 MARV SOSNA 86-87 FRED FERRARINI 87-88 DO NEEDHAM

1998 Sosna WRT A New Hope For The Royal Theater

Embedded SQL in PostgreSQL · PDF fileEmbedded SQL in PostgreSQL Dr. Michael Meskes, michael.meskes@credativ.de 7. Dezember 2010 Dr. Michael Meskes, michael.meskes@credativ.de Embedded

PostgreSQL Functions By Example - joeconway.com · Overview Function Basics By Example PostgreSQL Functions By Example Joe Conway joe.conway@credativ.com credativ Group January 20,

The KeanCast Kean University Media Initiative Dr. Paul J. Croft Cathleen Londino, Scott McHugh, Doug Black, Steve Sosna, Shane Brown, Victoria Schemelia.

Understanding Evidence-Based Practices Ventura County Behavioral Health April 29, 2009 Presented by Todd Sosna, Ph.D.