Taming 100+ PostgreSQL instances on the CloudNithya NatarajanKrishnaswamy Subramanian

LET’S UNDERSTAND THE EPHEMERAL NATURE OF CONTAINER / POD

ephemeral | ɪˈfɛm(ə)r(ə)l, ɪˈfiːm(ə)r(ə)l |

adjectivelasting for a very short time

STATELESS BY NATUREAny data saved within the

pod, is available till its lifetime

IP ADDRESSEverytime a pod is

scheduled, it comes with a new IP

PID 1Container lives as long as

PID 1 lives

PEERSA pod can be disturbed

by its neighbours

THINGS NEEDS TO RUN AN INSTANCE OF POSTGRESQL

NON ROOT USERA non root user is need

to run PostgreSQL

HOST BASED CONFIGURATIONNeed to configure

pg_hba.conf file to allow other hosts to communicate

POSTGRES CONFIGURATIONConfigurations to tweak PostgreSQL for custom

needs

OPERATING SYSTEMTo run PostgreSQL

database

POSTGRES BINARYBased on the operating system, corresponding

package manager is used to install PostgreSQL

EXTERNAL DISKExternal disk to store the

PostgreSQL data

LET’S SEE IT IN ACTION

ADDITIONAL THINGS NEEDS TO RUN HIGHLY AVAILABLE POSTGRESQL

REPLICATION USERA non root user is need

to run PostgreSQL

HOST BASED CONFIGURATIONNeed to configure

pg_hba.conf file to allow other secondary hosts to

communicate

POSTGRES CONFIGURATIONMake necessary changes

to PostgreSQL configuration for

replication

PERFORM BASE BACKUPCopy the data dir to all the secondary instances to begin the replication

then on

LET’S SEE IT IN ACTION

■ Kubernetes offers all the necessary building block but it’s still difficult to run and maintain PostgreSQL on it.

■ Given that Kubernetes is meant to run ephemeral loads, downtime could happen at any time if we run it with just the basic blocks offered by Kubernetes.

WHAT IS MISSING?

“To be clear Kubernetes does support stateful workloads.

Kubernetes has StatefulSets, Volumes, and many other

features to ease the burden of running stateful workloads. But

you still require expertise to effectively manage stateful

services on Kubernetes.”

— Kelsey Hightower, Google

SO WHAT’S THE SOLUTION?

We need a centralized store to store the PostgreSQL status and configuration

CENTRALIZED STORE1

We need a process manager to- generate PostgreSQL configuration for all the instance based on its role (primary vs

secondary)- restart / reload PostgreSQL based on the data available on the store

POSTGRES PROCESS MANAGER2

We need a process to check if primary db is up and healthy

MONITOR PRIMARY DB HEALTH3

We need a proxy to hide which one is primary, which one is secondary so that client doesn’t have to keep switching

PROXY4

OVERALL REQUIREMENTS

PROXY4

CENTRALIZED STORE1

POSTGRES PROCESS MANAGER2

MONITOR MASTER HEALTH3

LET’S INTRODUCE SOME TERMINOLOGY

PROXY -> PROXY4

CENTRALIZED STORE -> CONFIG STORE1

POSTGRES PROCESS MANAGER -> KEEPER2

MONITOR MASTER HEALTH -> SENTINEL3

REQUIREMENTS FOR CONFIG STORE

■ It should highly available by default without any additional effort

■ It should be distributed by default

■ It should be highly performant

CONFIG STORE

Raft based distributed KV store from CoreOS (now part of

Redhat, which is now part of IBM)

ETCDRaft based distributed KV

Store from Hashicorp

CONSULWe can leverage

Kubernetes Config Map as distributed store

K8s CONFIG MAP

ARCHITECTURE

Sentinel

Keeper(secondary)

Proxy

Client ClientClient

Keeper(secondary)

Keeper(primary)

FINAL ARCHITECTURE

Sentinel(elected)

Keeper(secondary)

Proxy

Client ClientClient

Keeper(secondary)

Keeper(primary)

ProxyProxy

SentinelSentinel

OPEN SOURCE SOLUTION

Is a template for you to create your own customized, high-availability solution using Python and - for maximum

accessibility

PATRONIIs a cloud native

PostgreSQL manager for PostgreSQL high

availability

STOLON

LET’S SEE IT IN ACTION

OTHER USE CASES

Latest version of Kubernetes support dynamic resizing of Persistent Volume

INCREASING DISK SPACE1

Steps:1. Setup streaming replication with standby cluster created with SSD2. Once the replication fully sync get rid of the old stateful set and underlying PVC

MIGRATING FROM HDD -> SSD2

Steps:1. Scale down the stateful set2. Backup disk3. Create a job to mount the PVC used by the statefulset4. Run the pg_upgradecluster to migrate

UPGRADING POSTGRESQL VERSION3

OUR LEARNING

K8s config store availability is greatly dependent on how K8s cluster is setup

USE K8s CONFIG STORE WITH CAUTION1

With VMs we generally run single instance of PostgreSQL, in case of Kubernetes there could be other workloads scheduled by the orchestrator and it could interfere with PostgreSQL performance

NEIGHBOUR NOISE2

Kubernetes scheduler needs this information to schedule your pod accordingly

ALWAYS SPECIFY POD RESOURCE REQUEST3

If you run them together there will too much neighbour noise

HAVE DIFFERENT K8s CLUSTER FOR STATELESS AND STATEFUL APPS4

Even though it is possible to setup PostgreSQL HA, ensure to backup the disk in regular interval

TAKE DAILY BACKUP OF DISK5

QUESTIONS?

Thanks to

◂ Presentation template by Slidesgo◂ Icons by Flaticon◂ Infographics by Freepik◂ Images created by Freepik - Freepik

CREDITS

RESOURCES

https://github.com/nithyanatarajan/pgconf2020workshop

THANK YOU

NITHYA NATARAJAN@nithyanatarajan

KRISHNASWAMY S@jskswamy