Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL...
Transcript of Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL...
![Page 1: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/1.jpg)
Deploying BDR
Simon RiggsCTO, 2ndQuadrant &Major Developer, PostgreSQL
February 2015
![Page 2: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/2.jpg)
What is BDR?
BDR is Bi Directional Replication
BDR is the newest replication technologyfor core PostgreSQL
BDR is open source, using PostgreSQL Licence
BDR is a code submission to the PostgreSQL project
BDR is a working, production quality solution
![Page 3: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/3.jpg)
Trigger Based Replication
It works! Selective replication Allows online upgrade
2006
![Page 4: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/4.jpg)
Trigger Based Replication
It works! Selective replication Allows online upgrade
Not Integrated in Core Complex design & code Difficult to Understand High Maintenance Slow, high latency High overhead Serialization problems
![Page 5: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/5.jpg)
Physical Streaming Replication
Integrated in core Simple design & code Ease of Use Low Maintenance (DDL) Fast, low latency Low overhead No serialization problems
2010
![Page 6: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/6.jpg)
Physical Streaming Replication
Integrated in core Simple design & code Ease of Use Low Maintenance (DDL) Fast, low latency Low overhead No serialization problems
Read-only access Same schema No temp tables Same users/security Physical effects Whole database only No Multi-Master Complex code No online upgrade
![Page 7: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/7.jpg)
Ideal Replication
Integrated in core Simple design & code Ease of Use Low Maintenance (DDL) Fast, low latency Low overhead No serialization problems
Read-write access Allows different schema Allows temp tables Allows different security Limited Physical effects Selective replication Allows Multi-Master Backwards compatible Online Upgrade
![Page 8: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/8.jpg)
2ndQuadrant designed Logical Replication in 2009
2ndQuadrant started buildingBDR in 2011
….we went production in 2014
![Page 9: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/9.jpg)
BD
R
![Page 10: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/10.jpg)
BD
R
![Page 11: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/11.jpg)
BD
R
![Page 12: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/12.jpg)
BD
R
![Page 13: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/13.jpg)
Database Access
US
E C
AS
E
RW
![Page 14: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/14.jpg)
Database Access
US
E C
AS
E
RWRW
RW
![Page 15: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/15.jpg)
Database Copies
US
E C
AS
E
RWRO
RO
![Page 16: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/16.jpg)
Multi-Master Database
US
E C
AS
E
RWRW
RW
![Page 17: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/17.jpg)
Force Majeure
BA
D N
EW
S
RWRW
RW
![Page 18: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/18.jpg)
Logical & Physical
TE
CH
A R
CH
ITE
CT
UR
E
M
S
S
S
S
M
M
![Page 19: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/19.jpg)
Physical Streaming Replication
WALSender
DatabaseWAL
WALRecvr Startup
DatabaseWAL
User
Master Standby
![Page 20: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/20.jpg)
Logical Streaming Replication
WALSender
Database
Apply
DatabaseWAL
User
UpstreamMaster
DownstreamMaster
![Page 21: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/21.jpg)
PostgreSQL 9.4 Infrastructure
9.3 Background Workers 9.4 Dynamic Shared Memory/Locks 9.4 Logical Decoding 9.4 pg_xlogdump 9.4 Replication Slots 9.4 REPLICA IDENTITY 9.4 user_catalog_table
![Page 22: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/22.jpg)
BDR 0.8.0.7 Production NOW!
Integrated in core Simple design & code Ease of Use Low Maintenance (DDL) Fast, low latency Low overhead No serialization problems Global Sequences
Read-write access Allows different schema Allows temp tables Allows different security Limited Physical effects Selective replication Allows Multi-Master Co-exists with Physical Rep Online Upgrade
![Page 23: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/23.jpg)
BDR 0.9 1Q2015
Integrated in core Simple design & code Ease of Use Low Maintenance (DDL) Fast, low latency Low overhead No serialization problems Global Sequences
Read-write access Allows different schema Allows temp tables Allows different security Limited Physical effects Selective replication Allows Multi-Master Co-exists with Physical Rep Online Upgrade
![Page 24: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/24.jpg)
Deliverables
BDR 0.8.0.7– Full function, already in use in production– Variant of PostgreSQL, open source– All changes submitted to PostgreSQL core– Production Now, merged later 9.6
BDR 0.9– Extension for 9.4+– One-way replication
Integration into PostgreSQL (patches)– 9.5: Online Upgrade from 9.4 to 9.5+– 9.6: More features
![Page 25: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/25.jpg)
Integration Schedule
Postgres Core Extension Variant Distro
9.4 – now! +Logical Decoding+Dynamic Shm+Replication Slots+REPLICA IDENTITY
UDR Extension Full BDR –up to 48 nodes
9.5 +Commit Timestamps+Sequence AM API+Replication Identifiers+DDL Event Triggers
UDR Extension (Faster!) Slim BDR
9.6 +Cluster Metadata+UDR Plugin+Replication Sets+Global Sequences
(Gone!) Multi-master BDR
9.7 +Multi-Master+New conflict models
(Gone!)
![Page 26: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/26.jpg)
BDR
10 years ago, Slony delivered practical solution for PostgreSQL replication that works outside of core
– but it had problems 10 years later, BDR delivers a practical solution for
PostgreSQL replication that works outside of core– nearly all problems solved– and submitted changes for core PostgreSQL
BDR will become “PostgreSQL replication”– Slony no longer being developed full-time– Londiste no longer being developed– It needs your support
![Page 27: Deploying BDR - PgConf.Russia · Logical Streaming Replication WAL Sender Database Apply WAL Database User Upstream Master Downstream Master. PostgreSQL 9.4 Infrastructure 9.3 Background](https://reader036.fdocuments.us/reader036/viewer/2022090604/6057d866d8659f43c413267d/html5/thumbnails/27.jpg)
Futures
Many more features – we ain't done yet!
More Practical Solutions
Quorum Sync Rep Replication Transforms Auto Sharding Massively Parallel Query Distributed Optimisation