14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR...
Transcript of 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR...
![Page 1: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/1.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
14th Feb 2019
Nikhil Sontakke
![Page 2: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/2.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
PostgreSQL to PostgreSQL replication
● Logical replication (one direction only)● De-facto tool for
○ Data Movement○ Migration○ Low Downtime Upgrade
● Pglogical 2.2 available as binary and also now available on Amazon RDS by popular demand
pglogical
![Page 3: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/3.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
Releases and Plans
● Pglogical 2.2 available now○ Supports PostgreSQL 11 and earlier○ Will continue to be supported for 3 years○ Much community interest and patches - thanks!
● Pglogical 3○ Future architecture for PostgreSQL 12+○ Many new features
Pglogical
![Page 4: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/4.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
● New architecture for performance○ Split Receiver/Apply processes, so now 2 procs○ Similar to WALReceiver/Startup process in
physical streaming replication● Pluggable Architecture
○ Allows heterogenous Source(s)/Target(s)○ Kafka, RabbitMQ targets available○ Bulk-mode for high volume data loads
● Parallel initial copy on roadmap for pglogical 3
Pglogical 3
![Page 5: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/5.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
Future of multi-master database for PostgreSQL
● BDR is Bi-Directional Replication (BDR)2-way/Multi-way logical replication
● Postgres-BDR3 runs on PgLogical3● Coordinated code bases, operating as a full stack● Streamlines feature development and support● New feature releases quarterly
● BDR and pglogical 3.5.1 now available
Postgres-BDR3
![Page 6: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/6.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
Trusted multi-master database for PostgreSQL
● Postgres-BDR 3○ Future architecture
● Postgres-BDR 2○ Upgrade path to BDR3 due 3Q2019○ De-supported by mid 2020
● Postgres-BDR 1○ Upgrade path to BDR3 due 1Q2019○ De-supported by end 2019
Postgres-BDR
![Page 7: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/7.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
Future of multi-master database for PostgreSQL
● Postgres-BDR Standard Edition○ Runs as an Extension on
Community PostgreSQL 10+
● Postgres-BDR Enterprise Edition○ Runs as an Extension,
using minimal additional features in 2ndQuadrant PostgreSQL 11+
Postgres-BDR3
![Page 8: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/8.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
Transparent multi-master database for PostgreSQL
● Connect to any node● Read and Write SQL like standard PostgreSQL● Create/Manage tables like standard PostgreSQL
● Options to control which tables are replicated● Some minor restrictions on DDL● Design considerations for distributed database
Using Postgres-BDR
![Page 9: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/9.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
![Page 10: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/10.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
Distributed database options
● Post-Commit SynchronizationResolve issues after COMMIT○ Conflict-Free Custom Datatypes (CRDTs)○ Row-level Conflict Handling by default○ Column-level Conflict Handling option○ Logging and resolution of issues○ Conflict Triggers
Writing to Postgres-BDR
![Page 11: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/11.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
Distributed database options
● Pre-Commit SynchronizationEager Replication avoids conflicts○ All Nodes
● Avoids issues at COMMIT○ Additional latency not desirable in many cases○ Some transaction aborts in conflict cases
Writing to Postgres-BDR
![Page 12: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/12.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
● Node Roles○ Send/Receive Node (default)○ Receive-Only Node
● Rep Sets (node subsets)● Rolling Cross-Version Upgrades
○ One-by-one upgrades with zero downtime○ Minimal APIs designed to allow upgrades○ Across BDR releases and Postgres major releases○ Fully tested to ensure compatibility
BDR Cluster ManagementEnhancements for Distributed PostgreSQL
![Page 13: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/13.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
BDR SecurityCloud and High Security use cases
● BDR changes are applied as the table-owning user● BDR functions do not need to be executed as
superuser, except for installation● Default roles allow principle of least privilege● Secure access to BDR catalog tables
for non-admin users
![Page 14: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/14.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
BDR Robustness and SupportabilityWhen it’s critical, you can count on us!
● Active-Active architecture offers Very High Availability● Each master has multiple protection options
○ Logical Standby(s)○ Physical Standby(s) (or both at once)
● Synchronous and asynchronousReplication options
● Commit At Most Once option● Trace facility for investigation of
complex bugs in distributed system
![Page 15: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/15.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
BDR Application MaintenanceDevOps Ease of Use
● Transparent handling of DDL for partitioned tables● Optional modes for very detailed specification of node
targets, allowing DDL even with down nodes● Allows differing indexes on different nodes,
allowing nodes for Business Intelligence andNew Application-Version Testing
● Stream Triggers for filtering and transformation
![Page 16: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/16.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
BDR OperationsPractical experience
● Clearly identify node state, even when disconnected● Monitor current lag time and lag bytes, even for down nodes● Options for controlling down-node effects● Error handling and controls
● Postgres-BDR plugin forOmniDB also available
● Postgres Cloud Managerwith cluster visualization
![Page 17: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/17.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
Postgres-BDR Plugin for OmniDBVisual Administration
● Clearly identify node state, even when disconnected
![Page 18: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/18.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
● Efficient logical replication● Streaming of large transactions● Efficient distributed sequences● Fast node addition● Choice of options for selecting appropriate
robustness and performance trade-offs
● Put data where its needed,keep it fresh easily and automatically
BDR PerformanceReal-World Production Performance
![Page 19: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/19.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
● Timestamp-based snapshots● Allow consistent queries across nodes
even with real-time replication of data● Data verification between nodes
● Multi-node parallel query (MPP) across○ Local clusters with remote DR nodes○ Geo-distributed clusters
BDR Multi-node QueryConsistency and Performance
![Page 20: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/20.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
Full Stack IntegrationWorking together as one
● Fully integrated stack○ Full automatic testing of operations and upgradability○ 258 pages of detailed documentation
● Postgres-BDR, pglogical, 2QPG● Repmgr, Barman● PCISecurityPack and other tools● OmniDB
● TPAexec Cloud/On-Premise Orchestration
![Page 21: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/21.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
Major new functionality, born out of production experience
● Cluster Management● Security● Robustness and Supportability● Application Maintenance● Operations● Performance
● Rapid, agile development from many user requests
Postgres-BDR3 Summary
![Page 22: 14th Feb 2019 - PGConf India · PostgreSQL Solutions for the Enterprise Pglogical and Postgres-BDR 3.5.1 BDR Application Maintenance DevOps Ease of Use Transparent handling of DDL](https://reader033.fdocuments.us/reader033/viewer/2022042208/5eaba1fd2e7ba075b72477a4/html5/thumbnails/22.jpg)
https://www.2ndQuadrant.com
2ndQuadrantPostgreSQL Solutions for the Enterprise
Pglogical and Postgres-BDR 3.5.1
Website https://www.2ndquadrant.com/Blog https://blog.2ndquadrant.com/ Email [email protected]
2ndQuadrantPostgreSQL Solutions