Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update...

29
Skytools3 PostgreSQL trigger-based replication Dimitri Fontaine [email protected] July, 18 2013 Dimitri Fontaine [email protected] Skytools3 July, 18 2013 1 / 27

Transcript of Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update...

Page 1: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Skytools3PostgreSQL trigger-based replication

Dimitri Fontaine [email protected]

July, 18 2013

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 1 / 27

Page 2: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Dimitri Fontaine

2ndQuadrant France

PostgreSQL Major Contributor

X pgloader, prefix, skytools, . . .

X apt.postgresql.org

X CREATE EXTENSION

X CREATE EVENT TRIGGER

X MySQL migration tool, new pgloader version

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 2 / 27

Page 3: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Dimitri Fontaine

2ndQuadrant France

PostgreSQL Major Contributor

X pgloader, prefix, skytools, . . .

X apt.postgresql.org

X CREATE EXTENSION

X CREATE EVENT TRIGGER

X MySQL migration tool, new pgloader version

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 2 / 27

Page 4: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Dimitri Fontaine

2ndQuadrant France

PostgreSQL Major Contributor

X pgloader, prefix, skytools, . . .

X apt.postgresql.org

X CREATE EXTENSION

X CREATE EVENT TRIGGER

X MySQL migration tool, new pgloader version

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 2 / 27

Page 5: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Skytools

Skype Tools for Replication

X PGQ

X londiste

X walmgr

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 3 / 27

Page 6: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Skytools Architecture

PGQ is organized into 3 components

Producer, Consumer, Ticker

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 4 / 27

Page 7: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

PGQ3

Some things did change in PGQ version 3

X The ticker is now pgqd

X The network topology is now known pgq nodes

X And we have Cooperative Consumers pgq coop

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 5 / 27

Page 8: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Londiste3 concepts

Londiste relies on PGQ nodes

X Root

X Branch

X Leaf

X Leaf –merge=’qname’

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 6 / 27

Page 9: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Operating londiste

Basic commands

X status

X members

X change-provider

X takeover –all –dead

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 7 / 27

Page 10: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Londiste Add Table

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 8 / 27

Page 11: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

londiste add-table

Plenty new options in add-table

X --wait-sync

X --dest-table

X --skip-truncate

X --create, --create-full

X --trigger-flags

X --trigger-arg

X --no-triggers

X --copy-node,

--copy-condition

X --merge-all, --no-merge

X --max-parallel-copy

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 9 / 27

Page 12: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

DDL

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 10 / 27

Page 13: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

DDL Handling

X londiste conf.ini execute

X EXECUTE queue events

X System Table londiste.applied execute

X SQL meta-data attributes

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 11 / 27

Page 14: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

execute Meta-Data attributes

--*--

--*-- Local-Table: mytable, othertable,

--*-- thirdtable

--*-- Local-Sequence: thisseq

--*--

X Local-Table Table must be added to local node with add-table.

X Need-Table Physical table must exist in database. It does not matterif it is replicated or not.

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 12 / 27

Page 15: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

DDL and table renaming

--*-- Local-Table: mytable

ALTER TABLE @mytable@ ...;

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 13 / 27

Page 16: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Londiste Handlers

It’s possible to register handlers to deal with specific needs

X add --handler [ --handler-arg ... ]

X show-handlers

X pl/proxy sharding, re-sharding

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 14 / 27

Page 17: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Sharding pluging

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 15 / 27

Page 18: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Handler: shard

X Event filtering by hash, for partitioned databases.

X key=COLUMN column name to use for hashing

X hashfunc=FUNCNAME function to use for hashing

X (default: partconf.get hash raw)

X ev extra3=’hash=’||partconf.get hash raw(key column)

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 16 / 27

Page 19: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Preparing for part: hashlib

> ... add-table pgbench_accounts \

> --handler=part --handler-arg=key=aid

psql rootdb < /usr/share/postgresql/8.4/contrib/hashlib.sql

psql sharddb_0 < /usr/share/postgresql/8.4/contrib/hashlib.sql

psql sharddb_1 < /usr/share/postgresql/8.4/contrib/hashlib.sql

> psql rootdb -c ’create extension hashlib;’

> psql sharddb_0 -c ’create extension hashlib;’

> psql sharddb_1 -c ’create extension hashlib;’

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 17 / 27

Page 20: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Preparing for part: setup

CREATE SCHEMA partconf;

CREATE TABLE partconf.conf (

part_nr integer,

max_part integer,

db_code bigint,

is_primary boolean,

max_slot integer,

cluster_name text

);

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 18 / 27

Page 21: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Preparing for part: get hash raw

CREATE FUNCTION partconf.get_hash_raw

( i_input integer)

RETURNS integer

LANGUAGE sql

AS $$

-- used to wrap hashtext so that we can replace it in 8.4

-- with older implementation to keep compatibility

select hash_string(\$1::text, ’lookup2’);

$$;

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 19 / 27

Page 22: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Preparing for part: local part

> psql rootdb < partconf.sql

> psql sharddb_0 < partconf.sql

> psql sharddb_1 < partconf.sql

> psql sharddb_0

=> insert into partconf.conf(part_nr, max_part)

values (0,1);

> psql sharddb_1

=> insert into partconf.conf(part_nr, max_part)

values (1,1);

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 20 / 27

Page 23: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Preparing for part: add-table

> londiste3 st3partsplit/st3_rootdb.ini

add-table pgbench_accounts

--handler=part --handler-arg=key=aid

> londiste3 st3partsplit/st3_sharddb_0.ini

add-table pgbench_accounts --create

--handler=part --handler-arg=key=aid

> londiste3 st3partsplit/st3_sharddb_1.ini

add-table pgbench_accounts --create

--handler=part --handler-arg=key=aid

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 21 / 27

Page 24: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Other Handlers

X applyfn

X bulk

X dispatch

X multimaster

X part

X qtable

X vtable

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 22 / 27

Page 25: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Handler: bulk

bulk loading with 3 options

X correct: COPY, COPY temp + UPDATE, COPYtemp + DELETE

X delete: correct + update devient DELETE +COPY

X merged: merge insert rows with update rows

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 23 / 27

Page 26: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Handler: dispatch

bulk monthly batch

X bulk

X hourly daily monthly yearly

X event batch field time

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 24 / 27

Page 27: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Handler: dispatch

X table mode part, direct, ignore

X part mode batch time, event,time, date field, current time

X part field date field

X period hour, day, month, year

X row mode plain, keep latest,keep all

X event types I,U,D

X load mode direct, bulk

X method correct, delete,merged, insert

X fields field name mapping,no COPY support

X skip fields

X table

X pre part, post part

X encoding

X analyze

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 25 / 27

Page 28: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Extras

X check

X fkeys

X compare

X repair

X wait-sync

X wait-provider

X wait-root

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 26 / 27

Page 29: Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update devient DELETE + COPY X merged: merge insert rows with update rows Dimitri Fontaine dimitri@2ndQuadrant.fr

Questions?

Now is the time to ask!

Dimitri Fontaine [email protected] Skytools3 July, 18 2013 27 / 27