Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update...
Transcript of Skytools3 - PostgreSQL trigger-based replication · temp + DELETE X delete: correct + update...
Skytools3PostgreSQL trigger-based replication
Dimitri Fontaine [email protected]
July, 18 2013
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 1 / 27
Dimitri Fontaine
2ndQuadrant France
PostgreSQL Major Contributor
X pgloader, prefix, skytools, . . .
X apt.postgresql.org
X CREATE EXTENSION
X CREATE EVENT TRIGGER
X MySQL migration tool, new pgloader version
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 2 / 27
Dimitri Fontaine
2ndQuadrant France
PostgreSQL Major Contributor
X pgloader, prefix, skytools, . . .
X apt.postgresql.org
X CREATE EXTENSION
X CREATE EVENT TRIGGER
X MySQL migration tool, new pgloader version
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 2 / 27
Dimitri Fontaine
2ndQuadrant France
PostgreSQL Major Contributor
X pgloader, prefix, skytools, . . .
X apt.postgresql.org
X CREATE EXTENSION
X CREATE EVENT TRIGGER
X MySQL migration tool, new pgloader version
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 2 / 27
Skytools
Skype Tools for Replication
X PGQ
X londiste
X walmgr
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 3 / 27
Skytools Architecture
PGQ is organized into 3 components
Producer, Consumer, Ticker
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 4 / 27
PGQ3
Some things did change in PGQ version 3
X The ticker is now pgqd
X The network topology is now known pgq nodes
X And we have Cooperative Consumers pgq coop
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 5 / 27
Londiste3 concepts
Londiste relies on PGQ nodes
X Root
X Branch
X Leaf
X Leaf –merge=’qname’
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 6 / 27
Operating londiste
Basic commands
X status
X members
X change-provider
X takeover –all –dead
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 7 / 27
Londiste Add Table
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 8 / 27
londiste add-table
Plenty new options in add-table
X --wait-sync
X --dest-table
X --skip-truncate
X --create, --create-full
X --trigger-flags
X --trigger-arg
X --no-triggers
X --copy-node,
--copy-condition
X --merge-all, --no-merge
X --max-parallel-copy
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 9 / 27
DDL
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 10 / 27
DDL Handling
X londiste conf.ini execute
X EXECUTE queue events
X System Table londiste.applied execute
X SQL meta-data attributes
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 11 / 27
execute Meta-Data attributes
--*--
--*-- Local-Table: mytable, othertable,
--*-- thirdtable
--*-- Local-Sequence: thisseq
--*--
X Local-Table Table must be added to local node with add-table.
X Need-Table Physical table must exist in database. It does not matterif it is replicated or not.
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 12 / 27
DDL and table renaming
--*-- Local-Table: mytable
ALTER TABLE @mytable@ ...;
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 13 / 27
Londiste Handlers
It’s possible to register handlers to deal with specific needs
X add --handler [ --handler-arg ... ]
X show-handlers
X pl/proxy sharding, re-sharding
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 14 / 27
Sharding pluging
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 15 / 27
Handler: shard
X Event filtering by hash, for partitioned databases.
X key=COLUMN column name to use for hashing
X hashfunc=FUNCNAME function to use for hashing
X (default: partconf.get hash raw)
X ev extra3=’hash=’||partconf.get hash raw(key column)
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 16 / 27
Preparing for part: hashlib
> ... add-table pgbench_accounts \
> --handler=part --handler-arg=key=aid
psql rootdb < /usr/share/postgresql/8.4/contrib/hashlib.sql
psql sharddb_0 < /usr/share/postgresql/8.4/contrib/hashlib.sql
psql sharddb_1 < /usr/share/postgresql/8.4/contrib/hashlib.sql
> psql rootdb -c ’create extension hashlib;’
> psql sharddb_0 -c ’create extension hashlib;’
> psql sharddb_1 -c ’create extension hashlib;’
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 17 / 27
Preparing for part: setup
CREATE SCHEMA partconf;
CREATE TABLE partconf.conf (
part_nr integer,
max_part integer,
db_code bigint,
is_primary boolean,
max_slot integer,
cluster_name text
);
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 18 / 27
Preparing for part: get hash raw
CREATE FUNCTION partconf.get_hash_raw
( i_input integer)
RETURNS integer
LANGUAGE sql
AS $$
-- used to wrap hashtext so that we can replace it in 8.4
-- with older implementation to keep compatibility
select hash_string(\$1::text, ’lookup2’);
$$;
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 19 / 27
Preparing for part: local part
> psql rootdb < partconf.sql
> psql sharddb_0 < partconf.sql
> psql sharddb_1 < partconf.sql
> psql sharddb_0
=> insert into partconf.conf(part_nr, max_part)
values (0,1);
> psql sharddb_1
=> insert into partconf.conf(part_nr, max_part)
values (1,1);
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 20 / 27
Preparing for part: add-table
> londiste3 st3partsplit/st3_rootdb.ini
add-table pgbench_accounts
--handler=part --handler-arg=key=aid
> londiste3 st3partsplit/st3_sharddb_0.ini
add-table pgbench_accounts --create
--handler=part --handler-arg=key=aid
> londiste3 st3partsplit/st3_sharddb_1.ini
add-table pgbench_accounts --create
--handler=part --handler-arg=key=aid
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 21 / 27
Other Handlers
X applyfn
X bulk
X dispatch
X multimaster
X part
X qtable
X vtable
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 22 / 27
Handler: bulk
bulk loading with 3 options
X correct: COPY, COPY temp + UPDATE, COPYtemp + DELETE
X delete: correct + update devient DELETE +COPY
X merged: merge insert rows with update rows
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 23 / 27
Handler: dispatch
bulk monthly batch
X bulk
X hourly daily monthly yearly
X event batch field time
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 24 / 27
Handler: dispatch
X table mode part, direct, ignore
X part mode batch time, event,time, date field, current time
X part field date field
X period hour, day, month, year
X row mode plain, keep latest,keep all
X event types I,U,D
X load mode direct, bulk
X method correct, delete,merged, insert
X fields field name mapping,no COPY support
X skip fields
X table
X pre part, post part
X encoding
X analyze
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 25 / 27
Extras
X check
X fkeys
X compare
X repair
X wait-sync
X wait-provider
X wait-root
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 26 / 27
Questions?
Now is the time to ask!
Dimitri Fontaine [email protected] Skytools3 July, 18 2013 27 / 27