Dbvisit replicate: logical replication made easy

Dbvisit replicate: logical replication made easy

Infrastructure at your Service.

Dbvisit replicate:logical replication made easy

Dbvisit replicate: logical replication made easy 07.11.2014


About me

Franck Pachot

Senior consultant

Oracle Technology Leader

Mobile +41 79 963 27 22

[email protected]

www.dbi-services.com

Presentations, Articles, blogs, forums

mailto:[email protected]

http://www.dbi-services.com/


Experts At Your Service

> 35 specialists in IT infrastructure

> Certified, experienced, passionate

Based In Switzerland

> 100% self-financed Swiss company

> Over CHF 4 mio. turnover

Leading In Infrastructure Services

> More than 100 customers in CH, D, & F

> Over 30 SLAs dbi FlexService contracted

07.11.2014

dbi servicesWho we are


1. Why logical replication?

2. Demo (setup)

3. Configuration alternatives

4. Demo (run)

5. Why Dbvisit?

07.11.2014

Agenda


Logical vs. Physical

07.11.2014

Why Logical replication

REDO

DB

SQLstatements

REDO LOGrecords

REDO LOGrecords

Applied on tables

Applied on files Applied on files

APPLY

Physical replication

> Redo based

> Oracle EE or Dbvisit standby

redo

redo



07.11.2014


REDO

DB

SQLstatements

REDO LOGrecords

REDO LOGrecords

Applied on tables


SQLstatements

Applied on tables

APPLY

trigger

mview log

sql

Logical replication

> Application based

> Oracle Streams



07.11.2014


REDO

DB

REDO LOGrecords

Applied on tables


SQLstatements

Applied on tables

sql

Logical replication

> Redo based

> Golden Gate or Dbvisit replicate

SQLstatements

MINE

redo

REDO LOGrecords

APPLY


Physical replication

> High performance, exact copy (HA)

> Same version (platform, database, application)

> No filter or transformation (same tables, columns, indexes, partitions)

> Oracle Data Guard (EE), Dbvisit standby, Manual standby and scripts

Logical replication based on redo

> More complex (supported datatypes, conflict resolution)

> Destination can differ (version, data model, data)

> We choose what we replicate (filter, change deletes to inserts)

> Destination is Read-Write, can be 2-ways

> Oracle Golden Gate, Dbvisit replicate, Streams


07.11.2014



Offloaded query environments

> Reporting, Data Quality

Distributed databases

> local proxies replicated to master

Data warehousing and Audit

> Extract data for ETL, can be real time

Audit

> Stores all modifications (old an new values)

Migration projects

> Oracle Edition downgrade (go to Standard Edition)

> Oracle upgrades, OS platform, to the Cloud, etc.

> 11g to PDB (without non_cdb_to_pdb.sql)

What are the use cases?

07.11.2014

Why logical replication


1. Why logical replication

2. Demo (setup)


4. Demo (run)

5. Why Dbvisit?

07.11.2014

Agenda


http://www.dbvisit.com/repattack/

07.11.2014

Demo (setup)


Two VirtualBox VM:

> source

> Linux Centos 6.5, Oracle XE 11g with XE database

> Swingbench schema, data and application

> target

> Linux Centos 6.5, Oracle XE 11g with XE database (empty)

Downloaded and installed dbvrep on both machines

> As root, on source and target:

> Run wizard:

http://screencast-o-matic.com/u/nTb6/dbvisit-repattack-setup

Demo environment

07.11.2014

Demo (setup)

root@source# rpm -ivh dbvisit_replicate-2.7.06-1.x86_64.rpm

oracle@source# dbvrep

Dbvrep> setup wizard

http://screencast-o-matic.com/u/nTb6/dbvisit-repattack-setup


Setup wizard and initialization

07.11.2014

Demo (setup)

Setup Wizard

•Describe databases (ttorcl_src and ttorcl_trg)

•Define replication pairs (initialization with impdp flashback_scn)

•Define replicated tables (all REPOE schema)

•Configure processes (MINE on source, APPLY on target)

Initialization

•Run all.sh (creates dbvrep schema)

•Copy scripts (see Nextsteps.txt)

•Create DBLINK

•run apply.sh (impdp)

Start replication

•Start MINE process

•Start APPLY process

•Run start-console.sh



2. Demo (setup)


4. Demo (run)

5. Why Dbvisit?

07.11.2014

Agenda


Editions and Features

07.11.2014

Configuration alternatives

LTD

• Replicate data for <200 tables

• For reporting, datawarehouse

XTD

• up to 2000 tables. + fetcher, DDL, CLOB,RAC

• For migration (can be rented by month)

MAX

• Unlimited tables, 2-way

• Database distribution

CHF 3000 to 7000x SOCKET


MINE + APPLY

07.11.2014


Source server Target server

DBDB

REDO

SQLPLOG

Dbvisitinternalformat PLOG

APPLY

MINE


FETCHER + MINE + APPLY

07.11.2014



DBDB

PLOG

REDO

REDO

SQL

APPLY

FETCHER

MINE


FETCHER + MINE + APPLY through SQL*Net

07.11.2014


Source server Target serverReplication server

DBDB

FETCHER

PLOG

REDO

REDO

MINE

APPLYSQL


Archivelog only MINE + APPLY

07.11.2014


Source server Target serverReplication server

DBDB

PLOG

REDOMINE

APPLYSQL

ARCHIVEDLOGS

ARC(not real time)


Target can be outside (no connect to source)

Apply can be outside (no connect to source)

TCP/IP Connections

07.11.2014



FETCHER

REDOMINE

APPLY

CONSOLE

DBDB


dp_networklink

> The simplest: impdp network_link= from target

dp_exp

> Create a dump file with expdp on source

exp

> Obsolete exp/imp (old versions)

ddl_run

> Get metadata as DDL

Other alternatives:

> RMAN duplicate, Storage snapshots, etc.

Initialization alternatives to import data

07.11.2014


sql


> Which changes are replicated by import?

> Which changes are replicated by MINE?

> We need to MIN complete transactions

We need to lock in Share mode to stop transactions when getting the SCN

Initialization alternatives to get SCN

07.11.2014


Transaction 1 Transaction 3 Transaction 5Transaction 2 Trans. 4 Trans. 6

?

Transaction 1 Transaction 3 Trans. 5

Transaction 2 Trans. 4 Trans. 6S

CN

lock

Data transferred by initialization Data transferred by replication

Data transferred by initialization Data transferred by replication


We must MINE complete transactions: cannot initialize when we have current transaction -> Lock in Share mode.

one-by-one

> Locks table one by one and get SCN for each table -> import one by one

single-scn

> Locks all tables to get one SCN -> easier import

ddl-only

> no lock, no data -> starts with empty tables (for audit)

resetlogs

> Using standby or duplicate instead of import

no-lock (since 2.7)

> Get SCN for the oldest transaction, no lock

> When supplemental logging must be already set

Initialization alternatives to get SCN

07.11.2014




2. Demo (setup)


4. Demo (run)

5. Why Dbvisit?

07.11.2014

Agenda


Setup wizard and initialization

07.11.2014

Demo (setup)

Setup Wizard

•Describe databases (ttorcl_src and ttorcl_trg)

•Define replication pairs (initialization with impdp flashback_scn)

•Define replicated tables (all REPOE schema)

•Configure processes (MINE on source, APPLY on target)

Initialization

•Run all.sh (creates dbvrep schema)

•Copy scripts (see Nextsteps.txt)

•Create DBLINK and run apply.sh (impdp)

Start replication

•Start MINE process

•Start APPLY process

•Run start-console.sh


Issue

> Table has no primary key and has duplicates (all columns have same values)

> By default, a redo record replication must touch one and only one row

> When we delete on source, the sql generated affect 2 rows

Conflict ‘too many rows’

> The first redo record received touches 2 rows

> We force do delete those 2 rows

> Then the second redo record touches no rows, we ignore it

Manual conflict resolution

07.11.2014

Additional demos

dbvrep> list conflict

dbvrep> resolve conflict <id from list conflict> resolve as FORCE

dbvrep> list conflict

dbvrep> resolve conflict <id from list conflict> resolve as IGNORE


Conflict handlers

> For automatic resolution, we choose to add a ‘where rownum=1’ in the generated sql.

> Syntax is available from online help

Conflict happen in logical replication, especially

> With delete cascade constraints

> With triggers doing DML

> With 2-way replication

Solution: either disable on target, or set conflict handlers

Automatic conflict resolution

07.11.2014

Additional demos

dbvrep> set_conflict_handlers for table REPOE.DEMODUP for DELETE on

TOO_MANY to sql s/$/ and rownum = 1/

dbvrep> help

dbvrep> help set_conflict_handlers


Reasons:

> Reorganization on source

> Bad conflict resolution, easier to restart then resolve

1. Stop replication

> The prepare gives the SCN for the initialization

2. Initialize the target table

3. Restart replication

Resynchronize a table

07.11.2014

Additional demos

dbvrep> unprepare table REPOE.DEMODUP

dbvrep> pause apply

dbvrep> prepare table REPOE.DEMODUP

SQL> truncate table REPOE.DEMODUP

SQL> insert /*+ append */ into REPOE.DEMODUP select * from

REPOE.DEMODUP@ttorcl_src as of scn <SCN given by the PREPARE>

dbvrep> resume



2. Demo (setup)


4. Demo (run)

5. Why Dbvisit?

07.11.2014

Agenda


Dbvisit Replicate vs. GoldenGate vs. Streams

07.11.2014

Why Dbvisit?

What Dbvisit says

Main differences we see:

> GoldenGate currently has more features but

> Dbvisit is per socket (CHF 2500 to 6000) Monthly rental possible

> Dbvisit optimistic apply: mine and apply don’t wait for commit

> Dbvisit is easy to start (setup wizard)


Know if you need logical replication

> It’s not the magic solution. It’s a project.

Testing (POC) is easy

> Install a Dbvisit replicate configuration in one hour

> test it on your data (datatypes, operations,etc.)

Logical replication is a DevOps project

> Operations is involved

supplemental logging, redo/archived, filesystem storage, network

> Application is involved

define conflict rules, trigger behavior, supportd datatypes,…

Core message

07.11.2014

Why Dbvisit?

Dbvisit replicate: logical replication made easy 07/11/2014

Any questions? Please do ask.


We look forward to working with you!

dbi services is the Dbvisit partner in Switzerland> We offer fixed prices for Dbvisit standby

> A POC for Dbvisit replicate is few days

> We have direct access to support

Repattack on 20th Nov. in Nuremberg (DOAG)> Dbvisit replicate on your laptop

Dbvisit replicate: logical replication made easy

Documents

Transcript of Dbvisit replicate: logical replication made easy