Dbvisit replicate: logical replication made easy

Infrastructure at your Service.

Dbvisit replicate:logical replication made easy

Dbvisit replicate: logical replication made easy 07.11.2014

About me

Franck Pachot

Senior consultant

Oracle Technology Leader

Mobile +41 79 963 27 22

franck.pachot@dbi-services.com

www.dbi-services.com

Presentations, Articles, blogs, forums

Experts At Your Service

> 35 specialists in IT infrastructure

> Certified, experienced, passionate

Based In Switzerland

> 100% self-financed Swiss company

> Over CHF 4 mio. turnover

Leading In Infrastructure Services

> More than 100 customers in CH, D, & F

> Over 30 SLAs dbi FlexService contracted

07.11.2014

dbi servicesWho we are

1. Why logical replication?

2. Demo (setup)

3. Configuration alternatives

4. Demo (run)

5. Why Dbvisit?

07.11.2014

Agenda

Logical vs. Physical

07.11.2014

Why Logical replication

SQLstatements

REDO LOGrecords

Applied on tables

Applied on files Applied on files

Physical replication

> Redo based

> Oracle EE or Dbvisit standby

07.11.2014

SQLstatements

REDO LOGrecords

Applied on tables

SQLstatements

Applied on tables

trigger

mview log

Logical replication

> Application based

> Oracle Streams

07.11.2014

REDO LOGrecords

Applied on tables

SQLstatements

Applied on tables

Logical replication

> Redo based

> Golden Gate or Dbvisit replicate

SQLstatements

REDO LOGrecords

Physical replication

> High performance, exact copy (HA)

> Same version (platform, database, application)

> No filter or transformation (same tables, columns, indexes, partitions)

> Oracle Data Guard (EE), Dbvisit standby, Manual standby and scripts

Logical replication based on redo

> More complex (supported datatypes, conflict resolution)

> Destination can differ (version, data model, data)

> We choose what we replicate (filter, change deletes to inserts)

> Destination is Read-Write, can be 2-ways

> Oracle Golden Gate, Dbvisit replicate, Streams

07.11.2014

Offloaded query environments

> Reporting, Data Quality

Distributed databases

> local proxies replicated to master

Data warehousing and Audit

> Extract data for ETL, can be real time

> Stores all modifications (old an new values)

Migration projects

> Oracle Edition downgrade (go to Standard Edition)

> Oracle upgrades, OS platform, to the Cloud, etc.

> 11g to PDB (without non_cdb_to_pdb.sql)

What are the use cases?

07.11.2014

Why logical replication

1. Why logical replication

2. Demo (setup)

4. Demo (run)

5. Why Dbvisit?

07.11.2014

Agenda

http://www.dbvisit.com/repattack/

07.11.2014

Demo (setup)

Two VirtualBox VM:

> source

> Linux Centos 6.5, Oracle XE 11g with XE database

> Swingbench schema, data and application

> target

> Linux Centos 6.5, Oracle XE 11g with XE database (empty)

Downloaded and installed dbvrep on both machines

> As root, on source and target:

> Run wizard:

http://screencast-o-matic.com/u/nTb6/dbvisit-repattack-setup

Demo environment

07.11.2014

Demo (setup)

root@source# rpm -ivh dbvisit_replicate-2.7.06-1.x86_64.rpm

oracle@source# dbvrep

Dbvrep> setup wizard

Setup wizard and initialization

07.11.2014

Demo (setup)

Setup Wizard

•Describe databases (ttorcl_src and ttorcl_trg)

•Define replication pairs (initialization with impdp flashback_scn)

•Define replicated tables (all REPOE schema)

•Configure processes (MINE on source, APPLY on target)

Initialization

•Run all.sh (creates dbvrep schema)

•Copy scripts (see Nextsteps.txt)

•Create DBLINK

•run apply.sh (impdp)

Start replication

•Start MINE process

•Start APPLY process

•Run start-console.sh

2. Demo (setup)

4. Demo (run)

5. Why Dbvisit?

07.11.2014

Agenda

Editions and Features

07.11.2014

Configuration alternatives

• Replicate data for <200 tables

• For reporting, datawarehouse

• up to 2000 tables. + fetcher, DDL, CLOB,RAC

• For migration (can be rented by month)

• Unlimited tables, 2-way

• Database distribution

CHF 3000 to 7000x SOCKET

MINE + APPLY

07.11.2014

Source server Target server

SQLPLOG

Dbvisitinternalformat PLOG

FETCHER + MINE + APPLY

07.11.2014

FETCHER

FETCHER + MINE + APPLY through SQL*Net

07.11.2014

Source server Target serverReplication server

FETCHER

APPLYSQL

Archivelog only MINE + APPLY

07.11.2014

Source server Target serverReplication server

REDOMINE

APPLYSQL

ARCHIVEDLOGS

ARC(not real time)

Target can be outside (no connect to source)

Apply can be outside (no connect to source)

TCP/IP Connections

07.11.2014

FETCHER

REDOMINE

CONSOLE

dp_networklink

> The simplest: impdp network_link= from target

dp_exp

> Create a dump file with expdp on source

> Obsolete exp/imp (old versions)

ddl_run

> Get metadata as DDL

Other alternatives:

> RMAN duplicate, Storage snapshots, etc.

Initialization alternatives to import data

07.11.2014

> Which changes are replicated by import?

> Which changes are replicated by MINE?

> We need to MIN complete transactions

We need to lock in Share mode to stop transactions when getting the SCN

Initialization alternatives to get SCN

07.11.2014

Transaction 1 Transaction 3 Transaction 5Transaction 2 Trans. 4 Trans. 6

Transaction 1 Transaction 3 Trans. 5

Transaction 2 Trans. 4 Trans. 6S

Data transferred by initialization Data transferred by replication

We must MINE complete transactions: cannot initialize when we have current transaction -> Lock in Share mode.

one-by-one

> Locks table one by one and get SCN for each table -> import one by one

single-scn

> Locks all tables to get one SCN -> easier import

ddl-only

> no lock, no data -> starts with empty tables (for audit)

resetlogs

> Using standby or duplicate instead of import

no-lock (since 2.7)

> Get SCN for the oldest transaction, no lock

> When supplemental logging must be already set

Initialization alternatives to get SCN

07.11.2014

2. Demo (setup)

4. Demo (run)

5. Why Dbvisit?

07.11.2014

Agenda

Setup wizard and initialization

07.11.2014

Demo (setup)

Setup Wizard

•Describe databases (ttorcl_src and ttorcl_trg)

•Define replication pairs (initialization with impdp flashback_scn)

•Define replicated tables (all REPOE schema)

•Configure processes (MINE on source, APPLY on target)

Initialization

•Run all.sh (creates dbvrep schema)

•Copy scripts (see Nextsteps.txt)

•Create DBLINK and run apply.sh (impdp)

Start replication

•Start MINE process

•Start APPLY process

•Run start-console.sh

> Table has no primary key and has duplicates (all columns have same values)

> By default, a redo record replication must touch one and only one row

> When we delete on source, the sql generated affect 2 rows

Conflict ‘too many rows’

> The first redo record received touches 2 rows

> We force do delete those 2 rows

> Then the second redo record touches no rows, we ignore it

Manual conflict resolution

07.11.2014

Additional demos

dbvrep> list conflict

dbvrep> resolve conflict <id from list conflict> resolve as FORCE

dbvrep> list conflict

dbvrep> resolve conflict <id from list conflict> resolve as IGNORE

Conflict handlers

> For automatic resolution, we choose to add a ‘where rownum=1’ in the generated sql.

> Syntax is available from online help

Conflict happen in logical replication, especially

> With delete cascade constraints

> With triggers doing DML

> With 2-way replication

Solution: either disable on target, or set conflict handlers

Automatic conflict resolution

07.11.2014

Additional demos

dbvrep> set_conflict_handlers for table REPOE.DEMODUP for DELETE on

TOO_MANY to sql s/$/ and rownum = 1/

dbvrep> help

dbvrep> help set_conflict_handlers

Reasons:

> Reorganization on source

> Bad conflict resolution, easier to restart then resolve

1. Stop replication

> The prepare gives the SCN for the initialization

2. Initialize the target table

3. Restart replication

Resynchronize a table

07.11.2014

Additional demos

dbvrep> unprepare table REPOE.DEMODUP

dbvrep> pause apply

dbvrep> prepare table REPOE.DEMODUP

SQL> truncate table REPOE.DEMODUP

SQL> insert /*+ append */ into REPOE.DEMODUP select * from

REPOE.DEMODUP@ttorcl_src as of scn <SCN given by the PREPARE>

dbvrep> resume

2. Demo (setup)

4. Demo (run)

5. Why Dbvisit?

07.11.2014

Agenda

Dbvisit Replicate vs. GoldenGate vs. Streams

07.11.2014

Why Dbvisit?

What Dbvisit says

Main differences we see:

> GoldenGate currently has more features but

> Dbvisit is per socket (CHF 2500 to 6000) Monthly rental possible

> Dbvisit optimistic apply: mine and apply don’t wait for commit

> Dbvisit is easy to start (setup wizard)

Know if you need logical replication

> It’s not the magic solution. It’s a project.

Testing (POC) is easy

> Install a Dbvisit replicate configuration in one hour

> test it on your data (datatypes, operations,etc.)

Logical replication is a DevOps project

> Operations is involved

supplemental logging, redo/archived, filesystem storage, network

> Application is involved

define conflict rules, trigger behavior, supportd datatypes,…

Core message

07.11.2014

Why Dbvisit?

Dbvisit replicate: logical replication made easy 07/11/2014Page 32

Any questions? Please do ask.

We look forward to working with you!

dbi services is the Dbvisit partner in Switzerland> We offer fixed prices for Dbvisit standby

> A POC for Dbvisit replicate is few days

> We have direct access to support

Repattack on 20th Nov. in Nuremberg (DOAG)> Dbvisit replicate on your laptop

Dbvisit replicate: logical replication made easy

Documents

Transcript of Dbvisit replicate: logical replication made easy

© 2015 Dbvisit Software Limited | dbvisit.com Welcome to Dbvisit Replicate Overview and Architecture.

replication of viral DNA II - Microbe · DNA Replication •Viruses must replicate their genomes to make new progeny •This always requires expression of at least one virus protein,

mrsg- Web viewWhat is another word for replication??? _____ Why must DNA replicate? so when cell divides, each new cell

6.4 Data and File Replication Gang Shen. Why replicate Performance Reliability Resource sharing Network resource saving.

Replication quality control of metal and polymer micro ...€¦ · characterization is to replicate the optical components using polymer casting. The replication method has been used

© 2014 Dbvisit Software Limited | dbvisit.com Welcome to Dbvisit Replicate Migrations Overview.

DNA Replication and Repair. How does DNA replicate? Mitosis and meiosis require DNA to replicate itself so that the daughter cells will contain DNA But.

© 2014 Dbvisit Software Limited | dbvisitdocs.huihoo.com/oracle/openworld/2014/CON2282-Internals-of-the... · © 2014 Dbvisit Software Limited | dbvisit.com ... • A lot of things

Using Physical Replication and Oracle® Database Standard ... · Dbvisit Standby provides a cost-effective, reliable and proven physical replication solution for Oracle Standard Edition

SAP LT Replication Server Overviewdocshare04.docshare.tips/files/25824/258248016.pdf · SAP LT Replication Server is used to load and replicate data in scheduled or real-time mode

DNA Replication Section 4.3 Page 217 Why do we need to replicate our DNA? When does DNA replication occur in a cell?

Dbvisit White Paper Why Not Standard Edition Comp

DNA Replication. Warm-UP What is the cell cycle? When does DNA replicate during the cell cyle?

Security in a Hacker’s World - Progress.com · •Off-site backup storage •Cloud storage is quickly growing in popularity Database Replication & Replication Plus •Replicate

Epidemic Techniques Chiu Wah So (Kelvin). Database Replication Why do we replicate database? – Low latency – High availability To achieve strong (sequential)

Quantact workshop on risk management of …Quantact workshop on risk management Why replicate hedge funds? Replication techniques Strong replication Weak replication Modeling of returns

Antiviral Agents. Understanding Viruses Viral Replication A virus cannot replicate on its own. A virus cannot replicate on its own. It must attach to.

Welcome to Dbvisit Replicate

HIGH VOLUME REPLICATION IN INGRESdownloads.actian.com/online/collaterals/wp/HVR-WP.pdf · replication, HVR can replicate files. This means file changes can be replicated alongside

GlusterFS: Arbiter based replication · GlusterFS: Arbiter based replication ... Software Engineer, Red Hat April 20th, VAULT- 2016 . 2 Agenda Introduction to replicate (AFR) volumes