Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

48
©Continuent 2014 Replicate Oracle to Oracle, Oracle to MySQL , and Oracle to Analytics Linas Virbalas, Senior Software Engineer Robert Hodges, CEO

description

Oracle is the most powerful DBMS in the world. However, Oracle's expensive and complex replication makes it difficult to build highly available applications or move data in real-time to data warehouses and popular databases like MySQL. In this webinar you will learn how Continuent Tungsten solves problems with Oracle replication at a fraction of the cost of other solutions and with less management overhead too – think "Oracle GoldenGate without the price tag!" We will demo constructing a highly available site using Oracle-to-Oracle replication. We will then show you how to replicate data in real time from Oracle to MySQL as well as load a data warehouse.

Transcript of Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

Page 1: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

Replicate Oracle to Oracle, Oracle to MySQL ,

and Oracle to Analytics

Linas Virbalas, Senior Software Engineer Robert Hodges, CEO

Page 2: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

Introducing Continuent

���2

• The leading provider of clustering and replication for open source DBMS

• Our Product: Continuent Tungsten

• Clustering - Commercial-grade HA, performance scaling and data management for MySQL

• Replication - Flexible, high-performance data movement

Page 3: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

What is Tungsten Replicator?

���3

Tungsten Replicator: a high-performance

open source database replication engine

!GPL V2 license - 100% open source

Download from https://code.google.com/p/tungsten-replicator/ Annual support subscription available from Continuent

“GoldenGate without the Price Tag”®

Page 4: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

Tungsten Replicator Overview

���4

Master

(Transactions + Metadata)

Slave

THL

DBMS Logs

Replicator

(Transactions + Metadata)

THLReplicator

Download transactions via network

Apply using JDBC

Page 5: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014 ���5

Extract Filter Apply

StageExtract Filter Apply

StageExtract Filter Apply

Stage

Pipeline

Master DBMS

Transaction History Log

QueueSlave

DBMS

Tungsten Replication Service

Page 6: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

Filters and Parallel Apply

���6

Extract Filter Apply

StageExtract Filter Apply

StageStage

Pipeline

Master DBMS

Transaction History Log

In-Memory Queue

Slave DBMS

Extract Filter ApplyExtract Filter ApplyExtract Filter Apply

Page 7: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

Multiple Services per Replicator

Service frommysql

Service fromoracle

frommysql

fromoracle

Replicator

Replicator

Replicator

Replicator

Aggregated

Page 8: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

star

master-slave Heterogeneous

fan-in slave all-masters

MySQL

Oracle

Oracle

MySQL Oracle

Oracle

MySQL MySQL

Page 9: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Heterogeneous Replication

Page 10: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Steps to Heterogeneous Replication

1. Prepare (translate) schema for the slave DBMS

2. Set up replication

3. Provision initial data

4. Continue real-time replication

ddlscan

tpm

Parallel ApplyMySQL Blackhole

Tungsten Replicator

Parallel Extract

Page 11: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Replicating from MySQL to Oracle

Page 12: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Use Case: Real-Time Sales/Order Fulfillment

Real-Time Replication

Web-based Sales Purchase Ordering

3M Order Items

~1M Xacts/ Day

Page 13: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

1. Translating schema for the slave

Page 14: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Translating Schema

���14

• Goal - set up heterogeneous replication

Replication

Page 15: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Translating Schema

���15

• Beginning - how to convert tables?

empty

Tables

Page 16: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Translating Schema

���16

• Beginning - how to convert tables?

empty

Tables

•Data types? •Column lengths? •Naming conventions? •Reserved words?

Page 17: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Translating Schema

���17

• ddlscan automates these conversions

empty

Tables

ddlscan

Page 18: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

ddlscan

���18

• Part of Tungsten Replicator, GPL v2

• Translates schema with replication in mind

• Provides errors and warnings

• Can rename schema/tables/columns

Page 19: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Usage (MySQL to Oracle Example)

���19

$ cd tungsten-replicator/bin !

$ ./ddlscan \ -db test \ -template ddl-mysql-oracle.vm \ -user tungsten \ -pass secret

Page 20: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Translating Schema

���20

• ddlscan looks into source schema

empty

Tables

Page 21: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Translating Schema

���21

• ddlscan translates and renders DDL commands

empty

Tables

oracle-ddl.sql

Page 22: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Result of ddlscan

���22

CREATE TABLE 3colors /* ERROR: table starts with a number, use rename option and a filter */

( id NUMBER(10, 0), color VARCHAR2(1) /* ENUM('R','G','B') */, enabled NUMBER(1) /* BIT(1) - no constraints for 0 or 1 */, acolumnthatdoesntfittooracleat NUMBER(10, 0)

/* WARN: truncated column name exceeding 30 characters (acolumnthatdoesntfittooracleatall) */

! /* ERROR: table must have a primary key! */ ); !CREATE TABLE talks ...

Page 23: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Translating Schema

���23

• You run resulting SQL file on Oracle

empty

Tables

oracle-ddl.sql

Page 24: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Translating Schema

���24

• Tables are ready!

Tables Translated tables (empty)

Page 25: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

2. Set Up Replication

Page 26: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Replicator Installation

���26

./tools/tpm configure my_ora \ --enable-heterogenous-service=true \ --install-directory=/opt/mysql_to_ora/continuent \ --members=alpha,bravo \ --master=alpha !./tools/tpm configure my_ora --hosts=alpha \ --replication-user=tungsten \ --replication-password=secret !./tools/tpm configure my_ora --hosts=bravo \ --replication-user=tungsten_frommysql \ --replication-password=secret \ --datasource-type=oracle \ --datasource-oracle-service=ORCL \ --svc-applier-filters=dropstatementdata !./tools/tpm install my_ora !./tools/tpm start my_ora

!

Page 27: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Replication Under the Hood

���27

MySQL Binlog

binlog_format=row

Service my_ora

Tungsten Master Replicator

MySQLExtractor Special Filters •Transform

ENUM to string •Transform SET

to string •Column names

and signed flag

Service my_ora

Tungsten Slave Replicator

Special Filters •Drop DDL •Map names to upper case •Rename too long or reserved

objects •Optimize updates to remove

unchanged columns •Ignore extra tables

OracleApplierMySQLExtractor

Page 28: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

3. Provisioning an Oracle slave

Page 29: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

table structure

table data

table structure

Provisioning (2)

���29

table data

table structuretable data

table structure table structuretable structureddlscan

Page 30: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

table structure

Provisioning (3)

���30

table structuretable structure

MySQL Sandbox

CONVERT to BLACKHOLE

table data

table structuretable data

table structuretable data

table structure

Page 31: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Provisioning (4)

���31

MySQL Sandbox

table data

table structuretable data

table structuretable data

table structure

not replicating

backup

Page 32: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Provisioning (5)

���32

MySQL Sandbox

backup

bin logs

THL

1

(ROW)

Master Tungsten

Replicator

Slave Tungsten

Replicator

2 3 4

Page 33: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

4. Continue Real-Time Replication

Page 34: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

Provisioning (6)

���34

MySQL Sandbox

table data

table structuretable data

table structuretable data

table structure

bin logs

THL

Master Tungsten

Replicator

Slave Tungsten

Replicator

Page 35: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

Replicating from Oracle to MySQL

Page 36: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

Use Case: Web Content Publishing

Real-Time Publication

Backend Office Web-Based Catalog

Page 37: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

How Does Tungsten Do That?

���37

Service ora_my

Tungsten Master Replicator

OracleCDCExtractor No Special Filters

Service ora_my

Tungsten Slave Replicator

Special Filters •Map names to lower case •Ignore extra tables •Heartbeat table renaming

demo schema

demo_pub schema

Oracle CDC (Synchronous or

Async Hotlog)

MySQLApplierOracleCDCExtractor

Page 38: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

Replicating from Oracle to Oracle

Page 39: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

How Does Tungsten Do That?

���39

Service ora_ora

Tungsten Master Replicator

OracleCDCExtractor No Special Filters

Service ora_ora

Tungsten Slave Replicator

Special Filters No Special Filters

demo schema

demo_pub schema

Oracle CDC (Synchronous or

Async Hotlog)

Tungsten Slave Replicator

OracleApplierOracleCDCExtractor

Page 40: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014©Continuent 2014

DEMO

DEMO.* demo.*

DEMO2.* demo2.*

Page 41: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

Replicating from Oracle to Vertica

Page 42: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

Use Case: Web Content Publishing

Near Real-Time Publication

Backend Office Analytic Reports

Feeds from other data sources

Page 43: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

How Does Tungsten Do That?

���43

Service ora_ver

Tungsten Master Replicator

OracleCDCExtractor No Special Filters

Service ora_ver

Tungsten Slave Replicator

Special Filters •Map names to lower case •Ignore extra tables •Rename data tables (if required)

demo schema

demo_pub schema

Oracle CDC (Synchronous or

Async Hotlog)

SimpleBatchApplierOracleCDCExtractor

Page 44: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

Batch Loading--The Gory Details

���44

Replicator

Transactions from master

CSV FilesCSV FilesCSV Files

Staging TablesStaging TablesStaging Tables

Base Tables

Base Tables

Base Tables

Merge Script

(or) COPY

directly to base tables

COPY to stage tables SELECT to

base tables

Service ora_ver

Page 45: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

DEMO

Page 46: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

Coming Attractions

Page 47: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014

New Replication Features in 2014

• Parallel extraction for fast provisioning

• Select data as of specific SCN

• Pull data across in multiple streams

• Matches parallel apply on slave side

!

• Hadoop data loading

• Direct loading from MySQL/Oracle to HDFS

• View data using Hive

• Generation of materialized views

���47

Page 48: Replicate Oracle to Oracle, Oracle to MySQL, and Oracle to Analytics

©Continuent 2014 ���48

Continuent Web Page: http://www.continuent.com

!Documentation:

https://docs.continuent.com !

Tungsten Replicator: http://code.google.com/p/tungsten-replicator

Our Blogs: http://flyingclusters.blogspot.com http://scale-out-blog.blogspot.com http://datacharmer.blogspot.com http://continuent-tungsten.blogspot.com

560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1 (866) 998-3642 Fax +1 (408) 668-1009 e-mail: [email protected]