GoldenGate CDR from UKOUG 2017

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Oracle GoldenGate 12.3:Conflict, Detection and ResolutionBobby Curtis, EMBASenior Principal Product Manager - CoreOracle GoldenGate

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 3

Speaker

@dbasolved

https://dbasolved.com

[email protected]

https://www.linkedin.com/in/bobbycurtis294/

https://www.slideshare.net/BobbyCurtisMBA

12/15/2017 4Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Agenda Oracle GoldenGate 12c Overview

Oracle GoldenGate 12c Architecture

Conflict Management

Requirements

New Feature

Examples

Monitoring

Oracle GoldenGate

Real time Performance

Extensible & Flexible

Proven & Reliable

Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of database transactions across homogeneous and heterogeneous environments in real-time with no distance limitations.

MostDatabases

DataEvents

Transaction Streams

Cloud

DBs

Big Data

Supports Databases, Big Data and NoSQL:

* The most popular enterprise integration tool in history

Copyright © 2017, Oracle and/or its affiliates. All rights reserved.

Where does GoldenGate fit?

BusinessData

ServingLayer

Apps

Analytics

EDWs

Batch Layer

Data Streams

Social and Logs

Enterprise Data

Highly AvailableDatabases

Analytics

Speed LayerPub / Sub

REST APIs

NoSQL

Bulk Data

Raw DataLayer

GoldenGate

Data Integrator

Stream Analytics

Event Hub Big Data Database

Metadata Management (for Data Governance)

Big Data SQL

Oracle Software can help customers Accelerate & Reduce Risk around adoption:• Ingest Data with lower latency, greater reliability and from any database using Oracle GoldenGate


Oracle GoldenGate Use Cases

New DB/HW/OS/APP

Fully Active Distributed DB

ApplicationOffloading

Big Data, DW & Marts

Message Bus& Data Grid

Global Data Centers

DataStreaming

GoldenGate

Databases

Applications

Big Data and No SQL

Mainframes

Mobile

Streaming & Messaging

Cap

ture

Del

iver

Dis

trib

ute

Rec

eiv

e

Secu

re

Oracle

SQL Server

MySQL

IBM DB2 Z

IBM DB2 i

IBM DB2 LUW

HP NonStop

Informix

Sybase

Messaging

High Availability

Zero Downtime Migration

Data Synchronization

Big Data Ingestion

Query/Report Offloading

Stream Analytics

Data Bus (Lambda/Kappa)

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. 12/15/2017 8Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |



Conflict Management

Requirements

New Feature

Examples

Monitoring


Oracle GoldenGate ArchitectureClassic

Capture Trail

FilesPump DeliveryTrail

Files

Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction logs. As of V.11.2.1, GoldenGate offers two options for capture for Oracle; Classic & Integrated Capture

Trail: stages and queues data for routing.

Pump: distributes data for routing to target(s).

Route: data is compressed, encrypted for routing to target(s).

Delivery: applies data with transaction integrity. New with GoldenGate 12c, Integrated Delivery.

12.2-


Oracle GoldenGate ArchitectureMicroservices

Capture Trail Files Trail Files DeliveryDistribution

ServiceNetwork Receiver

Service

Administration Service

Metrics Service

ServiceManager

>HTML>>SQL >>CLI >>API

Customizable Interactive Experience

Deploy -- Embed -- Automate

Proxy/Reverse Proxy

12.3+




Conflict Management

Requirements

New Feature

Examples

Monitoring


Conflict Management• Most Active/Active or Multi-Master systems

will have some form of conflict

• Conflicts happen due to at least one row being modified on more than one system


Conflict ManagementExample - Financial Institution: Active/Active

ACCTINFO

ID (PK)

NAME

ADDRESS

CITY

BALANCE

ACCTINFO

ID (PK)

NAME

ADDRESS

CITY

BALANCE

AcctInfo Table: General details related to a person’s bank account at a national bank.

ServiceManager ServiceManager

Distribution

Service

Receiver

Service

Distribution

Service

Receiver

Service

Admin. Service

Admin. Service

EXT1

EXT2

REP1

REP2


Conflict ManagementExample - Financial Institution: Active/Active

ACCTINFO

ID (PK)

NAME

ADDRESS

CITY

BALANCE

CHANGE_TS

ACCTINFO

ID (PK)

NAME

ADDRESS

CITY

BALANCE

CHANGE_TS

ServiceManager ServiceManager

Distribution

Service

Receiver

Service

Distribution

Service

Receiver

Service

Admin. Service

Admin. Service

EXT1

EXT2

REP1

REP2

CHANGE_TS CHANGE_TS

• 12.2 and earlier - Tables should contain timestamp columns• 12.3 and later - Timestamp column can be added without effecting the application or code

base - leverages invisible columns/RDBMS 12.2 kernel


Conflict ManagementSupported Conflicts

Insert Conflict ✔ ✔

Update Conflict ✔ ✔

Delete Conflict ✔ ✔

Delta Resolution ✔ ✔

Insert-Update/Delete Conflict ✖ ✔

Column Group Conflict ✖ ✔

Piecewise LOB Update ✖ ✔

Conflict Type Classic CDR Auto CDR


Conflict ManagementSupported Data Types

Datatypes that can be compared are supported:

• NUMERIC• DATE• TIMESTAMP• CHAR/NCHAR• VARCHAR/NVARCHAR

Typically these data types are used with parameters like COMPARECOLS, GETBEFORECOLS, and in resolution parameters using RESOLVECONFLICT ([USEMIN] | [USEMAX])


Conflict ManagementBefore and After Images

ACCTINFO BEFORE AFTER

ID (PK) 10 10

NAME 32 32

ADDRESS 96 Smith Rd 96 Smith Rd

CITY Atlanta Atlanta

BALANCE 1500 1550

CHANGE_TS2014-10-29 00.00.00.000000 AM

2014-10-30 01.32.05.000000 PM

• Transaction logs are needed for recovery

• Before Images • Deletes and Updates

• After Images• Inserts and Updates

Note: LogDump utility is useful for reading trail files to identify before/after images




Conflict Management

Requirements

New Feature

Examples

Monitoring


RequirementsEnvironment

Full before image of each record

Transactional Dataadminclient> dblogin user <gg user> password <password>

adminclient> add trandata SFAA.ACCTINFO, allcols

Behind the scene:SQL> ALTER TABLE SFAA.ACCTINFO ADD SUPPLEMENTAL LOG GROUP

GGS_94879(id, name, address, city, balance, change_ts)

ALWAYS;

Note: System generated log group names can be located in DBA_LOG_GROUPS


• Sequences need to be unique on each system• Use starting value and increment by

number of systems• Applies to existing sequences and identity

columns<site_number>+3

<site_number>+3 <site_number>+3

RequirementsEnvironment - Sequences


• LOGALLSUPCOLS

Extract captures before images for UPDATE operationsExtract captures before images of supplemental logged columns for both UPDATE and DELETE operations

• GETBEFORECOLS

Ensures certain columns are loggedGETBEFOREUPDATES (DB2 only)

TABLE option in extractGETBEFORECOLS (ON UPDATE ALL, ON DELETE ALL)

Example:TABLE SFAA.ACCTINFO, GETBEFORECOLS(ON UPDATE ALL, ON DELETE ALL);

RequirementsEnvironment - Extract Parameter File


• TRANSLOGOPTION EXCLUDETAG

• Tag supplied to transaction as it is extracted to prevent receiving replicat from trying to send it back to source• Classic/Integrated Extract (primary)

• Tag can be any number/letter [0-9 A-Z]

Example:TRANSLOGOPTION EXCLUDETAG 0294

RequirementsEnvironment - Extract Parameter File


• REPERROR

Used to control how Replicat responds to errors

Default:REPERROR(default, abend)

For CDR:REPERROR(default, exception)

REPERROR(default2, [ abend | discard ])

REPERROR(-1, exception)

RequirementsEnvironment - Replicat Parameter File


• COMPARECOLS

Used by Replicat to detect and resolve update/delete conflicts

• RESOLVECONFLICT

Used by Replicat in bi-directional/multi-master to handle conflicts for DML operations

Example:

MAP SFAA.ACCTINFO, TARGET ATLAA.ACCTINFO,

COMPARECOLS (ON UPDATE ALL, ON DELETE ALL),

RESOLVECONFLICT (INSERTROWEXISTS, (DEFAULT,USEMAX(CHANGE_TS)));

RequirementsEnvironment - Replicat Parameter File


• Exception Table[s]

• User defined

• Used for troubleshooting or handling errors

• Referenced in REPERROR and MAPparameters

• Best if defined through macros

RequirementsEnvironment - Exception Table


RequirementsExtract Parameter File

-- CHECKPARAMSEXTRACT EXTAAUSERID <gg user>, PASSWORD <pwd>TRANLOGOPTIONS DBLOGREADERTRANLOGOPTIONS EXCLUDETAG 0294SETENV (ORACLE_HOME="/u01/app/oracle/product/12.1.0/db12cr1")SETENV (ORACLE_SID="oragg")WARNLONGTRANS 10m, CHECKINTERVAL 5mLOGALLSUPCOLSEXTTRAIL ./dirdat/laWILDCARDRESOLVE IMMEDIATE

TABLE SFAA.ACCTINFO, GETBEFORECOLS(ON UPDATE ALL, ON DELETE ALL);


-- CHECKPARAMSREPLICAT REPABSETENV (ORACLE_HOME="/u01/app/oracle/product/12.1.0/db12cr1")SETENV (ORACLE_SID="oragg")USERID ggate, PASSWORD <pwd>DISCARDFILE ./dirrpt/REPAB.discard APPEND MEGABYTES 100REPERROR (default, exception)REPERROR (default2, abend)ASSUMETARGETDEFSWILDCARDRESOLVE IMMEDIATEINCLUDE ./dirmac/exceptions.mac

MAP ATLAA.ACCTINFO, target SFAA.ACCTINFO,COMPARECOLS (ON UPDATE ALL, ON DELETE ALL),RESOLVECONFLICT (insertrowexists, (default, usemax(change_ts))),RESOLVECONFLICT (updaterowexists, (default, usemin(change_ts))),RESOLVECONFLICT (deleterowmissing, (default, discard));

MAP ATLAA.ACCTINFO, #exception_handler(ggate);

RequirementsReplicat Parameter File




Conflict Management

Requirements

New Feature

Examples

Monitoring


New FeatureAutomatic Conflict Detection & Resolution Highlights

• Invisible timestamp columns automatically created and maintained by database kernel

• Delete tombstone log to support updates-delete conflicts (new capability)

• Enables supplemental logging on required columns• Allows ability to detect conflicts for LOB column changes• No need to specify CDR/REPERROR parameters• Last record wins (highest timestamp)

No Visible Application Changes

Requirements

• Requires source and target to be Oracle Database 12.2• Requires Integrated Extract (IE)

• Classic Extract (CE) does not support invisible columns

• Requires Integrated Replication (IR) or Integrated Parallel Replicat (IPR)

12.3


New Feature Simple to Enable

• Add parameter “MAPINVISIBLECOLUMNS” in your replicat parameter file

– Classic Replicat does not support invisible columns

• Run procedure ADD_AUTO_CDR or ADD_AUTO_CDR_COLUMN_GROUP

– Sets up Latest Timestamp CDR for a table

12.3


New FeatureADD_AUTO_CDR Example

SQL> connect ggadmin/***@inst1

Connected

SQL> BEGIN

dbms_goldengate_adm.add_auto_cdr(

schema_name => ′HR′,

table_name => ’EMPLOYEES’);

END;

/

PL/SQL procedure successfully completed.

12.3


Key Name CityEnable Auto CDR

Key Name City CDRTS$ROW

Statement TS

PKey Delete TS

Statement TS

Invisible

DT$_TAB1

TAB1

TAB1

• Automatically creates an invisible timestamp column to associate with the row• Invisible timestamp column automatically

maintained

• Automatically enables Delete Tombstone Log Table• Tombstone table automatically maintained

• Automatically enables all column supplemental logging

+


12.3


DML driver will automatically populate all theROW_TS timestamp columns with the currenttimestamp

Key Name City ROW_TS

1 Alice San Mateo @T2


DML driver will automatically populate the ROW_TS timestamp columns associated with the modified columns with the current timestamp

Automatically Record a tombstone record in the Delete Tombstone log. Another timestamp is used to record when the row was deleted.


1 Alice San Mateo @T2


1 Alice South SF @T5

Key Delete TS

1 @T12


1 Alice South SF @T5

Inserts Updates Deletes

Denotes fields automatically populated and maintained by database


12.3


• DBA_APPLY_ERROR_MESSAGES tracks Error Resolution

• Win/Loss-Monitoring

CONFLICT_TYPE APPLIED_STATE CONFLICT_INFO

-------------------------- ------------------ ----------------------

UPDATE ROW EXISTS PARTIAL DRTS$C:L,CDRTS$ROW:W

• Custom Conflict Handlers

– Apply DML (Error) Handler with own sophisticated business logic

– PL/SQL Procedure is linked to the Apply process

Additional Features12.3




Conflict Management

Requirements

New Feature

Examples

Monitoring


ExamplesInsert Row Exists



RESOLVECONFLICT (INSERTROWEXISTS, (DEFAULT, USEMAX(CHANGE_TS)));

ACCTINFO SF ATL

ID 944 944

NAME 32 32

ADDRESS 55 5th Street 55 5th Street

CITY Symrna Symrna

BALANCE 1100 1200

CHANGE_TS31-OCT-14 10.40.36.000000000 PM

31-OCT-14 10.40.40.000000000 PM

INSERTROWEXISTS

• Violates the unique constraint on the target


ExamplesUpdate Row Exists



RESOLVECONFLICT (UPDATEROWEXISTS, (DEFAULT, USEMIN(CHANGE_TS)));

ACCTINFO BEFORE AFTER

ID 944 944

NAME 32 32

ADDRESS 55 5th Street 55 5th Street

CITY Symrna Symrna

BALANCE 1150 1100

CHANGE_TS31-OCT-14 10.40.36.000000000 PM

31-OCT-14 10.40.40.000000000 PM

UPDATEROWEXISTS

• Updated row exists on target side

• One or more columns have a before image different from current value


ExamplesDelete Row



RESOLVECONFLICT (DELETEROWMISSING, (DEFAULT, [ DISCARD | IGNORE ]));

DELETEROWMISSING

• Deleted row does not exist on target

ACCTINFO SF ATL

ID 944 null

NAME 32 null

ADDRESS 55 5th Street null

CITY Symrna null

BALANCE 1100 null

CHANGE_TS31-OCT-14 10.40.36.000000000 PM null




Conflict Management

Requirements

New Feature

Examples

Monitoring


MonitoringStatistics

Replicating from ATLAA.ACCTINFO to SFAA.ACCTINFO:

*** Total statistics since 2014-11-02 00:30:43 ***Total inserts 719.00Total updates 93.00Total deletes 524.00Total discards 0.00Total operations 1336.00Total CDR conflicts 377.00CDR resolutions succeeded 377.00CDR INSERTROWEXISTS conflicts 257.00CDR UPDATEROWEXISTS conflicts 93.00CDR DELETEROWMISSING conflicts 27.00

adminclient> stats replicat <group name>, reportcdr


Additional Information

@DBASOLVED

https://dbasolved.com

[email protected]


CDR Info: https://docs.oracle.com/goldengate/c1230/gg-winux/GIORA/automatic-conflict-detection-and-resolution2.htm#GIORA-GUID-DF68483B-1F77-4464-B5E3-BB42B360E813

Installing Oracle GoldenGate: https://docs.oracle.com/goldengate/c1230/gg-winux/GIORA/toc.htm

http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

https://dbasolved.com/


https://docs.oracle.com/goldengate/c1230/gg-winux/GIORA/automatic-conflict-detection-and-resolution2.htm#GIORA-GUID-DF68483B-1F77-4464-B5E3-BB42B360E813

https://docs.oracle.com/goldengate/c1230/gg-winux/GIORA/toc.htm

http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

GoldenGate CDR from UKOUG 2017

Technology

Transcript of GoldenGate CDR from UKOUG 2017