Introduction Storage Failure Recovery Logging Undo Logging Redo Logging ARIES.
Data Replication With Oracle GoldenGate Looking Behind The ... München... · 7 8 9 n ....
Transcript of Data Replication With Oracle GoldenGate Looking Behind The ... München... · 7 8 9 n ....
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
– Looking Behind The Scenes –
Robert Bialek Principal Consultant Partner
Data Replication With Oracle GoldenGate
Who Am I
Principal Consultant, Partner and Trainer at Trivadis GmbH in Munich
Focus:
– Oracle Database High Availability
– Database Architecture/Internals
– Backup/Recovery
– Troubleshooting/Performance Tuning
– Linux Administration
Trainer for the following Trivadis courses
– Oracle Grid Infrastructure (O-GRINF)
– Oracle Real Application Cluster (O-RAC)
– Oracle Data Guard (O-DG)
2 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Our company.
Trivadis is a market leader in IT consulting, system integration, solution engineering
and the provision of IT services focusing on and technologies
in Switzerland, Germany, Austria and Denmark. We offer our services in the following
strategic business fields:
Trivadis Services takes over the interactive operation of your IT systems.
O P E R A T I O N
16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes 3
COPENHAGEN
MUNICH
LAUSANNE
BERN
ZURICH
BRUGG
GENEVA
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
VIENNA
With over 600 specialists and IT experts in your region.
Data Replication With Oracle GoldenGate - Looking Behind The Scenes 4 16.07.2015
14 Trivadis branches and more than
600 employees
200 Service Level Agreements
Over 4,000 training participants
Research and development budget:
CHF 5.0 / EUR 4 million
Financially self-supporting and
sustainably profitable
Experience from more than 1,900
projects per year at over 800 customers
16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes 5
Technology on its own won't help you. You need to know how to use it properly.
Agenda
6
1. Introduction
2. Oracle GoldenGate Architecture
3. Oracle GoldenGate Setup
4. Core Messages
16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Introduction
7 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
What is Oracle GoldenGate?
Oracle strategic Middleware software product used for data extraction and replication
– Use-cases: data/service high-availabillity, “zero-downtime” upgrades/migrations,
easy fallback (minimize risk)
Supports
– asynchronous data transfer (DML, DDL)
– heterogeneous sources and targets (databases, operating systems)
– different extraction, propagation and apply methods
– mappings, transformations, filtering, error handling, …
Topologies
– Active/passive, active/standby, active/active
– Broadcasting, cascading, consolidation
Source as well as the target environment requires a license
– Includes (EE): Oracle Active Data Guard and Oracle Global Data Services
(12.1.0.1+)
8 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Why Oracle GoldenGate and not Streams?
Oracle Streams is deprecated in the Oracle 12c version (but still supported!)
– https://docs.oracle.com/database/121/UPGRD/deprecated.htm
According to the Oracle GoldenGate Statement of Direction
– http://www.oracle.com/technetwork/middleware/goldengate/overview/statement-of-
direction-gg-132120.pdf
Beginning with the version 12.1.0.2, the non-CDB architecture is also deprecated
– As of now, Streams is not supported with the new CDB architecture
Oracle Streams continues to be supported but will not be enhanced to
support any new database features in future releases. This means that
Streams will continue to support exactly what it supports today (11.2)
in future releases. It also means that any new database features, data
types, or capabilities will not be supported by Streams.
9 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Oracle GoldenGate Architecture
10 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Oracle GoldenGate Architecture: Big Picture
11 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Source DB Target DB
Primary
(Regular)
Extract
Primary
(Passive)
Extract
Secondary
Extract
Data Pump
Collector
Replicat
Local Trail Remote Trail
Primary
(Alias)
Extract
Op
tio
nal
Oracle GoldenGate Instance Oracle GoldenGate Instance
Static
Configuration
Runtime
Configuration
Runtime
Configuration
Static
Configuration
Optional Optional
Manager
(VIP) Manager
(VIP)
MONITOR MONITOR
Primary Extract Group: Capture
Primary Extract can run in either
– Classic Capture mode (all database versions)
– Integrated Capture mode (full support: 11.2.0.3+)
Extracts and optionally filters/transforms the transactions from the source database
– Only for configured objects
Many Extracts per source database can operate at the same time
Only committed transactions (CSN) will be propagated to the target system (in
contrast to Oracle Streams…)
– Optionally written to a local trail (recommended)
– Without a local trail, the primary Extract send the data to the target system
– Not committed transactions will be discarded from memory or dirtmp directory
12
Primary
(Regular)
Extract
Primary
(Passive)
Extract
16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Source DB
Primary Extract Group: Classic Capture
Classic Capture mode
– Traditional mode, reads directly Oracle database redo logs
– Able to mine redo logs in ASM (two methods)
– All database versions supported
With some restrictions. For example no support for:
– BASIC, OLTP, Exadata Hybrid Columnar
compression
– PDML on RAC database
– CDB architecture!
DDL, TDE support – no out-of-the box support
Not transparent to RAC configuration changes (#threads)
13 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Source DB
Primary
Extract
Secondary
Extract
Data Pump Local Trail
Redo Logs
OS
System Calls
Primary Extract Group: Integrated Capture
Integrated Capture mode
– Interacts with the database logmining server
– Required to capture data from a CDB architecture
– Fully transparent with RAC, TDE, ASM
– Full support for databases 11.2.0.3+
– Source database 11.2.0.4+ does not require
supplementary objects for DDL support
– Supports more complex data types than classic
capture
– Integrated capture deployments
• Local or a downstream deployment
The best choice for most deployments
14 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Source DB
Primary
Extract
Secondary
Extract
Data Pump Local Trail
Redo Logs Logmaining
Server
LCR LCR
Replicat Process: Apply
The Replicat process can run in either
– Classic Replicat mode (nonintegrated) - all database versions
– Coordinated Replicat mode (nonintegrated) - all database versions, GoldenGate
12c required
– Integrated Replicat mode – database versions 11.2.0.4+
The Replicat process reads the remote trail, reconstructs
the SQL and applies them to the target database
– Optionally filters/transforms the data
The application of the data can be optionally delayed (DEFERAPPLYINTERVAL
parameter)
15 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Target DB
Replicat
Remote Trail
Replicat Group: Classic/Coordinated Mode
Classic Replicat
– Constructs SQL statements from the trails
and applies the SQL through OCI
– Supported for all database versions
– Transactions are applied sequentially
(single-threaded)
Coordinated Replicat
– Transactions are applied in parallel (multi-threaded)
– There is 1 coordinator plus
many apply threads
– Supported for all database
versions
16 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
OCI
SQL
Target DB
Replicat
Remote Trail
OCI
SQL
Target DB
Remote Trail
Coordinator
Thread 1
Thread 2
Thread n
Replicat Group: Integrated Mode
Integrated Replicat
– Supported for target database
versions 11.2.0.4+
– Constructs LCRs and transmits
DMLs to the database inbound
server
Some changes (e.g. DDL) are not supported directly by the inbound server
– In those cases the Replicat process disconnects from the inbound server and
executes the transaction in direct apply mode through OCI
With many transactions applied in direct apply mode it might be necessary to switch
to nonintegrated Replicat
17 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
LCR
Target DB
Replicat
Remote Trail
Database
Inbound
Server
SQL
OCI
Capture/Apply: CDB & Pluggable Databases
Extract must operate in integrated mode
– Extract must connect to the CDB$ROOT container as a common user (c##)
– One Extract can capture from one or many PDBs to a single trail
Replicat can connect and apply only to one PDB (one Apply/PDB, connect using
service)
Source DB – CDB
CDB$ROOT PDB1 PDB2
Target DB – CDB
CDB$ROOT PDB1 PDB2
Primary Extracts
INTEGRATED Data Pumps Collectors
Replicats
18 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
common_user_prefix
(12.1.0.2)
Oracle GoldenGate Setup
19 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Oracle GoldenGate: Setup Flow
20 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Architecture
Design
Software Installation/
Patches Database Preparation
Manager Configuration
Primary Extract, Data
Pump, Replicat
Configuration Files
Target Database
Instantiation
Register/Start Primary
Extract, Data Pump
Register/Start Replicat Cluster Integration /
INIT Scripts
Tuning/Customizing/
Maintenance/…
1 2 3
6 5 4
7 8 9
n
Supplemental Logging: Source
Activate minimal database level supplemental logging on the source database
– To add row chaining information to the redo stream
Add schema or table level supplemental logging on the source database
– Schema level logging is required for DDL support
For all schema tables
– Activates unconditional supplemental logging for PK and conditional for unique/foreign
keys (scheduling columns) – all existing an future tables
For a table
– Depending on the use-case, different parameters are necessary
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;
Database Preparation
3
21 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
ADD SCHEMATRANDATA <schema> [ALLCOLS]
ADD TRANDATA <schema>.<table> [COLS (…) | ALLCOLS]
Manager Configuration: Target/Source
Manager configuration (tune the values according to your needs)
– <GG_HOME>/dirprm/mgr.prm
Manager Configuration
4
PORT 7809
DYNAMICPORTLIST 7810-7910
AUTOSTART ER *
AUTORESTART ER *, RETRIES 4, WAITMINUTES 4, RESETMINUTES 60
PURGEOLDEXTRACTS <...>/dirdat/*ext*/*t*, USECHECKPOINTS, &
MINKEEPHOURS 4
LAGREPORTMINUTES 30
LAGINFOMINUTES 5
LAGCRITICALMINUTES 10
unreserved and unrestricted
22 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Primary Extract Configuration
Create the configuration file for the primary Extract (source)
Primary Extract, Data
Pump, Replicat
Configuration Files
5
23 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
EXTRACT app1ext1
SETENV (ORACLE_HOME = "/u00/app/oracle/product/11.2.0.4")
USERIDALIAS ggadm_app1_11204
LOGALLSUPCOLS
UPDATERECORDFORMAT COMPACT
DDL INCLUDE MAPPED EXCLUDE OBJNAME APP1.T4 &
EXCLUDE OBJNAME APP1.T5
ENCRYPTTRAIL AES256
EXTTRAIL <...>/dirdat/app1ext1/lt
SEQUENCE APP1.*;
TABLE APP1.*;
A Master-Key Wallet is
necessary
You need GoldeGate
Credential Store
Secondary Extract/Replicat Configuration Files
Create the configuration file for the Data Pump Extract (source)
Create the configuration file for the Replicat group (target)
24 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
EXTRACT app1pmp1
USERIDALIAS ggadm_app1_11204
RMTHOST ggmgr-cl121.trivadis.com, MGRPORT 7809,ENCRYPT AES256,&
COMPRESS, COMPRESSTHRESHOLD 1200
RMTTRAIL <...>/dirdat/app1rep1/rt
SEQUENCE APP1.*;
TABLE APP1.*;
Not necessary if the
trail is already
encrypted
REPLICAT app1rep1
SETENV (ORACLE_HOME = "/u00/app/oracle/product/12.1.0.2")
DBOPTIONS INTEGRATEDPARAMS(parallelism 6)
USERIDALIAS ggadm_app1_12102
DDLOPTIONS REPORT
ASSUMETARGETDEFS
MAP APP1.*, TARGET APP1.APP1.*;
Mapping: non-CDB CDB
Primary Extract, Data
Pump, Replicat
Configuration Files
5
PDB
Primary/Secondary Extract Group
Register the primary Extract with the database (integrated)
Create the primary Extract group (integrated)
Add a local trail an associate it with the primary Extract
Add the Data Pump Extract
25 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
DBLOGIN USERIDALIAS gg_app1_11204
REGISTER EXTRACT app1ext1 DATABASE [CONTAINER (APP1)]
Extract APP1EXT1 successfully registered with database at SCN 3630440.
GoldeGate Credential
Store
ADD EXTRACT app1ext1, INTEGRATED TRANLOG, BEGIN NOW
Register/Start Primary
Extract, Data Pump
6
ADD EXTTRAIL <...>/dirdat/app1ext1/lt, EXTRACT app1ext1
ADD EXTRACT app1pmp1, EXTTRAILSOURCE <...>/dirdat/app1ext1/lt
Secondary Extract/Replicat Group
Add a remote trail and associate it with the secondary Extract
Add the Replicat group (integrated)
Start the Replicat on the target with the correct CSN
26 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
ADD RMTTRAIL <...>/dirdat/app1rep1/rt, EXTRACT app1pmp1
ADD REPLICAT app1rep1, INTEGRATED,
EXTTRAIL <...>/dirdat/app1rep1/rt
GGSCI> START REPLICAT app1rep1, AFTERCSN 3693221
Register/Start Replicat
8
Oracle GoldenGate: Demo Environment
27 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Source DB: 2-nodes RAC 11.2.0.4
Schema: APP1
GoldenGate version: 12.1.2.1.2
Active/Passive data replication
CDB$ROOT APP1 APP2
Primary Extract
INTEGRATED
Target DB: CDB 2-nodes RAC 12.1.0.2
PDB: APP1, Schema: APP1
GoldenGate version: 12.1.2.1.2
Active/Passive data replication
APP3
Schema: APP1
Local Trail
Secondary Extract
Data Pump
RemoteTrail
Collector
Replicat
INTEGRATED
Core Messages
28 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Core Messages
GoldenGate is a very powerful replication software framework
For most deployments the integrated Extract/Replicat is the best choice
Supplemental logging generates more undo and redo
– In some cases you need even to supplementally log all columns!
Script all the configuration steps
– Very useful if you need to re-setup the whole environment
You don’t need to replace your current Streams installation ASAP by Oracle
GoldenGate , but
– Streams is deprecated in the version 12c and will not be enhanced in future
releases
– There is a framework to ease the migration from Streams to GoldenGate
(streams2ogg)
– As of now Streams is not supported with CDB architecture
– For new projects it might be a better choice (licensing)!
29 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes
Questions and answers…
Robert Bialek
Principal Consultant
Partner
Tel. +49 89 99275930
30 16.07.2015 Data Replication With Oracle GoldenGate - Looking Behind The Scenes