GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS...
Transcript of GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS...
![Page 1: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/1.jpg)
GoldenGateGoldenGate
Zbigniew Baranowski
![Page 2: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/2.jpg)
Outline
• What is GoldenGate?• ArchitectureArchitecture• Performance• GoldenGate vs StreamsGoldenGate vs. Streams• Monitoring• Summary• Summary
![Page 3: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/3.jpg)
What is GoldenGate?
• Real-time data integration solutions• Continuous data synchronization acrossContinuous data synchronization across
heterogeneous environments– Oracle, DB2, SQL Server, MySQL and more
• Project started in early 90s• Purchased by Oracle in late 2009y
![Page 4: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/4.jpg)
Motivation for testing
• Need of stable and reliable replicarion service• Streams require frequent interventions (at leastStreams require frequent interventions (at least
once per week)– Blocking sessions– Memory pools shortage– Human errors
• Streams administration is difficult– T1s administrators need our help
• Streams project has been abandoned by Oracle
![Page 5: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/5.jpg)
Data integration using GG
![Page 6: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/6.jpg)
GoldenGate architecture
![Page 7: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/7.jpg)
GoldenGate components
• Manager– Runs on both databases– Starts and monitors GG processes– Manages trail files– Reporting
![Page 8: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/8.jpg)
GoldenGate components
• Extract (capture process)– Runs on the source database– Extracts data changes from redo logs– Writes transactional and DDL changes (in common GG
f ) il filformat) to trail files
![Page 9: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/9.jpg)
GoldenGate components
• Collector– Background process which runs on the target systemg p g y– Two modes: static and dynamic (managed by manager) – Receives data changes from TCP/IP and writes to remote
il filtrail files
![Page 10: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/10.jpg)
GoldenGate components
• Replicat (apply process)– Runs on the target systemg y– Reads transactional data from trail files and applies them
on target databaseC b ll li d– Can by parallelized
![Page 11: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/11.jpg)
GoldenGate components
• Trail files– Contains data changes written in GG common formatg– Series of files that GoldenGate temporarily stores on
disksT– Two types
• Extract Trail (located on source - optional)• Remote Trail (located on destination)( )
![Page 12: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/12.jpg)
GoldenGate components
• Data pump– Sends data changes from extract trail via TCP/IP to g
remote trail on the target– Optional component
E t t d h di tl t t t il• Extract process can sends changes directly to remote trail
– Optional encryption and compression over TCP/IP
![Page 13: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/13.jpg)
GoldenGate architecture (overview)
EXTRACTPROCESS
LCsTRAIL FILES
SourceSource
PROCESS
capture changes
TRAIL FILES
File streaming
DestinationDestination
DELIVERYPROCESS
log changes
(parsing logs) File streaming
(data pumping)
LCs apply changesPROCESS
REDO LOGTRAIL FILES
TARGET DATABASE (replica)
SOURCE DATABASE
MANAGERMANAGERPROCESS MANAGER
PROCESS
![Page 14: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/14.jpg)
Streams architecture (overview)
CAPTURE PROCESS
LCRs
SourceSource
PROCESS
capture changes
SOURCE QUEUE
propagate
DestinationDestination
APPLYPROCESS
log changes
(using Logminer)propagate
events
LCRs apply changesPROCESS
REDO LOG DESTINATION QUEUE TARGET
DATABASE (replica)
SOURCE DATABASE
![Page 15: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/15.jpg)
Discovered problems
• Supplemental logging groups are created to late to replicate the following DML operationsg– Workaround: add global supplemental logging for primary
and foreign keys• Cannot use create as select – select statement is
executed on the destination database as GGADMIN useruser
• Delivery parallelism: lack of DMLs and DDLs synchronization – problem with temporal tablessynchronization – problem with temporal tables
![Page 16: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/16.jpg)
Performance
• Simple workload (inserts, updates, deletes)– With basic configuration (no parallelism): 7K LCRs /sg ( p )– With BatchSQL optimization 15K LCRs /s and can be
moreWi h ll li l 15K LCR / ( h ll l– With parallelism: at least 15K LCRs /s (each parallel process increases throughput linearly)
• COOL workload• COOL workload– ~3K LCRs /s– BatchSQL does not improve performanceBatchSQL does not improve performance– Unable to use parallelism
• Delivery process is the bottlenecks due to limits y pof resource utilization by single session.– Possible solution: per schema delivery parallelism
![Page 17: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/17.jpg)
Where are we now?
![Page 18: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/18.jpg)
GoldenGate vs Streams (setting up)
Streams GoldenGate
Installation •Embedded in Oracle DBMS •Unzipping binaries
Management •Using SQL or EM •Mainly with GGSCI command tool•Director application
Replication setup
•TNS + database links config•Executions of few procedures
•Editing parameter files + executions of few command•Additional port required
RAC N dditi l t i d Sh d t CRS fi tiRAC environment
•No additional steps required •Shared storage, CRS configuration, additional parameters
Schemasselection
•Setting rules for process – SQLprocedure
•Editing mapping filesselection procedure
Supervisor account
•STRMADMIN – owner of processes, jobs, queues and links•Executes transactions on target
•GGADMIN – keeps metadata about replicated schema•Executes transactions on targetg g
Watch •None •Manager process
![Page 19: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/19.jpg)
GoldenGate vs Streams (performance)
Streams GoldenGate
Bottleneck •Capture process – log mining •Delivery process – limitation of single session resourcesresources
Parallelism of apply / delivery
•By setting process’s parameter•Apply coordinator takes care about serialization
•By manual addition of processes and specification of filters•No coordinator process – no serialization pguaranteed
Big parallel transactions
•Serialized capture •Parallel capture
Replication processes impact
on the system
•Quite significant •Minimal
Potential improvements
•None •Parallelism of schemas + BatchSQL + compression
Stability •Not stable. A lot of aborts due to •Very stable so farydatabase memory issues•Hangs due to session blocking
![Page 20: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/20.jpg)
GoldenGate vs Streams (functionality and maintenance)
Streams GoldenGate
Initial load •Data pump •Initial load configuration
Monitoring •SQL + EM + Custom tools •Director
Administration •All task can be done through SQLPlus•No direct access to machines required
•Using Director (most of tasks)•Using GGSCI requires direct access to node
Error handling •Errors well documented •Errors not well documented and not fully Error handling•Errors expressed in database language•All error handling procedures fully understood
yunderstandable•Handling procedures not recognized yet
Data replication •Replication of data definitions and •Special handling of data definitions.Data replication(DDL+DML)
pmodifications
p g•Only delivery without parallelism can guarantee smooth replication
Specific data handling
•Apply handlers •Nonehandling
![Page 21: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/21.jpg)
GoldenGate vs Streams (functionality and maintenance)
Streams GoldenGate
Schemasversioning
•Multi Version Object Dictionary •Single Version Object Dictionary•Potential problems with recapturing of olderversioning •Potential problems with recapturing of older changes
Resuming of replication
•Slow - capture process needs to re-init dictionary from the last checkpoint
•Very fast•Using checkpoint files
Failingtransactions
•Mechanism for re-execution of failed transaction
•No extra mechanism.•Rollback the sequence change number (SCN) and restart the delivery process
![Page 22: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/22.jpg)
Monitoring
• 3 applications– Admin tool (defines managers location)( g )– Director web application (status, logs, start/stop, email
notifications)Di d k li i ( l /– Director desktop application ( status, logs, start/stop, email notifications, replication configuration and management, setup topology )g , p p gy )
• Written in JAVA• Requires Java 1.6 SDK and Weblogic serverRequires Java 1.6 SDK and Weblogic server• Wraps GGSCI command line interface
![Page 23: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/23.jpg)
Admin tool
![Page 24: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/24.jpg)
Director - web app
![Page 25: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/25.jpg)
Director – desktop app
![Page 26: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/26.jpg)
Monitoring
• Intuitive interface layout (fast information display – no nested views)
• Access to process statistics and reports• Email notification
Lag (floods mail box sends every minute)– Lag (floods mail box – sends every minute)– Event (errors, warnings)
• Difficult to monitor many replication setups• From time to time problems with refreshing• Not all operations works after first attempt (retries are
d d)needed)• No history data• No plots except lags• No plots except lags
![Page 27: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/27.jpg)
Summary
• DBMS independent software– Deployment requires more afford (‘attaching’ to DB)p y q ( g )– Minimal machine resource utilization– Minimal impact on the database
• Golden Gate is stable and efficient.• Technology focused on data modification
operations – CERN’s data profile is definitions + modifications
![Page 28: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/28.jpg)
Summary
• Performance (safe mode) better than Streams10g– There is big potential of improvements but we need to g p p
avoid ‘driving without breaks’• Errors not well documented• Lack of experience (administration)
– Errors handling• Fair monitoring (in multiple setups context)• Strategic replication solution for Oracle
![Page 29: GoldenGate - CERN openlab€¦ · Streams GoldenGate Installation •Embedded in Oracle DBMS •Unzipping binaries Management •Using SQL or EM •Mainly with GGSCI command tool](https://reader036.fdocuments.us/reader036/viewer/2022081515/5f077fdf7e708231d41d4970/html5/thumbnails/29.jpg)
Future plans
• More performance tests– Per schema delivery parallelismy p– Real PVSS’s and Condition's data replication tests
• Establishing contact with GG development team– Arranged meeting at Oracle Open World next week
• Evaluation new features on GG 11g (new version just has been released)