Kevin D. Smith Oracle Open World Review Golden Gate / ODI ...Oracle GoldenGate 10g to 12c Certified...
Transcript of Kevin D. Smith Oracle Open World Review Golden Gate / ODI ...Oracle GoldenGate 10g to 12c Certified...
1
November 10-13, 2015
Kevin D. Smithemail: [email protected]: +1 813-340-0335
A Forsythe Division
Kevin D. Smith
Oracle Open World Review
Golden Gate / ODI & Big Data
2
Thank you for joining us today
33
DAY’S AGENDA
10:15 - 10:30 Welcome / Intro / Meta7 / OOW (Kevin Imm)
10:30 - 11:30 Systems and Storage Announcements (Chris Brillante)
• M7/T7 Chip and Systems
• Engineered Systems
• Storage
11:30 - 12:15 Oracle Cloud Offerings (Eric Frank)
12:15 - 12:45 Lunch and Discussion
12:45 - 1:30 GoldenGate / ODI / Big Data Tools (Kevin Smith)
1:30 - 2:00 Database and Red Stack Security (Kevin Smith)
44
SPEAKER PROFILE – KEVIN SMITH
Kevin Smith - Master Consultant, Meta7
Over twenty years of business & technical experience in Information Technology encompassing Service Oriented Architecture (SOA), Application & System Integration for large implementations across multiple platforms & operating systems
Expertise in Oracle, DB/2, Sybase, & MS SQL
Multiple platform support: HP, SunOS, AIX, and Linux Unix; Windows; MVS, OS/390, z/OS
Oracle GoldenGate 10g to 12c
Certified GoldenGate Implementation Specialist GoldenGate
Co-Founder of International GoldenGate Oracle Users Group – www.iggoug.org
55
META7 is a division of Forsythe Technologies
Platinum Oracle partner
Deeply skilled, uniquely experienced
Focused on the Oracle Red Stack
Flexible, creative and accommodating
Reliable on-time and on-budget delivery
State of the Art Technology Evaluation Center
Secure hosting and Managed Services
Flexible financial support
META7
66
Consulting and Managed Services
12c DB Upgrade & Migrations
SPARC Refreshes
IBM Power -> Oracle Systems Migrations
Oracle-on-Oracle Storage
DB Security
Engineered Systems Deployments
Oracle Cloud Strategy
META7 - AREAS OF FOCUS
7Oracle Open World 2015
NoETL Engine100% Native Data
Transformation
DataIntegrator
Big DataPreparation
GoldenGate
DataQuality
Data ServiceIntegrator
MetadataManagement
Non-invasive CDC, Realtime streaming data
delivery
Profile, Cleanse, Match, and
Remediate Data
Prepare, Secure, Enrich and Publish Unstructured Data
Catalog, Trace and View Models across the Enterprise
Federate Data Across DBs, Services and Applications
88
Part of a Family of Oracle Data Integration Technologies
• Comprehensive Integration
• ELT/ETL for Bulk Data
• Service Bus
• Process Orchestration
• Human Workflow
• Data Grid
• Business Data / Metadata
• Statistical Analysis
• Time Series Reporting
• Integrated Data Quality
• Metadata Harvest
• Metadata Catalog
• Data Lineage
• Interactive Search
• Interactive Browse
• Impact Analysis
• Heterogeneous E-LT & ETL
• High-speed Transformations
• OLAP Data Loading
• Data Warehouse Loading
• Real Time Data Replication
• Changed Data Capture
• DBMS High Availability
• Disaster Tolerance
• Data Service Modeling
• XQuery Data Federation
• Data Security/Redaction
• XA Compliance
• Cleansing & Parsing
• De-duplication
• High Performance
• Integrated w/ODI
99
Oracle Database Replication Technologies
Data Guard
Physical
Logical
Usable only when the source and the target are identical
Operating System and Database Version
Streams
While still supported and used, Streams had a number of issues (which is why Oracle acquired GoldenGate) and is no longer Oracle's strategic replication product, it's on a continue and converge path and will continue to be supported but not extended to support multi-tenant container databases
Change Data Capture (CDC)
A subset of Streams capabilities: Support will continue but capabilities will not be enhanced for multi-tenant container databases
Materialized View, Database Links, 3rd Party Products
1010
What Makes An Ideal Replication Solution?
Replication apply time configurable from near-zero to a user defined pre-set delay
Scalable to handle high transaction volumes
Minimal footprint and impact to existing systems
Modular architecture supporting multiple data sources and targets
Fault tolerant
Maintains transactional integrity
Reliable
Secure
Stable
1111
What Is GoldenGate?
A software tool that creates a tunnel between two, or more physically separate systems such that they can behave as a single logical system
Middleware that provides a data communications channel between a source from which transactions will be read and a target to which the transactions will be written
GoldenGate connections canbe homogeneous, for exampleOracle Database to OracleDatabase, or heterogeneous,for example Teradata to Oracle Database
Supports data transformation
Can replicate DML and DDL
Unlike Oracle Service Bus ... understands transactions
1212
Logical Replication Architectures
UnidirectionalReporting Instance
Bi-directionalActive:Active Failover
Peer-to-PeerLoad Balancing, High Availability
BroadcastData Distribution
ConsolidationData Warehouse/Mart/Store
CascadingScaling, Database Tiering
13
Business Cases For Oracle GoldenGate
1414
GoldenGate Business Cases (1:3)
Load Distribution
The current server infrastructure is not sufficient to handle the demands placed upon it; for example it would save the organization both time and money if analytic queries were run against a separate database performance optimized for those queries
Upgrading
The organization wishes to upgrade from one version of a product to a different version of the same product; for example 11.2.0.3 to 12.1.0.2 with near-zero downtime
Homogeneous Migration
The organization wishes to migrate a database from one hardware platform to another; for example Oracle Database on Solaris to Oracle Database on Linux
Heterogeneous Migration
The organization wishes to migrate from one database product to another; for example from stand-alone SQL Server to Oracle RAC on an ODA
1515
GoldenGate Business Cases (2:3)
Data Distribution
The organization wishes to make data available at different physical locations without dependency upon the internet; for example the corporate head quarters are in London England but the manufacturing facilities are in Latin America
Data Consolidation
The organization has multiple locations where data is collected and wishes to have a single source that combines them all for management reporting and analysis, for example there are 8,200 retail stores in North America, each with its own on-site database and the corporate headquarters needs to be able to run a single query across data from one region's stores or all stores in all regions
Data Access Sharing
The organization wishes to have two separate locations with equal access to data with the ability to perform DML and DDL, for example the organization considers each office a DR site for the other
1616
GoldenGate Business Cases (3:3)
Security
The organization stores both public and confidential information some of which it does not wish to make vulnerable by making it visible through-out the organization; for example data collection from an e-commerce site that processes both orders and credit cards
Auditing
The organization wants a database of suspicious transactions for fraud investigators where the source might be multiple databases, from multiple vendors; for example the organization has both Oracle and DB2 databases and wants a single repository for internal auditors
Data Transformation
The organization wants to load data from OLTP (On-Line Transaction Processing) systems into one or more data warehouses, Operational Data Stores (ODS) or Decision Support Systems (DSS); for example the data warehouse collects data from only specific tables and columns and must load them into cubes
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Open World 2015 17
Announcing: Oracle GoldenGate New Capabilities
Zero Downtime Upgrades & Data Migration
High Availability& Disaster Recovery
Application andReport Offloading
Big Data, Data Warehouseand Data Marts
Message Bus, SOA/JMS, Coherence
Global Data SynchronizationAcross the Enterprise
Real-time Analytics& Data Streaming
New Capabilities GG Studio for easy design
Metadata in Trail File
Delivery to Kafka
Enhanced Monitoring w/Lag
Performance Improvements
MySQL 5.7. SQL Server SE, ALO
Key Benefits Reliability most proven CDC on
the planet – don’t lose data!
Performance when low latency matters most – GG is the best
Breadth of coverage on all your targets, Oracle, Non-Oracle, Big Data and NoSQL
Cap
ture
Trai
l
Ro
ute
Del
iver
Pu
mp
GoldenGate
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Streaming Big Data
Batch Big Data
NoSQL
Cap
ture
Trai
l
Ro
ute
Del
iver
Pu
mp
Oracle Open World 2015 18
NEW: Oracle GoldenGate for Big Data
GoldenGate
GoldenGate for Big Data Same trusted architecture used
by 1000’s of customers
Data delivery to Big Data
Lambda/Kappa streaming data
Key Benefits Less invasive on Source DBs
when compared to Batch Processing such as Sqoop or ETL processes
Simple ingestion for 1:1 data architecture for populating “raw data” zones
Real-time data delivery for streaming analytics/apps
Reliable, proven at scale
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Database as a Service
Hadoop as a Service
3rd Party Cloud
Cap
ture
Trai
l
Ro
ute
Del
iver
Pu
mp
Oracle Open World 2015 19
NEW: Oracle GoldenGate Cloud Service
GoldenGate Cloud Service Available in the Public Cloud via
Subscription or Hourly basis
Key Benefits Oracle Database Cloud Service
delivery via GoldenGate
Exadata Cloud Service delivery via GoldenGate
Big Data Cloud Service delivery to Hadoop and NoSQL
3rd Party Cloud More Choices run as BYOL on
other Clouds for delivery to any supported Database
On Premise
DatabaseCloud Service
ExadataCloud Service
Big DataCloud Service
GGCS
GGCS
GoldenGate
20
GoldenGate Supported Systems
2121
Supported Sources
HP SQL/MP (Tandem)
HP SQL/MX
IBM DB2 z/OS
IBM DB2 (LUW)
IBM DB2 for i (AS/400)
IBM Informix
Java Messaging Queues (JMS)
Microsoft SQL Server
Oracle Database
Oracle MySQL
Oracle TimesTen
Sybase ASE
Teradata
2222
Supported Targets
Base24
EMC Greenplum
GG For BigData: Hadoop File System (HDFS, Hive, HBase, MongoDB, Kafka)
HP Enscribe
HP SQL/MP (Tandem)
HP SQL/MX
IBM DB2 (LUW)
IBM DB2Remote
IBM DB2 System i (AS/400)
IBM DB2 System z (mainframe)
IBM Informix
IBM Netezza
Java Message Queue (JMS)
Microsoft SQL Server
Oracle Database
Oracle MySQL
Oracle TimesTen
PostgreSQL
Sybase ASE
Teradata
Flat Files
XML Files
JSON File
AVRO File (across the BigDatastack)
2323
Supported Operating Systems
HPUX IA64
HP NonStop Itanium
IBM AIX
IBM iSeries
IBM z/Linux
IBM z/OS
Linux x86-64
Oracle Solaris Sparc
Oracle Solaris x86-64
Windows x86-64Copyright © Puget Sound Oracle Users Group
2424
Complete Logical Flow with Managers
ReplicatPump Trail File
TargetSource
Extract Trail File
Network
Network
Manager Manager
The Pump process communicates with the Target's Manager and requests a Collector process to transfer the trail file
2525
Logical Flow: Bi-directional
ReplicatExtract Trail File Pump Trail File
TargetSource
Target Source
Replicat Trail File Pump Trail File Extract
ManagerManager
26
Troubleshooting and Debugging
2727
Troubleshooting Considerations
The most common problems are network stability issues
The second most common problems are caused by DBAs and Developers making schema changes that create collisions
For example inserting rows into the target that will later cause a failure when an identical key is replicated from the source
The third most common issues relate to adding and altering tables creating supplemental logging failures
Help your GoldenGate admin identify processing -intensive tables so they can be replicated using separate processes
There are special considerations for ASM and RAC dependent upon GoldenGate and Database version ... do your research
Bidirectional replication is hard ... not because of the database and not because of GoldenGate but because of application designs not capable of supporting it
28
Basic GoldenGate Security
2929
Credential Store
Deploy GoldenGate version 12.1 with a Credential Store to hold login useridand passwords to protect the database
[oracle@gg00a gghome_1]$ cd dircrd
[oracle@gg00a dircrd]$ ggsci
Oracle GoldenGate Command Interpreter for Oracle
Version 12.1.2.1.0 OGGCORE_12.1.2.1.0_PLATFORMS_140727.2135.1_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Aug 7 2014 09:14:25
Operating system character set identified as UTF-8.
Copyright (C) 1995, 2014, Oracle and/or its affiliates. All rights reserved.
GGSCI (gg00a) 1> ADD CREDENTIALSTORE
Credential store created in ./dircrd/.
GGSCI (gg00a) 1> exit
[oracle@gg00a dircrd]$ ls -l
total 4
-rw-r----- 1 oracle oinstall 324 May 9 12:06 cwallet.sso
GGSCI (gg00a) 1> ALTER CREDENTIALSTORE ADD USER ggadmin PASSWORD ora123 ALIAS ggadm
Credential store in ./dircrd/ altered.
GGSCI (gg00a) 1> dblogin useridalias ggadm
Successfully logged into database.
3030
OOW Highlights: Oracle Data Integrator
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Open World 2015 31
Announcing: Oracle Data Integrator New Capabilities
On Any DB/DWOn Exadata On Big Data
0
50
100
150
200
250
300
350
42XFASTER
ETL
INFA
/ IB
M
Best ETL for any Exadata App
Run ODI on the Exadata Host
ODI Optimized for SmartScan
Proven w/1000’s of customers
Native SQL for 3rd Party DB’s
Faster ETL and lower TCO
Oracle Data Integrator
First to market with Spark
Reuse maps across languages
Executable runs anywhere
New Capabilities Lifecycle Management updates
to core versioning systems
Simplify setup and deployment of Big Data transformations
Performance improvements across all E-LT platforms
Key Benefits Open Source friendly – run E-LT
on Apache Spark/Hadoop
Best Performance – take processing to the data
No Hardware required – run on the DB host or on Apache big data cluster
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Open World 2015 32
Oracle Data Integrator Advanced Big Data Edition
SQOOP
OGG
Oracle Data Integrator
SQOOP+ NativeLoaders
API/File
Flume/Kafka
OLH/OSCH
Big Data SQL
Data Integrator for Big Data Batch data ingestion with
Sqoop or native loaders
Generate data transformations in Hive, Pig or Spark
Extract data into external DBs, Files or Cloud
Key Benefits NoETL Engine designed from
scratch for E-LT execution
Non-invasive does not require any install on cluster
Decoupled mappings allow for portable runtime execution, saving development time, and simplifying debugging
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Open World 2015 33
Oracle Data Integrator for Cloud
Data Integrator for the Cloud Deploy ODI anywhere and
execute ETL on the data
Key Benefits Integrated with OPC (Oracle
Public Cloud)
Flexible Deployment architecture that can run on-premise, Java Cloud Service or Big Data Cloud Service
Broad Connectvity for sourcing data from any Apps or DBs
3rd Party Cloud
Flexible run ODI with BYOL on other clouds
On Premise
ODI
SaaS Apps
Oracle Data Integrator
ODI/EDQ
JavaCloud Service
ODI
Big DataCloud Service
DatabaseCloud Service
ExadataCloud Service
StorageCloud Service
Business IntelligenceCloud Service
3434
META7 - AREAS OF FOCUS
3535
OOW Highlights: Oracle Metadata Management
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Open World 2015 36
Announcing: Oracle Metadata Management 12.2.1
1. Business Glossary – simple tools to catalog, link and collaborate on business terms
2. Metadata Management – horizontal and semantic data lineage for all big data sources
Business/Metadata Catalog Report to Source Lineage
Impact Analysis
Audit, Versioning & Diff Reports
Social/Collaboration Features
Annotations and Tagging
Automatic Stitching
Comprehensive Harvesting 3rd Party BI Metadata
3rd Party ETL Metadata
3rd Party DB Metadata
3rd Party Modeling Tools
Hadoop & NoSQL Metadata
Metadata Standards
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Open World 2015 37
Open Source, Standards and Third Party Integrations
Adaptive Altova Apache HCatalog Apache Hive/HQL Borland CA ERwin Cloudera Impala COBOL Copybook DataStax Embarcadero EMC ProActivity GentleWare Google BigQuery Grandite Hadapt Hive Hortonworks Hive IBM Cognos IBM DB2 IBM DataStage IBM Discovery IBM Federation Server IBM Lotus Notes IBM Netezza IBM Rational Rose IBM Rational Architect Informatica Metadata Manager
Informatica PowerCenter CoSORT ISO SQL Standard (DDL) MapR Hadoop Hive MicroFocus Microsoft Access Microsoft Office Excel Microsoft Visio Microsoft SQL Server Microsoft SSIS Microsoft Visual Studio Microstrategy Magic Draw OMG CWM Standard OMG UML Standard Oracle BI Answers Oracle BI Enterprise Edition Oracle BI Server Oracle DAC Oracle Data Integrator Oracle Data Modeler Oracle Database Oracle Designer Oracle Hyperion Applications Oracle Hyperion Essbase Oracle Warehouse Builder
Pivotal Greenplum PostgreSQL QlikView SAP BO Crystal Reports SAP BO Designer SAP BO Desktop Intelligence SAP BO Repository SAP BO Data Integrator SAP BO Data Steward SAP Master Data Management SAP Sybase PowerDesigner SAP Sybase ASE Database SAS Data Integration Studio SAS BI Server SAS Information Map SAS Metadata Management SAS OLAP Server Select Sparx Architect Syncsort Tableau Talend Teradata Tigris Visible W3C DTD & XSD Schema
Metadata Harvesting (Glossary, Lineage & Impact Analysis)Key Standards
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |38
OOW Highlights: Oracle Data Quality
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Oracle Open World 2015 39
Announcing: Oracle Enterprise Data Quality 12.2.1
New Capabilities Enhanced High Availabilty
WebLogic Clustering
DB Failover Support
Improved Matching Flexibility
Scoring for Matching
Easier Definitions
Hadoop/Hive Certification
Audit Framework Support
REST Web Services
Key Benefits Simpler tools to very efficiently
create match rules
Accuracy of scoring models and easier access to parameters
Oracle Enterprise Data Quality
Fusion DQ (EDQ)
EDQ 12c
Profile Audit Transform Match Parse
WebLogic 12c
DaaS DQ (EDQ)
EDQ
40
Thoughts and Conclusions
4141
Conclusion (1:4)
In my opinion, GoldenGate is the best homogeneous and heterogeneous replication tool on the market
Very small cpu, memory, and storage footprint
Capable of solid security lockdown
Robust capabilities for subsetting and transformation
Substantial tools for troubleshooting and debugging
Substantial capabilities for performance tuning
Easy to patch
Easy to upgrade
Stable
4242
Conclusion (2:4)
Oracle Data Integrator
Provides a new declarative design approach to defining data transformation and integration processes, resulting in faster and simpler development and maintenance.
Based on a unique E-LT architecture (Extract - Load Transform), Oracle Data Integrator not only guarantees the highest level of performance possible for the execution of data transformation and validation processes but is also the most cost-effective solution available today.
Oracle Data Integrator provides a unified infrastructure to streamline data and application integration projects.
4343
Conclusion (3:4)
Oracle & Big Data
Platform for Big Data
Faster time to actionable insights
Big Data Preparation Cloud Service
Data visualization tools
Big Data Discovery Cloud Service
Common Customer approach to Big Data
Internal use to derive more value from their data assets
Add on services to end-users to foster greater customer affinity
Stand-alone clients offerings on a paid basis that allow monetization of Big Data assets
4444
Conclusion (4:4)
Metadata Management
Metadata Harvest
Metadata Catalog
Data Lineage
Interactive Search
Interactive Browse
Impact Analysis
Data Quality
Cleansing & Parsing
De-duplication
High Performance
Integrated w/ODI
Hadoop/Hive Certification
4545
How can we help you be successful?
In my opinion GoldenGate is the best homogeneous and heterogeneous replication tool on the market
Very small cpu, memory, and storage footprint
Capable of solid security lockdown
Robust capabilities for subsetting and transformation
Substantial tools for troubleshooting and debugging
Substantial capabilities for performance tuning
Easy to patch
Easy to upgrade
Stable
If you have any questions about
how Meta7 can help or on what
Meta7 provides, we would like to
talk with you after the
presentations.
46
Thank you for your time
Questions ?Address questions directly to: [email protected]
Next EventTuesday, November 17, 2015:
Update on Oracle GoldenGate for DBA’s
12:00 ET / 11:00 CT
All Technical Briefingshttp://meta7.forsythe.com/Resources
contact: Kevin D. Smithemail: [email protected]: +1 813-340-0335skype: kevindsmith-Tampa
47
www.meta7solutions.com
Kevin D. Smith 813-340-0335