6. real time integration with odi 11g & golden gate 11g & dq 11g 20101103 - share

Post on 10-May-2015

1.578 views 0 download

Tags:

description

Alessandro Cagnetti

Transcript of 6. real time integration with odi 11g & golden gate 11g & dq 11g 20101103 - share

<Insert Picture Here>

High Performance, High Productivity Data Integration

Alessandro Cagnetti – EMEA Business development - Data Integration Solutions

Middleware Solution Specialist Team

2

Data Integration BuzzWhat the Analysts are saying about Data Integration

“Data services will grow in adoption as organizations recognize the importance of data and its integration for success of SOA initiatives.”- Gartner

“Most business functions can benefit from automation, but data integration problems

must first be solved .”- CIO Insight

“By 2012, 75% of organizations will have implemented some

form of modern data integration architecture.”

- Gartner

“Increasingly, the value-added advantage of a company isn't

what it makes but what it knows.”

- Chief Executive Magazine

2

3

Oracle Data IntegrationThe solution for enterprise-wide real-time data

Dramatically improve the accessibility, reliability, and quality of critical data across enterprise systems

Web Services

Databases

Distributed systems

Legacy systems

OLAP systems

OLTP systems

Mission critical systems and data

Business Intelligence, Performance Management

Data Warehouses,

MDM

Data Access

Real-time Data

Data Quality

SOA

4

Oracle Data IntegrationBest-in-class Real-time Data Integration

Oracle Data Quality and Data Profiling

Oracle Data Integrator EE

MDMApplications

SOAPlatforms

OracleApplications

BusinessIntelligence

ActivityMonitoring

Fusion Applications

Data Cleansing

Data Profiling

File SystemReal-Time

Data

OLTPSystem

Data Warehouse/ Data Mart OLAP Cube

Data Migration ServicesBulk Data Services Data Quality Services Data Access Services

ELT/ETL

Oracle GoldenGate

Zero Downtime Migration

Data Replication

Real-time CDC

Data Services Layer

Enterprise Service BusBPEL Process Manager Data Services

Data Transformation

Bulk Data Movement Match and Merge

Oracle Data Integration Solution

5

Move and transform data. Mixed sources

and targets.

BENEFITS KEY DIFFERENTIATED FEATURES

1. Performance >>> Heterogeneous “E-LT”

2. Flexibility >>> Event-Driven Platform

3. Productivity >>> Declarative Design

4. Open >>> 100% Java and SOA Native

5. Hot-Pluggable >>> Knowledge Modules

ODI-EE Value PropositionAgile Data Integration

5

6

SOA Suite

BPEL & OSB

ODI Integrated

Oracle Fusion ERP Applications

Other Apps … (+more)Business Intelligence

Enterprise Performance Management and Financials

Reduce TCO with Pre-Built SolutionsLower-Cost Adoption of Data Integration Within Oracle Products

Fusion MDM

ODIEmbedded

Pre-builtContent

BAM & CEP

ODIEmbedded

BIEE Suite

ODI Integrated

BI Apps

ODIEmbedded

Pre-builtContent

Agile PLM

ODIEmbedded

Pre-builtContent

Retek

ODIEmbedded

Pre-builtContent

HFM

ODIEmbedded

Planning

ODIEmbedded

Spend Analyzer

ODIEmbedded

MDM/DRM

ODI Integrated

Essbase

ODI Integrated

Governance& Risk

ODIEmbedded

Siebel CRM

GoldenGateIntegrated

Fusion Pillar Replication

OGGEmbedded

Pre-builtContent

Fusion HCM

ODIEmbedded

Pre-builtContent

Fusion AIA

ODIEmbedded

Pre-builtContent

Fusion CRM

ODIEmbedded

Pre-builtContent

Fusion Financials

ODIEmbedded

Pre-builtContent

Fusion Migration

ODI Embedded

Pre-builtContent

Pre-builtContent

Pre-builtContent

Pre-builtContent

ID Mgmt

Identity Analytics

ODIEmbedded

Pre-builtContent

Pre-builtContent

ODI Integrated

New

New New

Differentiator: E-LT ArchitectureHigh Performance with lower TCO

Transform Benefits in Existing RDBMS

• Eliminates the high cost of conventional ETL

• No New Servers Required

• Leverage Compute Resources & Partition Workload efficiently

• Exploits Database Optimizer

• Optimized Performance and Scalability

Transform in Separate ETL Server

Poor Performance

High Costs

POC: 40% performance

increase vs traditional ETL

Proprietary Engine

Proprietary Language

Leverage RDBMS

Native SQL

8

88

Differentiator: Declarative DesignIncreased Productivity & lower Maintenance costs

Conventional: Specify ETL Data Flow

• Developer must define every step of Complex ETL Flow Logic

• Traditional approach requires specialized ETL skills

• Significant development & maintenance cost

Oracle: Declarative Set-based Design

• Abstracts Modeling to High Level Design

• Simplifies the Number of Steps

• Automatically generates the Data Flow whatever the sources and target DB

• Easy to Handle Customizations: eg. Extension Columns in ERP Schemas

Benefits

• Significantly reduce the learning curve

• Shorter implementation times

• Streamline access to non-IT pros

Conventional ETL DesignConventional ETL Design

ODI Declarative Design

Define How: Built-in Templates

Define

WhatYou Want

Automatically GenerateDataflow

11 22

ODI Declarative Design

Define How: Built-in Templates

Define

WhatYou Want

Automatically GenerateDataflow

11 22

Define How: Built-in Templates

Define

WhatYou Want

Automatically GenerateDataflow

11 22

Define

WhatYou Want

Automatically GenerateDataflow

11 2211 22

9 9

Bloor Research: Lowest TCO over competitors

While this survey was commissioned by Pervasive, Oracle fared well in a number of scenarios, especially against IBM and Informatica especially at TCO and Speed

of Implementation

Note: this survey is not available for re-distribution on our Oracle website

10

Tailor to existing best practices

Reduce cost of ownership

Library of hot-pluggable templates

Open and extensible

Declarative set-based mapping• Spent 2-weeks at project initiation to

tailor KMs to best practice

• Now can do development to add a table to their DW in 20 seconds, assured of Best Practice Implemetation

Customer Benefits:

Differentiator: Knowledge ModulesEngineer once, Reuse everywhere

SAP/R3

Siebel

Log Miner

DB2 Journals

SQL Server Triggers

Oracle DBLink

DB2 Exp/Imp

JMS QueuesCheck MS

Excel

Check Sybase

Oracle SQL*Loader

TPump/ Multiload

Type II SCD

Oracle Merge

Siebel EIM Schema

Oracle Web Services

DB2 Web Services

1-11

ODI Design-Time Environment ODI Runtime Environment

User Interfaces

Architecture: from Development to Production

1. Retrieve/Enrich metadata

2. Design transformations

3. Orchestrate data flows

4. Generate/Deploy data flows

5. Monitor executions

6. Analyze impact / data lineage

Development

Development Servers and Applications

Agent

Data Flow Conductor

CRM

Legacy

ERP

Data

Warehouse

Files / XML

User Interfaces

Administrators

Designers

ESB

Design-time RepositoriesDesign-time

Repositories

Production

Production Servers and Applications

Agent

Data Flow Conductor

CRM

Legacy

ERP

Data

Warehouse

Files / XML

Operator

Metadata Navigator

ESB

Runtime Repository

12

13

14

15 15

Comprehensive Data QualityFor Data Integration with Oracle Data Integrator

Sta

nd

ard

Qu

ali

ty

Data Movement & Transformation

Advanced

Data

Profiling

Standard

Profiling

Ongoing Audit & Monitoring

Ad

va

nc

ed

Da

ta Q

ua

lity

Comprehensive Data Quality

Sources

Targets

Oracle Data Quality and Oracle Data Profiling

(OEM Agreement with Trillium)

Example of Data Quality IssuesA Simple Customer Table Sample

Name Address City State Zip Phone Email

Bob Williams 36 Jones Avenue Newton MA 02106 617 555 000 bob.williams@yahoo.com

Robert Williams 36 Jones Av. MA 02106 617555000

Burkes, Mike and Ilda 38 Jones av. Nweton MA 02106 617-532-9550 mburkes@gmail.com

Jason Bourne,

Bourne & Cie.76 East 51st Newton MA 617-536-5480 6175541329

… … … … … … …

Mis-fielded data

Matching Records

TyposMixed business and contact names

Multiple Names

Non Standard formats

Missing Data

17

How can this…

= = this?

Name1 Flugtaggen GMBH

Name2 rhamer strasse 20

Address dus

City/Town 40489

Post Code

Country

Business Name Flugtaggen GMBH

Address Rhamer Str. 20

City/Town Düsseldorf

Post Code 40489

Country DE

The Importance of Data Quality

18

Original Record 1 Original Record 2

Standardized Record 1 Standardized Record 2Name Flugtaggen GMBH

Contact

Address Rhamer Str. 20

City Düsseldorf

Post Code 40489

Country DE

Name Flugtaggen GMBH

Contact Klaus Webersinke

Address Rhamer Str. 20

City Düsseldorf

Post Code 40489

Country DE

Name1 Flugtaggen GMBH

Name2 rhamer strasse 20

Address dus

City/Town 40489

Post Code

Country

Name1 Klaus Webersinke

Name2 c/o Flugtaggen GMBH

Address 20 ramerstr.

City/Town Dusseldorf

Post Code

Country DE

How Standardization Helps Matching

• Time Series controls data quality over time

• Set right timeframe, right business thresholds

• Alert if DQ metrics and business rules are not met

Monitor Data Quality over time

20

Oracle GoldenGate provides low-impact capture, routing, transformation, and delivery of transactional data across heterogeneous environments in

real time

Key Differentiators:

Non-intrusive, low-impact, sub-second latency

Open, modular architecture - Supports heterogeneous sources and targets

Maintains transactional integrity - Resilient against interruptions and failures

What is Oracle GoldenGate?

Performance

Flexible and Extensible

Reliable

21

Traditional ETL + CDC

• Invasive Capture on OLTP systems

using complex Adapters

• Transformations in ETL engine on

expensive middle tier servers

• Bulk load to the data warehouse with

large nightly/daily batch

• Continuous feeds from operational

systems

• Non-invasive data capture

• Thin middle tier with transformations

on the database platform (target)

• Mini-batches throughout the day or

bulk processing nightly

ODI + Oracle GoldenGate

Staging

Trickle

LookupData

Load

Extract

LookupData

Xform Xform Bulk

GG

+ O

DI

GG

+ O

DI

Heterogeneous

Real-time CDC Integration Best-in-class solution for real-time with Oracle GoldenGate

22

Oracle for Real-Time Data Warehousing Fastest Real-Time Data Integration, Fastest Bulk Data Transformation

BI Application

Production

OLTP Databases

DW

tx6 tx5 tx4 tx2 tx1tx3

Real-time streaming of transactions

LAN / WAN /

InternetCapture

Source TrailTarget Trail

Deliver

• Sub-second data latency

• Minimal overhead and no batch windows

• High-performance, in-database transformations

• Read-consistent changed data with referential integrity

• Complete data recoverability via Trail files

ODI

23

Extreme Performance for Data WarehousingExadata V2 is the Platform for Real Time Data Warehousing

Oracle Data Integrator

EMP DEPT

DIM

FACT

DIM

DIMDIM

ODS Schema DW Schema

Load Exadata with GoldenGate; Generate Transforms from ODI-EE

• New Exadata platform optimized for OLTP and Data Warehousing suits Real Time DW

• Extreme Exadata performance can handle huge data volumes that come with Real Time DW

OracleGoldenGate

Oracle Database

24

Oracle GoldenGate OverviewEnterprise-wide Solution for Real Time Data Needs

Log Based, Real-Time Change Data

Capture

Heterogeneous Source Systems

EDWODS

EDW

Disaster Recovery, Data Protection

Zero Downtime Migration and

Upgrades

Operational Reporting

Real-time BI

Standby

(Open & Active)

Reporting

DatabaseOGG

ETL

ETL

Query Offloading

Data Distribution

• Standardize on Single Technology for Multiple Needs

• Deploy for Continuous Availability and Real-time Data

Access for Reporting / BI

• Highly Flexible

• Fast Deployments

• Lower TCO & Improved ROI

25

Oracle GoldenGate Topologies

UnidirectionalQuery Offloading

Zero-Downtime Migration

Bi-Directional

Hot Standby or

Active-Active for HA

Peer-to-Peer

Load Balancing,

Multi-Master

Broadcast

Data Distribution

Integration/Consolidation

Data Warehouse

Data Distribution

via Messaging

BPM

26

Energy/Industrial Insurance / Health Retail Other

Communications Finance / Banking Media Public Sector

ODI: an horizontal product

Ireland's Department

of Social & Family Affairs

27

Benefits of Real-time DI with OracleCut Costs, Reduce Risk, and Revolutionize Business Insight

• Move only the changed data from redo logs and reduce source and network overhead

• Shorten implementation times from months to weeks using pre-packaged integrations to well-known applications, sources and targets.

Cut Costs and Improve Efficiencies.

• Eliminate performance impact on source systems

• Reduce the risk of missed orders, poor customer interactions, missed opportunities through improved recoverability, data quality

Reduce Risk, Ensure Continuity

• Enable near real-time decision making with real-time data flows

• Combine real-time data with historical context for better insights

Improve Business Insight

28

Quote Attribution

Title, Company

• Visit the Oracle Fusion Middleware 11g

web site at

oracle.com/goto/fmw11g/index.html

• Oracle Data Integration on oracle.com

oracle.com/goto/odi

• Oracle GoldenGate on oracle.com

oracle.com/goto/goldengate

• Oracle BI/EPM on oracle.com

oracle.com/solutions/business_intelligenc

e/index.html

Get Started

• Oracle GoldenGate Resource Kit:

oracle.com/goto/goldengate

• Blog:

http://blogs.oracle.com/dataintegration

• Technical information available at:

oracle.com/technology/products/oracle-

data-integrator/goldengate/index.html

• Data Integration Events

http://www.oracle.com/events

Resources

29

31

32

For More Information

search.oracle.com

or

oracle.com