UKOUG BIRT SIG 2014 – ODI for OWB Developers

41
T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India) E : [email protected] W : www.rittmanmead.com Rittman Mead Consulting Oracle Data Integrator for Oracle Warehouse Builder Developers Jérôme Françoisse UKOUG BIRT SIG - 17th September 2014 Wednesday, 10 September 14

Transcript of UKOUG BIRT SIG 2014 – ODI for OWB Developers

Page 1: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Rittman Mead ConsultingOracle Data Integrator forOracle Warehouse Builder Developers

Jérôme FrançoisseUKOUG BIRT SIG - 17th September 2014

Wednesday, 10 September 14

Page 2: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Jérôme Françoisse

•Consultant for Rittman Mead‣Oracle BI/DW Architect/Analyst/Developer

•ODI Trainer•Providing ODI support on OTN Forums•ODI 12c Beta Program Member•Blogger at http://www.rittmanmead.com/blog/•Email : [email protected]•Twitter : @JeromeFr

Wednesday, 10 September 14

Page 3: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Rittman Mead

•Oracle Gold partner with offices in Europe, US (Atlanta), Australia, India and South Africa

•World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI

•Provide consulting, training, global managed services for customers around the world

•120+ consultants including 1 Oracle ACE Director, 3 Oracle ACEs and 1 Oracle ACE Associate

•All expert in Oracle BI, DW, EPM and Analytics tech

•Skills in broad range of supporting Oracle tools: OBIEE, OBIA, ODIEE, Essbase, Oracle OLAP, GoldenGate, Exadata, Endeca

•Blog : http://www.rittmanmead.com/blog/•Twitter : @rittmanmead

Wednesday, 10 September 14

Page 4: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Agenda

•History•Similarities•Differences•Overview and Architecture•Tour of ODI 12c‣ODI Studio‣Mappings‣Knowledge Modules

•OWB to ODI‣Execution‣Migration

Wednesday, 10 September 14

Page 5: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

History of OWB & ODI

•2000 - First release of OWB•2006 - Oracle buys Sunopsis - ODI 10g - OWB Standard and Enterprise Editions•2008 - Roadmap to merge the products•2009 - Oracle Data Integrator Enterprise Edition•2010 - ODI 11g•2013 - ODI 12c•2015 - End of support for OWB

Wednesday, 10 September 14

Page 6: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Similarities

•Leverage database capabilities•ETL objects are stored in the database•Flow-based development•Can use the ELT paradigm

Wednesday, 10 September 14

Page 7: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Flow-based

Wednesday, 10 September 14

Page 8: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Flow-based

Wednesday, 10 September 14

Page 9: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

ETL vs ELT

•Reduce network transfer•Reduce hardware costs•Set-based operations

•Leverage target platform capabilities•Referential Integrity Check

Wednesday, 10 September 14

Page 10: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

ETL vs ELT

•Reduce network transfer•Reduce hardware costs•Set-based operations

•Leverage target platform capabilities•Referential Integrity Check

Wednesday, 10 September 14

Page 11: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

ETL vs ELT

•Reduce network transfer•Reduce hardware costs•Set-based operations

•Leverage target platform capabilities•Referential Integrity Check

Wednesday, 10 September 14

Page 12: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

ODI Differences

•Heterogeneous (Oracle, IBM DB/2, SQL Server, Flat files, Webservices, Hadoop, ...)•No deployment, the code is generated at run-time•Code executed by agents•Repository stored on any RDBMS (not only Oracle)•Separation between Physical and Logical implementation•Possibility to customise the code thanks to Knowledge Modules•Parallelism•Real-time

Wednesday, 10 September 14

Page 13: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Extract and load to many data sources

•Structured Data‣Oracle, SQL Server, IBM DB/2 etc

•Multi-dimensional Data‣Essbase, Oracle OLAP, MS AS

•Big Data‣Hadoop, Hive

•Packaged Applications•Legacy‣CICS, ISM, VSAM etc

•B2B‣EDIFACT, X12, RosettaNet

•Financial Management‣Hyperion, XBRL

•Broad range of protocols‣ JCA, JDBC, ODBC, FTP, HTTP etc

Wednesday, 10 September 14

Page 14: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

ODI 12c Key Components

•ODI Studio (the main development IDE)‣Designer Navigator‣Operator Navigator‣Topology Navigator‣Security Navigator

•ODI Agents (used for executing mappings and packages)

•ODI Console (web-based console for running published packages)

•ODI Repositories (contain definitions of models, interfaces and other objects)

DesignerModel data sources

Define projectsCreate interfaces

Release scenarios

OperatorView running interfaces and

packagesOperate production

RepositoryStandaloneAgent

Orchestrateinterfaces and scenarios

Information Systems(Databases, files, applications)

Wednesday, 10 September 14

Page 15: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

ODI 12c Key Components

•ODI Studio (the main development IDE)‣Designer Navigator‣Operator Navigator‣Topology Navigator‣Security Navigator

•ODI Agents (used for executing mappings and packages)

•ODI Console (web-based console for running published packages)

•ODI Repositories (contain definitions of models, interfaces and other objects)

Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository(Development)

Execution

Execution Repository(Production)

Wednesday, 10 September 14

Page 16: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

ODI Architecture

Wednesday, 10 September 14

Page 17: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Physical and Logical Topology

DEV_SCOTT

SH

HR

SCOTT

HR

SCOTT

SH

HUM_RES

DEV_OP

TEST_OP

PROD_OP

Physical Contexts Logical

Wednesday, 10 September 14

Page 18: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Physical and Logical Topology

DEV_SCOTT

SH

HR

SCOTT

HR

SCOTT

SH

HUM_RES

DEV_OP

TEST_OP

PROD_OP

Physical

SCOTT

SH

HR

Contexts Logical

Wednesday, 10 September 14

Page 19: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Physical and Logical Topology

DEV_SCOTT

SH

HR

SCOTT

HR

SCOTT

SH

HUM_RES

DEV_OP

TEST_OP

PROD_OP

Physical

SCOTT

SH

HR

Contexts Logical

DEV

TEST

PROD

Wednesday, 10 September 14

Page 20: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Physical and Logical Topology

DEV_SCOTT

SH

HR

SCOTT

HR

SCOTT

SH

HUM_RES

DEV_OP

TEST_OP

PROD_OP

Physical

SCOTT

SH

HR

Contexts Logical

DEV

TEST

PROD

Wednesday, 10 September 14

Page 21: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Physical and Logical Topology

DEV_SCOTT

SH

HR

SCOTT

HR

SCOTT

SH

HUM_RES

DEV_OP

TEST_OP

PROD_OP

Physical

SCOTT

SH

HR

Contexts Logical

DEV

TEST

PROD

Wednesday, 10 September 14

Page 22: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Physical and Logical Topology

DEV_SCOTT

SH

HR

SCOTT

HR

SCOTT

SH

HUM_RES

DEV_OP

TEST_OP

PROD_OP

Physical

SCOTT

SH

HR

Contexts Logical

DEV

TEST

PROD

Wednesday, 10 September 14

Page 23: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Separation between logical and physical implementation

•ODI splits data interfaces into business rules (logical), and technical implementation (physical)

•Business rules define what goes where,and using which transformation rules

•Technical implementation defineshow data is moved‣Changed Data Capture‣SQL to SQL‣File to SQL‣Slowly Changing Dimension

•Allows you to split mapping role intobusiness analyst and technical specialist

Wednesday, 10 September 14

Page 24: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Mappings

PROPERTY INSPECTOR

COMPONENTS PALETTE

Wednesday, 10 September 14

Page 25: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Mappings

Wednesday, 10 September 14

Page 26: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Mappings

Wednesday, 10 September 14

Page 27: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Knowledge Modules

Wednesday, 10 September 14

Page 28: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Knowledge Modules

Wednesday, 10 September 14

Page 29: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Knowledge Modules

create table<%=odiRef.getTable("L", "INT_NAME", "A")%>(

<%=odiRef.getColList("", "[COL_NAME]\t[DEST_WRI_DT] " + odiRef.getInfo("DEST_DDL_NULL"), ",\n\t", "", "INS or UPD")%>

)

Wednesday, 10 September 14

Page 30: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Debugger

1

2

Wednesday, 10 September 14

Page 31: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Debugger

1

2

Wednesday, 10 September 14

Page 32: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Debugger

4

3

Wednesday, 10 September 14

Page 33: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Execution OWB jobs from ODI

•Register in Topology•ODI Tool : OdiStartOwbJob•Monitor in Operator

Wednesday, 10 September 14

Page 34: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Migration Utility

•Supported‣Module / Location Model and Topology‣All data objects (except sequences) Datastores‣Mapping (Classic, Code-template and Pluggable) Mapping

•Unsupported‣Process flows‣PL/SQL objects (procs, packages, ...)‣Oracle Discoverer / OBIEE metadata‣Queues, Streams, CDC, User Defined Types‣Dimension and cube, cursors‣Data quality, data profiles, data auditors‣Configuration details (security, OMB scripting, ...)

Wednesday, 10 September 14

Page 35: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Migration Utility

•Needs OWB 11.2.0.3+•Based on ODI SDK•ODI 12.1.2 : Patch 17053768•ODI 12.1.3 : No Patch•Patch OWB 17830453•Command Line Interface•3 modes :‣Fast Check‣Dry Run‣Run

Wednesday, 10 September 14

Page 36: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Migration Utility

ODI_MASTER_USER=DEV_ODI_REPOODI_MASTER_URL=jdbc:oracle:thin:@localhost:1521/orclODI_MASTER_DRIVER=oracle.jdbc.OracleDriverODI_USERNAME=SUPERVISORODI_WORK_REPOSITORY_NAME=WORKREP_DEVOWB_WORKSPACE_OWNER=OWB_REPOSOWB_URL=localhost:1521:orclOWB_WORKSPACE_NAME=OWB_REPOS

MIGRATION_MODE=DRY_RUN

MIGRATION_OBJECTS=PROJECT.MY_RPOJECT; PROJECT.PROJECT_1.MODULE.MODULE_1;

MIGRATION_STRATEGY=CREATE

Wednesday, 10 September 14

Page 37: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Migration UtilityODI_MASTER_USER=DEV_ODI_REPOODI_MASTER_URL=jdbc:oracle:thin:@localhost:1521/orclODI_MASTER_DRIVER=oracle.jdbc.OracleDriverODI_USERNAME=SUPERVISORODI_WORK_REPOSITORY_NAME=WORKREP_DEVOWB_WORKSPACE_OWNER=OWB_REPOSOWB_URL=localhost:1521:orclOWB_WORKSPACE_NAME=OWB_REPOS

MIGRATION_MODE=DRY_RUN

MIGRATION_OBJECTS=PROJECT.MY_RPOJECT; PROJECT.PROJECT_1.MODULE.MODULE_1;

MIGRATION_STRATEGY=CREATE

Wednesday, 10 September 14

Page 38: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Migration Utility: Report

Wednesday, 10 September 14

Page 39: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Migration Utility: Result

Wednesday, 10 September 14

Page 40: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Further reading•Blog post series by Peter Scott (@dw_pete) on Rittman Mead blog:http://www.rittmanmead.com/2011/10/oracle-warehouse-builder-and-data-integrator/

•Blog post series on Rittman Mead blog:http://www.rittmanmead.com/2013/10/oracle-data-integrator-12c-release-part-1/

•OTN article by Stewart Bryson (@stewartbryson) : Making the Move from Oracle Warehouse Builder to Oracle Data Integrator 12c : http://www.oracle.com/technetwork/articles/datawarehouse/bryson-owb-to-odi-2130001.html

•Blog Post by David Allan (@im_dave) and Julien Testut (@JulienTestut) on ODI12c migration : https://blogs.oracle.com/dataintegration/entry/planning_the_journey_from_oracle

•“OWB – Making the Move to Oracle Data Integrator” by Michael Rainey (@mRainey) at UKOUG 2012:http://www.rittmanmead.com/files/ukoug2012_rainey_odi.pdf

Wednesday, 10 September 14

Page 41: UKOUG BIRT SIG 2014 –  ODI for OWB Developers

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] : www.rittmanmead.com

Wednesday, 10 September 14