OracleDataIntegrator12c:GettingStarted

MichaelRainey|UTOUGTrainingDays2017

• MichaelRainey-TechnicalAdvisor• Spreadingthegoodwordaboutgluentproductswiththeworld

• OracleDataIntegrationexpertise

• OracleACEDirector

• mRainey.co

• gluent.com

2

Introduction

we liberate enterprise data

AboutGluent

We are long term Oracle Database & Data Warehousing guys – long history of performance & scaling

About 20 people in distributed teams in US & UK

The world is changing – we help customers to get the best out of both worlds!

we liberate enterprise data

Gluent Data Platform enables all applications to access all enterprise data, at anytime, on demand, and without any code migration or rewrites

• GluenttobefeaturedintheStartupShowcaseatStrata+HadoopWorld(Tomorrow!)

• HadoopforDatabaseProfessionals one-daytrainingcourse(March21st)

4

Quickannouncements

gluent.com/event/hadoop-for-database-professionals-2

Why“GettingStarted”

6

Manyconferencepresentationsgettootechnical,toofast

WhatisODI?

• E-LT(Extract,Load,Transform)vsETL• Heterogeneoussourcesandtargets• Builttointegratealldata

• Flexibility• Reusablecodetemplates(KnowledgeModules)

• OpenAPI

• ODIwilladapttoyourdatawarehouse-andnottheotherwayaround

• Flowbasedmappings• LogicalvsPhysical

8

WhatisODI?

9

WheredidODIcomefrom?

Early2000s:Sunopsisformed

2006:OracleacquiresSunopsis.

2010:OracleDataIntegrator11greleased.Oracle’sfirstrealupdatetoODI.

Late2013:OracleDataIntegrator12creleased,combiningODIandOWBfeatures.

2012:ODIforBigDatafirstreleased

10

ODI12cArchitectureandComponents

*imagesource:https://docs.oracle.com/middleware/1221/odi/concepts/img/architecture.png

11

TypicalODIrepositorysetup-thebasics

Source:ateam-oracle.com/how-many-odi-master-repositories-should-we-have

12

TypicalODIrepositorysetup-full-blownenvironment

Source:ateam-oracle.com/how-many-odi-master-repositories-should-we-have

InstallationandConfiguration

• Knowthedifferenttypesofinstalls• Standard-usedforStandaloneAgentoronclientmachine(ODIStudioonly)

• Enterprise-JEEAgentorColocatedAgent,ODIConsole,FMCplugin

14

Chooseyourinstallationtypewisely

15

Configurationishalfthebattle

CONFIGURATION

• ODI_HOME/odi/studio/odi.sh|exe• Connecttoexistingrepository• Ifnoneexists,runRCUfirst

• Boom,you’rein!

16

ODIStudioConfiguration

• ODI_HOME/oracle_common/common/bin/config.sh|exe• RuntheWebLogicServerconfigurationwizard

• StandaloneAgent• CreatedomainoutsideofODIOracleHome

• JEEAgent• Addcredentialsmapping

• Create/configureadminserver,nodemanager,topology,etc.

17

StandaloneorJEEAgentConfiguration

18

ODIAgent-howitworks

ODIAgent

ODIrepository

Sourcedatabase

Targetdatabase

1.Executionrequestviahttp(s)

2.AgentcommunicateswithODIrepositorytogeneratecodeandgathersource/targetconnection

3.Agentpassescodetosource/targetdatabasesandupdatesrepositoryexecutionlogs

GettingStarted

• DataServer• Connectiontothedatasource

• PhysicalSchema• Databaseschema

• Workschema

• LogicalSchema• Abstractsthephysicalconnection/schemafromthelogical

• DevelopmentoccursagainsttheLogicalSchema

• Model• MetadataobjecttiedtoLogicalSchema

• Representsdatabaseschema,filedirectory,XMLdocument,etc

• Datastore• Logicalrepresentationoftable,file,XMLelement,etc.

20

Topology,Contexts,Models,andDatastores-Ohmy!

Alloftheseobjectsmustbecreatedandsetuppriortoanymappingdevelopmentcanoccur.

• There’sanotherwaytogetstarted-quickly!• NewModelandTopologyObjectswizard-ODI12.1.3• InODI11g,theODIQuickStartwasa10stepchecklist,whereeachstepledtoanothersectioninthedocumentation

21

Developer’sQuickstart

22

Topology,Contexts,Models,andDatastores

23

LogicalConstraintsonthemetadata

23

LogicalConstraintsonthemetadata

23

LogicalConstraintsonthemetadata

23

LogicalConstraintsonthemetadata

MappingsandKnowledgeModules

25

Mappings-LogicalDatastorecomponent

Joincomponent

Splitoutputtomultipletargettables

MappingsarebuiltfromdatastoreandcomponentstologicallyformtheETL

26

Mappings-Physical

Physicalimplementationsofthemappingsarerepresentedbasedonlocationofthedatastore.KnowledgeModulesareselectedtogeneratethecode.

26

Mappings-Physical

Physicalimplementationsofthemappingsarerepresentedbasedonlocationofthedatastore.KnowledgeModulesareselectedtogeneratethecode.

ODIwilladapttoyourdatawarehouse-andnottheotherwayaround

27

KnowledgeModules-thesecretsauce

*imagesource:FusionMiddlewareDevelopingKnowledgeModuleswithOracleDataIntegrator

27

KnowledgeModules-thesecretsauce

*imagesource:FusionMiddlewareDevelopingKnowledgeModuleswithOracleDataIntegrator

27

KnowledgeModules-thesecretsauce

*imagesource:FusionMiddlewareDevelopingKnowledgeModuleswithOracleDataIntegrator

OrchestrationPackagesandLoadPlans

• Grouplikeobjectstogether• Performspecializedtasks(looping,etc)

29

Packages

• ExecutesScenarios(executionobjectsthatareessentiallycompiledcode)• Stepscanberuninparallelorserially• If-then-elsestepcanevaluateavariabletodeterminepath• Exceptionstepscanbecustomizedandexecutedonfailure

30

LoadPlans-thetoplevelorchestrationobject

MoreflexibilityProcedures

• Reusablecomponentsforadmintasks• DDL,filearchivescripts,exceptionhandling,etc

• Movingortransformingdata,itprobablyshouldbeamapping• Creatingacomplex,recurringtaskfordatamanipulation,probablyaKnowledgeModule

32

Proceduresformaintenancetasksandone-offfunctions

Imagessource:http://ritt.md/9E

• IntegrationwithGoldenGate(changedatacapture)andEnterpriseDataQuality(for…dataquality)• ODIforBigData• ODICloudService

33

Othernotablefeatures

• OracleDataIntegratoris…• anE-LT/dataintegrationtool

• builttointegratealldata

• flexible.ODIwilladapttoyourdatawarehouse-andnottheotherwayaround

• Chooseyourinstallationwiselyandknowconfigurationismoreinvolved• ManyobjectsmustbecreatedpriortoMappingdevelopment

• UsetheDeveloper’sQuickstart!

• Flowbasedmappingsseparatebusinesslogicfromphysicalimplementation• KnowledgeModulesarethe“secretsauce”ofODI• Moreinfo: mRainey.co/odi12c-getting-started ateam-oracle.com/?cat=64 blogs.oracle.com/dataintegration

34

Recap

35

we liberate enterprise data

thank you!