UKOUG Tech 15 - ODI Lifecycle and Data Governance

[email protected] www.rittmanmead.com @rittmanmead

Jerome Francoisse | UKOUG Tech 2015

Oracle Data Integrator – Development Lifecycle & Data Governance


Jérôme Françoisse• Consultant for Rittman Mead- Oracle BI/DW Architect/Analyst/Developer• ODI Trainer

• Providing ODI support on OTN Forums• ODI 12c Beta Program Member• Blogger at http://www.rittmanmead.com/blog/• Email : [email protected]• Twitter : @JeromeFr

http://www.rittmanmead.com/blog/

mailto:[email protected]


About Rittman Mead

•World’s leading specialist partner for technical excellence, solutions delivery and innovation in Oracle Data Integration, Business Intelligence, Analytics and Big Data

•Providing our customers targeted expertise; we are a company that doesn’t try to do everything… only what we excel at

•70+ consultants worldwide including 1 Oracle ACE Director and 2 Oracle ACEs

•Founded on the values of collaboration, learning, integrity and getting things done

Optimizing your investment in Oracle Data Integration

•Comprehensive service portfolio designed to support the full lifecycle of any analytics solution

•4 UKOUG Partner of the Year Award 2015


User Engagement

Visual Redesign Business User Training

Ongoing SupportEngagement Toolkit

Average user adoption for BI platforms is below 25%

Rittman Mead’s User Engagement Service can help


Successful ODI project

• Having good developers is great• … but it’s not enough!

• We also need :- Assurance that the code deployed in production is

correct- The right data- Good data


Agenda

• Lifecycle Management- Architecture- Promotion- Versioning • Data Governance- Oracle Enterprise Metadata Management- Oracle Enterprise Data Quality


ODI 12c Key Components

Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository(Development)

Execution

Execution Repository(Production)

• ODI Studio (the main development IDE)- Designer Navigator- Operator Navigator- Topology Navigator- Security Navigator• ODI Agents (executing mappings and

packages)• ODI Console (web-based interface)• ODI Repositories (contain definitions of

models, mappings and other objects)


Physical and Logical Topology

DEV_SCOTT

SH

HR

SCOTT

HR

SCOTT

SH

HUM_RES

DEV_OP

TEST_OP

PROD_OP

Physical

SCOTT

HR

Contexts

Logical

DEV

TEST

PROD

SH


Architecture

Security

Topology

Versioning

Master Repository

Models

Projects

Execution


Execution



How many repositories do I need?

Security

Topology

Versioning

Master Repository

Models

Projects

Execution


Execution


Execution

Execution Repository(Test)



Security

Topology

Versioning

Master Repository

Models

Projects

Execution


Execution


Execution


Security

Topology

Versioning

Master Repository



Security

Topology

Versioning

Master Repository

Models

Projects

Execution


Execution


Execution


Security

Topology

Versioning

Master Repository

Execution

Execution Repository(Pre-Production)

Security

Topology

Versioning

Master Repository



Models

Projects

Execution


Execution


Execution


Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository(Hotfix)

Execution


Security

Topology

Versioning

Master Repository

Security

Topology

Versioning

Master Repository



Models

Projects

Execution


Execution


Execution


Security

Topology

Versioning

Master Repository

Models

Projects

Execution

Work Repository(Hotfix)

Execution


Security

Topology

Versioning

Master Repository

Security

Topology

Versioning

Master Repository

Execution

Execution Repository(Smoke Test)


Problems• As a developer, how do I create a consistent set of objects for a

release?- For development environments- For execution environments

• As a release manager, how do I easily promote code from one environment to another?• As a release manager, how do I easily keep track of these sets of

objects?• As a release manager, how do I roll back to a previous release?


The Solution – Deployment Archive (DA)

• Archive file containing a set of consistent ODI objects• Applied to Deployed Repositories (test/production)• 3 types :- Initial- Patch- Rollback• Development or Execution


Example

Development environment

QA environment (Deployed repository)

Developer

Release Manager

Bug

Creates Initial Release 1.0.0.0

Initialize repository with Release 1.0.0.0

Creates Patch Release 1.1.0.0

to add a features

Apply Patch Release 1.1.0.0

and create rollback DA

Apply Rollback Release 1.1.0.0

Creates Patch Release 1.1.0.1

to fix the previous bug

Apply Patch Release 1.1.0.1

and create rollback DA


Developer / Administrator

• Create a deployment archive



• Create a deployment archivefrom VCS label

• Best Practice :Link documentation and DDL statements to be applied



• Create a deployment archive from selected objects

• Best Practice :Mention the bug number in the description



• No need to think about development or execution-only environment


Release Manager

• Apply Deployment Archives


Release Manager

• Initial DA• Can choose from

development or execution-only archive


Release Manager

• Patch DA• Can choose from

development or execution-only archive

• Best Practice :Always create a Rollback DA!


Release Manager

• See applied DAs – Inventory

• See available DAs – View Details


Release Manager

• Rollback DA


Automation

• Available through ODI SDK

• Scenarios :- Import and be sure you always generate a rollback

DA- Import daily into a smoke test environment- Import the same release in Hotfix and Production at

the same time


Other Problems

• How to rollback on the Development environment?• How to restore delete objects in the Development

environment?


The Solution - VCS

• Integration with Subversion (SVN)- Everything is done through ODI Studio- Git support in the next release- Tagging- Branching / Merging


VCS - Features

• Add an object to VCS


VCS - Features• Create a new version of an object


VCS - Features• View and compare versions


VCS - Features• Restore a version


VCS - Features• Tags- Snapshot of consistent set of versioned objects- Ready for promotion – Deployment Archive- … or for branching- Created from

‣ Whole repository (Full Tag)‣ Selected Objects and dependencies (Partial Tag)


VCS - Features• Branches- Created from a Tag- Populate a new environment – Master and Work

repository- For a new feature or a new team- Can be merged back in the trunk


VCS - Features• Branches

Trunk

Branch 1

BranchingRelease 1

101 2 3 5

4 6 7 9

Merging

11

Bug Fixes for Release 1

New features for Release 2

New features for Release 2New features for Release 1

8


Data Governance• The right data• The good data

People

Process Tools


Data Governance• The right data

Which reports usesmy application

data?

Application User Sysadmin

Which reports use the mainframe

data?

ETL Developer

What is the impact if I change this

mapping?

DBA

Who is using this database I need to

upgrade?

Data Steward

Where is this data entered?

Who should I notify if there is a

problem?

Which data are available?

Data Scientist BI Developer

Where does this data come from?

Executive

How are sales figures

calculated?


Oracle Enterprise Metadata Management• Lineage

- From reports to source• Impact Analysis

- From source to reports• DB - ETL - BI Metadata• Business Glossary• Semantic Linking

• Supporting features- Harvesting- Automatic Stitching- Versioning- Collaboration - Annotations


Oracle Enterprise Metadata Management


• The good dataOracle Enterprise Data Quality


Oracle Enterprise Data Quality - Profiling


Oracle Enterprise Data Quality - Cleansing/Matching


Oracle Enterprise Data Quality - ODI Integration


Summary• ODI - Deployment Archives- Promote, Rollback and keep track of your code- For development or execution-only environment- Easy for Developers and Release Managers• ODI - VCS- Be able to get back to any version to object- Branching for Team development• OEMM- Lineage, impact analysis and business glossary• OEDQ- Profile, Cleanse, Match, Monitor, Govern


Further Reading / Testing• ODI Lifecycle- Managing versions in SVN

https://blogs.oracle.com/dataintegration/entry/oracle_data_integrator_12_21- Parallel Development using SVN Branches

https://blogs.oracle.com/dataintegration/entry/oracle_data_integrator_12_22- ODI Getting Started VM 12.2.1

http://www.oracle.com/technetwork/middleware/data-integrator/odi-demo-2032565.html

• OEMM- Oracle By Example

http://www.oracle.com/technetwork/middleware/oemm/learnmore/index.html

• OEDQ Getting Started 12.2.1http://www.oracle.com/technetwork/middleware/oedq/downloads/edq-vm-download-2424092.html

https://blogs.oracle.com/dataintegration/entry/oracle_data_integrator_12_21

https://blogs.oracle.com/dataintegration/entry/oracle_data_integrator_12_22

http://www.oracle.com/technetwork/middleware/data-integrator/odi-demo-2032565.html

http://www.oracle.com/technetwork/middleware/oemm/learnmore/index.html

http://www.oracle.com/technetwork/middleware/oedq/downloads/edq-vm-download-2424092.html


Questions?

UKOUG Tech 15 - ODI Lifecycle and Data Governance

Data & Analytics

Transcript of UKOUG Tech 15 - ODI Lifecycle and Data Governance