Limor Fledel-Vagman BI Practice Manager Oracle Israel OBIEE Data Federation.

Limor Fledel-VagmanBI Practice ManagerOracle Israel

OBIEE Data Federation

OBIEE Federation

• Vertical Federation– Integrating two or more disparate data source having different

levels of granularity joined by one or more conformed dimensions– Example: Essbase Sales cube to relational Sales detail data (drill-

through)

• Horizontal Federation– Integrating two or more disparate data source having the same

level of granularity joined by one or more conformed dimensions– Example: Essbase Sales cube with HR relational database

LTS ???

Logical table sources (LTS’s) are a key feature within the OBIEE semantic model

OBIEE Aggregate Navigation

Why Aggregate Navigation?

• Problem: Summary questions are slow– Example, “All sales worldwide for last 2 years by quarter by

family”– This will potentially sum/group by millions of rows!– The disk spin time to scan all these rows will be very long

• Solution: ETL to batch build high level aggregates– Oracle BI EE has “aggregate navigation” to use them– Aggregates can be in the same database as the source– They can also be in a different relational source or Essbase– Often orders of magnitude performance improvement

Oracle BI EE “Architecture”

Relational

Tables

BI ServerSemantic Layer Metadata

SQL

Oracle BI EE Aggregate Navigation

Relational

Detail

Summary

Semantic Layer Metadata

Metadata describes aggregate mappings

SQL

SQL

At query time, BI Server queries the fastest source that has enough detail to satisfy the user request

Load time

aggregation

Performance of highly summarized requests is dramatically improved

BI Server

Federation Enables External Aggregates

Relational

Detail

Summary


SQL

SQL

Load time

aggregation

Summary

SQL

Aggregates can be in other federated sources

BI Server

External Aggregates Can Be in Essbase

Relational

Detail

SummaryEssbase

Semantic Layer Metadata Aggregate navigation

can include multidimensional

SQL

MDX

Load time

aggregation

BI Server

External Aggregates Can Be Oracle OLAP

Relational

Detail

Summary

Oracle OLAP AW

Semantic Layer Metadata Aggregate navigation

can include multidimensional

SQL

MDX

Load time

aggregation

BI Server

"This information is not a commitment to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle"

How is Aggregate Navigation Set Up?

• Database Design time:– DBA defines aggregate stars at various grain combinations

– Builds ETL to load the aggregates each night

Star Time Product

Base Day SKU

Agg 1 Month Brand

Agg 2 Qtr -

Mapping a Base-Level Physical Star

Business Model and Mapping Layer

Physical LayerMappings

D0 Time

D4 Product

F0 Rev Base Measures

D_Day

D_Prod_SKU

F_Bill

Adding a Moderately Aggregated Star


D0 Time

D4 Product


D_Day

D_Prod_SKU

F_Bill

D_Month

D_Brand

F_Bill_agg2


Adding a Highly-Aggregated Star


D0 Time

D4 Product


D_Day

D_Prod_SKU

F_Bill

D_Month

D_Brand

F_Bill_agg2

D_Qtr

F_Bill_agg1


Essbase as an Aggregate Source


D0 Time

D4 Product


D_Day

D_Prod_SKU

F_Bill


Month

Brand

Billed Amt

Adding a Highly-Aggregated Star


D0 Time

D4 Product


D_Day

D_Prod_SKU

F_Bill

D_Month

D_Brand

F_Bill_agg2

D_Qtr

F_Bill_agg1

How does the query planner decide which FJC to use as its source for a given request?


Selecting an Aggregate or Base Star at Query Time

How does the query planner decide which star to use as its source for a given LSQL request?

• First, it eliminates sources that don’t have enough detail to answer the question

– Query grain < star grain star not qualified– Query grain = star grain star qualified– Query grain > star grain star qualified

• Second, it estimates the fastest/prioritized source

How BI Server Knows the Grain of a Star

• At query time, the BI Server checks the available mappings to find which ones are qualified by grain

How Does It Know Which Star is Fastest? Number of elements

• Be sure to correctly populate the Dimension Level’s “Number of elements at this level”

OBIEE 10g

How Does It Know Which Star is Fastest?LTS Priority Group Order

• LTS Priority Group Order• Lower value = higher priority• LTS Priority Group becomes

main decider in which LTS to use

OBIEE 11g New Feature

Aggregate Persistence

Aggregate Persistence Feature

• Managing aggregates manually has high TCO• Oracle BI EE automates creation of aggregates• Leverages existing metadata• Orders of magnitude performance boost; low TCO

“Aggregate Persistence” Automates Create & Load

• Define agg navigation metadata

Detail

Summary


Admin Tool: Aggregate build automation wizard

• Create & schedule scripts

• Create Tables

• Aggregation

• Run scripts • Create Tables• Aggregation

BI Server

Aggregate Persistence – Manual Option

Detail

Summary



• Create scripts • Create Table DDL• Aggregation

• Run scripts • Create Tables• Aggregation

DBA

BI Server


Aggregate Persistence with Essbase & Oracle OLAP

Detail



• Define cube schema• Create load rules & schedule

SummaryEssbase or AW

BI Server


"This information is not a commitment to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle"

Administration Steps

Design aggregate facts and grains

Wizard to create metadata

1 Target database for create/load

3

2

Aggregate Persistence Wizard

Aggregate Persistence Wizard

• Automates the creation of physical aggregate tables and their corresponding objects in the repository

Use wizard to build script to generate aggregate tables.

Script generates physical tables,

repository objects, and mappings.

Fact aggregate

Dimension aggregates

Aggregate Persistence Wizard Steps

1. Open Aggregate Persistence Wizard.2. Specify the file name and location.3. Select business model and measures.4. Select dimensions and levels5. Select connection pool, container, and name.6. Review aggregate definition.7. View complete aggregate script.8. Verify script is created.9. Create and run a batch file.10. Verify aggregates in the Physical layer.11. Verify aggregates in the BMM layer.12. Verify aggregates in the database.13. Verify results in Answers.

1. Open Aggregate Persistence Wizard

• Select Tools > Utilities > Aggregate Persistence Wizard and click the Execute button.

2. Specify File Name and Location

• Specify a file and location where the output script should be saved.

3. Select Business Model and Measures

Select business model.

Select fact table.

Select measures.

4. Select Dimensions and Levels

• Select corresponding aggregate dimensions and levels.

5. Select Connection Pool, Container, and Name

Select the database object.

Select the schema.

Select the connection pool.

Name the aggregate table.

6. Review Aggregate Definition

View the aggregate definition.

7. View Complete Aggregate Script

Confirmation of script creation and location

Script

8. Verify that the Script Is Created

• Navigate to the directory where the file was saved and verify that the script was created as expected.

9. Create and Run a Batch File

• Create and run a batch file with the following format to drive the aggregate creation process:

nqcmd Oracle BI Server command utility

-d Oracle BI Server data source name

-u Repository username-p Repository password-s Path to the create

aggregate SQL script

10. Verify Aggregates in the Physical Layer

• Verify that the aggregates are created in the Physical layer of the repository as expected.

Fact aggregate

Dimension aggregates

11. Verify Aggregates in the BMM Layer

• Verify that the aggregates are created in the Business Model and Mapping layer of the repository as expected.

Fact aggregate

Dimension aggregate

12. Verify Aggregates in the Database

• Verify that the aggregates are created in the database.

13. Verify Results in Answers

– Activate the aggregate tables:

– Run a query in Answers:

– Check the log and verify that the aggregate tables are accessed as expected:

Considerations

• Using aggregates comes with a price:– Additional time is required to build and load these tables.– Additional storage is necessary.

• Build only the aggregates you need:– Look at query patterns and build aggregates to speed up

common queries that require summarized results.– Ensure that enough data is combined to offset the cost of

building aggregates.– Monitor and adjust to account for changing query patterns.

Using Partitions and Fragments

Business Challenge

– Data is often partitioned into multiple physical sources for a single logical table.

– Organizations need to seamlessly and efficiently access and process data from multiple sources to satisfy user requests.

– Business applications must “know” where to go for what type of data and under what conditions.

Business Solution: Oracle BI Server

– Oracle BI repository can be configured so that Oracle BI Server handles the navigation to the appropriate source.

– Oracle BI Server seamlessly and efficiently accesses and processes data from multiple sources to satisfy user requests.

Partition

– Is a database element that contains part of the data for a fact or a dimension

– Combines with other data fragments as necessary – May be:

• Fact-based• Value-based• Level-based• Complex

Partitioning by Fact

– Data is partitioned by fact when different fact data is stored in different tables.

– Example: Actual sales versus quota targets

1000

Product Sale

1100

Sales Rep

2000

Product Quota

1100

Sales Rep

Actual sales Quota targets

Partitioning by Value

– Data is partitioned by value when the data is split into separate tables according to the values of the data.

– Example: Invoice data is stored separately for each region.

1000

Dollars

Central1135293

RegionInvNbrInvoices for Central Region

Invoices for West Region 200

Dollars

West114444

RegionInvNbr

Partitioning by Level

– Data is partitioned by level when the same facts are stored in separate tables at different levels of aggregation.

– Example: Detailed sales data is summarized and stored by year and region.

19981010

19981001

19980105

Date

100001100

250001100

10000

Product Sale

1100

Sales Rep

Sales detailed data Sales by year and region

Central3000001999

200000

Total Dollars

Central1998

RegionYear

Complex Partitioning

– Data is partitioned using more than one technique.– Example: Invoice sales data is partitioned by value and level.

Central25000199802

10000

Total Dollars

Central199801

RegionMonth

Invoices by month for Central Invoices by year for Central

200000

Total Dollars

Central1998

RegionYear

Invoices by month for West Invoices by year for West

West350000199802

300000

Total Dollars

West199801

RegionMonth

3000000

Total Dollars

West1998

RegionYear

ABC Example: Fact-Based (Quota)

• Allow users to query for actual sales data and quota data in a single query.

1000

Product Sale

1100

Sales Rep

2000

Product Quota

1100

Sales Rep

Actual sales Quota targets

ABC Example: Value-Based (Customer)

• Replace the current, single source for customer data with two value-based partitions.

Clifton Lunch

Name

1000

NewKeyCustomers with names starting with letters A–M

Customers with names starting with letters N–Z Tong’s Wok

Name

1002

NewKey

ABC Example: Value-Based (Inventory)

• Build a business model for inventory data that is fragmented into multiple tables.

Eight quarters of inventory data stored in eight separate tables

Implementation Steps

– Import physical sources.– Create physical joins.– Add sources to the Business Model and Mapping layer.– Specify fragmentation content.– Test the results.

New step

Specify Fragmentation Content

– Use the Expression Builder to define the type of content that the fragment contains.

– Set the flag to specify whether to combine this fragment with other data.

Specifies that customer data from A–M is contained in this fragment

Specifies combining this fragment with the other

Summary

Identify reasons for segmenting data and

implement it – benefit is promised!

Question: • The end user needs to know what data is contained in which

partition before submitting a query. True or false?

Real-time BIFederated OLTP/EDW Reporting

Horizontal Federation for Real-Time BI

• Using horizontal fragmentation in OBIEE, we can map a single logical fact table to multiple LTS’s.

• Physical fact table in our EDW called SALES_FACT• Physical fact table in our source system to represent

real-time data

• One logical fact table in the BMM — called “Sales Fact Realtime”


Fact table mapping


Customer dimension mapping


Use variable as the threshold between reporting against the EDW schema and the source system schema


LTS mapping using threshold date


OBIEE does the logical union


When the BI Server has enough information to know that the entire result set will come from a single source, then the SQL will be issued against only one of the LTS’s.

Oracle by Example and Relevant Blogs

Web tutorial :Oracle by Example

“Federating Essbase and Relational Data Sources in OBIEE”

http://www.artofbi.com/index.php/2009/07/federation-in-obiee-what-are-you-talking-about/

http://odtugspconference.com/Slides/OBIEE%20Integration%20with%20Essbase.pdf

http://www.rittmanmead.com/2011/05/real-time-bi-federated-oltpedw-reporting/

http://www.artofbi.com/index.php/2009/07/federation-in-obiee-what-are-you-talking-about/

http://odtugspconference.com/Slides/OBIEE%20Integration%20with%20Essbase.pdf

http://www.rittmanmead.com/2011/05/real-time-bi-federated-oltpedw-reporting/

Limor Fledel-Vagman BI Practice Manager Oracle Israel OBIEE Data Federation.

Documents

Transcript of Limor Fledel-Vagman BI Practice Manager Oracle Israel OBIEE Data Federation.