Oracle Business Intelligence 11g Incrementally Loading Exalytics … · 2017-01-25 ·...

Oracle Business Intelligence 11g

Incrementally Loading Exalytics using Notepad

Antony Heljula April 2013

© Peak Indicators Limited 2

Incrementally Loading Exalytics using Notepad Why Notepad?

• The current method of reloading Exalytics:


Incrementally Loading Exalytics using Notepad #1 Did you know?

• Using the NQCMD command-line you can initiate various types of SQL against the BI Server: • Logical SQL • Business Model SQL • Physical SQL (including PL/SQL blocks)

• Example Physical SQL to drop a table:

EXECUTE PHYSICAL CONNECTION POOL "Aggregates"."Aggregates Connection Pool" BEGIN EXECUTE IMMEDIATE 'DROP TABLE AGG_STG.AG_FACT_ORDERS_LN'; END;



#2 Did you know?

• When you run a “Create Aggregates” command, behind the scenes the BI Server runs a series of “populate” commands to load each dimension/fact table

• The code can be obtained from the BI Server log file (nqquery.log)

populate "ag_Fact_Orders_Lin" mode ( create table connection pool "Aggregates"."Aggregates Stage Connection Pool") as select_business_model "Sales DW"."Dim - Organization"."Organization Id" as "Organizati00001A59", "Sales DW"."Dim - Time"."Month" as "Month00001C1C", "Sales DW"."Fact - Orders Lines"."# Order Lines" as "Z_Order_Li00006355", "Sales DW"."Fact - Orders Lines"."# Orders" as "Z_Orders00006356", "Sales DW"."Fact - Orders Lines"."Quantity" as "Quantity00006357", "Sales DW"."Fact - Orders Lines"."Order Amount" as "Order_Amou00006358", "Sales DW"."Fact - Orders Lines"."Return Amount" as "Return_Amo00006359", "Sales DW"."Fact - Orders Lines"."Cost of Sales" as "Cost_of_Sa0000635A", "Sales DW"."Fact - Orders Lines"."Avg Unit Price" as "Avg_Unit_P0000635B", "Sales DW"."Fact - Orders Lines"."Order Amount (Primary Markets)" as "Order_Amou0000635C", "Sales DW"."Fact - Orders Lines"."Order Amount (Secondary Markets)" as "Order_Amou0000635D" from "Sales DW"

Table Name

“Create” to create target table and then insert rows or

“Append” to just insert records (if table already exists)

The connection pool containing the target aggregate table



#3 Did you know?

• The “populate” command initiates a “Business Model” SQL query • It is possible to add a “WHERE” clause to filter the rows extracted from the

source system

populate "ag_Fact_Orders_Lin" mode ( create table connection pool "Aggregates"."Aggregates Stage Connection Pool") as select_business_model "Sales DW"."Dim - Organization"."Organization Id" as "Organizati00001A59", "Sales DW"."Dim - Time"."Month" as "Month00001C1C", "Sales DW"."Fact - Orders Lines"."# Order Lines" as "Z_Order_Li00006355", "Sales DW"."Fact - Orders Lines"."# Orders" as "Z_Orders00006356", "Sales DW"."Fact - Orders Lines"."Quantity" as "Quantity00006357", "Sales DW"."Fact - Orders Lines"."Order Amount" as "Order_Amou00006358", "Sales DW"."Fact - Orders Lines"."Return Amount" as "Return_Amo00006359", "Sales DW"."Fact - Orders Lines"."Cost of Sales" as "Cost_of_Sa0000635A", "Sales DW"."Fact - Orders Lines"."Avg Unit Price" as "Avg_Unit_P0000635B", "Sales DW"."Fact - Orders Lines"."Order Amount (Primary Markets)" as "Order_Amou0000635C", "Sales DW"."Fact - Orders Lines"."Order Amount (Secondary Markets)" as "Order_Amou0000635D" from "Sales DW" where "Sales DW"."Dim – Time"."Year / Month" = VALUEOF(CURR_YEARMONTH)

Business Model SQL

Example WHERE clause filter referencing a Repository Variable



#4 Did you know? • The difference between a full load and an incremental load is:

1. A Staging Table

2. A “WHERE” clause



Step 1

• We need to create an empty “Staging” aggregate table

• It should be an identical copy of the target aggregate table Oracle

Business Intelligence 11g

Target Aggregates

Staging Aggregates

1



Step 1 - Code

EXECUTE PHYSICAL CONNECTION POOL "Aggregates"."Aggregates Connection Pool"

BEGIN

EXECUTE IMMEDIATE 'DROP TABLE AG_FACT_ORDERS_STG'; EXECUTE IMMEDIATE 'CREATE TABLE AG_FACT_ORDERS_STG AS SELECT * FROM AG_FACT_ORDERS WHERE 1=2'; END;



Step 2

• Obtain the relevant “populate” command from the BI Server query log file

• Modify it to “append” into the Staging aggregate table instead of the Target aggregate

• Add a WHERE clause filter to limit the rows extracted e.g. WHERE Quarter = CURR_QTR

Oracle Business Intelligence

11g

Target Aggregates

Staging Aggregates

2



Step 2 - Code

SET VARIABLE DISABLE_CACHE_HIT=1, DISABLE_CACHE_SEED=1, DISABLE_SUMMARY_STATS_LOGGING=1, INACTIVE_SCHEMAS='"Aggregates".."AGG"':

populate "AG_FACT_ORDERS_STG" mode (append table connection pool "Aggregates"."Aggregates Connection Pool") as select_business_model

"Sales DW"."Dim - Organization"."Organization Id" as "Organizati00001A59",

"Sales DW"."Dim - Time"."Month" as "Month00001C1C",

"Sales DW"."Fact - Orders Lines"."# Order Lines" as "Z_Order_Li00006355",

"Sales DW"."Fact - Orders Lines"."# Orders" as "Z_Orders00006356",

"Sales DW"."Fact - Orders Lines"."Quantity" as "Quantity00006357",

"Sales DW"."Fact - Orders Lines"."Order Amount" as "Order_Amou00006358",

"Sales DW"."Fact - Orders Lines"."Return Amount" as "Return_Amo00006359",

"Sales DW"."Fact - Orders Lines"."Cost of Sales" as "Cost_of_Sa0000635A",

"Sales DW"."Fact - Orders Lines"."Avg Unit Price" as "Avg_Unit_P0000635B",

"Sales DW"."Fact - Orders Lines"."Order Amount (Primary Markets)" as "Order_Amou0000635C",

"Sales DW"."Fact - Orders Lines"."Order Amount (Secondary Markets)" as "Order_Amou0000635D"

from "Sales DW"

WHERE "Sales DW"."Dim - Time"."Quarter" = VALUEOF(CURR_QTR);



Step 3

• Incrementally update the Target Aggregate table from the Staging Aggregate. Performed in 2 stages:

1. Delete any records from the Target aggregate which need to be reloaded from the Staging aggregate (e.g. all records for the current quarter)

2. Insert all records from the Staging aggregate

• Both statements performed as part of a

single database transaction

Oracle Business Intelligence

11g

Target Aggregates

Staging Aggregates

3



Step 3 - Code

EXECUTE PHYSICAL CONNECTION POOL "Aggregates"."Aggregates Connection Pool"

BEGIN

DELETE FROM AGG_FACT_ORDERS WHERE Quarter IN (SELECT Quarter FROM AGG_FACT_ORDERS_STG);

INSERT INTO AGG_FACT_ORDERS SELECT *

FROM AGG_FACT_ORDERS_STG;

COMMIT;

DBMS_STATS.GATHER_TABLE_STATS(ownname => 'AGG', tabname =>

'AG_FACT_ORDERS' , estimate_percent => 30 );

END;

Delete all records from the Target aggregate where the Quarter exists in

the Staging aggregate

Both commands done in a single database transaction

Once loaded, gather statistics



That’s it!

• Put all the commands into a single script (using Notepad) and then run it against the BI Server using nqcmd: nqcmd –d AnalyticsWeb –u weblogic –p welcome1 –s inc_load.txt



Problem

• This script is for one table:

• How many dim/fact aggregate tables do you have?



Inmarsat have lots!

• How do you cater for: ? Scheduling ? Handling dependencies ? Parallel running ? Error handling ? Surrogate keys ? Alerting ? Automatic code generation


Peak ETA

http://www.peakindicators.com/index.php/peak-eta

http://www.peakindicators.com/index.php/peak-eta

Helping Your Business Intelligence Journey

Oracle Business Intelligence 11g Incrementally Loading Exalytics … · 2017-01-25 ·...

Documents

Transcript of Oracle Business Intelligence 11g Incrementally Loading Exalytics … · 2017-01-25 ·...