In-Memory Oracle BI Applications (UKOUG Analytics Event, July 2013)

T : +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com

In-Memory Analytics with Oracle BI Apps and Oracle ExalyticsUKOUG Analytics Event, London, July 2013Mark Rittman, Technical Director, Rittman Mead

1Friday, 5 July 13

mailto:[email protected]


http://www.rittmanmead.com



About the Speaker

• Mark Rittman, Co-Founder of Rittman Mead• Oracle ACE Director, specialising in Oracle BI&DW• 14 Years Experience with Oracle Technology• Regular columnist for Oracle Magazine• Author of two Oracle Press Oracle BI books• Oracle Business Intelligence Developers Guide• Oracle Exalytics Revealed• Writer for Rittman Mead Blog :

http://www.rittmanmead.com/blog• Email : [email protected]• Twitter : @markrittman

2Friday, 5 July 13





http://www.rittmanmead.com/blog

http://www.rittmanmead.com/blog




About Rittman Mead

• Oracle BI and DW platinum partner• World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI• Approximately 50 consultants worldwide• All expert in Oracle BI and DW• Offices in US (Atlanta), Europe, Australia and India• Skills in broad range of supporting Oracle tools:‣OBIEE‣OBIA‣ODIEE‣Essbase, Oracle OLAP‣GoldenGate‣Exadata ‣Endeca

3Friday, 5 July 13






Oracle Business Intelligence 11g

• Oracle’s business intelligence platform, 11.1.1.7 release came out in April 2013 • Fourth major release of OBIEE 11g, with many new features + updated look and feel• Enterprise BI platform centered around the Common Enterprise Semantic Model (RPD)• Mobile BI apps, MS Office integration, ad-hoc,

dashboard and published reporting• Built around Oracle Fusion Middleware• Deployable on Windows, Unix, Linux• Accessing a range of enterprise data sources‣Oracle and other RDBMSs‣Essbase and other OLAP servers‣ Files, XML, web services‣ADF and SOA sources‣ TimesTen in-memory database

4Friday, 5 July 13






Oracle BI Applications

• Packaged version of OBIEE that includes a data warehouse, and ETL mappings,from E-Business Suite, Siebel, SAP and Peoplesoft

• Covers areas such as Financial Analytics, HR Analytics, Sales Analytics etc• Built on the same technology as OBIEE 11g, plus ETL and administration tools

5Friday, 5 July 13






Pre-Defined ETL Routines from Oracle EBS, Peoplesoft, Siebel, JDE, SAP

• Integrated, conformed dimensional data warehouse• Deployable on Oracle, MS SQL, IBM DB/2 and Teradata• Uses Informatica PowerCenter for ETL, or now ODI11g• Staging tables and presentation tables• Allows modular deployment• Lowest grain of information• Prebuilt aggregates • History tracking• Indexing

6Friday, 5 July 13






BI Apps Data Warehouse Limitations

• Designed for “lowest common denominator” DB features‣No “out of the box” partitioning, MVs, compression optimization for PQ

• Based on traditional disk-based RBDMS technology• Can often lead to slow reports, dashboards, limiting user acceptance• Common issue - what can we do about it?

7Friday, 5 July 13






Oracle Exalytics : First In-Memory Engineered System for Analytics

•Relational, Multi-Dimensional and Unstructured data analysis available as a single engineered system•Combination of in-memory hardware and optimized software versions•Supports the Exadata and Big Data Appliance data management systems

Exalytics In-Memory Machine

Spans Relational, Multi-Dimensional, and Unstructured analysis, combined with Financial & Operational Planning‣ In-Memory Optimized Hardware‣ In-Memory Oracle BI, TimesTen, Essbase, and Endeca‣ Many In-Memory Software Innovations

Tightly-Integrated with Exadata, and Big Data Appliance

8Friday, 5 July 13






Oracle Exalytics Benefits vs. Standard Hardware + Software

• Single supported stack of hardware + software : patching synchronized and tested across all components• OBIEE, Essbase, TimesTen etc optimizations that are only available when deployed on Exalytics hardware• Optimal selection of CPUs, RAM (DRAM), network connectors for a BI application tier• Automatic in-memory caching of commonly-used aggregates - no manual tuning and selection• Future platform for all Oracle BI products - EPM Suite, BI, Endeca, BI Apps

9Friday, 5 July 13






Exalytics as the Exa-Machine for OBIEE

• Runs the BI layer on a high-performance, multi-core, 1TB server• In-memory cache used to accelerate the BI part of the stack• If Exadata addresses 80% of the query performance, Exalytics addresses

the remaining 20%‣Consistent response times for queries‣ In-memory caching of aggregates‣ 40 cores for high concurrency‣Re-engineered BI and OLAP software

that assumes 40 cores and 1TB RAM

ERP/Apps DW

Oracle BI

In-Memory DB/Cache

10Friday, 5 July 13






Exalytics Under the Covers - How Does it Work?

• Exalytics brings together different technologies, which are still standalone products in their own right• To harmonise and optimise their use within Exalytics, it utilises the following techniques:‣ In-Memory Adaptive Data Mart - Using Oracle TimesTen for Exalytics, an in-memory RDBMS‣ In-Memory Intelligent Result Cache‣ In-Memory Cubes

• Some of these are genuine "secret sauce"• New functionality and algorithms• You can only get them through licensing Exalytics• Others are descriptions of DW/BI strategies, or existing product functionality, extended to take advantage of

the capacity for processing in memory that Exalytics has

11Friday, 5 July 13






In-Memory Adaptive Data Mart

• Commonly-used aggregates are copied into Oracle TimesTen for Exalytics• Past query patterns are analyzed and suitable aggregates recommended• Oracle BI Server then uses these aggregates to make queries run faster• Aggregates change over time in response to

changes in query patterns• Tools are provided for managing and populating these aggregates

TimesTen BI Server

Exal

ytic

s

Aggregates

Data WarehouseDetail-level

Data

12Friday, 5 July 13






Oracle TimesTen for Exalytics

• New version of TimesTen specifically for Exalytics (and only available with Exalytics)• Support for analytic functions‣Perform all the processing at source‣Combine with being in-memory = should be very fast

• Column compression‣Whitepaper cites 5x‣Given the hardware capacity, we could seriously contemplate loading the whole Data Warehouse into

memory‣Opens up lots of interesting design potential

• We can load aggregates into TimesTen, leave base data at source, and use OBIEE’s Vertical Federation capability to seamlessly report across both‣All hidden from the end-user, all they will know is that their reports run fast!

13Friday, 5 July 13






TimesTen and OBIEE Architecture

• Oracle BI Server communicates with TimesTen through TimesTen Client• Summary Advisor, and nqcmd use Oracle BI Server to access TimesTen• Typical single TimesTen database per Exalyics machine‣Max TimesTen database size around 300MB

- Due to need to set aside equalTemp size for the Perm size selected

• Clustered Exalytics boxes can be daisy-chainedtogether using InfinBand connections‣ For HA scenarios, does not increase

available RAM‣Summary advisor scripts write to both TimesTen

databases, replicating aggregates‣ TimesTen databases can be “wired together”

for failover/HA purposes

TimesTenMemory-ResidentDatabase

CheckpointFiles

LogFiles

ODBC

Oracle BIServer

nqcmdSummaryAdvisor

14Friday, 5 July 13






Summary Advisor for Aggregate Recommendation & Creation

• Utility within Oracle BI Administrator tool that recommends aggregates• Bases recommendations on usage tracking and summary statistics data• Captured based on past activity• Runs an iterative algorithm that searches,

each iteration, for the best aggregate• Could we use this to cache commonly-used BI Apps

aggregations in TimesTen, automatically based on usage patterns?

15Friday, 5 July 13






Alternative Approach - Copy “Hot Data” into TimesTen for Exalytics

• Standard approach is to store aggregates in the TimesTen datamart‣Aggregated by the source DB, aggregates then cached in TT database

• Other approaches could be used, however‣Store whole detail-level dataset in the TT database‣Store just recent detail-level data in TT, and use OBIEE’s fragmentation feature‣Store aggregate layer from BI Apps DW entirely in TimesTen

• Would this be an option that we could use with BI Apps datasets?

16Friday, 5 July 13






Another Option - Oracle In-Memory Database Cache (IMDB)

• Automatically replicate “hot” transactional data from Oracle BI Apps DW tables into TimesTen for Exalytics• Use OBIEE fragmentation to enable automatic navigation between sources• Aggregation performed by both TimesTen,

and by source DB (as appropriate)• However - fairly intrusive approach, Oracle-only,

probably not attractive to mostBI Apps customers and DBAs

17Friday, 5 July 13






Running BI Apps In-Memory - Is it Possible and Practical?

• Can we use the Summary Advisor to automatically cache commonly-used aggregates in-memory?‣Similar to regular OBIEE caching, relies on query repeatability + use of aggregation

• Could we copy all, or part, of the BI Apps data warehouse directly into TimesTen?• How would we update the RPD to point to the in-memory tables?• How fast would TimesTen be to load, and to query, vs. Oracle/SQL Server/DB2 etc?• Here’s our thoughts and R&D to date....

18Friday, 5 July 13






Using the Summary Advisor on BI Apps 7.9.6.4 Step 1 : Start Advisor

• Server has to be an Exalytics server, in this example is patched-up to 11.1.1.6.9• Workstation has the 11.1.1.6.9 BI Administration tool installed• Select Tools > Utilities, then Oracle BI Summary Advisor from utility list

12

19Friday, 5 July 13






Using the Summary Advisor on BI Apps 7.9.6.4 Step 2 : Scope Source Queries

• By default, all queries registered in the usage tracking and summary statistics tables are in-scope• Refine the recommendations by limiting timeframe, and setting minimum accumulated time threshold• Still an opportunity later on to pick and choose from recommended aggregates• Once selected, then select the TimesTen connection pool and database as the aggregate table target

34

20Friday, 5 July 13






Using the Summary Advisor on BI Apps 7.9.6.4 Step 3 : Create Aggregates

• Summary Advisor then recommends a set of “candidate” aggregates, which you can choose to implement• Select all, none or some of the recommended aggregates• Then run the resulting logical SQL script using the nqcmd utility• Note - may need to clean-up BI Apps DW data to remove duplicates etc before script completes OK

21Friday, 5 July 13






Using the Summary Advisor on BI Apps 7.9.6.4 Step 4 : Review RPD and Data

• Aggregate Persistence process called by the “create aggregates” process also maps tables in RPD• Physical layer contains entries for the TimesTen tables• Business Model and Mapping later contains vertically-federated LTSs for the new TT tables

22Friday, 5 July 13






Summary Advisor for BI Apps : Pros and Cons

• Pros‣Automatically analyzes query patterns and recommends aggregates to accelerate them‣Handles the registering of the TimesTen tables in the RPD, including mapping back into business model‣Supports any data source that the BI Server supports

• Cons‣Queries have to have run before they’ll be considered for loading

into TimesTen for Exalytics‣Relies on subsequent queries being able to use those aggregates‣Could get unwieldy if many aggregates are registered in the RPD‣Summary Advisor process does not automatically clear down

tables that don’t feature in future recommendations‣ Inefficient refresh process, unless you use a process such ashttp://www.rittmanmead.com/2013/04/incremental-refresh-of-exalytics-aggregates-using-native-bi-server-capabilities/

23Friday, 5 July 13





http://www.rittmanmead.com/2013/04/incremental-refresh-of-exalytics-aggregates-using-native-bi-server-capabilities/







Alternative Approach - Copy “Hot Data” into TimesTen for Exalytics

• Standard approach is to store aggregates in the TimesTen datamart‣Aggregated by the source DB, aggregates then cached in TT database

• Other approaches could be used, however‣Store whole detail-level dataset in the TT database‣Store just recent detail-level data in TT, and use OBIEE’s fragmentation feature‣Store aggregate layer from BI Apps DW entirely in TimesTen

• Would this be an option that we could use with BI Apps datasets?

24Friday, 5 July 13






Loading data directly from Oracle into TimesTen with ttLoadFromOracle

• The procedure ttLoadFromOracle uses OCI to load data directly from Oracle into an existing TimesTen table

• Specify a whole table (SELECT *), or part (SELECT ... WHERE)• Target table must existing on TimesTen already‣Create it automagically using ttTableSchemaFromOraQueryGet

or createandloadfromoraquery‣However both these use Oracle data types and no compression,

so size in memory is going to be greater

REVENUE_F_TS

REVENUE_F_LARGE

ttIsql --ConnStr "DSN=BISAMPLE_TT;UID=SH;PWD=SH;OracleNetServiceName=orcl;OraclePWD=SH"

Command> call ttLoadFromOracle('A_TEST', 'REVENUE_F_TS', 'SELECT SHIPTO_ADDR_KEY, OFFICE_KEY, EMPL_KEY, PROD_KEY, ORDER_KEY, REVENUE, UNITS, DISCNT_VALUE, BILL_MTH_KEY, BILL_QTR_KEY, BILL_DAY_DT, ORDER_DAY_DT, PAID_DAY_DT, DISCNT_RATE, ORDER_STATUS, CURRENCY, ORDER_TYPE, CUST_KEY, SHIP_DAY_DT, COST_FIXED, COST_VARIABLE, SRC_ORDER_NUMBER, ORDER_NUMBER FROM BISAMPLE.SAMP_REVENUE_F_LARGE WHERE BILL_MTH_KEY=201012');< 7750 >1 row found.

-- this has loaded 7750 rows for a given month

TimesTen is loaded from the results of a query on Oracle ttLoadFromOracle

25Friday, 5 July 13






Generating TimesTen load DDL and scripts with ttImportFromOracle

• For ttLoadFromOracle to work, the target table must exist• The utility ttImportFromOracle is useful here. ‣ It can map Oracle data types to optimal TimesTen ones‣Optionally, it can aggressively limit column sizes based on data to reduce TimesTen footprint‣ It can evaluate compression effectiveness and apply it only where most useful‣Given a set of tables, it will generate:

- TimesTen DDL for requires schemas/tables/indexes- A script to load all the tables into TimesTen in parallel (ttPDL.sh)

CreateIndexes.sql

ttImportFromOracle

REVENUE_F

CreateTables.sqlCreateUsers.sql

DropIndexes.sql

DropTables.sql

LoadData.sql TableList.txt

ttPDL.sh ttSizing.sh

UpdateStats.sql

26Friday, 5 July 13







• ttImportFromOracle is new in TimesTen 11.2.2.5‣Not an official production utility - best efforts support only‣But it’s only generating scripts, which contain standard (supported) TimesTen functionality

• The executable is located in $TT_HOME/support• It uses OCI, so make sure LD_LIBRARY_PATH is set to include Oracle DB lib‣ export LD_LIBRARY_PATH=$ORACLE_HOME/lib

• Feature-rich syntax, but at its simplest can just be invoked for a single table, with compression: $ ttImportFromOracle -oraconn SH/SH@orcl -tables REVENUE_F_TS -compression 1 Beginning processing Resolving any tablename wildcards Eliminating any duplicate tables Getting metadata from source Generating database user list Assigning TimesTen datatypes Analyzing source tables Analyzing table 'SH.REVENUE_F_TS' ... Estimating table sizes Evaluating parallel data load Generating output files Finished processing

27Friday, 5 July 13







• ttImportFromOracle writes a set of scripts that are subsequently executed to :

‣Create the target tables and indexes on TimesTen, using optimised data types and compression

‣ Load the target tables on TimesTen, still via ttLoadFromOracle

REVENUE_F

ttPDL.sh

REVENUE_F_TS

ttLoadFromOracle

REVENUE_F_TS

CREATE TABLE ...CreateTables.sql

28Friday, 5 July 13






Comparing Oracle and TimesTen as data sources

• Our testing has shown that in general, ‣ For base level data, Oracle outperforms TimesTen‣ For aggregated data, TimesTen outperforms Oracle

• Therefore entire lift + shift of OBIA Data Warehouse into TimesTen is possibly not going to give optimal response times

-- same query over OBIA-- 0.68 secondsSelect * from samp_revenue_f_large f, samp_customers_d cd, samp_addresses_d ad, samp_products_d pdwhere cd.cust_key = f.Cust_key and cd.address_key = ad.address_keyand pd.prod_key = f.prod_keyand ad.city = 'San Francisco'and pd.type = 'Cell Phones'and f.bill_mth_key = 201012;

-- using full sized fact table (native data types)-- 0.78 seconds query timeSelect * from revenue_f_native f, customers_d cd, addresses_cd ad, products_d pdwhere cd.cust_key = f.Cust_key and cd.address_key = ad.address_keyand pd.prod_key = f.prod_keyand ad.city = 'San Francisco'and pd.type = 'Cell Phones'and f.bill_mth_key=201012;0.68 seconds 0.78 seconds

29Friday, 5 July 13






Comparing Oracle and TimesTen as data sources

• One option would be to put just “Hot Data” (eg current month) into TimesTen, and then update the RPD to use fragmentation‣ This has an overhead in terms of RPD updates (and support - added

complexity), as well as an additional “ETL” process to manage

-- using full sized fact table (native data types)-- 0.78 seconds query timeSelect * from revenue_f_native f, customers_d cd, addresses_cd ad, products_d pdwhere cd.cust_key = f.Cust_key and cd.address_key = ad.address_keyand pd.prod_key = f.prod_keyand ad.city = 'San Francisco'and pd.type = 'Cell Phones'and f.bill_mth_key=201012; 0.78 seconds

-- using Time slice table -- 0.25 secondsSelect * from revenue_f_ts f, customers_d cd, addresses_cd ad, products_d pdwhere cd.cust_key = f.Cust_key and cd.address_key = ad.address_keyand pd.prod_key = f.prod_keyand ad.city = 'San Francisco'and pd.type = 'Cell Phones'and f.bill_mth_key=201012; 0.25 seconds

30Friday, 5 July 13






Conclusions

• The scale of most BI Apps implementations means that query performance can be an issue• Exalytics’ TimesTen In-Memory Database could be a potential solution to this issue• Several approaches to putting all or part of the BI Apps DW into TimesTen / In-Memory• Copying all or sections of the BI Apps DW into TimesTen is a potential approach‣But current version of TimesTen best suited to smaller tables and datasets

• Exalytics’ Summary Advisor now works with Oracle BI Apps 7.9.6.4• Automatically detects and recommends suitable aggregates, builds and maps into RPD‣ Though custom solutions are probably more efficient for their later incremental refresh

• A work in progress - speak to Rittman Mead for more details on how this can work• Offers the potential of “speed-of-thought” business analytics dashboards, with minimum additional work

31Friday, 5 July 13






Thank You for Attending!

•Thank you for attending this presentation, and more information can be found at http://www.rittmanmead.com•Contact us at [email protected] or [email protected]•Look out for our book, “Oracle Business Intelligence Developers Guide” out now!•Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)

32Friday, 5 July 13












In-Memory Analytics with Oracle BI Apps and Oracle ExalyticsUKOUG Analytics Event, London, July 2013Mark Rittman, Technical Director, Rittman Mead

33Friday, 5 July 13





In-Memory Oracle BI Applications (UKOUG Analytics Event, July 2013)

Technology

Transcript of In-Memory Oracle BI Applications (UKOUG Analytics Event, July 2013)