Advanced Development Using OBIEE Plus

T : +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com

Advanced Development Using OBIEE PlusMark Rittman, Director, Rittman Mead ConsultingODTUG Kaleidoscope 2008, New Orleans

mailto:[email protected]


http://www.rittmanmead.com



Who Am I?

• Oracle BI&W Architecture and Development Specialist• Co-Founder of Rittman Mead Consulting‣ Oracle BI&W Project Delivery Specialists

• 10+ years with Discoverer, OWB etc• Oracle ACE Director, ACE of the Year 2005• Writer for OTN and Oracle Magazine• Longest-running Oracle blog‣ http://www.rittmanmead.com/blog

• Chair of UKOUG BIRT SIG• Co-Chair of ODTUG BI&DW SIG• Speaker at IOUG and BIWA events





http://www.rittmanmead.com/blog

http://www.rittmanmead.com/blog


Rittman Mead Consulting

• Oracle BI&DW Project Specialists• Consulting, Training, Support• Works with you to ensure OBIEE project success• Small, focused team• OWB, Oracle BI, DW technical

specialists• Clients in the UK, Europe, USA




Oracle Business Intelligence Enterprise Edition






Hot-Pluggable and Standards-Based






Connectivity to Most Applications and Data Sources

• Oracle Business Intelligence Enterprise Edition is a “federated query server”• Connectivity to Oracle, MS SQL Server, Sybase, IBM DB2, Teradata etc• MS AS, Essbase, Oracle OLAP• Files, XML• Microsoft Excel• etc






Common Enterprise Information Model

• Enables consistency, security, reuse, flexibility• Role-based views of the organization• Single metadata model of the business• Consistent definition of business measures,

metrics, calculations• Drill paths, summaries, security• Model once, deploy anywhere• Across any data source






Pervasive Information Access through a Unified BI Foundation






This Morning’s Agenda

1. Techniques to develop the Common Enterprise Information Model2. Creating a Logical Star Schema from a Normalized Model3. Adding Additional Data Sources (Federated Queries)4. Fragmentation (Partitioned Measures)5. Time-Series Analysis6. Adding Row-Level and Subject-Area Security7. Aggregate Persistence (Level-Based Measures)8. Cache Management






Web Server (IIS, Tomcat, Websphere, iPlanet)

Oracle BI ServerOracle BI Server

Intelligent Request GenerationIntelligent Request Generation

Data Source AdaptersData Source Adapters

NavigatorNavigator

Logical Request GenerationLogical Request Generation

AggregateNavigator

FragmentationOptimization

Multi-Pass /Sub-Request Logic

Optimized Query RewritesOptimized Query Rewrites

Session ManagementSession Management

Met

adat

a In

ter c

hang

eM

etad

a ta

Inte

r cha

nge

Logical SQL ODBC/JDBC (Logical Business Model)Logical SQL ODBC/JDBC (Logical Business Model)

CacheCacheServicesServices

Oracle BI AdministrationOracle BI Administration

Analytical and Operational Data Sources

Metadata Documentation ServicesMetadata Documentation Services

Metadata Management ServicesMetadata Management Services

Multi-User Development ServicesMulti-User Development Services

Server Management ServicesServer Management Services

ODBC over TCP/IP (SSL)

Sys

tem

/ S

yste

m /

Pe r

fP

e rf

Mon

it ori n

g M

onit o

ri ng

ODBC, CLI, OCI, XML, MDX

Oracle BI Presentation ServicesOracle BI Presentation ServicesSOAP Web Services, XML and URL InterfaceSOAP Web Services, XML and URL Interface

Cache Services (Web) & Connection Cache Services (Web) & Connection MngmtMngmt

XMLXMLFrameworkFramework

WebWebCatalogCatalogServiceService

User Profiling, Security and Session User Profiling, Security and Session MngmtMngmt

Oracle InteractiveOracle InteractiveDashboardsDashboards

Oracle AnswersOracle Answers

Load BalancerLoad Balancer

Security S

ervicesS

e cur ity Se rvi ce s

Quer y G

o ver n.Q

uer y Gov er n.

TCP/IP(SSL)

Javascriptfor Usability & Interactivity

Web BrowserXML, HTML,

XLS, PDF, TXTover

HTTP/HTTPS

External Applications and Portals

HTML, SOAPover

HTTP/HTTPS

Execution EngineExecution Engine

SAW Bridge (J2EE/ISAPI)SAW Bridge (J2EE/ISAPI)

TCP/IP (SSL)Oracle BI PublisherOracle BI PublisherDelivery ServerDelivery Server

Layout InterfacesLayout Interfaces

Data LogicData LogicXSL

Externalized Authentication

LDAP

DB Authentication

Custom Authenticator

Oracle BIOracle BICluster ControllerCluster Controller

Oracle Delivers ServerOracle Delivers ServerScheduling/Event ServicesScheduling/Event Services

Agent Execution LogicAgent Execution Logic

Device Adaptive ContentDevice Adaptive Content

OBIEE 10.1.3.2 Architecture






Oracle BI Server

• Calculation and Data Integration Engine‣ “Virtual Data Warehouse”‣ Allows reporting across multiple sources‣ Data accessed though metadata layer

• Connects to RDBMS and OLAP data‣ Oracle Database (including Oracle OLAP)‣ IBM DB2‣ Microsoft SQL Server‣ Teradata‣ Microsoft Analysis Services (OLAP)‣ SAP BW 'info cubes'

• Security, Summary Management• Translates incoming “logical SQL”






Oracle BI Administration

• MS Windows application for managing BI Server• Creates and manages Common Enterprise Information Model‣ Physical, logical and presentation layers

• Manage other aspects of BI Server‣ Security‣ Scheduler (Delivers server)‣ Jobs‣ Joins‣ Marketing‣ Sessions‣ Cache

• Utilities and Tools• Connect to BI Server via ODBC DSN

Oracle BI AdministrationOracle BI Administration

Metadata Documentation ServicesMetadata Documentation Services

Metadata Management ServicesMetadata Management Services

Multi-User Development ServicesMulti-User Development Services

Server Management ServicesServer Management Services






Core BI Server Functionality

• Data Integration through Business Model View• Calculation and analytical engine• Support for RDBMS and OLAP data sources• Intelligent Request Generation and

Optimized Data Access• Security and Data Scoping• ODBC Interface out, native RDBMS and

OLAP interfaces in• Clustering, replication, support

for Windows and Unix/Linux hosts,32-bit and 64-bit






Data Access

• Reporting data may be sourced frommultiple databases, applications‣ Oracle Database (including Oracle OLAP)‣ IBM DB2‣ Microsoft SQL Server‣ Teradata‣ Microsoft Analysis Services (OLAP)‣ SAP BW 'info cubes'‣ SAP, Peoplesoft, Siebel, E-Business Suite

• Oracle BI Server joins this sourcedata together and presents alogical business model‣ Facts‣ Dimensions

Customer






Common Enterprise Information Model

• Metadata layer for OBIEE tools• Three layers of metadata‣ Physical layer, connections to sources‣ Logical layer, denormalized facts

and dimensions‣ Presentation layer, “report-centric”

folders• Maintained using BI Administrator• Model is the “center” of the OBIEE

universe






Data Modeling using Oracle BI Server

• Design goal for the BI Server metadata layer is to create a Star Schema‣ Fact tables‣ Dimension tables‣ Drill paths

• Source data can either be pre-integrated (data mart,data warehouse) or integrated using the BI Server (“federated”)‣ Federated can be faster to deliver (report in place)

- Assumes data sources share common key values‣ Data marts are simpler and faster to query

• MDX (OLAP, Multi-dimensional) data is converted into RDBMS row-sets

• Logical layer is main integration point• Presentation layer allows “report centric” folders






Information Model Layers

• Physical Layer contains connections, physical tables‣ Connections start as ODBC, switch to native (OCI etc)‣ Primary keys and foreign keys

• Logical layer is where the data is integrated‣ Logical fact tables, logical dimension tables‣ Dimensions‣ Calculations‣ Complex joins to define relationships

• Presentation layer is where data is presented‣ Can be a simple copy of the logical layer‣ Or can be made more personalized‣ Report centric






Star Schema Design Approach

• Classic “Kimball” approach• Fact tables contain measures, and joins to dimensions‣ No attributes except for degenerate dimensions‣ Leave IDs in, remove in Presentation layer

• Dimension tables contain lookup data‣ Customer, Product, Time etc‣ Dimensions can however be facts

- Use alias tables to disambiguate• Dimension objects define drill path‣ Multiple hierarchies supported‣ Ragged hierarchies possible (see later)‣ Time dimensions enable time-series queries






“Model First” Development

• Business Model is centre of the OBIEE “universe”• First element of design that is produced• Integrated view of the organization’s data• Option 1: Star with model first, then map on physical sources‣ “purest” approach, uses theoretical model as basis‣ Tricky to get joins and integration working‣ Lots of manual logical table and column definition steps

• Option 2: Produce model on paper, then construct it using elements of physical model‣ Quicker and faster, just drag and drop‣ May end up with redundant columns‣ Still needs a design phase at the start






Sample Business Model






Data Modeling Workflow Steps

1. Define logical model (on paper or using BI Administrator)2. Import physical data sources & define primary and foreign keys3. Map logical model to physical model4. Create initial calculations5. Create dimensions6. Rename and tidy-up logical model7. Create initial presentation model8. Create complex calculations, time dimensions etc9. Add additional columns to presentation model






Seminar Reporting Scenario

• Global Company has a data warehouse, containing order and customer data• Requirement is to take the existing warehouse (normalized)

and turn it into a star schema (denormalized)• For now, OBIEE will be used to map the data






Step 1: Create Logical Layer

• Firstly, create new logical model• Then create logical tables and columns‣ Either create from scratch, then connect to physical

(best practice)‣ Or define by dragging and dropping physical tables

• Start from fact table and work outward to the dimensions• Do one fact/dimension combination first• Then add more facts• ... and more dimensions• Add details on datatypes etc later on






DemonstrationCreating the Logical Model






Step 2: Create Physical Layer

• Define ODBC connection from client or server to database‣ ODBC only needed to define initial connection, import metadata‣ Native connection (OCI etc) takes over from then

• Import tables, views, keys etc from data source

• Define keys, foreign keys if needed• Update row counts• Define table aliases






Physical Join Types

• If source data has foreign key joins, they will be imported• If not, they can be defined in the physical model• Two types of join‣ Foreign key joins (e.g. CUSTID = CUSTID)

- 1:N joins only‣ Complex joins (TRAN_DATE between

STARTDATEID and ENDDATEID)- also supports 1:1, O:1, 1:O etc

• Important to create physical joins before importing in to logical layer‣ Logical joins rely on physical joins

to define paths‣ Makes life a lot simpler

• Create joins using Join Manager,






DemonstrationCreating the Physical Model






Step 3: Mapping the Logical Model to the Physical Model(s)

• Once you have your logical model and one or more physical models, you can map them together‣ The simpler the mapping, the better

• Drag and drop physical columns on to the logical model

• Logical model will inherit physical data types






Logical Table Sources

• Logical tables can have more than one physical source• Can create >1 Logical Table Source‣ BI Server creates two physical queries‣ Useful when mapping data from

different levels‣ and when using “fragmentation”

• Each Logical table source can have >1 table mapped‣ BI server creates one query per LTS, joining

as per physical foreign key joins‣ Used when you need columns from other

joined tables in your logical table (join only happens if additional columns requested)






Consider Where to Join Data

• Again, consider where to integrate and join data• Whilst the BI Server can join data, it may be faster to load into a data mart‣ ETL tools can integrate data at time of load‣ Denormalized data structures can be faster than joining at report time‣ Indexing, Summary Management, Query Optimization

• Creating a data warehouse still is the optimal solution‣ BIEE data source joining features are most appropriate when bringing

new data in that’s not yet in the data warehouse






Logical Join Types

• Logical tables are by default not joined• Logical layer supports both foreign key

and complex joins• Foreign key joins not recommended‣ Restrict choices that BI server

can make over join paths‣ Only there for backward compatibility

• Complex joins are preferred‣ Indicates relationship between

tables‣ Allows BI Server to determine

best join path- Useful when more than one

Logical Table Source






DemonstrationMapping the Logical Model






Step 4: Create Initial Calculations• Calculations are created as additional logical columns• Sit alongside existing logical columns derived from physical layer• Calculations can also

be created within report






OBIEE SQL and the Expression Editor

• Calculations within the repository are defined using the Expression Editor• Uses OBIEE SQL Syntax• Reference functions,

expressions, logicalcolumns, constants,variables

• Same SQL syntax acrossall supported databases

• Ability to pass-throughfunction calls usingEVALUATE function






Create Calculations in the Optimal Location

• Whilst the BI Server can create calculations, it may be more appropriateto do this in the underlying database‣ Oracle Database analytic functions (LAG, LEAD, RANK etc)‣ Oracle Database OLAP functions (time-series, forecast, allocation)

• Use the available architecture efficiently‣ Perform calculations as close to the source data as possible‣ Use dedicated OLAP technology if available‣ Use BI Server to provide features not found in the sources

• Just because the BI Server can do something, doesn’t mean it should do something






Step 5: Create Dimensions and Hierarchies

• Dimensions define hierarchical relationships between columns‣ Also used when defining level-based measures

and summaries• Level-based, starts with Grand Total level‣ One-way export out to Oracle dimensions (DDL)

also possible• Enables hyperlinks on columns within

dimension levels‣ Drill-down through the hierarchy‣ No drill-up or drill to related (yet)






Multiple Hierarchies and Ragged / Unbalanced Hierarchies

• Dimensions with multiple hierarchies are possible‣ All hierarchies must share common top and bottom levels

• Ragged hierarchies, unbalanced hierarchies only supported formulti-dimensional data sources‣ MS Analysis Services, SAP BW,

Oracle Hyperion Essbase‣ For relational, need to balance

hierarchy beforehand






DemonstrationCreating Dimensions






Step 6: Renaming and Tidying Logical Model

• If logical model was created through dragging-and-dropping physical objects, logical table and column names may be CAPITALIZED or HAVE_UNDERSCORES_IN_NAME

• Utilities accessing from Tools menu can be used to tidy up names‣ Many other useful utilities as well






Step 7: Create Initial Presentation Models

• Business models themselves are not visible to users‣ Users see presentation models whose columns are

defined in business models.‣ Appear as “subject areas”

in the Oracle BI Answers‣ Appear as “databases” to

client tools using ODBC• Can create multiple

Presentation Catalogs from single Logical Model






Step 8: Creating Time Dimension and Time-Series Calculations

• OBIEE 10.1.3.2 supports limited time-series functions• AGO – value of a measure x months, quarters, years ago

AGO(Videostore."Sales fact view".Sales, Videostore.TimesDim."Month", 1)

• TODATE – value of a measure to the end of a month, quarter, yearAGO(Videostore."Sales fact view".Sales, Videostore.TimesDim."Quarter", 1)

• Requires at least one “time dimension” in the metadata layer‣ Requires at least one or more Chronological keys‣ Monotonically increasing value (DAY etc)






The Calculation Wizard

• Wizard that generates share, percentage change etc calculations• Useful in conjunction with

Time Dimension‣ % Change since last month‣ Change since last month etc‣ Index‣ Percent

• Wraps ISNULL around calcs






DemonstrationCreating Time-Series Calculations






Reporting Across Multiple Data Sources

• OBIEE can combine and join data from multiple data sources• Data is retrieved in separate SQL, MDX calls and joined using the BI Server• Allows you to perform federated queries• Watch out for performance though‣ End goal should be to eventually move reporting data to a data warehouse






Federated Query Scenario

• Additional data exists in an Excel spreadsheet• Contains quotas, plus information on products and dates• Quotas are at the product category and month level‣ Existing data is at the product and day level

• Requirement is to join data, but only show quotas the relevant level






Joining A Second Data Source : Step 1

• Import the new schema into the physical model, ensure FK and PKconstraints are set







• Use the Join Manager, or Physical Model Diagrammer, to create joins between tables in the two schemas







• Add the new dimension tables as additional logical table sourcesto the exisiting logical dimension tables

• Set the dimension hierarchy level at which they are applicable







• Add the additional measures to the fact table• Set the dimension levels at which the measures become relevant






DemonstrationAdding an Additional Data Source






The Oracle BI Server Cache

• Stores local copy of data that is requested• Can be pre-seeded by iBots (agents)• Off by default, enabled using NQSConfig.ini file• Requires that you put a cache management strategy in place‣ Particularly suited to DW

environments‣ Cache is not aware of when

data becomes “stale”






Turning on Caching

• Caching is not enabled by default• Enabled by editing NQSConfig.INI• Can set additional cache options‣ Where cache data is stored‣ Max size of cache (>2GB)‣ Max entries‣ Whether it holds aggregations‣ etc






Cache Administration

• Cache usage can be monitored from the Cache Manager• Caching can be enabled at a physical table level‣ Whether cacheable or not‣ Cache persistence time






Aggregation and Level-Based Measures

• By default, logical columns (measures) are mapped to lowest-level, detail-level source data‣ Oracle BI Server provides aggregation as needed, shipped back to RDBMS

• Additional, summary data sources can however be introduced to logical model‣ Plug in summary tables, data from OLAP servers

• Depends on dimensions having been defined‣ Oracle BI Server knows that summary table can return data faster

• An alternative to query rewrite and MVs‣ Useful when aggregating federated data






Adding Summary Data to the Logical Model

• Summary data is added as an additional Logical Table Source• Then set level at which additional Logical Table Source is used‣ Oracle BI Server will use original LTS before this level‣ Oracle BI Server will use new LTS at this level and higher






The Aggregate Persistence Wizard

• Automates the process of creating and mapping summary tables• Pick one or more measures and dimension levels to aggegate on• Generates internal scripting code, Execute script using nqCMD.exe• Generates RDBMS tables, inserts summary data, maps to logical model• Aggregates can be stored in different RDBMS than source data






Using the Aggregate Persistence Wizard Step 1

• Select measures to aggregate• Can be from different physical databases• Must be from the same logical table







• Select the dimension levels to aggregate on• Must have been already defined as

dimensions in the logical model







• Select RDBMS to store aggregates in• Must be one of‣ Oracle RDBMS‣ MS SQL Server‣ IBM DB/2‣ Teradata

• Can be a different database than thesource physical database







• Execute the script that the wizard generates• Either through nqCmd.exe or through Scheduler• Generates RBDMS tables• Populates with summary data• Maps new physical tables to logical model• Sets aggregation levels

C:\%ORACLEBIHOME%\server\bin\nqCmd.exe –d AnalyticsWeb –u Administrator –p Administrator –s c:\agg_wiz_script.sql






DemonstrationThe Aggregate Persistence Wizard






Oracle OLAP within Oracle BI EE

• Oracle OLAP 11g can be used along with Oracle BI EE• Either through Cube Organized Materialized View, or through SQL Views• Cube Organized MVs

are simple to set up, but donot expose calculations‣ No relational column to

rewrite from• SQL Views provide full

capabilities‣ Tricky to manipulate though

• At present, no metadata import‣ Coming in OBIEE 11g+






Oracle Essbase and Oracle BI EE

• Essbase is a native data source for Oracle BI Enterprise Edition• Uses XMLA compatibility feature of OBIEE• Requires OBIEE 10.1.3.3.2, or 10.1.3.3.1 with registry edit‣ HKEY_CURRENT_USER\Software\Oracle Corporation\Oracle BI\Oracle BI

Administration Tool\Hidden Features ‣ Create new DWORD value called “Essbase”, set to “1”

• Some restrictions in 10.1.3.3.1 release, treat as “for evaluation only”







OBIEE 10.1.3.3.x Guidelines & Limitations

• Measure hierarchies not supported, all measures shown on same level• Import process does not know that dimension member names may be unique,

you will need to inform OBIEE of this in the logical dimension member properties• EVALUATE function can now work against MDX data sources• EVALUATE_AGGREGATE function will need to be used for Essbase to natively

handle time-series functions (not function shipped by default)• Essbase UDAs suppressed by default, can be enabled using EVALUATE• Requires Essbase 9.3.1 stack‣ Essbase Server 9.3.1‣ Essbase Provider Services 9.3.1






Registering an Essbase Cube in OBIEE Step 1

• Use Oracle BI Administrator to Import from XMLA• Requires Essbase Provider Services to be installed and running







• Enter the connection details for the Essbase server• URL = <server_name>:13080/aps/XMLA• Press “Update” to pre-populate the Data Source field• Username = Essbase login ID• Password = Essbase password‣ Not Shared Services login







• Select the cube that you want to import• Essbase cubes are hypercubes, contain measures of the same

dimensionality• Cube brings across measures

and dimensions







• Cube appears as new databasedata source in OBIEE Physical layer

• Note different icons for dimensionsand measures

• All “joins” are created automatically







• Drag physical cube into logical layer• All dimensions, joins, “tables” are automatically

created• Note that level names do not get imported







• Rename levels to correct names







• Drag logical model into Presentation Layer• Delete all the Level 0 levels‣ Not required, if selected will

cause request to error







• Presentation layer should now only contain thetidied-up column names, as if it was a relationalschema

• Now ready for reporting against







• Create request as normal, results are returned super-fast






DemonstrationOracle OLAP and Oracle Essbase in OBIEE






Advanced Development Using OBIEE PlusMark Rittman, Director, Rittman Mead ConsultingODTUG Kaleidoscope 2008, New Orleans





Advanced Development Using OBIEE Plus

Documents

Transcript of Advanced Development Using OBIEE Plus