Perfect solution outstanding experience. ORACLE MATERIALZIED VIEWS A PERFORMANCE TUNING GEM.

perfect solution outstanding experience


ORACLE MATERIALZIED VIEWSA PERFORMANCE TUNING GEM


Agenda

• A Game Changer• What is a Materialized View• Purposes of Materialized Views• Enhancing SQL Performance• Query Rewrite• Real World Example• Advanced Query Rewrite Concepts• Q/A


A Game Changer

• Provide SQL Tuning Services for OLTP and DSS (Data Warehouse)

• Most Tuning Methods Consist of• SQL Redevelopment• Converting Correlated Sub Queries to Simply Sub Queries • Partitioning/Compression• Index Development• SQL Baselines

• Data warehouse tuning cases started to pop up where none of the conventional methods would provide the solution


A Game Changer

• Data warehouse performance goals

• The Report that Started it All• Local Fund Budget Report• Heavily Utilized Report Running almost 4 minutes• Conventional methods would not work

Type of Report General Performance Goal

Heavily Utilized 5 seconds or less

Moderately Utilized 30 seconds or less

Occasionally Utilized 2 Minutes or less

Maximum Time Allowed 3 Minutes


A Game Changer

• Local Fund Budget Report was tuned to return results in about 4 seconds, down from almost 4 minutes

• How did it happen?• The SQL in the report was not touched• New indexes were not added• The existing table structure behind the report remained the

same

SOLUTION:Query Rewrite using Materialized Views


What is a Materialized View

First, let’s define a View• A View stores a SELECT statement and executes the stored

SELECT statement when referenced via another SQL statement.

View: HR_DEPT_VW

SELECT DEPTID,MANAGER_IDFROM DEPT_TBLWHERE CATEGORY = ‘HR’

SELECT COUNT(1)FROM HR_DEPT_VW

Table: DEPT_TBL

DEPTIDMANAGER_IDCATEGORY



Now, let’s define a Materialized View• Like a View, a Materialized View stores a SELECT statement, in

addition the results of the SELECT are stored in a table.

MView: HR_DEPT_MV


Table: DEPT_TBL


MView Table: HR_DEPT_MV

DEPTIDMANAGER_ID



Now, let’s define a Materialized View• Like a View, a Materialized View stores a SELECT statement, in

addition the results of the SELECT are stored in a table.

SELECT COUNT(1)FROM HR_DEPT_MV

MView: HR_DEPT_MV


Table: DEPT_TBL


MView Table: HR_DEPT_MV

DEPTIDMANAGER_ID


Purposes of Materialized Views

• Replication• Easily replicate tables from one database to another

• Enhancing SQL Performance• Facilitate Incremental Warehouse Loading Operations (ETL)

• Use Materialized View Logs to track data changes in tables, which will be used to identify records needing to be updated, inserted, and deleted in the data warehouse.


Enhancing SQL Performance

• How do Materialized Views help SQL Perform Better?• Results of the associated query are stored in a table• Data can be pre-aggregated and pre-joined, cutting down the

resources and time needed to execute the query

TABLE: GL_ENTRIES

FISCAL_PERIOD_SIDENTRY_AMT

38,020,294 rows

MV: MV_GL_ENTRIES

SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID

95 rows


95 rows2:40.03 Min

SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM MV_GL_ENTRIESGROUP BY FISCAL_PERIOD_SID

95 rows0:0.03 Min


Enhancing SQL Performance

• There are two ways to utilize MVs to enhance SQL performance• Directly referencing the MV as in the example below• Using the Oracle Query Rewrite functionality

TABLE: GL_ENTRIES


38,020,294 rows

MV: MV_GL_ENTRIES


95 rows


SELECT FISCAL_PERIOD_SID,SUM(ENTRY_AMT)FROM MV_GL_ENTRIESGROUP BY FISCAL_PERIOD_SID

95 rows2:40.03 Min

95 rows0:0.03 Min


Query Rewrite

What is Query Rewrite?• Query rewrite transforms a SQL statement expressed in terms of

tables or views into a statement accessing one or more materialized views that are defined on the detail tables.

• The transformation is transparent to the end user or application, requiring no intervention and no reference to the materialized view in the SQL statement.

• Because query rewrite is transparent, materialized views can be added or dropped just like indexes without invalidating the SQL in the application code.


Query RewriteExample of Query Rewrite

TABLE: GL_ENTRIES


38,020,294 rows

MV: MV_GL_ENTRIES


95 rows


Generate Plan

Generate Plan

WINNER


Query RewriteExample of Query Rewrite



Query RewriteMaterialized View DDL

CREATE MATERIALIZED VIEW MV_GL_ENTRIES BUILD IMMEDIATEREFRESH COMPLETEENABLE QUERY REWRITE ASSELECT FISCAL_PERIOD_SID FISCAL_PERIOD_SID,SUM(ENTRY_AMT) ENTRY_AMTFROM GL_ENTRIESGROUP BY FISCAL_PERIOD_SID

REFRESH COMPLETETells Oracle that when this MV is refreshed all data will first be removed and then a complete reload will occur. Most MVs created for Query Rewrite will be a complete refresh.

ENABLE QUERY REWRITETells Oracle that this MV can be considered for Query Rewrite. If this Query Rewrite is Disabled then Oracle will not consider this MV as an option for Query Rewrite.

BUILD IMMEDIATETells Oracle to run the SELECT statement and load the MV table right away. The Build can also be deferred.


Query RewriteThe STALENESS Factor• A MV can be in one of several STALENESS states

• FRESH – The data in the MV accurately reflects the data in the base tables

• STALE – Data in one or more of the base tables has changed and the MV may no longer accurately reflect the data in the base tables

• UNUSABLE – The MV is currently being refreshed• NEEDS_COMPILE – The table structure of one or more of the base

tables has changed and the MV needs to be compiled• UNKNOWN – One of two situations cause an UNKNOWN status

• An “ALTER MATERIALIZED VIEW….CONSIDER FRESH” has been run• The MV is based on a PREBUILT TABLE (Advanced Topic)

A Materialized View will only be considered for

Query Rewrite if it’s in one of the two STALENESS

states

OR STALENESS = STALE and

the following database parameter is set

query_rewrite_integrity=stale_tolerated


Query RewriteThe STALENESS Factor

TABLE: GL_ENTRIES


38,020,294 rows

MV: MV_GL_ENTRIES


95 rows


Generate Plan

Generate Plan

WINNER

UPDATE GL_ENTRIESSET ENTRY_AMT = 3WHERE FISCAL_PERIOD_SID = 12

STALE

WINNER


Query RewriteThe STALENESS Factor

TABLE: GL_ENTRIES


38,020,294 rows

MV: MV_GL_ENTRIES


95 rows


Generate Plan

Generate Plan

WINNER

UPDATE GL_ENTRIESSET ENTRY_AMT = 3WHERE FISCAL_PERIOD_SID = 12

STALE

WINNER

BEGIN DBMS_MVIEW.REFRESH (‘MV_GL_ENTRIES’);END;

FRESH


Query RewriteThe STALENESS Factor• Query Rewrite works best for Materialized Views that are

based on tables that remain static throughout the day• Transactional or OLTP databases typically do not work well

due to the constant updating of data that will require constant refreshing of the MV to maintain a FRESH state.

• Query Rewrite works well with data warehouses


A Real World Example

• First, a little lingoSubject Area

Fact

Dimension

Dimension

Dimension

Dimension

Dimensions have a one-to-many

relationship with the Fact

Star Schema

Basically, just a bunch of tables

with data



Subject Area

Fact

Dimension

Dimension

Dimension

Dimension



Local Fund Budget Report



Generate PlanCost 207,0003:53 Minutes


Materialized View

A Real World ExampleLet’s Create an Materialized View

Create Materialized View

CREATE MATERIALZIED VIEW KUALI_ADMIN.KF_MVA_BUDGET_BALANCE…SELECT BUD.FISCAL_PERIOD_SID…, SUM(BUD.UA_BUD_CY_ANN_EXP)FROM SYSADM.PS_UA_EMPL_JOB_DTL EMP, SYSADM.PS_UA_BUD_ACNT_TYP ACNT, KUALI_ADMIN.KF_D_ORG ORG, KUALI_ADMIN.KF_D_FISCAL_PERIOD FP, KUALI_ADMIN.KF_D_PROJECT PROJ, KUALI_ADMIN.KF_D_OBJECT_CODE OC, KUALI_ADMIN.KF_D_ACCOUNT ACCT, KUALI_ADMIN.KF_F_BUDGET_BALANCE BUDWHERE BUD.ACCOUNT_SID = ACCT.ACCOUNT_SIDAND BUD.OBJECT_CODE_SID = OC.OBJECT_CODE_SIDAND BUD.PROJECT_SID = PROJ.PROJECT_SIDAND BUD.FISCAL_PERIOD_SID = FP.FISCAL_PERIOD_SIDAND BUD.UA_BUD_ACCT_DEPT_ORG_SID = ORG.ORG_SIDAND BUD.BUD_ACCT_TYP_SID = ACNT.BUD_ACCT_TYP_SID (+)AND BUD.EMPL_JOB_DETAIL_SID = EMP.EMPL_JOB_DTL_SID (+)GROUP BY BUD.FISCAL_PERIOD_SID…

Pre-Aggregate

Pre-Join


Materialized View



Generate PlanCost 483

4 Seconds

WINNER




Materialized View

Generate PlanCost 483

3 Seconds

WINNERExplain Plan without MV

Explain Plan with MV

Expensive Joins

NO Joins

MV used 98.5% Less Temp Space

for Sorting

CBO Cost207,000 vs 483



• Another alternative would have been to create a new, summarized Subject Area

• Benefits of Materialized Views over building additional summarized Subject Areas• Materialized Views are more agile and quicker to

implement• Existing reports need not be updated to point to a new

set of tables• Report writers will not have to be concerned with

choosing the best Subject Area from which to create reports


Advanced Concepts


In the Beginning…

• Due to differences in tables and joins in each report a different MV was created

• Every time a change was made to the report the MV had to be updated

• Became a maintenance nightmare

• Something had to change or MVs were not going to be a viable option

Budget MV1

Budget Report 1

Budget Report 2

Budget Report 3

Budget MV2

Budget MV3

Budget MV4


Budget Balances

A New Methodology…

Budget Balances

MV1

Budget Report 1

Budget Report 2

Budget Report 3

GL Entries

GL Entries MV1

GL Entries Report 1

GL Entries Report 2

GL Entries Report 3

Labor Entries

Labor Entries MV1

Labor Entries Report 1




How did we do it…

• Must Know Advanced Concepts• Query Delta Joins• Materialized View Delta Joins• Nested Materialized Views


Query Delta Joins

• A query delta join is a join that appears in the query but not in the materialized view.

• A query delta join will also occur if a field from a dimension is included in the query but is not included in the MV.

• In order for the join to work, the materialized view must contain the joining key.


Example Materialized View

Example Query Delta Join

MV Includes Budget Balance Fact joined to Account Dimension

MV Includes FK columns to the Object Code and Account Dimensions



Budget Balances- OBJECT_CODE_SID- ACCOUNT_SID- SUM(UA_BUD_CY_ANN)

Account- ACCOUNT_SID- ACCOUNT_BUDGET_SHELL_CD

MV ERD

Example Query #1

Consists of columns, tables, and joins that all exist in the MV



Query ERD





MV ERD

Example Query #2

This query adds the ACCOUNT_NBR field which does not exist in our test MV


Account- ACCOUNT_SID- ACCOUNT_BUDGET_SHELL_CD- ACCOUNT_NBR

Query ERD

The Delta Join is made possible by including the joining key from the FACT in the MV





MV ERD

The Delta Join is made possible by including the joining key from the FACT in the MV

Example Query #3

OBJECT_CD and the Object Code Dimension was added to the query



Query ERDObject Code- OBJECT_CODE_SID- OBJECT_CD


Materialized View Delta Joins

• A materialized view delta join is a join that appears in the materialized view but not the query.

• All delta joins in a materialized view are required to be lossless with respect to the result of common joins. (We’ll get to this concept a little later.)


Example Materialized View Delta Join



MV ERD

MV Delta Join is made possible by guaranteeing a “lossless” join between the FACT and DIMENSION

Budget Balances- SUM(UA_BUD_CY_ANN)

Query ERD

Query Example

This query removes any reference to the Account Dimension, which exists in the MV definition.


Combined MV and Query Delta Join



MV ERD- Removes the Account Dimension (MV Delta Join)- Adds the Object Code Dimension (Query Delta Join)

Budget Balances- OBJECT_CODE_SID- SUM(UA_BUD_CY_ANN)

Query ERDObject Code- OBJECT_CODE_SID- OBJECT_CD

Query Example


How are Delta Joins Possible?

• Table Relationships Must be Configured Properly• Lossless Joins• Non-Duplicating Joins

• Dimension Join Keys must be included in the SELECT of the MV definition


Lossless Joins

• A lossless join guarantees that the result of common joins is not restricted

Budget Balances- ACCOUNT_SID- COUNT(1)

Fact

Account- ACCOUNT_SID

Dimension

ACCOUNT_SID = ACCOUNT_SID

No rows were lost when Budget Balance was joined to Accounts

LosslessJoin


Lossless Joins



Fact

Employee Detail- EMPL_JOB_DETAIL_SID

Dimension

EMPL_JOB_DETAIL_SID = EMPL_JOB_DTL_SID

Not aLossless

Join

Rows were lost when Budget Balance was joined to Employee Detail

Not aLossless

Join


Lossless Joins



Fact

Employee Detail- EMPL_JOB_DETAIL_SID

Dimension

EMPL_JOB_DETAIL_SID = EMPL_JOB_DTL_SID

Not aLossless

Join

Outer Joins will guarantee a Lossless Join

EMPL_JOB_DETAIL_SID = EMPL_JOB_DTL_SID (+)

LosslessJoin


Non-Duplicating Join

• A non-duplicating join guarantees that the result of common joins will not cause duplicates

• There must be a many-to-one relationship between the joins from the Fact to the Dimensions

Many-to-OneMany-to-Many


How to let Oracle Know

• To let Oracle know of the Lossless and Non-Duplicating joins the following must be configured on the tables in the subject area:• Primary Key constraints must be created on all

Dimensions• Foreign Key constraints must be create on Fact join

columns• Join columns on the Fact must be set to NOT NULL

unless that column will be outer joined to the Dimension


Example Primary Key Constraint

• PK and FK constraints can cause havoc and slowness in a data warehouse

• Oracle has provided a way to create the constraints while avoiding the baggage

DISABLE – Does not enforce the constraint on rows created after the constraint is created. Good for ETL.

NOVALIDATE – Does not enforce the constraint on rows that exist in the table when the constraint is created.


Example Primary Key Constraint

• If the PK and FK constraints are in DISABLE and NOVALIDATE mode there is one more step that must be taken in order to let Oracle know we can RELY on these constraints

RELY – Tells Oracle that regardless of the state of the constraint, Oracle can assume the constraint is valid. Without this MV Rewrite functionality will not work properly if constraints are not enabled.


Include Join Columns in MV

Subject Area

Fact

Dimension

Dimension

Dimension

Dimension


Nested Materialized Views

Budget Balances

Budget Balances

MV1

Budget Report 3

Budget Report 4

Budget Report 5

Budget Balances

Nested MV1

Budget Balances

Nested MV2

Budget Report 1

Budget Report 2

Budget Report 6

Budget Report 7

60 Minute Refresh

2 Minute Refresh

4 Minute Refresh


Compression

• Compressing data improves I/O because less blocks of data are read from or written to disk and sent back and forth between disk and database servers

• I/O is the biggest bottleneck in most Databases so this improves the performance of SELECTs

• Compression also saves disk space• Basic compression is standard with Oracle

Enterprise Edition and works well with MVs• Compress your MVs


Compression

Data Uncompressed Compressed

Size in MB (76.5% Compression rate) 829.44 194.56

Seconds to Query Shell Cd = C2 5.20 3.80

Seconds to Query Shell Cd = A41 21.87 12.99

Seconds to Query all Shell Cd 30.37 22.64


Document your MVs

• We document our MVs in wiki and include:• Reports tuned by MV• Data Model of the MV• Change History• Constraints required along with DDLs to create them• FK Index Requirements• Required Database Grants• Materialized View DDL• Materialized View Index DDLs


To Sum it Up

• Materialized View Query Rewrite is a powerful tool that can dramatically increase query performance

• Tuning your Warehouse with MVs instead of summarized subject areas is far more efficient and flexible

• Make sure to understand a few advanced concepts (Delta Joins and Nested MVs) before starting your adventure

• Use Compression• Document your MVs


Q/A

Sean KelleherEmail: [email protected]: 480.444.7714Cell: 602.363.9563Web: www.tbginc.com

Please feel free to contact Sean with any questions

mailto:[email protected]

http://www.tbginc.com/

Perfect solution outstanding experience. ORACLE MATERIALZIED VIEWS A PERFORMANCE TUNING GEM.

Documents

Transcript of Perfect solution outstanding experience. ORACLE MATERIALZIED VIEWS A PERFORMANCE TUNING GEM.