Carl Dudley – University of Wolverhampton, UK Materialized Views Carl Dudley University of...

Carl Dudley – University of Wolverhampton, UK

Materialized Views

Carl DudleyUniversity of Wolverhampton, UK

Oracle ACE DirectorUKOUG Committee

[email protected]

2Carl Dudley – University of Wolverhampton, UK

IntroductionIntroduction

Working with Oracle since 1986

Oracle DBA - OCP Oracle7, 8, 9, 10

Oracle DBA of the Year – 2002

Oracle ACE Director

Regular Presenter at Oracle Conferences

Consultant and Trainer

Technical Editor for a number of Oracle texts

UK Oracle User Group Director

Member of IOUC

Day job – University of Wolverhampton, UK


TopicsTopics

Materialized View Features and Syntax

Query Rewrite

Refreshing Materialized Views

Nested Materialized Views

Implementing Constraints via Materialized Views

Fast and Complete Refresh Performance

Multiple Materialized Views

Estimating the Size of Materialized Views


Materialized View FeaturesMaterialized View Features

Data Warehouses typically contain a few very large tables— Joins and aggregate queries can be very expensive

MVs often contain summary tables of aggregations of a table's data — Pre-calculated results enhance query performance— Usually defined as aggregate views and/or join views

• For example, a view containing sums of salaries by job within department

Sometimes act as replica snapshots of tables on remote databases — Not covered in this presentation


Materialized Views Features (continued)Materialized Views Features (continued)

Not normally accessed directly— Queries on base tables are automatically re-written to use MVs

MVs can be nested— MVs built on other materialized views

Transparent to SQL applications

— Can be created or dropped without affecting the validity of SQL statements

Consume storage space

Contents must be updated whenever underlying detail tables are modified


Creating Materialized ViewsCreating Materialized Views

CREATE MATERIALIZED VIEW emvBUILD IMMEDIATEREFRESH COMPLETE ON DEMAND ASSELECT deptno ,SUM(sal) FROM emp GROUP BY deptno;

This view is populated with data upon creation

To keep the view synchronised with emp, it must be completely refreshed— This will be done manually on demand (the default)

Queries against emp will not be re-written to use the view (default)


Creating Materialized Views (continued)Creating Materialized Views (continued)

CREATE MATERIALIZED VIEW emvBUILD DEFERREDREFRESH FAST ON COMMITENABLE QUERY REWRITE ASSELECT deptno ,SUM(sal) ,COUNT(*) ,COUNT(sal) FROM emp GROUP BY deptno;

Not populated with data until first refresh action takes place— The first refresh must be a complete refresh

To keep the view synchronised, it will undergo fast refresh operations— Refresh happens every time a transaction on emp commits

Queries against emp will be re-written to use the view— Unless the parameter QUERY_REWRITE_ENABLED is set to false


TopicsTopics


Query Rewrite








Query RewriteQuery Rewrite

The optimizer compares costs when deciding to use a materialized view — Query is rewritten to use the materialized view— Underlying base tables are not used and performance is normally improved

A few carefully chosen MVs can support a large range of queries


Query RewriteQuery Rewrite

Transformation of a query based on tables to an equivalent query based on one or more materialized views

Several factors affect whether query rewrite will occur1. Enabling or disabling query rewrite can be achieved via the

• CREATE or ALTER statement for materialized views• REWRITE and NOREWRITE hints in individual SQL statements

SELECT /* +NOREWRITE */... • The REWRITE_OR_ERROR hint will prevent the processing of a

statement if it is not rewritten

2. Rewrite integrity levels• Some levels allow the use of stale data

3. Dimensions and constraints• Outside the scope of this talk but details are in the Oracle docs


Controlling Accuracy of Query Rewrite with QUERY_REWRITE_INTEGRITYControlling Accuracy of Query Rewrite with QUERY_REWRITE_INTEGRITY

ALTER SESSION SET QUERY_REWRITE_INTEGRITY = ENFORCED|TRUSTED|STALE_TOLERATED;

ENFORCED MV data must be fresh and any constraints used must be enabled and validated

TRUSTED Trusts data conforms to constraints that are not validated

STALE_TOLERATED Allows stale MVs to be used

Only the ENFORCED level guarantees totally accurate results— MVs will not be used if they are not refreshed after the base table changes


Checking on the MV SituationChecking on the MV Situation

DML on underlying objects causes NEEDS_COMPILE when REFRESH ON DEMAND is specified

A refresh or compile fixes the state

SELECT mview_name,refresh_mode,staleness,compile_state FROM user_mviews

MVIEW_NAME REFRESH_MODE STALENESS COMPILE_STATE------------------------ ------------ ------------- -------------CAL_MONTH_SALES_MV DEMAND NEEDS_COMPILE NEEDS_COMPILEFWEEK_PSCAT_SALES_MV DEMAND STALE VALIDSUM_SALES_PSCAT_WEEK_MV COMMIT FRESH VALID

ALTER MATERIALIZED VIEW cal_month_sales_mv COMPILE;

SELECT mview_name,refresh_mode,staleness,compile_state FROM user_mviews;

MVIEW_NAME REFRESH_MODE STALENESS COMPILE_STATE------------------------ ------------ ------------- -------------CAL_MONTH_SALES_MV DEMAND STALE VALIDFWEEK_PSCAT_SALES_MV DEMAND STALE VALIDSUM_SALES_PSCAT_WEEK_MV COMMIT FRESH VALID


Query Rewrite Performance Query Rewrite Performance

CREATE MATERIALIZED VIEW sales_prod_mvBUILD IMMEDIATEENABLE QUERY REWRITEASSELECT prod_id ,SUM(amount_sold)FROM salesGROUP BY prod_id;

SELECT prod_id ,SUM(amount_sold)FROM salesGROUP BY prod_id;

Sample MV

Sample query


Observing Query RewriteObserving Query Rewrite

Queries may be rewritten to use the materialized viewSELECT prod_id ,SUM(amount_sold)FROM salesGROUP BY prod_id;

Elapsed time : 0.03 seconds

---------------------------------------------------------------------------|Id | Operation | Name |Rows|Bytes | Cost(%CPU)|---------------------------------------------------------------------------| 0| SELECT STATEMENT | | 10| 1872 | 3 (0)|| 1| MAT_VIEW REWRITE ACCESS FULL| SALES_PROD_MV| 72| 1872 | 3 (0)|---------------------------------------------------------------------------

-----------------------------------------------------------------|Id | Operation | Name |Rows | Bytes | Cost(%CPU) |-----------------------------------------------------------------| 0 | SELECT STATEMENT | | 73 | 657 | 1182 (17)|| 1 | HASH GROUP BY | | 73 | 657 | 1182 (17)|| 2 | TABLE ACCESS FULL| SALES | 932K| 8196K| 1038 (5)|-----------------------------------------------------------------

When the view is not present, the base table is scanned

Elapsed time : 16.21 seconds


Did Query Rewrite Occur and Why?Did Query Rewrite Occur and Why?

Query rewrite occurs transparently— Check for query rewrite

• AUTOTRACE • dbms_mview.explain_rewrite

dbms_mview.explain_rewrite— Gives reasons WHY the rewrite did not occur

• The rules governing this are extremely complex— Populates the rewrite_table with information

• Which materialized views will be used• Comparative costings of the rewritten and un-rewritten query

— Each account must have its own rewrite_table• Created by running the utlxrw script in the admin directory


Explaining the rewrite: Example OneExplaining the rewrite: Example One

BEGIN DBMS_MVIEW.EXPLAIN_REWRITE ('SELECT prod_id,SUM(amount_sold) FROM sales GROUP BY prod_id');END;/SELECT message ,original_cost ,rewritten_cost FROM rewrite_table;MESSAGE ORIGINAL_COST REWRITTEN_COST------------------------------------------------------------ ------------- --------------QSM-01009: materialized view,prod_sum_mv, matched query text 180 3

An efficient rewrite occurs because of the exact match of the query and the definition of the view— Most rewrites are more complex

Aliases can be used for MV columns to allow exact match with table columns

CREATE MATERIALIZED VIEW emv (e_deptno, d_deptno) ENABLE QUERY REWRITE AS SELECT e.deptno, d.deptno...


Explaining the rewrite: Example TwoExplaining the rewrite: Example Two

BEGIN DBMS_MVIEW.EXPLAIN_REWRITE ('SELECT prod_id,AVG(amount_sold) FROM sales GROUP BY prod_id');END;/

SELECT message ,original_cost ,rewritten_cost FROM rewrite_table;

MESSAGE ORIGINAL_COST REWRITTEN_COST------------------------------------------ ------------- --------------QSM-01086: dimension(s) not present or not 0 0used in ENFORCED integrity mode

QSM-01065: materialized view, prod_sum_mv, 0 0Cannot compute measure, AVG, in the query

Query rewrite cannot occur for the given reasons — No costs are calculated


Undocumented Query Rewrite ParametersUndocumented Query Rewrite Parameters

KSPPINM KSPPSTVL-------------------------- --------_explain_rewrite_mode FALSE_query_rewrite_or_error FALSE_query_cost_rewrite TRUE_query_rewrite_2 TRUE_query_rewrite_1 TRUE_query_rewrite_fudge 90_query_rewrite_expression TRUE_query_rewrite_jgmigrate TRUE_query_rewrite_fpc TRUE_query_rewrite_drj TRUE_query_rewrite_maxdisjunct 257_query_rewrite_vop_cleanup TRUE_mmv_query_rewrite_enabled TRUE

KSPPINM KSPPSTVL------------------------------------ ------------_bt_mmv_query_rewrite_enabled TRUE_pre_rewrite_push_pred TRUE_union_rewrite_for_gs YES_GSET_MVS_query_rewrite_setopgrw_enable TRUE_force_rewrite_enable FALSE_query_mmvrewrite_maxpreds 10_query_mmvrewrite_maxintervals 5_query_mmvrewrite_maxinlists 5_query_mmvrewrite_maxdmaps 10_query_mmvrewrite_maxcmaps 20_query_mmvrewrite_maxregperm 512_query_mmvrewrite_maxmergedcmaps 50_query_mmvrewrite_maxqryinlistvals 500_enable_query_rewrite_on_remote_objs TRUE

COL ksppinm FORMAT a40COL ksppstvl FORMAT a40SELECT ksppinm,ksppstvlFROM x$ksppi i,x$ksppcv vWHERE i.indx = v.indxAND i.ksppinm like '%rewrite%‘AND SUBSTR(ksppinm,1,1) = '_';


TopicsTopics


Query Rewrite








Refreshing Materialized ViewsRefreshing Materialized Views

If data in base tables change, MVs need to be refreshed in one of two ways

1. On transaction commit (REFRESH ON COMMIT)• MV kept up to date but performance impact on base table transactions

2. On demand (REFRESH ON DEMAND)• Requires a procedure to be invoked

• The procedure can take an argument to specify the type of refresh COMPLETE - complete refresh by repopulating FAST - incrementally applies changes to the viewFORCE - fast refresh if possible, if not it forces a complete refresh

dbms_mview.refresh('emp_mv1')


Refreshing Materialized ViewsRefreshing Materialized Views

Refresh can be accelerated with ATOMIC_REFRESH set to 'FALSE'— MV is truncated rather then deleted (default value is TRUE on 10g and up)

• Less undo and redo but MV unavailable during refresh — Allows parallel refresh

Fast Refresh can be accelerated if MV logs are created 'WITH COMMIT SCN'— Snapshots do not have to be built during the refresh — Use is made of all_summap view — Can also set NOLOGGING for the materialized view logs


Materialized View LogsMaterialized View Logs

Materialized view logs are required for fast refresh

Defined using CREATE MATERIALIZED VIEW LOG on the base table— Not created on the materialized view— Each inserted and deleted row on the base table creates a row in the log— An update may generate two rows in the log for old and new values

Normally need to specify the ROWID clause

Logs that support aggregate materialized views must also contain:1. Every column in the table that is referenced in the materialized view2. The INCLUDING NEW VALUES clause

— Name of the log is mlog$_<base_table_name>— Specify SEQUENCE if the table is expected to have a mix of inserts/direct-

loads, deletes, and updates

CREATE MATERIALIZED VIEW LOG ON emp WITH ROWID (deptno,sal) INCLUDING NEW VALUES;


Tuning Materialized Views for Fast Refresh and Query RewriteTuning Materialized Views for Fast Refresh and Query Rewrite

dbms_advisor.tune_mview helps optimize CREATE MATERIALIZED VIEW statements

Shows how to implement materialized views and any required logs

The output statements for ‘IMPLEMENTATION’ include:— CREATE MATERIALIZED VIEW LOG statements

• Creates any missing materialized view logs required for fast refresh

— ALTER MATERIALIZED VIEW LOG FORCE statements• Fixes any materialized view logs for maybe fast refresh

— One or more CREATE MATERIALIZED VIEW statements• Could suggest just adding required columns

• For example, add ROWID column for materialized join view and add aggregate column for materialized aggregate view


Tuning Materialized Views – Using the AdvisorTuning Materialized Views – Using the Advisor

VARIABLE task_cust_mv VARCHAR2(30);VARIABLE create_mv_ddl VARCHAR2(4000);

BEGIN :task_cust_mv := 'emp_mv1'; :create_mv_ddl := 'CREATE MATERIALIZED VIEW emp_mv1 REFRESH FAST ENABLE QUERY REWRITE AS SELECT deptno,sum(sal) FROM emp GROUP BY deptno'; DBMS_ADVISOR.TUNE_MVIEW(:task_cust_mv, :create_mv_ddl);END;/

Tasks can be deleted

EXEC dbms_advisor.delete_task('emp_mv1')

Requires ADVISOR privilege


Tuning Materialized Views – Examining the OutputTuning Materialized Views – Examining the Output

SELECT statement FROM user_tune_mview WHERE task_name= :task_cust_mv AND script_type='IMPLEMENTATION';

STATEMENT------------------------------------------------------------------CREATE MATERIALIZED VIEW LOG ON "SCOTT"."EMP" WITH ROWID, SEQUENCE "SAL","DEPTNO") INCLUDING NEW VALUES

ALTER MATERIALIZED VIEW LOG FORCE ON "SCOTT"."EMP" ADD ROWID, SEQUENCE

"SAL","DEPTNO") INCLUDING NEW VALUES

CREATE MATERIALIZED VIEW SCOTT.EMP_MV1 REFRESH FAST WITH ROWID ENABLE QUERY REWRITE AS SELECT SCOTT.EMP.DEPTNO C1, SUM("SCOTT"."EMP"."SAL") M1, COUNT("SCOTT"."EMP"."SAL") M2, COUNT(*) M3 FROM SCOTT.EMP GROUP BY

SCOTT.EMP.DEPTNO

For the view to be fast refreshed1. A log is recommended because the view will be fast refreshed2. The view must include a count of the aggregate column and also a count of

the rows


Tuning Materialized Views – a Further ExampleTuning Materialized Views – a Further Example

BEGIN :task_cust_mv := 'emp_mv2'; :create_mv_ddl := 'CREATE MATERIALIZED VIEW emp_mv2 REFRESH FAST ENABLE QUERY REWRITE AS SELECT dept.deptno,dname,sum(sal) FROM emp,dept WHERE dept.deptno = emp.deptno GROUP BY dept.deptno,dname' DBMS_ADVISOR.TUNE_MVIEW(:task_cust_mv, :create_mv_ddl);END;/


Further Example OutputFurther Example Output

SELECT statement FROM user_tune_mview WHERE task_name= :task_cust_mv AND script_type='IMPLEMENTATION';

STATEMENT--------------------------------------------------------------------CREATE MATERIALIZED VIEW LOG ON "SCOTT"."DEPT" WITH ROWID, SEQUENCE("DEPTNO","DNAME") INCLUDING NEW VALUES

ALTER MATERIALIZED VIEW LOG FORCE ON "SCOTT"."DEPT" ADD ROWID, SEQUENCE ("DEPTNO","DNAME") INCLUDING NEW VALUES

CREATE MATERIALIZED VIEW LOG ON "SCOTT"."EMP" WITH ROWID, SEQUENCE("SAL","DEPTNO") INCLUDING NEW VALUES

ALTER MATERIALIZED VIEW LOG FORCE ON "SCOTT"."EMP" ADD ROWID, SEQUENCE ("SAL","DEPTNO") INCLUDING NEW VALUES

CREATE MATERIALIZED VIEW SCOTT.EMP_MV2 REFRESH FAST WITH ROWID ENABLE QUERY REWRITE AS SELECT SCOTT.DEPT.DNAME C1, SCOTT.DEPT.DEPTNO C2,SUM("SCOTT"."EMP"."SAL") M1, COUNT("SCOTT"."EMP"."SAL") M2, COUNT(*) M3 FROM SCOTT.EMP, SCOTT.DEPT WHERE SCOTT.DEPT.DEPTNO = SCOTT.EMP.DEPTNO GROUP BY SCOTT.DEPT.DNAME, SCOTT.DEPT.DEPTNO


Materialized View CapabilitiesMaterialized View Capabilities

dbms_mview.explain_mview is used to explain the capabilities (query rewrite, refresh modes) of an MV

— The optional value '6' is a user supplied statement identifier

Places information in mv_capabilities_table— Has one row for each reason why a certain action is not possible— Needs to be built in each investigating account by executing the script

utlxmv.sql found in the admin directory— The statement_id column can be used to fetch rows for a given MV

EXECUTE dbms_mview.explain_mview ('emp_mv1','6')


The mv_capabilities_tableThe mv_capabilities_table

EXECUTE dbms_mview.explain_mview('emp_mv1','6')

SELECT capability_name,possible,msgtxt FROM mv_capabilities_table WHERE statement_id = '6';

CAPABILITY_NAME P MSGTXT----------------------------- - -----------------------------------------------PCT NREFRESH_COMPLETE YREFRESH_FAST YREWRITE YPCT_TABLE N relation is not a partitioned tableREFRESH_FAST_AFTER_INSERT YREFRESH_FAST_AFTER_ONETAB_DML N SUM(expr) without COUNT(expr)REFRESH_FAST_AFTER_ONETAB_DML N COUNT(*) is not present in the select listREFRESH_FAST_AFTER_ANY_DML N mv log does not have sequence #REFRESH_FAST_AFTER_ANY_DML N see the reason why

REFRESH_FAST_AFTER_ONETAB_DML is disabledREFRESH_FAST_PCT N PCT is not possible on any of the detail tables

in the materialized viewREWRITE_FULL_TEXT_MATCH YREWRITE_PARTIAL_TEXT_MATCH YREWRITE_GENERAL YREWRITE_PCT N general rewrite is not possible or PCT is not possible on any of the detail tablesPCT_TABLE_REWRITE N relation is not a partitioned table


Restrictions on Fast Refresh on MVsRestrictions on Fast Refresh on MVs

There are many more special case restrictions and rules

Joins…— ROWIDs of all tables in FROM list must appear in the SELECT list of the query— Materialized view logs must exist on all tables involved in the join

Aggregates…— Only SUM, COUNT, AVG, STDDEV, VARIANCE, MIN and MAX are

supported for fast refresh• COUNT(*) must be included in the view

— For each aggregate such as AVG(expr), the corresponding COUNT(expr) must be present


sales sales_mv

mlog$_sales

2547

2547 25

47

62

62 62

materialized view

Materialized view log

build

commit62

Base table

62

REFRESH COMPLETE ON COMMIT does not require logs– Very unlikely to be used

Fast Refresh on CommitFast Refresh on Commit


Bulk Loading – Impact on Fast RefreshBulk Loading – Impact on Fast Refresh

1. INSERT /*+ append */

2. SQL*Loader DIRECT=TRUE

Improved performance as no data written to MV log— Much less redo and undo— Minimal tracking in all_sumdelta

• Summary information with ROWID ranges


Flexing the Materialized Views with ON PREBUILT TABLEFlexing the Materialized Views with ON PREBUILT TABLE

MVs cannot be ALTERed— What if a column width change is required?

• Could make use of ON PREBUILT TABLE to avoid MV rebuildCREATE TABLE j_g AS SELECT job, COUNT(*) ct FROM emp GROUP BY job;

CREATE MATERIALIZED VIEW j_g ON PREBUILT TABLE AS SELECT job, COUNT(*) ct FROM emp GROUP BY job;

OBJECT_ID DATA_OBJECT_ID OBJECT_NAME OBJECT_TYPE--------- -------------- ----------- ------------------- 86118 86118 J_G TABLE 86119 J_G MATERIALIZED VIEW

DROP TABLE j_g; -- fails due to presence of MV

DROP MATERIALIZED VIEW j_g; --succeeds but leaves table intact

ALTER TABLE j_g MODIFY job VARCHAR2(40);

CREATE MATERIALIZED VIEW j_g ON PREBUILT TABLE AS SELECT job, COUNT(*) ct FROM emp GROUP BY job;

— Can also help when fixing 'out of sync' errors with logs and base tables• ORA-12034: materialized view log younger than last refresh


TopicsTopics


Query Rewrite








Nested Materialized ViewsNested Materialized Views

Base table 1 Base table 2 Base table 3

Join Materialized View

Aggregate Materialized

View 1


View 2


View 3


View 4

Nested Views


prod_sales (master)

Prod_name Cust_id Amt_soldBolt 237 22.05Pin 120 9.23Bolt 194 3.87

Refreshing Nested MVsRefreshing Nested MVs

DBMS_MVIEW.REFRESH_DEPENDENT with NESTED = TRUE refreshes all views in a hierarchy that depend on one or more specified base tables

sum_prod_sales (nested)

Prod_name SUM(Amt_sold)

Bolt 25.92Pin 9.23

Fresh

products

Prod_id Prod_name

10 Bolt14 Pin

Sales

Prod_id Cust_id Amt_sold

10 237 22.0514 120 9.2310 194 3.87

Fresh

StaleRefresh nested viewProd_name SUM(Amt_sold)

Bolt 136.75Pin 9.23

Stale

sales

Prod_id Cust_id Amt_sold

10 237 22.0514 120 9.2310 194 3.8710 105 110.83

Update the sales table

Refresh the master

Prod_name Cust_id Amt_soldBolt 237 22.05Pin 120 9.23Bolt 194 3.87Bolt 105 110.83


Nested MVs in Query RewriteNested MVs in Query Rewrite

2. Every master object requires a log containing the ROWID

CREATE MATERIALIZED VIEW master_nest_mvBUILD IMMEDIATEENABLE QUERY REWRITEASSELECT p.prod_id, p.prod_name, c.channel_id, c.channel_descFROM sales s, products p, channels cWHERE s.prod_id = p.prod_idAND s.channel_id = c.channel_id;

1. Create a master materialized view

CREATE MATERIALIZED VIEW LOG ON sales WITH ROWID;CREATE MATERIALIZED VIEW LOG ON products WITH ROWID;CREATE MATERIALIZED VIEW LOG ON channels WITH ROWID;3. The master materialized view also requires a log containing ROWID, each

column in the materialized view itself, and all new values

CREATE MATERIALIZED VIEW LOG ON master_nest_mv WITH ROWID(prod_id, prod_name, channel_id, channel_desc)INCLUDING NEW VALUES;


Nested MVs in Query Rewrite (continued)Nested MVs in Query Rewrite (continued)

— AUTOTRACE shows evidence of use but does not show the hierarchy

-----------------------------------------------------------------------------------|Id| Operation |Name |Rows|Bytes|Cost(%CPU)| Time |-----------------------------------------------------------------------------------| 0|SELECT STATEMENT | | 225| 9450| 3 (0) |00:00:01|| 1|MAT_VIEW REWRITE ACCESS FULL|NESTED_PROD_CHAN_MV| 225| 9450| 3 (0) |00:00:01| -----------------------------------------------------------------------------------

CREATE MATERIALIZED VIEW nested_prod_chan_mvBUILD IMMEDIATEENABLE QUERY REWRITEASSELECT channel_desc, prod_name, COUNT(prod_id)FROM master_nest_mvGROUP BY channel_desc, prod_name;

4. Create the nested MV based on the master MV


Nested MVs in Query Rewrite (continued)Nested MVs in Query Rewrite (continued)

SELECT message FROM rewrite_table;

MESSAGE-----------------------------------------------------------------QSM-01151: query was rewrittenQSM-01033: query rewritten with materialized view, NESTED_PROD_CHAN_MVQSM-01336: the materialized view you specified (NESTED_PROD_CHAN_MV) was not used to rewrite the queryQSM-01033: query rewritten with materialized view, MASTER_NEST_MV

BEGIN DBMS_MVIEW.EXPLAIN_REWRITE( 'SELECT c.channel_desc, p.prod_name FROM sales s, products p, channels c WHERE s.prod_id = p.prod_id AND c.channel_id = s.channel_id GROUP BY channel_desc, prod_name' ,'NESTED_PROD_CHAN_MV');END;/


TopicsTopics


Query Rewrite








Foreign Keys Referencing Non-unique ColumnsForeign Keys Referencing Non-unique Columns

Problem :

Need to enforce a foreign key constraint on the occupation column in the empdep table based on data in the emp table— Requires a reference to a non-unique column (job) in the emp table

Reference a materialized view carrying only unique values of job

EMPNO JOB----- -------- 7366 SALESMAN 7500 MANAGER 7902 MANAGER 7566 CLERK 7934 SALESMAN

JOB--------SALESMANMANAGERCLERK

ENAME OCCUPATION-------- ----------WOODS CLERKJOHNSON MANAGERCOX CLERKPITT CLERKSPINK SALESMANDRAPER MANAGER

EMP

EMP_V1

EMPDEP

primary keyforeign key


Enforcing Foreign Keys Without Primary keysEnforcing Foreign Keys Without Primary keys

CREATE MATERIALIZED VIEW LOG ON emp WITH ROWID,PRIMARY KEY,SEQUENCE(job) INCLUDING NEW VALUES;

CREATE MATERIALIZED VIEW emp_v1REFRESH FAST ON COMMITENABLE QUERY REWRITEAS SELECT job,COUNT(*),COUNT(job) FROM emp GROUP BY job;

ALTER MATERIALIZED VIEW emp_v1 ADD PRIMARY KEY (job);

ALTER TABLE empdep ADD CONSTRAINT empdep_emp_v1 FOREIGN KEY (occupation) REFERENCES emp_v1;

UPDATE empdep SET occupation = 'X';ORA-02291: integrity constraint (SCOTT.EMPDEP_EMP_V1) violated - parent key not found

job is not unique in emp

job is unique in MV

empdep must have only jobs in the emp table


Enforcing Complex Constraints with Materialized ViewsEnforcing Complex Constraints with Materialized Views

Employee's appraisal date must be within their department's appraisal period

EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO APP_DATE----- ------ --------- ---- --------- ---- ---- ------ --------- 7369 SMITH CLERK 7902 17-DEC-80 800 20 15-MAR-11 7499 ALLEN SALESMAN 7698 20-FEB-81 1600 300 30 15-APR-11 7521 WARD SALESMAN 7698 22-FEB-81 250 500 30 15-APR-11 7566 JONES MANAGER 7839 02-APR-81 2975 20 15 MAR-11

DEPTNO DNAME LOC START_APP_PERIOD END_APP_PERIOD------ -------- -------- ---------------- -------------- 20 RESEARCH DALLAS 01-FEB-11 01-MAY-11 30 SALES CHICAGO 01-MAR-11 01-JUN-11

MV which returns app_dates with departmental appraisal periodsCREATE MATERIALIZED VIEW emp_dept_appr_mvREFRESH FAST ON COMMIT ASSELECT e.deptno ,e.app_date e_app_date ,d.start_app_period d_start_app_period ,d.end_app_period d_end_app_period ,e.ROWID e_rowid ,d.ROWID d_rowidFROM emp e ,dept dWHERE e.deptno = d.deptno

constraint checked when changes to the emp table are committed


Enforcing Complex Constraints with Materialized Views (continued)Enforcing Complex Constraints with Materialized Views (continued)

Prevent invalid appr_dates appearing in the view

ALTER TABLE emp_dept_appr_mvADD CONSTRAINT emp_dept_appr_mv_chk1CHECK (e_app_date BETWEEN d_start_app_period AND d_end_app_period)DEFERRABLE;

— Constraint is checked on commit rather than on update

Insert a row with an appr_date outside of the departmental period

INSERT INTO emp (empno,deptno,app_date)VALUES(1, 20, '01-jan-2013');

Constraint is not enforced until the commit (and refresh) takes place

SQL> COMMIT;ORA-12048: error encountered while refreshing materialized View "EMP_DEPT_APPR_MV"ORA-02290: check constraint (emp_dept_appr_mv_chk1) violated


Testing Complex Constraint Checking with Materialized Views Testing Complex Constraint Checking with Materialized Views

Could lead to large numbers of rows in the MV when constraint is repeatedly NOT violated— Hence, simply reverse the MV condition to prevent valid data in the view

— The MV never collects any data • Constraint prevents the invalid data• But the MV logs are still populated!

AND NOT (e.app_date BETWEEN d.start_app_period AND d.end_app_period)


Indexes on Materialized ViewsIndexes on Materialized Views

A single unique index is often created on an MV when it is built— If the MV is a grouped (aggregate) MV, the index is a function-based Btree

and has a name of I_SNAP$_<mv_name>— Built on virtual RAW column(s) built into the MV— Number of columns in the index is equal to number of grouping columns

• Names are similar to sys_nc00003$

CREATE MATERIALIZED VIEW sales_mv ASSELECT p.prod_subcategory, t.week_ending_day, SUM(s.amount_sold) AS sum_amount_soldFROM sales s, products p, times tWHERE s.time_id = t.time_id AND s.prod_id = p.prod_idGROUP BY p.prod_subcategory, t.week_ending_day

generates two hidden indexed columns in the MV


Indexes on Grouped Materialized ViewsIndexes on Grouped Materialized Views

Values in the hidden columns

SELECT sys_nc00004$ ,sys_nc00005$ ,prod_subcategory ,week_ending_day FROM sales_mv;

SYS_NC00004$ SYS_NC00005$ PROD_SUBCATEGORY WEEK_ENDING_DAY------------------------ ---------------- ---------------- ---------------43616D6572617300 77C6020101010100 Cameras 01-FEB-984D6F6E69746F727300 77C6031601010100 Monitors 22-MAR-984465736B746F702050437300 77C6030801010100 Desktop PCs 08-MAR-9843616D636F726465727300 77C6010B01010100 Camcorders 11-JAN-9843616D636F726465727300 77C6011901010100 Camcorders 25-JAN-984163636573736F7269657300 77C6020101010100 Accessories 01-FEB-984163636573736F7269657300 77C6030101010100 Accessories 01-MAR-98486F6D6520417564696F00 77C6010B01010100 Home Audio 11-JAN-98486F6D6520417564696F00 77C6031601010100 Home Audio 22-MAR-98 : : : :

— 43 is the ASCII value of ‘C’ in hexadecimal


Indexes on Non-grouped Materialized ViewsIndexes on Non-grouped Materialized Views

If base table has a PK and a one to one mapping of rows with the MV— A normal Btree index is built on the PK within the view

• Oracle recommends building an index on any ROWID column in the MVand a concatenated index on all key columns of the base tables on the MV (for fast refresh)

— If PK on the base table is explicitly named (e.g PK_EMP) • The normal Btree index on the MV is named PK_EMP1 and supports a

constraint named PK_EMP1

If no primary key on the base table — No index is built on the MV

Indexes are not built when the USING NO INDEX clause is used in MV creation


TopicsTopics


Query Rewrite








Managing Update and Refresh Operations Managing Update and Refresh Operations

No. of rows updated

Update Refresh

Fast Complete Fast Complete

1 00:00.2 00:00.1 00:01.4 00:13.2

100 00:00.2 00:00.2 00:02.2 00:07.3

500 00:00.5 00:00.1 00:00.2 00:09.3

1000 00:00.7 00:00.1 00:01.5 00:07.7

5000 00:06.5 00:00.3 00:01.6 00:08.0

50000 00:58.0 00:19.1 00:39.1 00:10.5

75000 00:35.7 00:22.3 00:53.5 00:08.5

100000 00:58.9 00:28.5 01:40.6 00:06.1

500000 08:52.9 02:47.7 12:40.8 00:07.2

Tests performed on a 918443 row sales table having one MV in fast or complete refresh mode— Updates performed in batch mode (single update statement) on base table


Updating Tables having Materialized Views Updating Tables having Materialized Views

Batch Update Performance on Table with Fast or Complete Refresh Materialized View

00:00.0

01:00.0

02:00.0

03:00.0

04:00.0

05:00.0

06:00.0

07:00.0

08:00.0

09:00.0

10:00.0

110

050

010

0050

00

5000

0

7500

0

1000

00

5000

00

No of Rows Updated

Tim

e T

ake

n (M

inut

es)

Fast

Complete

Fast refresh has the overhead of MV log maintenance


Fast and Complete Refresh PerformanceFast and Complete Refresh Performance

Performance of Fast Vs Complete Refresh

00:00.0

01:00.0

02:00.0

03:00.0

04:00.0

05:00.0

06:00.0

07:00.0

08:00.0

09:00.0

10:00.0

11:00.0

12:00.0

1 100

500

1000

5000

5000

0

7500

0

1000

00

5000

00

No of Rows Updated

Tim

e T

aken

to

Ref

resh

(M

inut

es)

Fast

Complete

MV is truncated during a complete refresh— Faster for large scale updates


Overheads on OLTP - Transaction TestOverheads on OLTP - Transaction Test

Repeated updates on a single row

BEGIN FOR i in 1..n LOOP UPDATE sales SET amount_sold = amount_sold + .01 WHERE prod_id = 14 AND cust_id = 50840 AND time_id = '03-NOV-01' AND channel_id = 3; COMMIT; END LOOP;END;

sales table has an index to avoid full table scans masking update timings

ALTER TABLE sales ADD CONSTRAINT pk_sales PRIMARY KEY (prod_id,cust_id,time_id,channel_id);


Transaction Overheads - TestTransaction Overheads - Test

Routine timed with and without MV log on sales— Log was truncated after each test

CREATE MATERIALIZED VIEW LOG ON sales WITH ROWID (cust_id ,prod_id ,channel_id ,time_id ,amount_sold ,quantity_sold)INCLUDING NEW VALUES;

Timings also produced with and without the COMMIT statement


Timings With and Without MV log and COMMITTimings With and Without MV log and COMMIT

Number of iterations (n)

Time taken (seconds)

With commit Without commit

No MV log

MV log present

No MV log

MV log present

500 0.21 0.25 0.03 0.15

1000 0.34 0.43 0.06 0.25

2500 0.62 0.96 0.23 0.51

5000 1.14 1.89 0.40 1.07

7500 1.67 2.81 0.61 1.58

10000 2.03 4.00 0.79 2.09

50000 11.10 18.59 3.93 10.62

Presence of MV log can result in up to 250% performance overhead


Contention Caused by Materialized ViewsContention Caused by Materialized Views

Applies to REFRESH ON COMMIT views in OLTP environments— Refresh takes place as part of the OLTP transaction — Aggregate MVs are prone to contention issues— Updates on base tables are 'condensed' into fewer rows in the MV— Causes locking and contention issues as transactions queue to update the

grouped row in the MV

EMPNO ENAME SAL DEPTNO----- ------- ---- ------ 7782 CLARK 2450 10 7839 KING 5000 10 7934 MILLER 1300 10 7369 SMITH 800 20 7876 ADAMS 1100 20 7902 FORD 3000 20 7788 SCOTT 3000 20 7566 JONES 2975 20 7521 WARD 1250 30 7499 ALLEN 1600 30 7654 MARTIN 1250 30 7698 BLAKE 2850 30 7844 TURNER 1500 30 7900 JAMES 950 30

CREATE MATERIALIZED VIEW m1REFRESH FAST ON COMMIT AS SELECT deptno, SUM(sal) ,COUNT(*), COUNT(sal) FROM emp GROUP BY deptno;

DEPTNO SUM(SAL) COUNT(*) COUNT(sal)------ -------- -------- ---------- 10 8750 3 3 20 10875 5 5 30 940 6 6

Possible contention for locks of type JI


TopicsTopics


Query Rewrite








Using Multiple Materialized ViewsUsing Multiple Materialized Views

CREATE MATERIALIZED view prod_high_mvBUILD IMMEDIATEENABLE QUERY REWRITEASSELECT prod_id, SUM(amount_sold)FROM salesWHERE prod_id > 15GROUP BY prod_id;

CREATE MATERIALIZED view prod_low_mvBUILD IMMEDIATEENABLE QUERY REWRITEASSELECT prod_id, SUM(amount_sold)FROM salesWHERE prod_id <= 15GROUP BY prod_id;

PROD_ID SUM(AMOUNT_SOLD)------- ---------------- 14 733738663 10 50236718.5 15 734982650 11 49879911.7 13 662255842 12 33032602.8

PROD_ID SUM(AMOUNT_SOLD)---------- ---------------- 17 808449750 16 220330955 19 78746741.5 18 1005874847

Two materialized views each covering only part of the base data

— Table contains sales data for prod_ids from 10 to 19


Using Multiple Materialized ViewsUsing Multiple Materialized Views

Execution Plan-----------------------------------------------------------------------------------------| Id | Operation | Name |Rows | Bytes | Cost (%CPU)| Time |-----------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 10 | 110 | 6 (50)| 00:00:01|| 1 | UNION-ALL | | | | | || 2 | MAT_VIEW REWRITE ACCESS FULL| PROD_LOW_MV | 6 | 66 | 3 (0)| 00:00:01|| 3 | MAT_VIEW REWRITE ACCESS FULL| PROD_HIGH_MV| 4 | 44 | 3 (0)| 00:00:01|

SELECT prod_id, SUM(amount_sold)FROM sales-- WHERE prod_id > 0GROUP BY prod_id;

A WHERE clause appears to be necessaryfor multiple MVs to be used in pre 11gR2


Overlapping Materialized ViewsOverlapping Materialized Views

CREATE MATERIALIZED VIEW prod_low_mvBUILD IMMEDIATEENABLE QUERY REWRITEASSELECT prod_id, SUM(amount_sold)FROM salesWHERE prod_id <= 15GROUP BY prod_id;

PROD_ID SUM(AMOUNT_SOLD)------- ---------------- 14 733738663 10 50236718.5 15 734982650 11 49879911.7 13 662255842 12 33032602.8

CREATE MATERIALIZED VIEW prod_high_mvBUILD IMMEDIATEENABLE QUERY REWRITEASSELECT prod_id, SUM(amount_sold)FROM salesWHERE prod_id >= 15GROUP BY prod_id;

PROD_ID SUM(AMOUNT_SOLD)------- ---------------- 15 734982650 17 808449750 16 220330955 19 78746741.5 18 1005874847



Duplicate row for prod_id = 15 is (correctly) not shown in query output

SELECT prod_id, SUM(amount_sold)FROM salesWHERE prod_id > 0GROUP BY prod_idORDER BY prod_id;

PROD_ID SUM(AMOUNT_SOLD)------- ---------------- 10 50236718.5 11 49879911.7 12 33032602.8 13 662255842 14 733738663 15 734982650 16 220330955 17 808449750 18 1005874847 19 78746741.5



But the plan shows a UNION-ALL operation across the two views— This is due to re-aggregation that is required across inline views --------------------------------------------------------------|Id| Operation | Name |--------------------------------------------------------------| 0| SELECT STATEMENT | || 1| VIEW | || 2| UNION-ALL | | | 3| MAT_VIEW REWRITE ACCESS FULL | PROD_HIGH_MV || 4| MAT_VIEW REWRITE ACCESS FULL | PROD_LOW_MV |--------------------------------------------------------------SELECT prod_id,SUM(sa) FROM (SELECT prod_id, SUM(amount_sold) sa FROM sales WHERE prod_id <= 15 GROUP BY prod_id UNION SELECT prod_id, SUM(amount_sold) sa FROM sales WHERE prod_id >= 15 GROUP BY prod_id)GROUP BY prod_id ORDER BY prod_id;

Equivalent direct use of MVs generates exact same planeven when ALL is not specified


Multiple MVs with Different Aggregate FunctionsMultiple MVs with Different Aggregate Functions

The two views generate same groups— Different aggregate functions

CREATE MATERIALIZED view prod_avg_mvBUILD IMMEDIATEENABLE QUERY REWRITEASSELECT prod_id, AVG(amount_sold)FROM salesGROUP BY prod_id;

CREATE MATERIALIZED view prod_sum_mvBUILD IMMEDIATEENABLE QUERY REWRITEASSELECT prod_id, SUM(amount_sold)FROM salesGROUP BY prod_id;

SELECT prod_id, SUM(amount_sold), AVG(amount_sold)FROM salesGROUP BY prod_id; ----------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |----------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 72 | 648 | 960 (3)| 00:00:12 || 1 | HASH GROUP BY | | 72 | 648 | 960 (3)| 00:00:12 || 2 | TABLE ACCESS FULL| SALES | 500K| 4394K| 940 (1)| 00:00:12 |----------------------------------------------------------------------------

— Neither view is used


Multiple MVs with Different Aggregate Functions - a WorkaroundMultiple MVs with Different Aggregate Functions - a Workaround

Encapsulate the views in the FROM clause— Results from views are hash-joined

SELECT a.prod_id, a.sum_prod, b.avg_prodFROM (SELECT prod_id ,SUM(amount_sold) sum_prod FROM sales GROUP BY prod_id) a, (SELECT prod_id ,AVG(amount_sold) avg_prod FROM sales GROUP BY prod_id) b WHERE a.prod_id = b.prod_id;

-----------------------------------------------------------------------------------------| Id | Operation | Name | Rows| Bytes| Cost (%CPU)| Time |-----------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 72| 2664| 7 (15)| 00:00:01 || 1 | HASH JOIN | | 72| 2664| 7 (15)| 00:00:01 || 2 | MAT_VIEW REWRITE ACCESS FULL| PROD_SUM_MV | 72| 792| 3 (0)| 00:00:01 || 3 | MAT_VIEW REWRITE ACCESS FULL| PROD_AVG_MV | 72| 1872| 3 (0)| 00:00:01 |-----------------------------------------------------------------------------------------

— Both views are used


TopicsTopics


Query Rewrite








Estimating Size of Materialized ViewsEstimating Size of Materialized Views

Materialized views can consume significant amounts of storage

Estimate size of MV with dbms_mview.estimate_mview_size— Estimates number of rows and bytes

Accuracy of the estimate procedure was tested on an original sales table of 918443 rows, joined with customers and products


The Oracle BlockThe Oracle Block

Block 17

Free Space

Row 4

Block header

Reserved for updates to existing rows

12Row

numbers

8K block — Row header occupies approximately 100 bytes— 819 bytes of free space reserved for updates (default PCTFREE of 10%)— Inserted data can occupy around 7300 bytes


Estimate RoutineEstimate Routine

The output from the routine is as follows :

Number of rows is : 1020125Number of bytes is : 126495500

DECLARE nr NUMBER; --number of rows nb NUMBER; --number of bytesBEGIN dbms_mview.estimate_mview_size (stmt_id => 'mv_est', select_clause => 'SELECT c.cust_id ,c.cust_city ,p.prod_id ,p.prod_name FROM sales s, customers c, products p WHERE s.cust_id = c.cust_id AND s.prod_id = p.prod_id', num_rows => nr, num_bytes => nb); dbms_output.put_line('Number of rows is : '||TO_CHAR(nr)); dbms_output.put_line('Number of bytes is : '||TO_CHAR(nb)); END;


Accuracy of EstimateAccuracy of Estimate

On building the view, the number of rows was actually 918443— This shows that the row estimate of 1020125 is reasonably accurate

Its ROWIDs were interrogated to find the actual block storage

SELECT COUNT(COUNT(*)) “No. of Blocks”FROM mv_estGROUP BY dbms_rowid.rowid_relative_fno(ROWID) ,dbms_rowid.rowid_block_number(ROWID);No. of Blocks------------- 6170

If each block has around 7300 bytes of data, then the number of bytes is 45041000 (7300*6170)— This is wildly different from the estimated value of 126495500


Checking the DictionaryChecking the Dictionary

EXEC dbms_stats.gather_table_stats('SH','<base_table_name>')

SELECT avg_col_len ,column_nameFROM user_tab_colsWHERE table_name IN ('CUSTOMERS','PRODUCTS')AND column_name IN ('CUST_ID','PROD_ID','CUST_CITY','PROD_NAME');

AVG_COL_LEN COLUMN_NAME----------- ------------ 5 CUST_ID 10 CUST_CITY 4 PROD_ID 26 PROD_NAME

So, on average, each row should consume 45 + 5 (row overhead) bytes The total number of occupied blocks should be

(45+5)*918443/7300 = 6290 blocks

— This is close to the result obtained from the actual ROWIDs of the MV (6170)

Perhaps a more accurate value can be obtained from the dictionary


Results and Conclusions – Query RewriteResults and Conclusions – Query Rewrite

MVs return queries in a fraction of the time

MVs are considered for rewrite even if they contain only part of the data required by the query

MVs are not considered for query rewrite if base table(s) are updated— Unless the session settings are altered to STALE_TOLERATED mode

• This emphasizes the need for refreshing the views

Multiple materialized views can be used to satisfy a single query— However if the aggregate materialized views do not include the same aggregate function then neither view will be considered

The estimate_mview_size procedure— Extremely accurate in estimating the number of rows an MV will contain — Not so accurate at predicting actual storage consumption


Results and Conclusions – Refresh MethodsResults and Conclusions – Refresh Methods

REFRESH FAST — Efficient refresh performance when the numbers of rows affected by a

single transaction was small— Due to the presence of the materialized view log— BUT if transaction rate is significant, could have significant effect

REFRESH COMPLETE ON DEMAND suits batch processing environments — Table updates are large and infrequent

REFRESH FAST ON COMMIT suits OLTP environments— WELL perhaps— Regular yet small table updates depending on real-time data

Carl Dudley – University of Wolverhampton, UK

Materialized Views

Carl DudleyUniversity of Wolverhampton, UK

Oracle ACE DirectorUKOUG Committee

[email protected]

Carl Dudley – University of Wolverhampton, UK Materialized Views Carl Dudley University of...

Documents

Transcript of Carl Dudley – University of Wolverhampton, UK Materialized Views Carl Dudley University of...