1 © 2010 Julian Dyke juliandyke.com Optimizer Statistics Julian Dyke Independent Consultant Web...

1 © 2010 Julian Dyke juliandyke.com

Optimizer Statistics

Julian Dyke

Independent Consultant

Web Version


Agenda

Statistics Strategies ANALYZE DBMS_STATS ANALYZE versus DBMS_STATS System Statistics Automatic Statistics Collection Statistics History Manual Statistics Partition Statistics Oracle 11g Enhancements


StatisticsStrategies


Statistics Strategies Introduction Only gather statistics when object has changed significantly

Consider generating manual statistics for: Very large objects Temporary tables Highly volatile tables

Try to collect statistics during a specified window Do not execute SQL statements during statistics collection Use DBMS_STATS parallelism Review sample sizes for very large objects

In complex databases do not rely on Auto job Unpredictable collection behaviour / duration Unpredictable execution plan changes

See Metalink Note 44961.1 - Statistics Gathering; Frequency and Strategy


ANALYZE


ANALYZEIntroduction SQL statement

Introduced in Oracle 7.0 Collects object statistics for Cost Based Optimizer

Initially collected statistics on tables and indexes Subsequently on histograms

Advantages Online analyze - Oracle 9.0 and above Optionally validates structure of tables, indexes and clusters Optionally detects chained / migrated rows

Disadvantages Serial statistics collection only Limited partition-awareness


ANALYZEVALIDATE STRUCTURE Syntax is

ANALYZE INDEX <index_name> VALIDATE STRUCTURE;

Populates session-level dynamic performance views INDEX_STATS INDEX_HISTOGRAMS

Reports index statistics including: height number of leaf blocks number of branch blocks number of distinct keys

Reports space released by deletions Use to determine when to coalesce or rebuild index

Recommends index compression columns Use to determine which indexes should be compressed


ANALYZEVALIDATE STRUCTURE ONLINE Introduced in Oracle 9.0.1 Syntax is

ANALYZE INDEX <index_name> VALIDATE STRUCTURE ONLINE;

ONLINE option only checks structure does not require TM (DML) lock does not populate INDEX_STATS or INDEX_HISTOGRAM


DBMS_STATS

10

© 2010 Julian Dyke juliandyke.com

DBMS_STATSIntroduction PL/SQL supported package

Introduced in Oracle 8.1.5 In Oracle 11.1 there are 107 subroutines

Defined in $ORACLE_HOME/rdbms/admin/dbmsstat.sql

Advantages Parallel statistics collection Partition-awareness

Disadvantages Cannot perform structure validation Cannot detect chaining / migration


DBMS_STATSGathering Statistics Statistics can be gathered using the following subroutines:

GATHER_DATABASE_STATS GATHER_SCHEMA_STATS GATHER_TABLE_STATS GATHER_INDEX_STATS

These routines allow you to specify: Sample size Granularity (partitioned tables only) Histogram policy Cursor invalidation policy

12


DBMS_STATSDeleting Statistics Statistics can be deleted using the following subroutines:

DELETE_DATABASE_STATS DELETE_SCHEMA_STATS DELETE_TABLE_STATS DELETE_INDEX_STATS DELETE_COLUMN_STATS

DELETE_TABLE_STATS includes CASCADE_PART - optionally delete partition statistics CASCADE_COLUMNS - optionally delete column statistics CASCADE_INDEXES - optionally delete index statistics

DELETE_INDEX_STATS, DELETE_COLUMN_STATS include: CASCADE_PART - optionally delete partition statistics

13


DBMS_STATSStatistics Tables Used to

export / import statistics store object statistics values PRIOR to GATHER operations

To create a statistics table use CREATE_STATS_TABLE. For example:

dbms_stats.create_stats_table (

ownname => 'SYSTEM',tabname => 'STATS1',tblspace=> 'SYSAUX'

);

To drop a statistics table use DROP_STATS_TABLE

14


DBMS_STATSExporting and Importing Statistics Statistics can be exported to a statistics table using:

EXPORT_COLUMN_STATS EXPORT_DATABASE_STATS EXPORT_INDEX_STATS EXPORT_SCHEMA_STATS EXPORT_TABLE_STATS

Statistics can be imported from a statistics table using: IMPORT_COLUMN_STATS IMPORT_DATABASE_STATS IMPORT_INDEX_STATS IMPORT_SCHEMA_STATS IMPORT_TABLE_STATS

To transfer the statistics to another database export and import the statistics table

15


DBMS_STATSOPTIONS parameter Specifies how to determine which objects to gather statistics for:

GATHER - gather statistics for all objects GATHER AUTO - gather statistics for all objects needing new statistics GATHER STALE - gather statistics for all objects having stale statistics GATHER EMPTY- gather statistics for all objects having no statistics LIST AUTO - list objects needing new statistics LIST STALE - list objects having stale statistics LIST EMPTY - list objects having stale statistics

The OBJLIST parameter must be specified as an OUT parameter for the LIST options

GATHER STALE and GATHER EMPTY can also return a list of objects if this parameter is specified

16


DBMS_STATSOPTIONS parameter If GATHER STALE is specified statistics will be gathered for objects where at

least 10% of rows have changed Total of INSERTS, UPDATES and DELETES since last statistics collection

In Oracle 10g percentage is fixed at 10% In Oracle 11g percentage is user-configurable

17


DBMS_STATSStale Statistics In Oracle 10.1 and above table modification statistics:

are enabled if STATISTICS_LEVEL is TYPICAL or ALL are disabled if STATISTICS_LEVEL is BASIC

are collected in the SGA

are automatically flushed to SYS.MON_MODS$ by SMON background process every 15 minutes to SYS.MON_MODS$ if a significant number of rows has changed (possibly 10%)

can be manually flushed to SYS.MON_MOD$ using: DBMS_STATS.FLUSH_DATABASE_MONITORING_INFO

are reported in DBA_TAB_MODIFICATIONS

18


DBMS_STATSStale Statistics SYS.MON_MODS$ has the following columns:

Column Name Data Type

OBJ# NUMBER

INSERTS NUMBER

UPDATES NUMBER

DELETES NUMBER

TIMESTAMP DATE

FLAGS NUMBER

DROP_SEGMENTS NUMBER

19


DBMS_STATSStale Statistics In Oracle 8i, table modification monitoring can only be enabled manually

using:

ALTER TABLE <table_name> MONITORING;ALTER TABLE <table_name> NOMONITORING;

In Oracle 9i, table modification monitoring can also be enabled using

DBMS_STATS.ALTER_SCHEMA_TAB_MONITORING ('<owner>',TRUE);DBMS_STATS.ALTER_DATABASE_TAB_MONITORING (TRUE);

Reported in DBA_TABLES.MONITORING

In Oracle 10.1 and above MONITORING clause is deprecated and ignored

20


DBMS_STATSESTIMATE_PERCENT Parameter Specified using ESTIMATE_PERCENT parameter in

GATHER_DATABASE_STATS GATHER_DICTIONARY_STATS GATHER_INDEX_STATS GATHER_SCHEMA_STATS GATHER_TABLE_STATS

Valid range is 0.000001 to 100

NULL means COMPUTE

Default (10.2) is DBMS_STATS.AUTO_SAMPLE_SIZE Oracle determines sample size automatically

Default can be changed using SET_PARAM procedure

21


DBMS_STATSGRANULARITY Parameter Only relevant for partitioned objects

ALL - gather global, partition and subpartition statistics AUTO - determine granularity based on partitioning type. DEFAULT - gather global and partition-level statistics. Now obsolete. Use

GLOBAL AND PARTITION instead GLOBAL - gather global statistics GLOBAL AND PARTITION - gather global and partition-level statistics PARTITION - gather partition-level statistics SUBPARTITION - gather subpartition-level statistics

In Oracle 10.2 default is AUTO (not DEFAULT)

22


DBMS_STATSNO_INVALIDATE parameter In earlier versions ANALYZE and DBMS_STATS gather procedures always

invalidated cursors referencing affected object

In Oracle 10.1 and above, invalidation can be controlled using NO_INVALIDATE parameter

TRUE - do not invalidate dependent cursors FALSE - invalidate dependent cursors DBMS_STATS.AUTO_INVALIDATE (default) - Oracle determines whether

to invalidate dependent cursors or not.

In Oracle 10.2 the default value can be changed using SET_PARAM procedure

Only relevant for DBMS_STATS gather procedures ANALYZE always invalidates analyzed objects

23


DBMS_STATSData Dictionary Statistics Oracle recommends that data dictionary statistics are collected in Oracle

9.0.1 and above

Subroutines are: GATHER_DICTIONARY_STATS DELETE_DICTIONARY_STATS EXPORT_DICTONARY_STATS RESTORE_DICTIONARY_STATS

In Oracle 9i data dictionary statistics should only be collected using:

dbms_stats.gather_schema_stats (ownname => 'SYS',cascade=> TRUE);

In Oracle 9i data dictionary statistics can be deleted using:

dbms_stats.delete_schema_stats (ownname => 'SYS');

See Metalink Note 245051.1

Note ANALYZE should not be used with tables owned by SYS. See Metalink Note 35272.1

24


DBMS_STATSFixed Object Statistics In Oracle 10.1 and above Oracle recommends that statistics are collected on

fixed objects These include X$ and V$ views

Subroutines include: GATHER_FIXED_OBJECTS_STATS DELETE_FIXED_OBJECTS_STATS EXPORT_FIXED_OBJECT_STATS RESTORE_FIXED_OBJECTS_STATS

For example:

dbms_stats.gather_fixed_objects_stats;

Only gather fixed object statistics after the database has been running a representative workload

See Metalink Note 272479.1

25


DBMS_STATSMETHOD_OPT Parameter Specifies histogram collection policy

Can be specified for: GATHER_DATABASE_STATS, GATHER_DICTIONARY_STATS, GATHER_SCHEMA_STATS GATHER_TABLE_STATS

Default is FOR ALL COLUMNS SIZE AUTO Uses data from COL_USAGE$ table

26


DBMS_STATSMETHOD_OPT Parameter Syntax is:

FOR ALL [ INDEXED | HIDDEN ] COLUMNS [ size_clause ]

FOR COLUMNS [ [ size_clause ] column|attribute [size_clause][, column|attribute [size_clause]..]

size_clause is:

SIZE { integer | REPEAT | AUTO | SKEWONLY }

where: integer - number of histogram buckets (1..254) REPEAT - only collect histograms for columns that already have

histograms AUTO - determine which columns need histograms automatically SKEWONLY - determine which columns need histograms based on data

distribution

27


DBMS_STATSColumn Usage Monitoring Column usage is recorded in SYS.COL_USAGE$

Column Name Data Type

OBJ# NUMBER

INTCOL# NUMBER

EQUALITY_PREDS NUMBER

EQUIJOIN_PREDS NUMBER

NONEQUIJOIN_PREDS

NUMBER

RANGE_PREDS NUMBER

LIKE_PREDS NUMBER

NULL_PREDS NUMBER

TIMESTAMP DATE COL_USAGE$ is Updated when statements are parsed Reset when statistics are gathered for table

28


DBMS_STATSHistograms Summary Do not collect histograms if not necessary

Expensive to collect

Check accuracy of histograms Watch for columns with high proportion of NULL values Watch for highly skewed columns

Height-balanced histograms Limited use if column is not highly skewed

Object oriented designs rarely work with histograms

Using histograms introduces bind variable peeking issues

29


DBMS_STATSLocking Statistics Statistics can be locked for schemas, tables, and partitions

LOCK_SCHEMA_STATS / UNLOCK_SCHEMA_STATS LOCK_TABLE_STATS / UNLOCK_TABLE_STATS LOCK_PARTITION_STATS / UNLOCK_PARTITION_STATS

Possible values for STATTYPE are NULL, DATA, CACHE and ALL

By default GATHER procedures will not overwrite locked statistics Specify FORCE => TRUE to overwrite

Statistics locking reported in STATTYPE_LOCKED column in DBA_TAB_STATISTICS DBA_IND_STATISTICS

dbms_stats.lock_table_stats (

ownname => 'USER1',tabname => 'T1',stattype => 'ALL'

);

30


DBMS_STATSDefault Parameters In Oracle 10.1 and above default parameters can be specified System-wide defaults can be set for

AUTOSTATS_TARGET can be ALL collect statistics for all objects in the database ORACLE collect statistics for all Oracle-owned objects AUTO determine which objects need new statistics

Column Name Factory Default

CASCADE DBMS_STATS.AUTO_CASCADE

DEGREE NULL

ESTIMATE_PERCENT DBMS_STATS.AUTO_SAMPLE_SIZE

METHOD_OPT FOR ALL COLUMNS SIZE AUTO

NO_INVALIDATE DBMS_STATS.AUTO_INVALIDATE

GRANULARITY AUTO

AUTOSTATS_TARGET AUTO

31


DBMS_STATSDefault Parameters To set default parameters use SET_PARAM:

dbms_stats.set_param ('ESTIMATE_PERCENT','NULL');

Sets the default to COMPUTE statistics

dbms_stats.set_param ('ESTIMATE_PERCENT',NULL);

Restores factory default (DBMS_STATS.AUTO_SAMPLE_SIZE)

dbms_stats.reset_param_defaults;

Take care with NULL values For example:

dbms_stats.set_param (pname => 'CASCADE',pval => TRUE);

To restore all factory default parameters use:

To check default parameters use GET_PARAM e.g.:

SELECT dbms_stats.get_param (pname => 'CASCADE') FROM dual;

32


ANALYZEversus

DBMS_STATS

33


ANALYZE versus DBMS_STATSDifferences ANALYZE updates the following columns in DBA_TAB_STATISTICS

EMPTY_BLOCKS AVG_SPACE

DBMS_STATS sets these columns to 0

Tables AVG_ROW_LEN differs - e.g.:

Indexes No obvious differences between ANALYZE and DBMS_STATS

Columns AVG_COL_LEN differs - e.g.:

ANALYZE 43

DBMS_STATS 39

ANALYZE 4

DBMS_STATS 5

34


ANALYZE versus DBMS_STATSDifferences ANALYZE <table> gathers statistics for related indexes

Prior to Oracle 10.1 DBMS_STATS requires CASCADE => TRUE

For empty indexes ANALYZE sets BLEVEL to 0 DBMS_STATS sets BLEVEL > 0

35


ANALYZE versus DBMS_STATSSummary Use ANALYZE to

Collect index statistics without updating data dictionary Identify candidates for index coalesce/rebuild Identify candidates for index compression Identify tables containing excessive chained rows

Use DBMS_STATS Gather statistics on tables / indexes

Do not mix ANALYZE / DBMS_STATS

36


SystemStatistics

37


System StatisticsIntroduction System statistics were introduced in Oracle 9.0.1

Subroutines are: GATHER_SYSTEM_STATS DELETE_SYSTEM_STATS GET_SYSTEM_STATS SET_SYSTEM_STATS EXPORT_SYSTEM_STATS IMPORT_SYSTEM_STATS RESTORE_SYSTEM_STATS

System statistics are stored in SYS.AUX_STATS$

38


System StatisticsWorkload versus No Workload Statistics Oracle 10.1 and above supports:

noworkload statistics - default values supplied during database creation workload statistics - collected from representative workload

Workload statistics include: CPUSPEED - CPU speed SREADTIM - Single block read time in milliseconds MREADTIM - Multi block read time in milliseconds MBRC - Multi block read count MAXTHR - Maximum I/O system throughput (parallel execution only) SLAVETHR - Average slave I/O throughput (parallel execution only)

No workload statistics include: CPUSPEEDNW - CPU speed IOSEEKTIM - IO Seek Time IOTFRSPEED - IO Transfer Speed

39


System StatisticsExample

dbms_stats.create_stat_table ('SYS','OLTP_STATS');

dbms_stats.gather_system_stats (

gathering_mode => 'INTERVAL',interval => 60, -- 60 secondsstattab => OLTP_STATS',statid => 'OLTP'

);

Gather system statistics for a typical period using:

Import system statistics into AUX_STATS$ using:

dbms_stats.import_system_stats (

stattab => OLTP_STATS',statid => 'OLTP',statown => 'SYS'

);

Create a statistics table using:

40


System StatisticsSummary Enable system statistics for single instance databases

Usually improve execution plans

Consider carefully before enabling system statistics in RAC databases System statistics are database-specific Watch for asymmetric nodes

Hardware differences Service configuration

System statistics are: Difficult to monitor Very difficult to update / remove

If exporting object statistics to another system for testing: Remember to export system statistics

41


AutomaticStatisticsCollection

42


Automatic Statistics CollectionIntroduction Oracle 10.1 and above

Statistics collected during Maintenance Window Monday - Friday 22:00 to 06:00 Saturday / Sunday All day

Note that weekend window effectively starts at 06:00 on Saturday morning

Scheduler job GATHER_STATS_JOB

Scheduler job class AUTO_TASKS_JOB_CLASS

Scheduler windows WEEKNIGHT_WINDOW WEEKEND_WINDOW

Scheduler window group MAINTENANCE_WINDOW_GROUP

43


StatisticsHistory

44


Statistics HistoryIntroduction In Oracle 10.1 and above, existing statistics are stored in the data dictionary

when new statistics are collected

Statistics can be restored using: RESTORE_DATABASE_STATS RESTORE_DICTIONARY_STATS RESTORE_FIXED_OBJECTS_STATS RESTORE_SCHEMA_STATS RESTORE_SYSTEM_STATS RESTORE_TABLE_STATS

Statistics history for tables only is reported in DBA_TAB_STATS_HISTORY

SELECT stats_update_timeFROM dba_tab_stats_historyWHERE owner = 'USER1'AND table_name = 'T1';

12-FEB-09 04.36.32.997000 PM +00:00

45


Statistics HistoryOptimizer Statistics Operations Statistics gathering operations are reported in DBA_OPTSTAT_OPERATIONS

SELECT operation, target, start_time, end_timeFROM dba_optstat_operationsORDER BY start_time;

gather_database_stats(auto) 07-FEB09 06.00.03 07-FEB-09 06.01.52gather_database_stats(auto) 09-FEB09 10.00.03 09-FEB-09 10.03.37gather_database_stats(auto) 10-FEB09 10.00.03 10-FEB-09 10.02.01

DBA_OPTSTAT_OPERATIONS includes: gather_database_stats(auto) gather_schema_stats

DBA_OPTSTAT_OPERATIONS does not include: gather_table_stats gather_index_stats

46


Statistics HistoryData Dictionary Tables Historic statistics are stored in tables created by

$ORACLE_HOME/rdbms/admin/catost.sql

Tables created are: WRI$_OPTSTAT_TAB_HISTORY WRI$_OPTSTAT_IND_HISTORY WRI$_OPTSTAT_HISTHEAD_HISTORY WRI$_OPTSTAT_HISTGRM_HISTORY WRI$_OPTSTAT_AUX_HISTORY WRI$_OPTSTAT_OPR OPTSTAT_HIST_CONTROL$

47


Statistics HistoryRetention Period To check statistics history retention period use:

SELECT dbms_stats.get_stats_history_retentionFROM dual;

Default is 31 days

To check earliest historic statistics use:

SELECT dbms_stats.get_stats_history_availabilityFROM dual;

12-JAN-09 11.17.50.176000000 PM +00:00

To set statistics history retention period to 90 days:

dbms_stats.alter_stats_history_retention(

retention => 90);

48


Statistics HistorySummary Statistics history rows contain previous values for optimizer statistics

Statistics history does not include current value SAVTIME column is time row was written ANALYZETIME column is time statistics were analyzed SAVTIME != ANALYZETIME

ANALYZE does not update statistics history

[UN]LOCK_TABLE_STATS do update statistics history

49


ManualStatistics

50


Setting Table StatisticsIntroduction Table statistics can be set manually using SET_TABLE_STATS

Values can be specified for: NUMROWS - number of rows NUMBLKS - number of blocks AVGRLEN - average row length

51


Setting Table StatisticsExample (1 of 4)

DECLAREl_numrows NUMBER;l_numblks NUMBER;l_avgrlen NUMBER;l_flags NUMBER;

l_total_blocks NUMBER;l_total_bytes NUMBER;l_unused_blocks NUMBER;l_unused_bytes NUMBER;l_last_used_extent_file_id NUMBER;l_last_used_extent_block_id NUMBER;l_last_used_block NUMBER;

52


Setting Table Statistics Example (2 of 4)

BEGINdbms_stats.get_table_stats(

ownname => 'GP',tabname => 'CAR',numrows => l_numrows,numblks => l_numblks,avgrlen => l_avgrlen,

);

l_numrows := 0;l_numblks := 0;

SELECT COUNT(*) INTO l_numrows FROM gp.car;

53



dbms_space.unused_space(

segment_owner => 'GP',segment_name => 'CAR',segment_type => 'TABLE',total_blocks => l_total_blocks,total_bytes => l_total_bytes,unused_blocks => l_unused_blocks,unused_bytes => l_unused_bytes,last_used_extent_file_id => l_last_used_extent_file_id,last_used_extent_block_id => l_last_used_extent_block_id,last_used_block => l_last_used_block

);

l_numblks := l_total_blocks;

54



dbms_stats.set_table_stats(

ownname => 'GP',tabname => 'CAR',numrows => l_numrows,numblks => l_numblks,avgrlen => l_avgrlen

);END;

55


Setting Index StatisticsIntroduction Index statistics can be set manually using SET_INDEX_STATS

Values can be specified for: NUMROWS - number of rows NUMBLKS - number of blocks NUMDIST - number of distinct values AVGLBLK - average leaf blocks per key AVGDBLK - average data blocks per key CLSTFCT - clustering factor INDLEVEL - index level (height)

Use ANALYZE INDEX VALIDATE STRUCTURE to determine values

56


Setting Column StatisticsIntroduction Column statistics can be set manually using SET_COLUMN_STATS

Values can be specified for: DISTCNT - number of distinct values DENSITY - 1 / (number of distinct values) NULLCNT - number of NULL values Low and high values Histograms

Note that DISTCNT usage differs for columns with histograms

57


Setting Column Statistics Data Types Requires additional data types declared in DBMS_STATS package See $ORACLE_HOME/rdbms/admin/dbmsstat.sql

TYPE numarray IS VARRAY(256) OF NUMBER;

TYPE chararray IS VARRAY(256) OF VARCHAR2(4000);

TYPE statrec IS RECORD(

EPC NUMBER,MINVAL RAW(2000),MAXVAL RAW(2000),BKVALS NUMARRAY,NOVALS NUMARRAY,CHVALS CHARARRAY,EAVS NUMBER

)

58


Setting Column Statistics Example (1 of 3)

PROCEDURE copy_col_stats

(p_owner VARCHAR2,p_table_name VARCHAR2) ISl_row DBA_TAB_COLUMNS%ROWTYPE;l_statrec dbms_stats.statrec;l_numvals dbms_stats.numarray := dbms_stats.numarray();l_charvals dbms_stats.chararray := dbms_stats.chararray();l_datevals dbms_stats.datearray := dbms_stats.datearray();

CURSOR c1 RETURN DBA_TAB_COLUMNS%ROWTYPE ISSELECT *FROM dba_tab_columnsWHERE owner = p_ownerAND table_name = p_table_name;

BEGINl_numvals.extend(2);l_charvals.extend(2);l_datevals.extend(2);

OPEN c1;LOOP

FETCH c1 INTO l_row;EXIT WHEN c1%NOTFOUND;

59


Setting Column Statistics Example (2 of 3)

IF l_row.num_buckets IS NOT NULL THENl_statrec.epc := 2;l_statrec.bkvals := NULL;l_numvals(1) := NULL; l_numvals(2) := NULL;l_charvals(1) := NULL; l_charvals(2) := NULL;l_datevals(1) := NULL; l_datevals(2) := NULL;

IF l_row.column_name = 'COL1' THENl_numvals(1) := get_min_col1 (p_owner,p_table_name);l_numvals(2) := get_max_col2 (p_owner,p_table_name);DBMS_STATS.PREPARE_COLUMN_VALUES (l_statrec,l_numvals);

ELSIF l_row.column_name = 'COL2' THENl_datevals(1) := TO_DATE (v_target_date,'YYYYMMDD');l_datevals(2) := TO_DATE (v_target_date,'YYYYMMDD');DBMS_STATS.PREPARE_COLUMN_VALUES (l_statrec,l_datevals);

ELSIF l_row.column_name = 'COL3' THENl_charvals(1) := get_min_col3 (p_owner,p_table_name);l_charvals(2) := get_max_col3 (p_owner,p_table_name);DBMS_STATS.PREPARE_COLUMN_VALUES (l_statrec,l_charvals);

END IF;

60


Setting Column Statistics Example (3 of 3) Setting minimum and maximum values (without histograms)

DBMS_STATS.SET_COLUMN_STATS(

ownname => p_owner,tabname => p_table_name,colname => l_row.column_name,distcnt => l_row.num_distinct,density => l_row.density,nullcnt => l_row.num_nulls,srec => l_statrec,avgclen => l_row.avg_col_len

);END IF;END LOOP;CLOSE c1;

END;

61


Setting Column Statistics - HistogramsExample (1 of 2)

DECLAREl_statrec dbms_stats.statrec;l_charvals dbms_stats.chararray := dbms_stats.chararray ();l_bkvals dbms_stats.numarray := dbms_stats.numarray ();

BEGINl_charvals.extend (11);l_bkvals.extend (11);

l_charvals(1) := 'MSCH'; l_bkvals(1) := 91;l_charvals(2) := 'APRO'; l_bkvals(2) := 51;l_charvals(3) := 'ASEN'; l_bkvals(3) := 41;l_charvals(4) := 'NMAN'; l_bkvals(4) := 31;l_charvals(5) := 'JSTE'; l_bkvals(5) := 27;l_charvals(6) := 'NLAU'; l_bkvals(6) := 25;l_charvals(7) := 'JCLA'; l_bkvals(7) := 25;l_charvals(8) := 'NPIQ'; l_bkvals(8) := 23;l_charvals(9) := 'FALO'; l_bkvals(9) := 22;l_charvals(10) := 'DHIL'; l_bkvals(10) := 22;l_charvals(11) := 'MHAK'; l_bkvals(11) := 20;

62


Setting Column Statistics - HistogramsExample (2 of 2)

l_statrec.epc := 11;l_statrec.bkvals := l_bkvals;l_statrec.eavs := 0;

DBMS_STATS.PREPARE_COLUMN_VALUES (l_statrec,l_charvals);

DBMS_STATS.SET_COLUMN_STATS(

ownname => 'GP',tabname => 'CAR',colname => 'DRIVER_KEY',distcnt => 11,density => 0.00210084, -- 1 / 476 rows

nullcnt => 0,srec => l_statrec,avgclen => 4

);END;/

63


Partition Statistics

64


Partition Statistics Introduction Partition statistics are reported by:

DBA_TABLES DBA_TAB_PARTITIONS DBA_TAB_STATISTICS DBA_TAB_SUBPARTITIONS (Oracle 10.1 and above)

Indexes DBA_INDEXES DBA_IND_PARTITIONS DBA_IND_STATISTICS DBA_IND_SUBPARTITIONS (Oracle 10.1 and above)

Columns DBA_TAB_COL_STATISTICS DBA_PART_COL_STATISTICS DBA_SUBPART_COL_STATISTICS

Histograms DBA_TAB_HISTOGRAMS DBA_PART_HISTOGRAMS DBA_SUBPART_HISTOGRAMS

65


Partition StatisticsExample (1 of 12) Create a range partitioned table containing data for last four seasons

CREATE TABLE car3PARTITION BY RANGE (season_key)(

PARTITION p2005 VALUES LESS THAN ('2006'),PARTITION p2006 VALUES LESS THAN ('2007'),PARTITION p2007 VALUES LESS THAN ('2008'),PARTITION p2008 VALUES LESS THAN ('2009')

)AS SELECT * FROM carWHERE season_key >= '2005';

CREATE INDEX car3_i1 ON car3(season_key,race_key,position) LOCAL;

dbms_stats.gather_table_stats(

ownname => 'GP',tabname => 'CAR3',estimate_percent => NULL,cascade => TRUE

);

66


Partition StatisticsExample (2 of 12) Table Statistics

SELECT num_rows,blocks,avg_row_len FROM dba_tablesWHERE owner = 'GP' AND table_name = 'CAR3';

SELECT partition_name,num_rows,blocks,avg_row_len FROM dba_tab_partitionsWHERE table_owner = 'GP' AND table_name = 'CAR3';

NUM_ROWS BLOCKS AVG_ROW_LEN

1518 24 37

PARTITION_NAME NUM_ROWS BLOCKS AVG_ROW_LEN

P2005 380 6 36

P2006 396 6 38

P2007 374 6 37

P2008 368 6 37

67


Partition StatisticsExample (3 of 12) Index Statistics

SELECT blevel, leaf_blocks, distinct_keys, clustering_factor, num_rowsFROM dba_indexesWHERE owner = 'GP' AND index_name = 'CAR3_I1';

BLEVEL LEAF_BLOCKS DIST KEYS CLUFAC NUM_ROWS

1 8 1518 15 1518

PARTITION_NAME BLEVEL LEAF_BLOCKS DIST KEYS CLUFAC NUM_ROWS

P2005 1 2 380 3 380

P2006 1 2 396 3 396

P2007 1 2 374 5 374

P2008 1 2 368 4 368

SELECT partition_name, blevel, leaf_blocks, distinct_keys, clustering_factor, num_rows

FROM dba_ind_partitionsWHERE index_owner = 'GP' AND index_name = 'CAR3_I1';

68


Partition StatisticsExample (4 of 12) Add a new partition for 2009 season

ALTER TABLE car3ADD PARTITION p2009 VALUES LESS THAN ('2010');

dbms_stats.gather_table_stats(

ownname => 'GP',tabname => 'CAR3',estimate_percent => NULL,cascade => TRUE

);

Gather statistics again

69






1518 24 37


P2005 380 6 36

P2006 396 6 38

P2007 374 6 37

P2008 368 6 37

P2009 0 0 0

70





1 8 1518 15 1518


P2005 1 2 380 3 380

P2006 1 2 396 3 396

P2007 1 2 374 5 374

P2008 1 2 368 4 368

P2009 0 0 0 0 0



71


Partition StatisticsExample (7 of 12) Copy table statistics from 2008 to 2009 partitions

dbms_stats.copy_table_stats(

ownname => 'GP',tabname => 'CAR3',srcpartname => 'P2008',dstpartname => 'P2009'

);

72






1518 24 37


P2005 380 6 36

P2006 396 6 38

P2007 374 6 37

P2008 368 6 37

P2009 368 6 37

73





1 8 1518 15 1518


P2005 1 2 380 3 380

P2006 1 2 396 3 396

P2007 1 2 374 5 374

P2008 1 2 368 4 368

P2009 1 2 368 4 368



74


Partition StatisticsExample (10 of 12) Copy rows from 2008 to 2009. For example:

INSERT INTO car3SELECT

'2009',race_key,driver_key,team_key,engine_key,position,laps_completed,classification_key,notes,driver_points,team_points

FROM gp.car3WHERE season_key = '2008';

Statistics are unchanged

75


Partition StatisticsExample (11 of 12) Execution plans for statement against 2008 and 2009 partitions:

SELECT SUM(team_points) FROM gp.car3WHERE season_key = '2009';

0 SELECT STATEMENT1 0 SORT AGGREGATE2 1 PARTITION RANGE (SINGLE)3 2 TABLE ACCESS (BY INDEX ROWID) OF 'CAR3'4 3 INDEX (RANGE SCAN) OF 'CAR3_I1'

SELECT SUM(team_points) FROM gp.car3WHERE season_key = '2008';

0 SELECT STATEMENT1 0 SORT AGGREGATE2 1 PARTITION RANGE (SINGLE)3 2 TABLE ACCESS (FULL) OF 'CAR3'

Plans are different even though statistics and data are theoretically identical

76


Partition StatisticsExample (12 of 12) In Oracle 10.2.0.4 column values are not updated by COPY_TABLE_STATS

SELECT partition_name,low_value,high_valueFROM dba_part_col_statisticsWHERE owner = 'GP'AND table_name = 'CAR3'AND column_name = 'SEASON_KEY';

PARTITION_NAME LOW_VALUE HIGH_VALUE

P2005 0x32303035 (2005)

0x32303035 (2005)

P2006 0x32303036 (2006)

0x32303036 (2006)

P2007 0x32303037 (2007)

0x32303037 (2007)

P2008 0x32303038 (2008)

0x32303038 (2008)

P2009 0x32303038 (2008)

0x32303038 (2008)

Caused by bug 5643297 - only affects 10.2.0.4 Fixed in patch 7381308

includes fixes for COPY_TABLE_STATS and APPROX_GLOBAL

77


Partition StatisticsSummary Range-partitioning

Only collect statistics for partitions undergoing change Specify partition names and GRANULARITY parameter to restrict

partitions that are analyzed

Consider setting statistics manually for new partitions COPY_STATS has limited functionality Watch for changing high values

Use DBMS_STATS NO_INVALIDATE option ANALYZE invalidates all cursors referencing table being analyzed Can result in hard-parse rates approaching 100%

78


Oracle 11gEnhancements

79


StatisticsPreferences

80


Oracle 11g EnhancementsStatistics Preferences In Oracle 10g preferences are:

set using SET_PARAM reported by GET_PARAM stored in SYS.OPTSTAT_HIST_CONTROL$

In Oracle 11.1 and above preferences are: set using:

SET_TABLE_PREFS SET_SCHEMA_PREFS SET_DATABASE_PREFS SET_GLOBAL_PREFS

reported by GET_PREFS DBA_TAB_STAT_PREFS

stored in SYS.OPTSTAT_USER_PREFS$

In Oracle 11.1 and above global preferences can be reset using RESET_GLOBAL_PREF_DEFAULTS

81


Oracle 11g EnhancementsStatistics Preferences

In Oracle 10g preferences can be set for: CASCADE DEGREE ESTIMATE_PERCENT METHOD_OPT NO_INVALIDATE GRANULARITY

In Oracle 11g preferences can also be set for: PUBLISH - if TRUE publish statistics; if FALSE store as pending STALE_PERCENT - threshold level at which statistics considered stale INCREMENTAL - if TRUE collect incremental global partition statistics

Allow automatic statistics collection to be customized for individual objects For example heavily skewed data distributions

Can be set at table, schema, database and global level

82


Oracle 11g EnhancementsStatistics Preferences Delete preferences using:

DELETE_TABLE_PREFS DELETE_SCHEMA_PREFS DELETE_DATABASE_PREFS

Export and import preferences using: EXPORT_TABLE_PREFS / IMPORT_TABLE_PREFS EXPORT_SCHEMA_PREFS / IMPORT_SCHEMA_PREFS EXPORT_DATABASE_PREFS / IMPORT_DATABASE_PREFS

83


PendingStatistics

84


Oracle 11g EnhancementsPending Statistics In Oracle 11.1 and above statistics can be

Published - current statistics used by optimizer Pending - private statistics not yet published

In Oracle 11.1 statistics for a set of objects can be collected as an atomic transaction

Published at same time Avoids inconsistencies

PUBLISH mode can be set at the following levels Global Database Schema Table

85


Oracle 11g EnhancementsPending Statistics To defer publication of all statistics collected by USER1 use:

ALTER SESSION optimizer_use_pending_statistics = TRUE;

To make pending statistics permanently available to the optimizer use:

dbms_stats.publish_pending_statistics(

ownname => 'USER1',tabname => NULL

);

Statistics can be published at database, schema or table level

To delete pending statistics use DELETE_PENDING_STATS

To export pending statistics use EXPORT_PENDING_STATS There is no corresponding import procedure

To make pending statistics temporarily available to the optimizer use:

dbms_stats.set_schema_prefs ('USER1','PUBLISH','FALSE');

86


ExtendedStatistics

87


Oracle 11g EnhancementsExtended Statistics Extended statistics attempt to reflect true selectivity of data

There are two types: multi-column statistics expression statistics

Extended statistics use the following procedures: CREATE_EXTENDED_STATS DROP_EXTENDED_STATS SHOW_EXTENDED_STATS_NAME

88


Oracle 11g Enhancements Multi-Column Statistics In Oracle 11g the following statistics can be gathered on multiple columns in a

table (column groups):

Number of distinct values Density Number of nulls Frequency histograms

Multicolumn statistics Appear to work for frequency histograms

<= 254 combinations of values Do not appear to work for height-based histograms

> 254 combinations of values

89


Oracle 11g Enhancements Multi-Column Statistics

SELECT COUNT(*) FROM gp.carWHERE team_key = 'FER'AND engine_key = 'FER';

COUNT(*)1410

Id

Operation Name Rows Bytes Cost (%CPU)

Time

0 SELECT STATEMENT 35 (100)

1 SORT AGGREGATE 1 13

2 TABLE ACCESS FULL CAR 137 1781 35 (0) 00:00:01

Incorrect Cardinality

90



BEGINdbms_stats.gather_table_stats(

ownname => 'GP',tabname => 'CAR',estimate_percent => NULL,method_opt => 'FOR ALL COLUMNS SIZE 254 '||'FOR COLUMNS (TEAM_KEY,ENGINE_KEY) SIZE 254'

);END;

DECLAREl_extension_name VARCHAR2(30);

BEGINl_extension_name := dbms_stats.create_extended_stats(

ownname => 'GP',tabname => 'CAR',extension => '(team_key,engine_key)'

);END;

91



Id


Time




SELECT COUNT(*) FROM gp.carWHERE team_key = 'FER'AND engine_key = 'FER';

COUNT(*)1410

Correct Cardinality

92



Extension Name Extension

SYS_STUWIHETSRHT#5P210Z$AO1ON9

("TEAM_KEY","ENGINE_KEY")

SELECT extension_name,extensionFROM dba_stat_extensionsWHERE owner = 'GP'AND table_name = 'CAR';

93



INTCOL#

COL# NAME

1 1 SEASON_KEY

2 2 RACE_KEY

3 3 DRIVER_KEY

4 4 TEAM_KEY

5 5 ENGINE_KEY

6 6 POSITION

7 7 LAPS_COMPLETED

8 8 CLASSIFICATION_KEY

9 9 NOTES

10 0 SYS_STUWIHETSRHT#5P210Z$AO1ON9

SELECT c.intcol#,c.col#,c.nameFROM sys.col$ c, sys.obj$ o, sys.user$ uWHERE c.obj# = o.obj#AND o.name = 'CAR'AND o.owner# = u.user#AND u.name = 'GP';

New Virtual Column in Tablefor Multi-Column

Statistics

94


Oracle 11g Enhancements Expression Statistics

CREATE OR REPLACE FUNCTION points(season_key VARCHAR2,race_key NUMBER,position NUMBER)RETURN NUMBERDETERMINISTIC IS

l_result NUMBER;BEGIN

l_result := CASE positionWHEN 1 THEN 10WHEN 2 THEN 8WHEN 3 THEN 6WHEN 4 THEN 5WHEN 5 THEN 4WHEN 6 THEN 3WHEN 7 THEN 2WHEN 8 THEN 1ELSE 0

END;RETURN l_result;

END;

Note: The real algorithm is MUCH more complicated for various reasons including:

• changes to number of points awarded

• disqualifications, penalties etc

• differences between drivers and team championship points systems

95



SELECT COUNT(*) FROM gp.car WHERE POINTS (season_key,race_key,position) = 10;

COUNT(*)709

Id


Time




Incorrect Cardinality

96



BEGINdbms_stats.gather_table_stats(

ownname => 'GP',tabname => 'CAR',estimate_percent => NULL,method_opt => 'FOR ALL COLUMNS SIZE 254 '||'FOR COLUMNS (points(season_key,race_key,position)) SIZE 254'

);END;

DECLAREl_extension_name VARCHAR2(30);

BEGINl_extension_name := dbms_stats.create_extended_stats(

ownname => 'GP',tabname => 'CAR',extension => '(points(season_key,race_key,position))'

);END;

97



SELECT COUNT(*) FROM gp.car WHERE POINTS (season_key,race_key,position) = 10;

COUNT(*)709

Id


Time




Correct Cardinality

98


Oracle 11g Enhancements Incremental Statistics For partitioned tables statistics can be collected:

For sub-partitions For partitions Globally

Global statistics should reflect underlying partitioned statistics

Prior to Oracle 11.1 calculation of global statistics required full table scan of each partition in table

For many applications using range partitioning only the most recent partition is subject to change

Older partitions contain historical data

99


Oracle 11g Enhancements Incremental Statistics For example:

CREATE TABLE car4PARTITION BY RANGE (season_key)(

PARTITION p2006 VALUES LESS THAN ('2005'),PARTITION p2007 VALUES LESS THAN ('2006'),PARTITION p2008 VALUES LESS THAN ('2007'),PARTITION p2009 VALUES LESS THAN ('2008')

)AS SELECT * FROM carWHERE season_key >= '2005';

CREATE INDEX i_car4_1 ON carp(season_key,race_key,position) LOCAL;

INSERT INTO car4 SELECT * FROM car WHERE season_key BETWEEN 2006 AND 2009;

Create table with

partitions for four years

data

Insert data

100


Oracle 11g Enhancements Incremental Statistics Gather statistics

DBMS_STATS.GATHER_TABLE_STATS(

ownname => 'GP',tabname => 'CAR4'

);

Requires full table scan for each partition

Full Table Scan2006

Full Table Scan2007

Full Table Scan2008

Full Table Scan2009

101


Oracle 11g Enhancements Incremental Statistics Update data in one partition:



);

Still requires full table scan for each partition

Full Table Scan2006

Full Table Scan2007

Full Table Scan2008

Full Table Scan2009

UPDATE car4 SET team_points = driver_pointsWHERE season_key = '2009';

Gather table statistics again:

102


Oracle 11g Enhancements Incremental Statistics In Oracle 11.1 and above statistics can be gathered incrementally for

partitioned tables

To gather incremental statistics on a specific table INCREMENTAL and PUBLISH preferences for table must be TRUE ESTIMATE_PERCENT must be AUTO_SAMPLE_SIZE GRANULARITY must be AUTO

For each partition a synopsis is created Contains data about distinct values for each column in partition Stored in

SYS.WRI$_OPTSTAT_SYNOPSIS_HEAD$ SYS.WRI$_OPTSTAT_SYNOPSIS$

If a partition has not been modified synopsis can be used to calculate global statistics

Synopsis must be generated for all partitions first time statistics are gathered after incremental statistics are enabled

103


Oracle 11g Enhancements Incremental Statistics Set INCREMENTAL preference to TRUE for table:

BEGINDBMS_STATS.SET_TABLE_PREFS(

ownname => 'GP',tabname => 'CAR4',pname => 'INCREMENTAL',pvalue => 'TRUE'

);END;/

Gather table statistics again to generate synopsis for each partition



);

All partitions will be scanned first

time to create synopsis

104


Oracle 11g Enhancements Incremental Statistics Update data in one partition:


ownname => 'GP',tabname => 'CARP'

);

Uses synopsis for unmodified partitions Requires full table scan for modified partition

2006

2007

2008

Full Table Scan2009

UPDATE car4 SET team_points = driver_pointsWHERE season_key = '2009';

Gather table statistics again:

Synopsis

Synopsis

Synopsis

105


Thanks to the following for corrections:

[email protected]

Greg Rahn Tony Hasler

1 © 2010 Julian Dyke juliandyke.com Optimizer Statistics Julian Dyke Independent Consultant Web...

Documents

Transcript of 1 © 2010 Julian Dyke juliandyke.com Optimizer Statistics Julian Dyke Independent Consultant Web...