Real-World Performance Training · –SQL Design –Physical Layout •Table layout, partitions,...

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |

Real-World Performance TrainingSQL Performance

Real-World Performance Team


Agenda

The Optimizer

Optimizer Inputs

Optimizer Output

Advanced Optimizer Behavior

Why is my SQL slow?

Optimizer Edges Cases and Top SQL Mistakes

1

2

3

4

5

6


TheOptimizer


The Optimizer Engine

Inputs

• SQL

• Physical Design

– Indexes

– Partitioning

The Rule Based Optimizer—determines execution plan based on rules

Execution Plan

• Index access if index is present

• Join type based on how data is stored

The rule based optimizer used a set of rules and rankings to determine the execution plan based on the SQL statement and how data was stored

Not available in Oracle 11 or 12


The Cost Based Optimizer

• What would we need to know?

– SQL Design

– Physical Layout• Table layout, partitions, indexes, constraints

– Data Layout = Statistics• How many rows in the table?

• How big is the table?

• For each column how many values?

• How good is the system at table scanning?

• If we are joining how many rows will one row on one table get from the join?

Introduces Data Awareness


The Cost Based Optimizer

• System Statistics

– Performance information for the system

– IO and CPU capability

• Table Statistics– Reflect a point in time view of the data

– Static

• Dynamic Statistics

– Run time sampling of the data

Different Types of Statistics



Inputs

• SQL

• Schema Design

• Physical Design

• Statistics

– Table

– Column

– Dynamic Statistics

– System Statistics

The cost based optimizer—determines execution plan based on inputs

Execution Plan

• Access Method

• Join Method

• Join Order

• Distribution Method


Agenda

The Optimizer

Optimizer Inputs

Optimizer Output


Why is my SQL slow?


1

3

4

5

6

2


Optimizer InputsSchema Design

Third Normal Dimensional Other


Optimizer Inputs

The design of the SQL statement and the operators used have a considerable impact on performance

• intersect

• minus

• exists

• not exists

• window functions

• multi-table inserts

• outer joins

• distinct vs group by

SQL Design


Optimizer Inputs

The physical design of the system determines which optimizations are available during execution

• Indexes—index access vs full table scans

• Partitioning—partition pruning and partition wise joins

Physical Design


Statistics

• The statistics contain information which the optimizer uses when developing the execution plan

• They do not change unless they are recollected

• Statistics are kept at different levels

– Table

– Partition

– Column

– Index

Basics


Statistics

Statistic Description

Rows Number of rows in the object

Blocks Number of blocks in the object

Empty blocks Number of empty blocks in the object

Avg Row Length Average Row Length in the object

Main Table and Partition Statistics

15


Statistics

Statistic Description

NDV Number of distinct values

Low Value Low value in column

High Value High value in column

Density Shows distribution of values

Num Nulls Number of nulls in the column

Main Column Statistics

16


Gathering Statistics

• “I cannot collect statistics it takes too long”

• “If I collect statistics things might change”

• “What sample size should I use?”

• “I don’t need to bother because Oracle automatically collects statistics every night”

Problems



• Use the DBMS_STATS procedure; do not use ANALYZE

• DBMS_STATS changes between releases and patches

• Use AUTO_SAMPLE_SIZE

– This uses a faster and more accurate NDV algorithm

– Allows the use of Incremental Statistics

– Allows the use of Concurrent Statistics

– Allows the use of new histogram types

Guidelines


Statistics Gathering Performance

• Auto Sample Size

– Fast and Accurate

• Incremental Stats

• Concurrent Stats

• Gather statistics on bulk loads

– New in 12c—gathers stats on a bulk load into an empty table

–On by default and can be controlled with the GATHER_OPTIMIZER_STATISTICSand NO_GATHER_OPTIMIZER_STATISTICS hints

–More for convenience than performance



• Partitioned tables typically need both partition level and global statistics

– For queries that prune to a single partition, the partition level stats are used

– For queries that prune to more than one partition, the global level stats are used

• With incremental stats– The statistics for a newly loaded partition are gathered and recorded in a synopsis

– Global level stats are computed from the synopsis

• 12c supports incremental stats on non-partitioned tables

– Creates synopsis on table to facilitate partition exchange

Incremental Stats


Incremental Stats


Object Column Values NDV

Partition #1 1,3,3,4,5 4

Partition #2 2,3,4,5,6 5


Incremental Stats



Partition #1 1,3,3,4,5 4

Partition #2 2,3,4,5,6 5

NDV by Addition WRONG 9


Incremental Stats



Partition #1 1,3,3,4,5 4

Partition #2 2,3,4,5,6 5

NDV by Addition WRONG 9

NDV by Synopsis CORRECT 6


Incremental Stats


• Comparison of Incremental vsnon-incremental stats• Simulates a daily load over 60

days• 1,000,000 Rows Per Day• 60 Partitions



Begin

DBMS_STATS.SET_TABLE_PREFS(USER, 'TABLE NAME',

'INCREMENTAL','TRUE');

End;

/

Incremental Stats



• By default, stats are gathered sequentially for each partition of a partitioned table

• Concurrent stats enables partition statistics to be gathered concurrently

– Enabled through job scheduler

– Stats jobs only run on one partitioned table at a time

– Full details:https://blogs.oracle.com/optimizer/entry/gathering_optimizer_statistics_is_one

Concurrent Stats

https://blogs.oracle.com/optimizer/entry/gathering_optimizer_statistics_is_one


Concurrent Stats


Job1Table 1Global Stats



Job3Table 3

Coord Job

Job3.1Table 3

Partition 1 Stats

Job3.2Table 3

Partition 2 Stats

Job3.3Table 3

Partition 3 Stats

Gather Schema Stats



Begin

DBMS_STATS.SET_GLOBAL_PREFS('CONCURRENT','TRUE');

End;

/

– Number of jobs limited by job_queue_processes

– Coordinate job_queue_processes and parallel setting on stats job

Concurrent Stats


Dynamic Statistics

• Previously called dynamic sampling

• Statistics gathered at parse time to augment the existing statistics

• Always invoked for parallel execution

• Used when regular statistics are not sufficient to get good quality cardinality estimates.

• Table level stats prior to 12—enhanced in 12c to include joins and group-by predicates

• Controlled by the OPTIMIZER_DYNAMIC_SAMPLING parameter

• Dynamic statistics can greatly enhance the quality of statistics and lead to better execution plans

• But there is a cost—better statistics are usually derived with larger sample sizes leading to increased parse times


System Statistics

1. Gather system statistics– “System statistics describe the system's hardware characteristics, such as I/O and CPU

performance”– Recommended in the Upgrade Guide• https://mikedietrichde.com/2010/03/09/gathering-workload-statistics

2. Set Exadata statistics– Exadata specific– MOS Note 1274318.1

3. Do not gather or set system statistics– RWP Recommendation– System statistics add another variable for the optimizer to consider, which can lead to

suboptimal plans

System Statistics provide the Optimizer information about the performance capabilities of the server CPU and IO

https://mikedietrichde.com/2010/03/09/gathering-workload-statistics


Agenda

The Optimizer

Optimizer Inputs

Optimizer Output


Why is my SQL slow?


1

3

4

5

6

2



Inputs

• SQL

• Schema Design

• Statistics

– Table

– Column



Execution Plan

• Access Method

• Join Method

• Join Order



The Execution Plan

• The optimizer takes the inputs and creates an Execution Plan, which contains the detailed steps the system will use to execute the SQL statement

• The quality of the execution plan is dependent on the quality of the inputs

– Poor Quality Inputs = Poor Execution Plan

– Good Quality Inputs = Good Execution Plan

A product of the Inputs


The Execution Plan

• Each line in an execution plan is a row source which produces rows and sends them to the next row in the plan

• The optimizer estimates how many rows will be produced by a row source and associates a cost with the operation

• The optimizer evaluates multiple execution plans and determines their cost

• The optimizer picks the plan with the lowest cost

Determining the execution Plan


Execution Plan

Execution Plan Shows the detailed steps used to execute a SQL statement

Row Source Execution plan operators that produce and/or consume rows

Access method The way in which the data is accessed

Join method The method used to join tables

Join order The order in which the tables are joined

Cardinality Number of rows produced by a row source

Basic Components

35


How to see the Execution Plan

• DBMS_XPLAN.DISPLAY_* procedures

• SQL Monitor Report

• SQL Trace

• SQL*Plus Auto Trace

• Explain Plan for… shows the expected execution plan which may be different from the actual execution plan


Execution Plan

Gives the actual execution plan from various sources

• Cursor cache– dbms_xplan.display_cursor

• AWR– dbms_xplan.display_awr

• SQL Tuning Set– dbms_xplan.display_sqlset

DBMS_XPLAN.DISPLAY_*


Execution Plan

Plan hash value: 2267213921

----------------------------------------------------------------------------------------

| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |

----------------------------------------------------------------------------------------

| 0 | SELECT STATEMENT | | | | 187K(100)| |

| 1 | SORT AGGREGATE | | 1 | 14 | | |

|* 2 | TABLE ACCESS STORAGE FULL| LINEORDER | 19951 | 272K| 187K (1)| 00:00:08 |

----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------

2 - storage(("LO_TAX"=0 AND "LO_ORDERDATE">=TO_DATE(' 1997-07-01 00:00:00',

'syyyy-mm-dd hh24:mi:ss') AND "LO_ORDERDATE"<=TO_DATE(' 1997-07-07 00:00:00',

'syyyy-mm-dd hh24:mi:ss')))

filter(("LO_TAX"=0 AND "LO_ORDERDATE">=TO_DATE(' 1997-07-01 00:00:00',

'syyyy-mm-dd hh24:mi:ss') AND "LO_ORDERDATE"<=TO_DATE(' 1997-07-07 00:00:00',

'syyyy-mm-dd hh24:mi:ss')))

DBMS_XPLAN.DISPLAY_CURSOR Output

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |7/16/2018

SQL Monitor Report

Execution Plan


An Example—You As The Optimizer


You As The Optimizer

• If a query retrieves 10 rows from a 50 million row table, what is the correct access method?

– INDEX

• If a query retrieves 40 million rows from a 50 million row table, what is the correct access method?

– TABLE SCAN

• Did you make this decision based on rules, or an estimate of cost?



• In the previous example we were given the number of rows to be retrieved.

• But usually the optimizer is not given this information, so let’s look at a query and discuss how the optimizer decides which access method to use

• The query will ask

What is the total quantity of orders in the week of July 4th 1997sold with a zero tax rate?

Determining the Access Method


SELECT SUM(lo_quantity)

FROM lineorder

WHERE lo_orderdate

BETWEEN to_date('19970701','YYYYMMDD')

AND to_date('19970707','YYYYMMDD')

AND lo_tax = 0 ;

You As The OptimizerWhat information do you need to determine the access method?



FROM lineorder

WHERE lo_orderdate



AND lo_tax = 0 ;

• Number of rows in the table

• Existence of indexes

• If the table is partitioned, how many partitions will need to be accessed

• Estimate of the number of rows to be retrieved from the table– NDV—Number of distinct values in a

column– How selective are the filter columns?• If there are lots of distinct values the filter

predicates will probably have fewer rows• If there are few distinct values the filter

predicates will probably have more rows

You As The OptimizerWhat information do you need to determine the access method?



FROM lineorder

WHERE lo_orderdate



AND lo_tax = 0 ;

You As The OptimizerDetermining the Access Method

lineorder rows 53,986,608

lineorder partitioned no

lo_orderdate indexed yes, btree

lo_tax indexed no

lo_orderdate NDV 2406

lo_tax NDV 9



--------------------------------------------------------------------------------------------------


--------------------------------------------------------------------------------------------------

| 0 | SELECT STATEMENT | | 1 | 14 | 47908 (1)| 00:00:02 |

| 1 | SORT AGGREGATE | | 1 | 14 | | |

|* 2 | TABLE ACCESS BY INDEX ROWID BATCHED| LINEORDER | 19951 | 272K| 47908 (1)| 00:00:02 |

|* 3 | INDEX RANGE SCAN | LO_DATE_N | 179K| | 480 (1)| 00:00:01 |

--------------------------------------------------------------------------------------------------

Determining the Access Method—Index Scan



--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

| 0 | SELECT STATEMENT | | 1 | 14 | 187K (1)| 00:00:08 |

| 1 | SORT AGGREGATE | | 1 | 14 | | |

|* 2 | TABLE ACCESS FULL| LINEORDER | 19951 | 272K| 187K (1)| 00:00:08 |

--------------------------------------------------------------------------------

Determining the Access Method—Table Scan



Access Method Cost

Index Scan 47,908

Table Scan 187,519

Determining the Access Method

48



• Now let’s look at a query that has more than one table

• We will ask the same question but instead of putting date predicates directly on the lineorder table, we join the lineorder table to date_dim and we specify our date range using date_dim

What is the total quantity of orders in the week of July 4th 1997sold with a zero tax rate?

Determining the Join Order and Method



FROM lineorder

JOIN date_dim

ON lo_orderdate = d_datekey

WHERE d_year = 1997

AND d_weeknuminyear = 27

AND lo_tax = 0

;

You As The OptimizerDetermining the Join Order and Method



• Consider the join order– If we retrieve X number of rows from lineorder, how many rows will we then

retrieve from date_dim

– And vice versa, if we retrieve X number of rows from date_dim, how many rows will we then retrieve from lineorders

– How many rows from each table overlap or join?




Table Statistic Value

lineorder rows 53,986,608

date_dim rows 2,556

lineorder lo_orderdate low 01-JAN-92

lineorder lo_orderdate high 02-AUG-98

date_dim d_datekey low 31-DEC-91

date_dim d_datekey high 29-DEC-98


52



Table Statistic Value

lineorder lo_orderdate NDV 2,406

lineorder lo_tax NDV 9

date_dim d_datekey NDV 2,556

date_dim d_year NDV 8

date_dim d_weeknuminyear NDV 53


53



• For each order, we then consider which join method to use

– Nested Loop

– Hash

–Merge




• Nested loops joins work best when retrieving a small number of rows for the join condition

• A nested loop join involves the following steps:

1. The optimizer determines the driving table and designates it as the outer table.

2. The other table is designated as the inner table.

3. For every row in the outer table, retrieve matching rows in the inner table.– If 10 rows are retrieved from the outer table we need to perform 10 lookups in the inner table

– If 10M rows are retrieved from the outer table we need to perform 10M lookups in the inner table

– These lookups can be performed via index or table scans

Determining the Join Order and Method—Nested Loops Join



------------------------------------------------------------------------------------------------------


------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 29 | | |

| 2 | NESTED LOOPS | | 17177 | 486K| 39480 (1)| 00:00:02 |

| 3 | NESTED LOOPS | | 157K| 486K| 39480 (1)| 00:00:02 |

| 4 | TABLE ACCESS BY INDEX ROWID BATCHED| DATE_DIM | 7 | 105 | 2 (0)| 00:00:01 |

|* 5 | INDEX RANGE SCAN | DATE_DIM_N1 | 7 | | 1 (0)| 00:00:01 |

|* 6 | INDEX RANGE SCAN | LO_DATE_N | 22438 | | 60 (0)| 00:00:01 |

|* 7 | TABLE ACCESS BY INDEX ROWID | LINEORDER | 2493 | 34902 | 5988 (1)| 00:00:01 |

------------------------------------------------------------------------------------------------------


Outer TableInner Table



---------------------------------------------------------------------------------------------


---------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 29 | | |

| 2 | NESTED LOOPS | | 17177 | 486K| 6187K (1)| 00:04:02 |

| 3 | NESTED LOOPS | | 5998K| 486K| 6187K (1)| 00:04:02 |

|* 4 | TABLE ACCESS FULL | LINEORDER | 5998K| 80M| 187K (1)| 00:00:08 |

|* 5 | INDEX UNIQUE SCAN | DATE_DIM_PK | 1 | | 0 (0)| 00:00:01 |

|* 6 | TABLE ACCESS BY INDEX ROWID| DATE_DIM | 1 | 15 | 1 (0)| 00:00:01 |

---------------------------------------------------------------------------------------------


Outer TableInner Table



• Hash joins are used for joining large data sets.

1. The optimizer uses the smaller of the two tables or data sources to build a hash table on the join key in memory, using a deterministic hash function to specify the location in which to store each row in the hash table. This is called the build side.

2. It then scans the larger table, called the probe table, probing the hash table to find the joined rows.

• This method is best used when the smaller table fits in available memory. The cost is then limited to a single read pass over the data for the two tables.

Determining the Join Order and Method—HASH Join



-----------------------------------------------------------------------------------------------------


-----------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 29 | | |

|* 2 | HASH JOIN | | 17177 | 486K| 187K (1)| 00:00:08 |

| 3 | TABLE ACCESS BY INDEX ROWID BATCHED| DATE_DIM | 7 | 105 | 2 (0)| 00:00:01 |

|* 4 | INDEX RANGE SCAN | DATE_DIM_N1 | 7 | | 1 (0)| 00:00:01 |

|* 5 | TABLE ACCESS FULL | LINEORDER | 5998K| 80M| 187K (1)| 00:00:08 |

-----------------------------------------------------------------------------------------------------


Build TableProbe Table



-------------------------------------------------------------------------------------------------------------

| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |

-------------------------------------------------------------------------------------------------------------

| 0 | SELECT STATEMENT | | 1 | 29 | | 194K (1)| 00:00:08 |

| 1 | SORT AGGREGATE | | 1 | 29 | | | |

|* 2 | HASH JOIN | | 17177 | 486K| 148M| 194K (1)| 00:00:08 |

|* 3 | TABLE ACCESS FULL | LINEORDER | 5998K| 80M| | 187K (1)| 00:00:08 |

| 4 | TABLE ACCESS BY INDEX ROWID BATCHED| DATE_DIM | 7 | 105 | | 2 (0)| 00:00:01 |

|* 5 | INDEX RANGE SCAN | DATE_DIM_N1 | 7 | | | 1 (0)| 00:00:01 |

-------------------------------------------------------------------------------------------------------------


Build TableProbe Table



• Merge joins are used for joining large data sets.

• Requires the data to be sorted—not as efficient as a hash join

• Used with inequalities where a hash join is not able to be used

• How it works:1. A sort merge join reads two data sets and sorts them when they are not already

sorted.

2. For each row in the first data set, the database finds a starting row in the second data set, and then reads the second data set until it finds a nonmatching row.

3. For large data sets that do not fit into memory the sorts will spill to temp, which has a noticeable performance impact while performing physical IO

Determining the Join Order and Method—Merge Join



--------------------------------------------------------------------------------------------------------------


--------------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 29 | | | |

| 2 | MERGE JOIN | | 17177 | 486K| | 217K (1)| 00:00:09 |

| 3 | SORT JOIN | | 7 | 105 | | 3 (34)| 00:00:01 |



|* 6 | SORT JOIN | | 5998K| 80M| 275M| 217K (1)| 00:00:09 |


--------------------------------------------------------------------------------------------------------------




--------------------------------------------------------------------------------------------------------------


--------------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 29 | | | |

| 2 | MERGE JOIN | | 17177 | 486K| | 217K (1)| 00:00:09 |

| 3 | SORT JOIN | | 5998K| 80M| 275M| 217K (1)| 00:00:09 |


|* 5 | SORT JOIN | | 7 | 105 | | 3 (34)| 00:00:01 |



--------------------------------------------------------------------------------------------------------------




• The optimizer will associate a cost with each of these based on it’s estimate of the amount of work

– The goal is to reduce the number of rows early so it will perform less work throughout the execution of the SQL statement

–We basically have a matrix of join orders and methods and the cost associated with each




Join Order

date_dim, lineorder lineorder, date_dim

Join Method

Nested Loops 39,480 6,187,540

Hash 187,528 194,909

Merge 217,129 217,129


65


Serial vs Parallel Execution


Serial and Parallel Execution

• Serial Execution– SQL is executed by one process– The correct solution when:• the query references a small data set• high concurrency • efficiency is important

• Parallel Execution– SQL is executed by many processes working together– The correct solution when:• the query references a large data set• low concurrency• elapsed time is important


Parallel Execution

Query Coordinator (QC) The “top level” process for the parallel query

Parallel Execution Server (PX) An (OS) process that operates on part of a parallel query

Degree of Parallelism (DoP) The number of parallel execution servers used in each parallel server group during parallel execution

Parallel server group The group of parallel server processes that operate on a row source

Distribution method The method by which data is sent from one set of PX servers to another

Basics

68


Distribution Methods


Lets look at 2 parallel execution plans for the same query:

Parallel Query

select count(*)

from yellow y,

green g

where y.deptno = g.deptno


Hash Distribution

Parallel Query

PQ1

PQ2

PQ1

PQ2

Scan

PQ3

PQ4

QC

Hash Distribution

Join andAggregate

Result SetScan

Hash Distribution

GreenTable

YellowTable

f(x)

f(x)

f(x)

f(x)


Hash Distribution

Parallel Query


Hash DistributionConsumers

4

3

2

1

4

3

2

1

Producers


Hash DistributionConsumers

4

3

2

1

4

3

2

1

Producers

Sent16 rows

Received16 rows


Broadcast Distribution

Parallel Query

PQ1

PQ2

Scan

PQ3

PQ4

QC

BroadcastDistribution

Scan, Join andAggregate

Result Set

GreenTable

YellowTable


Broadcast Distribution

Parallel Query


Broadcast DistributionConsumers

4

3

2

1

4

3

2

1

Producers


Broadcast DistributionConsumers

4

3

2

1

4

3

2

1

Producers

Sent16 rows

Received64 rows


Small Table Replication

4

3

2

1

Buffer Cache

If the table is not already in the buffer cache, one or more of the px processes first load it into the buffer cache

Table is loaded into the buffer cache


Small Table Replication

4

3

2

1

Buffer Cache

Each parallel process reads the table from the buffer cache

Table is read from the buffer cache


Parallel Execution

Skew A PX process will do much more work than the others

• SELECT DISTINCT when there are very few distinct values

• A hash join with 1 or 2 values having most of the rows

Distribution method Using broadcast when it should use hash

• Usually happens when row estimate is much lower than actual

Using hash when it should broadcast:

• Too few values, row distribution is skewed

Usually, better/extended statistics help solve the problem

What may go wrong



FROM lineorder

JOIN date_dim

ON lo_orderdate = d_datekey

WHERE d_year = 1997

AND d_weeknuminyear = 27

AND lo_tax = 0

;

• Size of the tables and rows retrieved

• Selectivity of join predicates

– How many rows do we expect to join from each table

– How many rows do we expect to retrieve, and thus distribute, from each table

• Skew in the data– Hash distribution does not work well

with data skew

You As The OptimizerDetermining the Distribution Method



---------------------------------------------------------------------------------------------------------------------

| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |

---------------------------------------------------------------------------------------------------------------------

| 0 | SELECT STATEMENT | | 1 | 29 | 104K (1)| 00:00:05 | | | |

| 1 | SORT AGGREGATE | | 1 | 29 | | | | | |

| 2 | PX COORDINATOR | | | | | | | | |

| 3 | PX SEND QC (RANDOM) | :TQ10002 | 1 | 29 | | | Q1,02 | P->S | QC (RAND) |

| 4 | SORT AGGREGATE | | 1 | 29 | | | Q1,02 | PCWP | |

|* 5 | HASH JOIN | | 17177 | 486K| 104K (1)| 00:00:05 | Q1,02 | PCWP | |

| 6 | JOIN FILTER CREATE | :BF0000 | 7 | 105 | 11 (0)| 00:00:01 | Q1,02 | PCWP | |

| 7 | PX RECEIVE | | 7 | 105 | 11 (0)| 00:00:01 | Q1,02 | PCWP | |

| 8 | PX SEND HASH | :TQ10000 | 7 | 105 | 11 (0)| 00:00:01 | Q1,00 | P->P | HASH |

| 9 | PX BLOCK ITERATOR | | 7 | 105 | 11 (0)| 00:00:01 | Q1,00 | PCWC | |

|* 10 | TABLE ACCESS FULL| DATE_DIM | 7 | 105 | 11 (0)| 00:00:01 | Q1,00 | PCWP | |

| 11 | PX RECEIVE | | 5998K| 80M| 104K (1)| 00:00:05 | Q1,02 | PCWP | |

| 12 | PX SEND HASH | :TQ10001 | 5998K| 80M| 104K (1)| 00:00:05 | Q1,01 | P->P | HASH |

| 13 | JOIN FILTER USE | :BF0000 | 5998K| 80M| 104K (1)| 00:00:05 | Q1,01 | PCWP | |

| 14 | PX BLOCK ITERATOR | | 5998K| 80M| 104K (1)| 00:00:05 | Q1,01 | PCWC | |

|* 15 | TABLE ACCESS FULL| LINEORDER | 5998K| 80M| 104K (1)| 00:00:05 | Q1,01 | PCWP | |

---------------------------------------------------------------------------------------------------------------------

Determining the Distribution Method—Hash Distribution



---------------------------------------------------------------------------------------------------------------------


---------------------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 29 | | | | | |

| 2 | PX COORDINATOR | | | | | | | | |





| 7 | PX RECEIVE | | 7 | 105 | 11 (0)| 00:00:01 | Q1,01 | PCWP | |

| 8 | PX SEND BROADCAST | :TQ10000 | 7 | 105 | 11 (0)| 00:00:01 | Q1,00 | P->P | BROADCAST |

| 9 | PX BLOCK ITERATOR | | 7 | 105 | 11 (0)| 00:00:01 | Q1,00 | PCWC | |

|* 10 | TABLE ACCESS FULL| DATE_DIM | 7 | 105 | 11 (0)| 00:00:01 | Q1,00 | PCWP | |



|* 13 | TABLE ACCESS FULL | LINEORDER | 5998K| 80M| 104K (1)| 00:00:05 | Q1,01 | PCWP | |

---------------------------------------------------------------------------------------------------------------------

Determining the Distribution Method—Broadcast Distribution



-------------------------------------------------------------------------------------------------------------------


-------------------------------------------------------------------------------------------------------------------


| 1 | SORT AGGREGATE | | 1 | 29 | | | | | |

| 2 | PX COORDINATOR | | | | | | | | |





|* 7 | TABLE ACCESS FULL | DATE_DIM | 7 | 105 | 11 (0)| 00:00:01 | Q1,00 | PCWP | |



|* 10 | TABLE ACCESS FULL| LINEORDER | 5998K| 80M| 104K (1)| 00:00:05 | Q1,00 | PCWP | |

-------------------------------------------------------------------------------------------------------------------

Determining the Distribution Method—Small Table Replication



• The optimizer makes decisions for every step in the plan based on available information

• For more complex queries, the matrix of decisions becomes larger and the difficulty of producing accurate cardinality estimates increases

• There are lots of tools which can be used to increase performance but they are only effective if the optimizer chooses the correct plan– Partitioning

– Compression

– Exadata

– Database In-Memory

Determining the Execution Plan


The Optimizer

• The vast majority of bad execution plans are the result of poor cardinality estimates

• Verification of cardinality estimates should always be the starting point for diagnosis

• Check for order of magnitude differences

• Consider whether gathering new statistics fits the scope of the problem

– Is there one problem query?

– Are there many problem queries?

• Not every poor cardinality estimate results in a bad plan

Poor plans


Agenda

The Optimizer

Optimizer Inputs

Optimizer Output


Why is my SQL slow?


1

2

3

4

5

6


Advanced Optimizer Engine

• Basic statistics provide basic information about the data

– Assumes even distribution of data values

– Does not account for correlation between predicate filters

• There are facilities to restrict and/or guide the optimizer

• Versions 11g and 12c introduced features to allow the optimizer to learn and adapt



Inputs

• Statistics

– Histograms

– Extended Statistics

• Plan Management

• Hints

• Adaptive Statistics

Execution Plan

• Adaptive Plans


The Optimizer

• The optimizer has become increasingly more dynamic with each release

• We can classify the information the optimizer uses into the following categories

1. Things to help get the right plan the first time

2. Things to help get the right plan during execution

3. Things to help get the right plan on subsequent executions

4. Things to help get the right plan if the others fail



Inputs

• SQL

• Schema Design

• Physical Design

• Statistics

– Table

– Column



1. Getting the plan right the first time

Execution Plan

• Access Method

• Join Method

• Join Order




Inputs

• SQL

• Schema Design

• Physical Design

• Statistics

– Table

– Column


• Do not rely on these


– Histograms

– Extended Statistics

1. Getting the plan right the first time—work on getting the inputs correct

Execution Plan

• Access Method

• Join Method

• Join Order



Handling Data Skew

• Designed to give more detail on data distribution

• Histograms are built based on SQL Usage– Hence the need to “seed” the optimizer

– sys.col_usage$ keeps track of columns used in predicates and joins

– These columns are then candidates for histograms

• New in 12c– Up to 2,048 buckets but default values still use 254

– Uses a full table scan to create histograms with AUTO_SAMPLE_SIZE—more accurate for unpopular values

• Types– Frequency—for columns with NDV <= 254

– Top-frequency—frequency histogram where NDV > 254 but majority of the rows <= 254 values

– Height Balanced—no longer used with AUTO_SAMPLE_SIZE—Replaced by Hybrid• Still created with estimate_percent

• Remain after upgrades

– Hybrid—for columns with NDV > 254 and lots of popular values

Histograms


Handling Data Correlation and Functions

• Two Types of Extended Statistics

– Column Groups• Provides the optimizer with information regarding the relationship (correlation) between the data

stored in different columns of the same table

– City and airport code

– Month and zodiac sign

• Allows the optimizer to compute a better cardinality estimate when several the columns from the same table are use d together in a where clause of a SQL statement

– Expression Statistics• Helps estimate the cardinality of a where clause predicate that has columns embedded inside

expressions

– UPPER(EMP_LAST_NAME)=:B1


Extended Statistics

• Use the dbms_stats.create_extended_stats function to create extended statistics

Select

dbms_stats.create_extended_stats(USER, ’AIRPORTS’,’(CITY,CODE)’)

from dual;

Select

dbms_stats.create_extended_stats(USER,’EMP’,’(UPPER(EMP_LAST_NAME))’)

from dual;


Extended Statistics

• Restrictions

– Only used when the SQL statement predicates are equalities or in-lists.

– Not used if there are histograms present on the underlying columns and there is no histogram present on the column group.



Inputs

2. Get the right plan during execution

Execution Plan

• Adaptive Plans



Inputs

• Adaptive Statistics

3. Get the right plan on subsequent executions

Execution Plan


Adaptive Query Optimization

• Introduced in 12.1

• Increase the likelihood of good plans by making use of execution statistics

– During execution

– As feedback for re-optimization

– During future compilations• As feedback for sampling

• As feedback to direct subsequent gathering of statistics

• LEARNING from your workload and persisting information in the data dictionary, increasing the chance of good plans first time

100

Goals


Adaptive Query Optimization

• Use the actual number of rows to choose appropriate join methods and parallel distribution methods during execution

– Adaptive Plans REACTING

• Use the actual number of rows to re-optimize on subsequent execution

– Automatic Re-Optimization LEARNING

• Use the actual number of rows to direct the gathering of statistics for future compilations of similar statements

– Adaptive Statistics LEARNING

101

What happens during the execution of a statement?


The Real World

• The Adaptive Plans features in 12.1 have very few reported problems

– These features is enabled by default in 12.2

• The Adaptive Statistics features in 12.1 have many reported problems

– Inefficient execution plans while the system is learning

– Unpredictable execution plans as Oracle searches for better plans

– Long parse times

– These features is disabled by default in 12.2

102

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 103

Changes Between 12.1 and 12.2

optimizer_adaptive_featuresDefault: TRUE

AdaptivePlans

AdaptiveStatistics

optimizer_adaptive_statistics

Default: FALSEoptimizer_adaptive_plans

Default: TRUE

Oracle 12.1

Oracle 12.2


Recommended Strategy for 12.1

• Install patch for BUG#22652097– Splits the parameter optimizer_adaptive_features into two parameters• optimizer_adaptive_plans—enabled by default

• optimizer_adaptive_statistics—disabled by default

• Install patch for BUG#21171382

– Disables the automatic creation of extended statistics by default

• Details in MOS Note 2187449.1

104



Inputs

• Plan Management

• Hints

4. Get the right plan if the other methods fail

Execution Plan


Execution Plan Management

• Several features guide or restrict the Optimizer to use specific plans

– SQL Plan Baselines—constrains the optimizer to only select from a set of accepted plans for a SQL statement

• Controlled through SQL Plan Management

– SQL Profiles—auxiliary information specific to a SQL statement which guides the optimizer to a better plan

• These features are useful to protect against plan changes which could adversely affect performance

• These features are widely misused to compensate for poor inputs to the optimizer engine


Hints

• Hints can be used to influence the execution plan chosen by the optimizer

• Hints are a useful tool during development and testing to check the performance impact of a different execution plan

• Using hints in a production environment is poor practice

– It is difficult to apply hints appropriately for a large number of SQL statements

– They do not allow enhancements to the optimizer and execution engine to be used

– It is safer and easier to correct deficiencies in the optimizer inputs


Changes to the Optimizer Engine

• The set of algorithms which make up the optimizer change between database releases

• Optimizer changes can also be introduced in patches and patch sets

• If the quality of inputs is poor, such as inaccurate statistics, this can lead to execution plan differences between versions

Database Versions and Patches


Changes to the Optimizer Engine

• Database parameters influence or restrict the behavior of the optimizer

– optimizer_* parameters

– Hidden underscore parameters

• DANGER!—changing these parameters is often done to compensate for poor inputs and they should be left at the default values

Database Parameters


Agenda

The Optimizer

Optimizer Inputs

Optimizer Output


Why is my SQL slow?


1

2

3

4

5

6


Why is My SQL Slow ?


Problem Query

Table has 1B rows and is 55 GB.


Problem Query

Query 1 consists of two subqueries. The first subqueryfinds all of the Ferraris.


Problem Query

The second subquery finds all of the Ferrari 458s.


Problem Query

Outer query joins the results of the subqueries.

Outer query performs aggregations.


Problem Query

Query 2 is the same but has different predicate values.


Default Statistics


Default Statistics

Query 1 takes 40 seconds with default statistics


Default Statistics


Default Statistics

Query 2 takes 3 seconds with default statistics


Default Statistics


Development Findings

• Baseline Performance for Query 1 exceeds target

• Baseline Performance for Query 2 meets target

Default Statistics


Initial Optimization Steps—More Predicate Values


More Predicate ValuesIncrease the list of predicate values

Now query 1 takes 2 seconds


More Predicate Values



• Query runs faster just by changing the list of values in the select list

• Plan changed from a broadcast to a hash distribution due to the higher but inaccurate cardinality estimate

• Get correct plan with wrong cardinality estimate—can lead to inconsistent plans and performance

More Predicate Values


Initial Optimization Steps—Increase Degree of Parallelism


Degree of Parallelism

Change DoP from 32 to 128

Now query takes 2 seconds



• Changing DoP from 32 to 128 improves performance and meets the target; 4X more resources yields a 20X performance improvement

• Plan has changed from a broadcast distribution to a hash distribution due to DoP change

• DoP is a resource management technique, not a query tuning tool



Indexes


Development Findings—Indexes

Indexes on columns:

• owner_id

• country

• make

• model

• country, make, model

Indexes


Indexes

Add indexes and query takes longer—58 seconds!


Indexes

Index lookups on millions of rows is slow

Query performance varies depending on whether the index is cached or not


Development Findings—Indexes

•Not understanding the big/little data challenge

• Indexes are not efficient for operations on a large numbers of rows

• Full table scan is faster with predictable performance

Indexes


To Index or Not

• Indexing is an OLTP technique for operations on a small number of rows

• A table scan may consume more resources but it will be predictable no matter how many rows are returned

– Indexes impact DML operations

– If I/O bandwidth went from 70MB/sec to 70GB/sec would you change your optimization/execution strategy?


To Index or Not

• Index driven query retrieving 1,000,000 rows

– Assume the index is cached and the data is not.• 1,000,000 random IOPS @ 5ms per I/O

• This would require 5000 Seconds ( or over 1 hour ) to Execute

– How much data could you scan in 5000 Seconds with a fully sized I/O system able to scan 25 GB/Sec ?• Over 100 TB !


Histograms


Histograms

Rerun stats to get histograms—no change in plan or run time


Histograms

Lots of wait time on temp IO



• Re-gathered stats to automatically create histograms

• Frequency histograms on country, make and model columns

• No change in plan—query still exceeds target

Histograms


Flash Temp


Flash Temp

Query time reduced to 23 seconds


Flash TempNow IO accounts for a smaller percentage of database time



•Most of the wait time was spent performing IO on temp, so move temp to flash disks

• Improved performance but still does not meet target

•Not a good use of flash

• Incorrect use of tools/products

Flash Temp


Manual Memory Parameters



Increased memory size manually—now there is no use of temp

Very little IO in database time

Poor cardinality estimate—256K estimated rows vs 40M actual rows



• Set sort_area_size and hash_area_size to 2G

• Eliminated temp usage but still did not meet target

•Memory is allocated per parallel server process, which can quickly exceed resources

•Moving to a solution before understanding the problem



Cardinality Estimates



Use cardinality hint to specify correct number of rows

Plan switches from a broadcast to a hash distribution


Cardinality Hint

• SQL Monitor showed poor cardinality estimates

•Cardinality hint gives optimizer the correct number of rows for the table scan

•Plan changed from a broadcast to hash distribution

•Query time now meets target

•Now temp is not an issue



Disable Broadcast Distribution



Disable broadcast distribution and now we have the hash distribution as with the cardinality hint



•Google reveals a hidden parameter to disable broadcast distribution

•Plan and run times are similar to cardinality hint, meeting target

•Moving to a solution before understanding the problem



Second Query with Broadcast Distribution Disabled


Query 2: Broadcast Distribution Disabled



Query 2 also uses a hash distribution but no longer meets the target



•Plan uses a hash distribution

• Exceeds target



Second Query with Broadcast Distribution Enabled


Query 2: Broadcast Distribution Enabled



Reset parameter to enable broadcast distribution—now query 2 uses a broadcast and meets the target



•Reset _parallel_broadcast_enabled

•Plan now uses a broadcast distribution

•Meets target

• Should not change system parameters to tune one query



Extended Stats


Extended Stats

Created column group but still have a poor cardinality estimate



•High correlation between Country, Make and Model columns

•Created column group

•Query still exceeds target

• Still have poor cardinality estimate

Extended Stats


Histogram on Column Group


Histogram on Column Group With a histogram on the column group we now have a good cardinality estimate

Now we get a hash distribution and meet the target



•Re-gathered stats after running the query with the column groups

• Frequency Histogram on the column group

•Accurate cardinality estimates

•Optimizer now uses a hash distribution

Histogram on Column Group


Second Query with Histogram on Column Group


Query 2: Histogram Column Group



And uses a broadcast distribution

Query 2 also has a good cardinality estimate



•Accurate cardinality estimates

•Optimizer uses a broadcast distribution on second query



Now we have the correct solution!

•Both queries have good cardinality estimates

•Correct plans

•Meet targets

Histogram on Column Groups


Auto Column Group Creation: Seed Column Usage


Auto Column Group Creation



Back to the default stats while seeding column usage—poor cardinality estimate as seen earlier and a broadcast distribution for query 1



• Start with default statistics

• Execute dbms_stats.seed_col_usage to monitor column usage

•Run query

Auto Column Group Creation: Seed Column Usage


Auto Column Group Creation: Create Extended Stats


Auto Column Group CreationWith the column group identified and created, we have a good cardinality estimate

And we get a hash distribution



•dbms_stats.report_col_usage shows column groups identified during Seed Column Usage

•dbms_stats.create_extended_stats creates column groups identified

•Automatically identifies usage of Country, Make and Model columns together and creates column group




•Regather stats

•Automatically creates Histogram on the column group

•Query meets target



What Did We Learn?


• Using DoP for query tuning• Indexes for large data sets• Temp on flash• Forcing use of more memory• Disable broadcast distribution

7/16/2018

Root Causes of Suboptimal Database Performance


Agenda

SQL and the Optimizer


Optimization Strategies

Why is my SQL slow?

Optimizer Edges Cases

Top SQL Mistakes

1

2

3

4

5

6


Optimizer Edge Cases

• Always stale statistics

• Arbitrary high and low values

• Correlation

• Functions



• Scenario:

– Data is loaded into a partitioned table as it is received

– Statistics are gathered by nightly maintenance process

– Resulting stats indicate 0 rows in partition

– Poor cardinality estimates and plans

• Resolution:

– Copy statistics from the previous partition, or set statistics to appropriate values

– Lock statistics to prevent updates by maintenance job

– Optimizer is able to use representative stats to get more accurate cardinality estimates

– Optional: Unlock and gather new statistics after the partition is fully populated

Always stale statistics




TABLE_NAME PARTITION_NAME INT GLOBAL_STATS NUM_ROWS SAMPLE_SIZE LAST_ANALYZED

-------------------- -------------- --- ------------ ---------- ----------- -------------

LINEORDER_STALE R1992 NO YES 8204594 8204594 24-OCT-14






LINEORDER_STALE R1998 NO YES 0 24-OCT-14

7 rows selected.


Representative statistics


TABLE_NAME PARTITION_NAME INT GLOBAL_STATS NUM_ROWS SAMPLE_SIZE LAST_ANALYZED

-------------------- -------------- --- ------------ ---------- ----------- -------------

LINEORDER_YR R1992 NO YES 8204594 8204594 24-OCT-14







7 rows selected.


Representative statistics



Arbitrary High and Low Values


C1 C2

1 date'0001-01-01'

2 date'2011-06-07'

3 date'2011-06-07'

4 date'2011-06-08'

5 date'2011-06-09'

6 date'2999-12-31'


With Arbitrary High Value for Unknown Dates


CREATE TABLE

LINEORDER

(

"LO_ORDERKEY" NUMBER NOT NULL ENABLE

,"LO_LINENUMBER" NUMBER

,"LO_CUSTKEY" NUMBER NOT NULL ENABLE

,"LO_PARTKEY" NUMBER NOT NULL ENABLE

,"LO_SUPPKEY" NUMBER NOT NULL ENABLE

,"LO_ORDERDATE" DATE NOT NULL ENABLE

. . .

,"LO_CLOSEDATE" DATE DEFAULT '31-DEC-2999'

)

;


With Arbitrary High Value for Unknown Dates



With Nulls to Represent Unknown Dates


CREATE TABLE

LINEORDER

(

"LO_ORDERKEY" NUMBER NOT NULL ENABLE

,"LO_LINENUMBER" NUMBER

,"LO_CUSTKEY" NUMBER NOT NULL ENABLE

,"LO_PARTKEY" NUMBER NOT NULL ENABLE

,"LO_SUPPKEY" NUMBER NOT NULL ENABLE

,"LO_ORDERDATE" DATE NOT NULL ENABLE

. . .

,"LO_CLOSEDATE" DATE

)

;


With Nulls to Represent Unknown Dates




• When different columns in a given table have values that are correlated

–Make and model of a car, a phone, ….

–Month and zodiac sign

– City and airport

• Correlation causes an under-estimate in the cardinality because predicates are seen as independent by the Optimizer

Correlation


Correlation




• Functions and wildcards make it very difficult to obtain good cardinality estimates

• In many cases the optimizer will simply guess at a cardinality estimate based upon 1% or 5% of the rows in a table

• Techniques such as dynamic sampling and extended statistics should be evaluated to improve the cardinality estimates.

Functions and Wildcards


Rounding Function


SELECT d_sellingseason, p_category, s_region

FROM lineorder

JOIN customer ON lo_custkey = c_custkey

JOIN date_dim ON lo_orderdate = d_datekey

JOIN part ON lo_partkey = p_partkey

JOIN supplier ON lo_suppkey = s_suppkey

WHERE lo_orderdate between to_date('15-JUN-1993','DD-MON-YYYY')

and to_date('15-JUN-1995','DD-MON-YYYY')

AND d_monthnuminyear in (12, 1)

AND p_container in ('JUMBO PACK')

AND p_color in ('red')

AND round(lo_extendedprice) > 3670944

ORDER BY d_sellingseason, p_category, s_region


Rounding Function



Substring Function


SELECT d_sellingseason, p_category, s_region

FROM lineorder








AND SUBSTR(p_container,-4,4) in ('DRUM')




Substring Function



Using Extended Statistics for Substring Function


BEGIN

DBMS_STATS.gather_table_stats(

ownname => ‘DW',

tabname => ‘PART',

method_opt => 'for all columns size skewonly for columns (SUBSTR(P_CONTAINER,-4,4))’);

END;

/

PL/SQL procedure successfully completed.

select extension_name, extension from user_stat_extensions where table_name='PART';

EXTENSION_NAME EXTENSION

---------------------------------------- ----------------------------------------

SYS_STUW_AMVY8N$KU59A__847#Z7P (SUBSTR("P_CONTAINER",(-4),4))


Using Extended Statistics for Substring Function



Agenda

SQL and the Optimizer


Optimization Strategies

Why is my SQL slow?

Optimizer Edges Cases

Top SQL Mistakes

1

2

3

4

5

6


Top SQL Mistakes

• Missing Joins

• Implicit or Wrong Data Type Conversions

• More ‘Top SQL Mistakes’ shown in the Reference Materials


Top SQL Mistakes

• There should be n-1 join conditions in a query, where n is the number of tables in the query block, otherwise Cartesian products will occur

• Often a problem with programmers new to SQL developing/testing on small datasets where a DISTINCT is used to reduce the rows

• This is often not seen as a performance problem until the datasets become large!

• If the query aggregates the data, the total number of rows returned would be unchanged, but the values are likely to be incorrect

Missing Joins


Missing Join: 5 tables, 3 joins

Top SQL Mistakes

SELECT d_sellingseason, p_category, s_region, sum(lo_extendedprice)

FROM lineorder, customer, date_dim, part, supplier

WHERE -- lo_custkey = c_custkey AND

lo_orderdate = d_datekey

AND lo_partkey = p_partkey

AND lo_suppkey = s_suppkey

AND d_year in (1993, 1994, 1995)




GROUP BY d_sellingseason, p_category, s_region



Missing Join: 5 tables, 3 joins

Top SQL Mistakes


With All Joins: 5 tables, 4 joins

Top SQL Mistakes

SELECT d_sellingseason, p_category, s_region, sum(lo_extendedprice)

FROM lineorder, customer, date_dim, part, supplier

WHERE lo_custkey = c_custkey

AND lo_orderdate = d_datekey

AND lo_partkey = p_partkey

AND lo_suppkey = s_suppkey

AND d_year in (1993, 1994, 1995)







With All Joins: 5 tables, 4 joins

Top SQL Mistakes


Top SQL Mistakes

• SQL does not enforce much data type checking on SQL statements

• Where there are data type mismatches SQL will automatically cast/convert data into the appropriate type to execute the SQL statement

• This may result in the following effects

– Increased resource usage converting data types

– Poor execution plans avoiding indexes and partition pruning

– Failed SQL statements as data values may not convert correctly

Implicit or Wrong Data Type Conversions


Top SQL Mistakes

• Common in columns that contain numeric data but are never used for arithmetic operations

– telephone numbers

– credit card numbers

– check numbers

• When a programmer references these columns care must be made to ensure the bind variables are type VARCHAR2 and not numbers




Top SQL Mistakes

SELECT d_sellingseason,

p_category,

s_region,

sum(lo_extendedprice)

FROM lineorder





WHERE d_year in ('1993', '1994', '1995')






Top SQL Mistakes

SQL> desc date_dim

Name Null? Type

-------------------- -------- --------------

D_DATEKEY NOT NULL DATE

D_DATE VARCHAR2(18)

D_DAYOFWEEK VARCHAR2(10)

D_MONTH VARCHAR2(9)

D_YEAR NUMBER

D_YEARMONTHNUM NUMBER

D_YEARMONTH VARCHAR2(7)

D_DAYNUMINWEEK NUMBER

D_DAYNUMINMONTH NUMBER

. . .


No Data Type Conversion

Top SQL Mistakes


p_category,

s_region,


FROM lineorder





WHERE d_year in (1993, 1994, 1995)






Top SQL Mistakes


p_category,

s_region,


FROM lineorder





WHERE d_year in (1993, 1994, 1995)

AND c_phone = 222417451619



This SQL statement is vulnerable to:

• Bad plans because index cannot be used

• 1722 errors if there is non numerical characters

• Increased resource usage converting data

• Poor cardinality estimates



Top SQL Mistakes

SQL> desc customer

Name Null? Type

--------------------- -------- --------------

C_CUSTKEY NOT NULL NUMBER

C_NAME VARCHAR2(25)

C_ADDRESS VARCHAR2(25)

C_CITY VARCHAR2(10)

C_NATION VARCHAR2(15)

C_REGION VARCHAR2(12)

C_PHONE VARCHAR2(15)

C_MKTSEGMENT VARCHAR2(10)



Top SQL Mistakes

SQL> @bad_data_type_q.sql


*

ERROR at line 1:

ORA-12801: error signaled in parallel query server P00E, instance

scao08adm01.us.oracle.com:imtst1 (1)

ORA-01722: invalid number



Top SQL Mistakes


p_category,

s_region,


FROM lineorder





WHERE d_year in (1993, 1994, 1995)

AND c_phone = ’22-241-745-1619’





Top SQL Mistakes



Top SQL Mistakes

SELECT /*+ MONITOR */

p_category,

s_region,


FROM lineorder_mon



WHERE lo_orderdate between to_timestamp('15-JUN-1993','DD-MON-YYYY')

and to_timestamp('15-JUN-1995','DD-MON-YYYY')


GROUP BY p_category, s_region

ORDER BY p_category, s_region

;

This SQL statement is vulnerable to:

• Bad plans because partition pruning may not take place

• Poor cardinality estimates



Top SQL Mistakes

SQL> desc lineorder

Name Null? Type

-------------------- -------- --------------

LO_ORDERKEY NOT NULL NUMBER

LO_LINENUMBER NUMBER

LO_CUSTKEY NOT NULL NUMBER

LO_PARTKEY NOT NULL NUMBER

LO_SUPPKEY NOT NULL NUMBER

LO_ORDERDATE NOT NULL DATE

LO_ORDERPRIORITY VARCHAR2(15)

LO_SHIPPRIORITY VARCHAR2(1)

LO_QUANTITY NUMBER

LO_EXTENDEDPRICE NUMBER

LO_ORDTOTALPRICE NUMBER

. . .



Top SQL Mistakes


Correct Data Type Conversion

Top SQL Mistakes

SELECT /*+ MONITOR */

p_category,

s_region,


FROM lineorder_mon





GROUP BY p_category, s_region

ORDER BY p_category, s_region


Correct Data Type Conversion

Top SQL Mistakes


Reference Section


Tools for SQL Statement Analysis


Tools For SQL Analysis

• Static Analysis

• Dynamic Analysis

• Tracing and Debugging


Tools for SQL Analysis

• Static Analysis– Explain Plan and dbms_xplan.display

– SQL*Plus Autotrace


Static Analysis

• Overview

– Provides an indication of the possible execution plan• DDL e.g. CREATE TABLE … AS SELECT

• DML

• Queries

• Strengths

– Available in every installation

– Does not require execution of the SQL statement

Explain Plan


Static Analysis

• Weaknesses

– Bind variables• All binds treated as VARCHAR

• No bind peeking

– Beware of PLAN_TABLE inherited from previous releases

– Based on expectations rather than reality

Explain Plan


Static Analysis

• Predicate Information– Filters– Transformations– Candidates for Offload– Bloom Filters

• Notes Section– Automatic Degree of Parallelism (aka Auto DoP)• Rationale for choice of DoP

– Dynamic Sampling• When no statistics are available• When the optimizer chooses to use dynamic sampling in the evaluation of parallel queries• Cardinality Feedback

Explain Plan


Explain Plan needs to be formatted

Static Analysis


--------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |--------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 3 | 114 | 2 (0)| 00:00:01 || 1 | TABLE ACCESS BY INDEX ROWID| EMP | 3 | 114 | 2 (0)| 00:00:01 ||* 2 | INDEX RANGE SCAN | EMP_N1 | 3 | | 1 (0)| 00:00:01 |--------------------------------------------------------------------------------------

Predicate Information (identified by operation id):---------------------------------------------------

2 - access("DEPTNO"=10)


Static Analysis

• Formatting– SET TAB OFF

– SET TRIMSPOOL ON

– Use fixed width font

– Preserve spaces

– Avoid truncating or wrapping long lines

Explain Plan


Explain Plan

Static Analysis


--------------------------------------------------------------------------------------


--------------------------------------------------------------------------------------


| 1 | TABLE ACCESS BY INDEX ROWID| EMP | 3 | 114 | 2 (0)| 00:00:01 |

|* 2 | INDEX RANGE SCAN | EMP_N1 | 3 | | 1 (0)| 00:00:01 |

--------------------------------------------------------------------------------------


---------------------------------------------------



Static Analysis

• More Background

– Explain the Explain Plan• Written by Maria Colgan

• http://blogs.oracle.com/optimizer/entry/explain_the_explain_plan_white

• http://www.oracle.com/technetwork/database/focus-areas/bi-datawarehousing/twp-explain-the-explain-plan-052011-393674.pdf

Explain Plan


SQL*Plus Autotrace

Static Analysis

SQL> set linesize 132 tab off

SQL> set autotrace traceonly explain

SQL> SELECT * FROM EMP WHERE DEPTNO = 10;

Execution Plan----------------------------------------------------------Plan hash value: 3324114979

--------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |--------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | 3 | 114 | 2 (0)| 00:00:01 || 1 | TABLE ACCESS BY INDEX ROWID| EMP | 3 | 114 | 2 (0)| 00:00:01 ||* 2 | INDEX RANGE SCAN | EMP_N1 | 3 | | 1 (0)| 00:00:01 |--------------------------------------------------------------------------------------

Predicate Information (identified by operation id):---------------------------------------------------



SQL*Plus Autotrace

Static Analysis

SQL> set autotrace traceonly explain statistics

SQL> SELECT * FROM EMP WHERE DEPTNO = 10;

Execution Plan

----------------------------------------------------------

…

Statistics

----------------------------------------------------------

0 recursive calls

0 db block gets

4 consistent gets

0 physical reads

0 redo size

1159 bytes sent via SQL*Net to client

525 bytes received via SQL*Net from client

2 SQL*Net roundtrips to/from client

0 sorts (memory)

0 sorts (disk)

3 rows processed


Static Analysis

• Overview

– Provides an indication of the possible execution plan• DML

• Queries

• Strengths

– Built in functionality with some useful shortcuts

–May not require execution of queries

• Weaknesses

–Much the same as Explain Plan

SQL*Plus Autotrace



• Dynamic Analysis– dbms_xplan.display_*

– SQL Monitor


Dynamic Analysis

• Overview

– Gives the actual execution plan from various sources• Cursor cache

– dbms_xplan.display_cursor

• AWR

– dbms_xplan.display_awr

• SQL Tuning Set

– dbms_xplan.display_sqlset

DBMS_XPLAN.DISPLAY_*


dbms_xplan.display_cursor()

Dynamic Analysis

SQL> set feedback off linesize 132 pagesize 0 tab off

SQL> SELECT * FROM EMP WHERE DEPTNO = :n;

SQL> select * from table(dbms_xplan.display_cursor());

SQL_ID 5xt3urx81f9th, child number 0

-------------------------------------

SELECT * FROM EMP WHERE DEPTNO = :n


--------------------------------------------------------------------------------------


--------------------------------------------------------------------------------------

| 0 | SELECT STATEMENT | | | | 2 (100)| |

| 1 | TABLE ACCESS BY INDEX ROWID| EMP | 3 | 114 | 2 (0)| 00:00:01 |

|* 2 | INDEX RANGE SCAN | EMP_N1 | 3 | | 1 (0)| 00:00:01 |

--------------------------------------------------------------------------------------


---------------------------------------------------

2 - access("DEPTNO"=:N)


Dynamic Analysis

• Various formatting options

– Default is usually sufficient

– ALLSTATS LAST can be useful when using the GATHER_PLAN_STATISTICShint to gather execution statistics

– Beware of overhead

• Ignore the following artifact in parallel queries

– storage(:Z>=:Z AND :Z<=:Z)

dbms_xplan.display_cursor()


Dynamic Analysis

• Overview

– Feature of SQL Tuning Pack

– Generation• Oracle Enterprise Manager

• dbms_sqltune.report_sql_monitor

– Text, HTML or Active• Active is preferred

SQL Monitor


SQL Monitor

Dynamic Analysis

set trimspool on

set trim on

set pagesize 0

set linesize 1000

set long 1000000

set longchunksize 1000000

spool sqlmon_previous.html

select dbms_sqltune.report_sql_monitor(

session_id=>sys_context('userenv','sid'),

report_level=>'ALL',

type=>'ACTIVE')

from dual;

spool off


Dynamic Analysis

• By default SQL Monitor is limited to 300 plan lines. If the report exceeds 300 lines, it will not show up in v$sql_monitor or the EM SQL Monitoring Page.

• This value can be increased using the_sqlmon_max_planlines parameteralter session set “_sqlmon_max_planlines”=500;

• 300 is usually sufficient and should only be changed if needed

SQL Monitor


SQL Monitor

Dynamic Analysis


Performance Diagnosis and Tuning

• SQL Monitor can be accesses by navigating

– Performance Tab

– “SQL Monitoring” Link

– Select SQL Statement of Interest to drill down

• Show recent SQL statements running for more than 5 seconds.

• SQL statement details can quickly highlight issues such as• skew in PQ execution

• non-representative statistics

(large difference between estimated rows and actual rows)

EM Access to Reports


EM Monitor Report

Select “SQL Monitoring”


EM Monitor Report

Select SQL of Interest


EM Monitor Report



• Tracing

– 10046

– Active Session History• Discussed in OLTP session


Tracing

• Overview– Tracing of calls and optionally wait events and bind values

– Raw tracefile can be useful to identify anomalies

– Generally post-process using tkprof (trace kernel profile)

• Strengths– Difficult to get some of the information any other way

• Weaknesses– Tracing needs to be enabled

– Multiple tracefiles generated for parallel execution, making 10046 almost useless

– Runtime and space overhead

10046


Tracing

• Embed an identifier in the name of the tracefileSQL> alter session set tracefile_identifier = my_10046_trace;

• Level 1 is equivalent to SQL_TRACE = TRUESQL> alter session set events = '10046 trace name context forever, level 1';

10046


Tracing

• Use dbms_monitor to set it

– Easy way to enable/disable tracing in other sessions

SQL> execute dbms_monitor.session_trace_enable()

SQL> execute dbms_monitor.session_trace_disable()

10046


10046

Tracing

Trace Level Functionality

• Off (specified in place of ‘forever’) • Disable Tracing

• 1 • Equivalent to SQL_TRACE=TRUE

• 4 • Include Bind Values

• 8 • Include Wait Events

• 12 • Include Bind Values and Wait Events


10046

Tracing

select * from emp where deptno = :n

call count cpu elapsed disk query current rows

------- ------ -------- ---------- ---------- ---------- ---------- ----------

Parse 2 0.00 0.00 0 0 0 0

Execute 2 0.00 0.00 0 0 0 0

Fetch 4 0.00 0.00 0 8 0 8

------- ------ -------- ---------- ---------- ---------- ---------- ----------

total 8 0.00 0.00 0 8 0 8

Misses in library cache during parse: 1

Optimizer mode: ALL_ROWS

Parsing user id: 88 (TEACHER)

Number of plan statistics captured: 2

Rows (1st) Rows (avg) Rows (max) Row Source Operation

---------- ---------- ---------- ---------------------------------------------------

3 4 5 TABLE ACCESS BY INDEX ROWID EMP (cr=4 pr=0 pw=0 time=152 us cost=2 size=114 card=3)

3 4 5 INDEX RANGE SCAN EMP_N1 (cr=2 pr=0 pw=0 time=120 us cost=1 size=0 card=3)(object id 64363)


10046

Tracing

Rows Execution Plan

------- ---------------------------------------------------

0 SELECT STATEMENT MODE: ALL_ROWS

3 TABLE ACCESS MODE: ANALYZED (BY INDEX ROWID) OF 'EMP' (TABLE)

3 INDEX MODE: ANALYZED (RANGE SCAN) OF 'EMP_N1' (INDEX)

Elapsed times include waiting on following events:

Event waited on Times Max. Wait Total Waited

---------------------------------------- Waited ---------- ------------

Disk file operations I/O 1 0.00 0.00

SQL*Net message to client 4 0.00 0.00

SQL*Net message from client 4 0.88 0.89

********************************************************************************



• Debugging

– 10053• Superseded by SQL_COMPILER trace

– SQL Test Case Builder

– SQLTXPLAIN• my.oracle.support Doc ID 215187.1


Debugging

• Embed an identifier in the name of the tracefile:SQL> alter session set tracefile_identifier = SQLCOMP_trace;

• For all SQL statements:SQL> alter session set events = 'trace [SQL_COMPILER.*]';

• For a specific SQL Identifier:SQL> alter session set events = 'trace [sql_compiler] [SQL:7h35uxf5uhmm1]';

SQL_COMPILER


Debugging

• Overview– Export the execution environment for a SQL statement

– Generation• Oracle Enterprise Manager• dbms_sqldiag.export_sql_testcase

• dbms_sqldiag.import_sql_testcase

– TIP: Avoid the need to quote quotes, for example:

SQL> variable i clob

SQL> exec :i := q'#select * from emp where ename = 'KING'#'

SQL Test Case Builder



Debugging

variable i CLOB

variable o CLOB

BEGIN

:i := q'#select * from emp where deptno = 10#';

END;

/

BEGIN

dbms_sqldiag.export_sql_testcase

(

directory => 'TEACHER_TCB_EXPORT'

,sql_text => :i

,testcase => :o

);

END;

/



Debugging

BEGIN

dbms_sqldiag.import_sql_testcase

(

directory => 'TEACHER_TCB_IMPORT'

,filename => 'oratcb1_171F07E50002main.xml'

);

END;

/


More ‘Top SQL Mistakes’

7/16/2018



• Sending SQL that is syntactically incorrect is very expensive to the database and very historically difficult to diagnose

– exception handling/throwing errors is expensive

– Bad SQL leaves no trace within the shared pool because it is bad and hence unshareable

– Statistic parse count(failures) indicates it may be happening

• The danger is clever programmer that sends bad SQL to see if it parsed to determine version or other information

Parse Errors and Incorrect SQL


Parse Errors and Incorrect SQL


SQL> select pk_id from history_version_2 where pk_id is NULL;

select pk_id from history_version_2 where pk_id is NULL;

*

ERROR at line 1:

ORA-00942: table or view does not exist


ANSI Outer Join



• By looking at a breakdown of the example, the difference between them can be understood.

• First consider the incorrectly translated outer join

ANSI outer joins


ANSI Outer Join

ANSI Syntax – Correct Translation

SELECT dname, d.deptno,

e.ename, e.mgr, d.loc

FROM dept d

LEFT OUTER JOIN emp e

on d.deptno = e.deptno

where( e.mgr is null

or

d.loc = ‘NEW YORK’)

Oracle Syntax



FROM dept d

,emp e

where d.deptno = e.deptno(+)

and ( e.mgr is null

or


ANSI Syntax – Wrong Translation



FROM dept d



and ( e.mgr is null

or



Incorrect Translation of Oracle syntax to ANSI

ANSI Outer Join

DNAME DEPTNO ENAME MGR LOC

-------------- ---------- ---------- ---------- -------------

ACCOUNTING 10 KING NEW YORK

SALES 30 BLAKE 7839 CHICAGO

ACCOUNTING 10 CLARK 7839 NEW YORK

RESEARCH 20 JONES 7839 DALLAS

RESEARCH 20 FORD 7566 DALLAS

RESEARCH 20 SMITH 7902 DALLAS

SALES 30 ALLEN 7698 CHICAGO

SALES 30 WARD 7698 CHICAGO

SALES 30 MARTIN 7698 CHICAGO

RESEARCH 20 SCOTT 7566 DALLAS

SALES 30 TURNER 7698 CHICAGO

RESEARCH 20 ADAMS 7788 DALLAS

SALES 30 JAMES 7698 CHICAGO

ACCOUNTING 10 MILLER 7782 NEW YORK

Consider just the inner join d.deptno=e.deptno




FROM dept d



and ( e.mgr is null

or




ANSI Outer Join


-------------- ---------- ---------- ---------- -------------















Now add the join mgr is null or loc = ‘NEW YORK’




FROM dept d



and ( e.mgr is null

or




ANSI Outer Join


-------------- ---------- ---------- ---------- -------------




The results so far are shown




FROM dept d



and ( e.mgr is null

or




ANSI Outer Join


-------------- ---------- ---------- ---------- -------------




RESEARCH 20 DALLAS

SALES 30 CHICAGO

OPERATIONS 40 BOSTON

Now add the LEFT OUTER to the DEPT table




FROM dept d



and ( e.mgr is null

or




ANSI Outer Join


-------------- ---------- ---------- ---------- -------------




RESEARCH 20 DALLAS

SALES 30 CHICAGO


Final six rows are shown




FROM dept d



and ( e.mgr is null

or



ANSI Outer Join

• Now consider the correctly translated outer join

Correct Translation of Oracle syntax to ANSI



ANSI Outer Join


-------------- ---------- ---------- ---------- -------------















Consider just the inner join d.deptno=e.deptno




FROM dept d




or




ANSI Outer Join


-------------- ---------- ---------- ---------- -------------
















Add the LEFT OUTER to the DEPT table




FROM dept d




or




ANSI Outer Join


-------------- ---------- ---------- ---------- -------------
















Now add the WHERE clause




FROM dept d




or




ANSI Outer Join


-------------- ---------- ---------- ---------- -------------





Final four rows are shown




FROM dept d




or


Real-World Performance Training · –SQL Design –Physical Layout •Table layout, partitions,...

Documents

Transcript of Real-World Performance Training · –SQL Design –Physical Layout •Table layout, partitions,...