OOUG - Oracle Performance Tuning with AAS

Oracle Performance Tuning

Kyle Hailey

[email protected]

kylehailey.com

1990 Oracle– 90 support– 92 Ported v6– 93 France– 95 Benchmarking – 98 ST Real World Performance

2000 Dot.Com 2001 Quest 2002 Oracle OEM 10g Success!

First successful OEM design

Who is Kyle Hailey

1990 Oracle– 90 support– 92 Ported v6– 93 France– 95 Benchmarking – 98 ST Real World Performance

2000 Dot.Com 2001 Quest 2002 Oracle OEM 10g 2005 Embarcadero

– DB Optimizer

Who is Kyle Hailey

Who is Kyle Hailey

• 1990 Oracle 90 support 92 Ported v6 93 France 95 Benchmarking 98 ST Real World Performance

• 2000 Dot.Com• 2001 Quest • 2002 Oracle OEM 10g• 2005 Embarcadero

DB Optimizer• Delphix

When not being a Geek- Have a little 4 year old boy who takes up all my time

Production Development QA UAT

Instance Instance Instance Instance

Database

File systemFile system

Database

File systemFile system

Database

File system

Database

File systemFile systemFile systemFile system

Database

Production

Database

Development

Database

QA

Database

UAT

Instance Instance Instance Instance

File system

My Goal

Simplify the information

and empower the DBA

How do you tune a Database?

Database is running slow!

• first step?

• questions?

• tools?

• repeatable method?

What statistics do you look at?

Disastrously difficult

Launch: Pressure

Midnight January 28, 1986 Lives are on the line

Thanks to Edward Tufte

Night before the Flight

Estimated launch temperature 29º

13 Pages Faxed

13 Pages Faxed

3 different types of names

Damage (in overwhelming detail) but No Temperatures

13 Pages Faxed

13 Pages Faxed

Test engines, not launches, fired horizontally

Missing temperatures for 5 erosion damage flights

Temperatures but limited Damage

13 Pages Faxed

“blow by”, not more important “erosion”, (at hottest and coldest launches)

13 Pages Faxed

Predict Temperature

Recommendation

55 65 7560 70 80

1

Original Engineering data

2

3

damages atdamages atthe hottest the hottest and coldest and coldest temperature temperature

- - managementmanagement

Would you launch?

Congressional Hearings Evidence

No Damage LegendDamage hard to read

Congressional Hearings Evidence

Temperature correlation difficult

55 65 7560 70 80

1

Original Data

2

3

Clearer

1. Y-Axis amount of damage (not number of damage)

55 65 7560 70 80

4

8

12


2. Include successes

55 65 7560 70 80

4

8

12

Clearer



3. Mark Differences

55 65 7560 70 80

4

8

12

Clearer



3. Mark Differences

4. Normalize same temp

55 65 7560 70 80

4

8

12

Clearer



3. Mark Differences


55 65 7560 70 80

4

8

12

Clearer

Damage on every flight below 65

No damage on every flight above 75



3. Mark Differences


55 65 7560 70 80

4

8

12

Clearer



3. Mark Differences


5. Scale known vs unknown

55 65 7560 70 80

4

8

12

4

8

12

30 40 5035 45

XX

Clearer

Difficult

NASA Engineers Fail Congressional Investigators Fail Data Visualization is Difficult

But …

Lack of Clarity can be devastating

Solutions

• Clear Identification• Know how to identify problems and issues

• Access to details• Provide solutions and/or information to address the issues

• Graphics• Easy understanding, effective communication and discussion

First Step: Graphics

“The humans … are exceptionally good at parsing visual information, especially when that information is coded by color and/or .”

Knowledge representation in cognitive science. Westbury, C. & Wilensky, U. (1998)

motion

Why Use Graphics

You can't imagine how many times I was told that nobody wanted or would use graphics …

-- Jef Raskin, the father of the Macintosh

Infocus – (overhead projectors) sited a study that humans can parse graphical information 400,000 times faster than textual data

Counties in US

3101 Counties 50 pages

“If I can't picture it, I can't understand it”

Anscombe's QuartetI II III IV

x y x y x y x y10 8.04 10 9.14 10 7.46 8 6.58

8 6.95 8 8.14 8 6.77 8 5.7613 7.58 13 8.74 13 12.74 8 7.71

9 8.81 9 8.77 9 7.11 8 8.8411 8.33 11 9.26 11 7.81 8 8.4714 9.96 14 8.1 14 8.84 8 7.04

6 7.24 6 6.13 6 6.08 8 5.254 4.26 4 3.1 4 5.39 19 12.5

12 10.84 12 9.13 12 8.15 8 5.567 4.82 7 7.26 7 6.42 8 7.915 5.68 5 4.74 5 5.73 8 6.89

Average 9 7.5 9 7.5 9 7.5 9 7.5

Standard Deviation 3.31 2.03 3.31 2.03 3.31 2.03 3.31 2.03Linear Regression 1.33 1.33 1.33 1.33

- Albert Einstein- Albert Einstein

Graphics for Anscombe’s Quartet

Do You Want?

Engineering Data?Engineering Data?

Pretty PicturesPretty Pictures

Do You Want?

Clean and Clear Clean and Clear

? ? ? ? ? ? ? ? ? ?? ?

Do You Want?

What is a day in the life look What is a day in the life look like for a DBA who has like for a DBA who has performance issues?performance issues?

Tuning the Database

Anscombe's QuartetI II III IV

x y x y x y x yAverage 9 7.5 9 7.5 9 7.5 9 7.5Standard Deviation 3.31 2.03 3.31 2.03 3.31 2.03 3.31 2.03Linear Regression 1.33 1.33 1.33 1.33

ComplexComplex

AveragesAverages

What statistics do you look at?

AAS

Imagine Trying to Drive your Car

And is updated once and hourAnd is updated once and hour

Or would you like it to Or would you like it to look …look …

Would you want your dashboard to look like :Would you want your dashboard to look like :

How Can We Open the Black Box?

Max CPU

(yard stick)

Top Activity Top Activity

SQLSQLSessionsSessions

LOADLOAD

How Can We Open the Black Box?OEM OEM ASHMON/SASHASHMON/SASH DB OptimizerDB Optimizer

• Powerful - Identifies issues quickly and indepth

• Interactive - Allows exploring the data

• Easy - Understandable by everyone, DBA, Dev and Managers !

Ideas for Today

ASHASH

SamplingSampling

WaitWaitss

AASAAS

Average Active Sessions (AAS)

The Most Important Metric

Kyle Hailey

[email protected]

http://kylehailey.com

4704/10/23

In this Session1. AAS

– Single Metric– DB Performance

2. Yardstick – CPU Count – Compare to AAS

3. AAS components– CPU – Waits– Time series

4804/10/23

Database Performance

How quick can you find

Bottleneck in DB

If DB is idle

Current DB Load what is DB Load ?

What do you use?

Statspack/AWR

V$active_session_history

Alerts what do you alert on ?

what the *!####!*!*? is the database doing ?!

Ideas for Today

ASHASH

SamplingSampling

WaitWaitss

AASAAS

Copyright 2006 Kyle Hailey

Statspack: Cheat SheetStatspack: Cheat Sheet

• Install

– Connect as SYSDBA– @?/rdbms/admin/spcreate.sql

• Run

– Exec statspack.snap;• Generate Reports

– @?/rdbms/admin/spreport.sql– (AWR : @?/rdbms/admin/awrrpt.sql )

Statspack = always available

Statspack Sections (10g)Statspack Sections (10g)Instance descriptionHost hardwareSnapshot headlineCache informationLoad profileInstance efficiencyShared pool statsTop timed eventsHost CPU loadHost / Instance CPU warningVM activityMemory usageTime model statsRAC statisticsAll wait eventsBackground wait eventsEvent histogramSQL x 9Instance activityLog switchesOS stats

Latch parent and childMutexSegment stats x 8Dictionary cacheLibrary cacheRAC (GES)RAC (CR and CUR

served)RAC (cache xfer x 2)RAC (Remastering)Streams x 7Shared pool advisorJava pool advisorSGA resizingSGA target advisorSGA summarySGA detailSQL memory summaryResource limitsParameters

Session details x 3Tablespace I/OFile I/OFile I/O histogramBuffer poolInstance RecoveryBuffer pool advisoryBuffer busy waitsVarious PGA

summariesPGA histogramPGA advisoryPGA allocation

summaryPGA allocation top

NEnqueue (lock)

activityUndo stats x 2Latch activityLatch miss details

Statspack MethodStatspack Method

Load profile good for having a feel for the application and comparing two periods for changes Efficiency ratios misleading carry over from version 6 days

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call Time---------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72CPU time 32 10.16free buffer waits 1,588 15 4.63write complete waits 10 8 2.51log buffer space 306 5 1.51 ----------------------------------------

Summary

Waits

Who/When

Big Picture

Statspack MethodStatspack Method

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call Time---------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72CPU time 32 10.16free buffer waits 1,588 15 4.63write complete waits 10 8 2.51log buffer space 306 5 1.51 ----------------------------------------

v$system_event - wait eventsv$sysstat – cpu stats

Old method:

compare wait time to CPU time

Current MethodCurrent Method

• 1000 lines of data / 30 pages• What do you look at ?• Top 5 Timed Events?

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time------------------------ ------------ ----------- --------CPU time 11,330 81.95log file sync 163,725 1,012 7.32enqueue 4,486 615 4.45latch free 28,786 236 1.71db file sequential read 1,420,355 149 1.08 -----------------------------------------

Jonathan Lewis - http://www.jlcomp.demon.co.uk/statspack_01.html

188 minutes CPUElapsed 15 minutes48 Processors or74% idle

Analysis requires•elapsed time •CPU core count

Average Active Sessions (AAS)

DB Time = wait time + CPU time

AAS = DB Time/Elapsed Time

AAS like run queue on UNIX

Ideas for Today

ASHASH

SamplingSampling

WaitWaitss

AASAAS

5704/10/23

AWR ReportAWR Report• AAS = DB Time/Elapsed Time

• 23.56/59.66 = 0.39

AAS= 0.39

ASHRPTASHRPTASH Report For TESTDB/testdbASH Report For TESTDB/testdb

DB Name DB Id Instance Inst Num Release RAC HostDB Name DB Id Instance Inst Num Release RAC Host

------------ ----------- ------------ -------- ----------- --- ------------------------ ----------- ------------ -------- ----------- --- ------------

TESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604aTESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604a

CPUsCPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size SGA Size Buffer Cache Shared Pool ASH Buffer Size

-------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ ------------------

22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%)

Analysis Begin Time: 21-Apr-06 12:00:01Analysis Begin Time: 21-Apr-06 12:00:01

Analysis End Time: 21-Apr-06 12:05:01Analysis End Time: 21-Apr-06 12:05:01

Elapsed Time: 5.0 (mins)Elapsed Time: 5.0 (mins)

Sample Count: 3,716Sample Count: 3,716

Average Active Sessions: 12.39Average Active Sessions: 12.39

Avg. Active Session per CPU: 6.19Avg. Active Session per CPU: 6.19

Report Target: None specifiedReport Target: None specified

Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)

Avg ActiveAvg Active

Event Event Class % Activity SessionsEvent Event Class % Activity Sessions

----------------------------------- --------------- ---------- --------------------------------------------- --------------- ---------- ----------

CPU + Wait for CPU CPU 67.98 8.42CPU + Wait for CPU CPU 67.98 8.42

enq: TX - row lock contention Application 23.98 2.97enq: TX - row lock contention Application 23.98 2.97

buffer busy waits Concurrency 4.66 0.58buffer busy waits Concurrency 4.66 0.58

latch: cache buffers chains Concurrency 2.26 0.28latch: cache buffers chains Concurrency 2.26 0.28

ASH Report For TESTDB/testdbASH Report For TESTDB/testdb


------------ ----------- ------------ -------- ----------- --- ------------------------ ----------- ------------ -------- ----------- --- ------------



-------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ ------------------

22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%)











----------------------------------- --------------- ---------- --------------------------------------------- --------------- ---------- ----------





5904/10/23

Statspack AAS Statspack AAS

Look forElapsed TimeTop 5 Timed Events

Start at line 52 of about 1300Start at line 52 of about 1300

6004/10/23

Statspack AASStatspack AAS• Elapsed Time

• Look at Top 5 Timed Events

Top 5 Timed EventsTop 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call TimeEvent Waits Time (s) Call Time

--------------------- --------- --------- ---------------------------- --------- --------- -------

buffer busy waits 2,748 250 78.72buffer busy waits 2,748 250 78.72CPU time 32 10.16CPU time 32 10.16free buffer waits 1,588 15 4.63free buffer waits 1,588 15 4.63write complete waits 10 8 2.51write complete waits 10 8 2.51log buffer space 306 5 1.51log buffer space 306 5 1.51

Top 5 Timed EventsTop 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call TimeEvent Waits Time (s) Call Time

--------------------- --------- --------- ---------------------------- --------- --------- -------

buffer busy waits 2,748 250 78.72buffer busy waits 2,748 250 78.72CPU time 32 10.16CPU time 32 10.16free buffer waits 1,588 15 4.63free buffer waits 1,588 15 4.63write complete waits 10 8 2.51write complete waits 10 8 2.51log buffer space 306 5 1.51log buffer space 306 5 1.51

STATSPACK report for

DB Name DB Id Instance Inst Num Release RAC Host------- ----------- -------- -------- ---------- --- -------

LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr

Snap Id Snap Time Sessions Curs/Sess

--------- ------------------ -------- ---------

Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 End Snap: 2 03-Apr-06 12:34:36 18 4.8 Elapsed: 1.00 (mins)

STATSPACK report for

DB Name DB Id Instance Inst Num Release RAC Host------- ----------- -------- -------- ---------- --- -------

LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr

Snap Id Snap Time Sessions Curs/Sess

--------- ------------------ -------- ---------

Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 End Snap: 2 03-Apr-06 12:34:36 18 4.8 Elapsed: 1.00 (mins)

61

Statspack AASStatspack AAS

• DBTIME= CPU + WAITS– CPU = 32

– WAITS = 250+15+8+5 = 278 secsDBTIME=320

• Elapsed Time = 60 secs

• 320 secs / 60 secs

AAS = 5.1

CPU core count = 2

Top 5 Timed EventsTop 5 Timed Events Event Time (s) Event Time (s) ----------------- ---------------------- -----buffer busy waits 250buffer busy waits 250CPU time 32CPU time 32free buffer waits 15free buffer waits 15write complete waits 8write complete waits 8log buffer spacelog buffer space 55

Top 5 Timed EventsTop 5 Timed Events Event Time (s) Event Time (s) ----------------- ---------------------- -----buffer busy waits 250buffer busy waits 250CPU time 32CPU time 32free buffer waits 15free buffer waits 15write complete waits 8write complete waits 8log buffer spacelog buffer space 55

AAS FormulasAAS FormulasUse CPU count as yardstick:

AAS < 1 Database is not blocked

AAS ~= 0 Database basically idleProblems are in the APP not DB

AAS < # of CPUsCPU availableAre any single sessions 100% active?

• AAS > # of CPUsCould have performance problems

AAS >> # of CPUSThere is a bottleneck

Ideal world – one databasesolution track CPU at OS

AAS > 1 still want to know if a single user is 100% active

AAS in OEMAAS in OEM

LOAD

AAS

6404/10/23 Copyright 2006 Kyle Hailey

WhatWhat’’s the DB Doing?!s the DB Doing?!

It’s 2am … your manager calls

Whip out the stethoscope:

AAS

what the *!####!*!*? is the database doing ?!

AAS ComponentsAAS Components

Performance Page

Available CPU AAS:CPU + WAIT

Real CPU available:

Max CPU - non instance CPU

OEMOEM

Relax Relax Get to Work!Get to Work!Looks Looks OK OK

But …But …

What if AAS What if AAS indicates indicates a problem?a problem?

Latch Free Latch Free Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time-------------------------------------------- ------------ ----------- --------latch free 9,652 760 66.10CPU time 248 21.62PL/SQL lock timer 41 123 10.72SQL*Net message from dblink 681 14 1.22log file parallel write 128 1 .13 -------------------------------------------------------------

What Latch? There are 100sLatch Sleep breakdown for DB: CDB Instance: cdb Snaps: 3 -4-> ordered by misses descLatch Name Requests Misses Sleeps Sleeps Sleeps 1->4-------------------------- -------------- ----------- ----------- ------------cache buffers chains 8,448,787 649,484 cache buffers chains 8,448,787 649,484 6,9306,930 0/0/0/0/0 0/0/0/0/0library cache pin 8,405,896 82,915 1,427 81537/1330/4 7/1/0library cache 8,435,488 55,645 1,294 54375/1247/2 2/1/0shared pool 58,626 7 1 6/1/0/0/0

Row Locks 10g+Row Locks 10g+op 5 Timed Events Avg %Total~~~~~~~~~~~~~~~~~~ wait CallEvent Waits Time (s) (ms) Time----------------------------------------- ------------ ----------- ------ ------enq: TX - row lock contention 59 160 2714 41.8PL/SQL lock timer 4 117 29291 30.6CPU time 28 7.2buffer busy waits 1,217 18 15 4.7log file parallel write 422 11 27 3.0

Who is waiting

Who is blocking

What is the SQL

What is the row?

Buffer Busy WaitBuffer Busy WaitTop 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call Time-------------------------------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72CPU time 32 10.16free buffer waits 1,588 15 4.63write complete waits 10 8 2.51log buffer space 306 5 1.51 ------------------------------------------------------------

Buffer Busy Wait – trying to modify a block

Who blocks? What object? what is the SQL?

Buffer wait Statistics DB/Inst: DB4/db4 Snaps: 2-3Class Waits Wait Time (s) Avg Time (ms)------------------ ----- ------------- -------------file header block 264 203 769data block 6,070 162 27undo header 355 0 1

Statspack fails for analysis

CPU in ProblemCPU in Problem

OEM Performance Page: Buffer Busy Waits

Dilema: AAS indicates problem but not solutionsProblem in •Statspack•AWR •OEM performance pageCan’t analyze•Locks•Buffer Busy Waits•LatchesAlso can’t see when demand for CPU is greater than available CPU

Ideas for Today

ASHASH

SamplingSampling

WaitWaitss

AASAAS

Example• Look at Statspack:

Top 5 Timed EventsTop 5 Timed Events

~~~~~~~~~~~~~~~~~~ % Total~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call TimeEvent Waits Time (s) Call Time--------------------------- ----------- ------------------------------------ ----------- ---------buffer busy waits 2,748 250 78.72buffer busy waits 2,748 250 78.72CPU time 32 10.16CPU time 32 10.16free buffer waits 1,588 15 4.63free buffer waits 1,588 15 4.63write complete waits 10 8 2.51write complete waits 10 8 2.51log buffer space 306 5 1.51log buffer space 306 5 1.51--------------------------------------------------------------

What do we do?

• How do we solve “buffer busy wait”• Need:

• SQL• Sessions• Objects • Wait details

– Type of buffer busy wait– File and block numbers

• How do we get that information?– Not from Statspack or AWR

– But from v$session• v$session_wait prior to 10g

V$session

• Shows who is waiting – For example lists who is waiting on “buffer busy waits”

– Data only exists while waits are happening

– Data includes• Type of buffer busy wait• File and block involved in buffer busy wait

• Problem: Once waits are over, data is gone• Solution: Sample data all the time

Sample Query

select nvl(s.username,s.program) username, s.sid sid, s.serial# serial, s.sql_hash_value sql_hash_value, substr(decode(w.wait_time, 0, w.event, 'ON CPU'),1,15) event , w.p1 p1, w.p2 p2, w.p3 p3from v$session s, v$session_wait wwhere w.sid=s.sidand s.status='ACTIVE'and s.type='USER';

This query works since v7This query works since v7

SQL identifierSQL identifier

Waiting or on CPU?Waiting or on CPU?

Sampling Output

USERNAME SID SERIAL SQL_HASH_V EVENT P1 P2 P3USERNAME SID SERIAL SQL_HASH_V EVENT P1 P2 P3---------- ----- -------- ---------- -------------------- -------- -------- -------------- ----- -------- ---------- -------------------- -------- -------- ----SYS 64 8717 4116021597 PL/SQL lock timer 300 0 0SYS 64 8717 4116021597 PL/SQL lock timer 300 0 0SYS 58 19467 961168820 ON CPU 16508152 1 0SYS 58 19467 961168820 ON CPU 16508152 1 0STARGUS 71 6251 1311875676 direct path write 201 2155902 127STARGUS 71 6251 1311875676 direct path write 201 2155902 127(CJQ0) 9 1 0 rdbms ipc message 500 0 0(CJQ0) 9 1 0 rdbms ipc message 500 0 0

Run sample query every second and

save into a table “v$ash”

Create table v$ash as select … ;

Insert into v$ash select … ;

Buffer busy wait typeSQL> Select count(*), p3 from v$ash where event = 'buffer busy waits' group by p3;

COUNT(*) P3---------- ----3423 1

P3 = block type

Buffer busy wait type

Block types come from

select rownum n,

class

from v$waitstat;

N CLASS

--- ------------------

1 data block 2 sort block 3 save undo block 4 segment header 5 save undo header 6 free list 7 extent map

8 1st level bmb 9 2nd level bmb 10 3rd level bmb 11 bitmap block 12 bitmap index block 13 file header block 14 unused 15 system undo header 16 system undo block 17 undo header 18 undo block

File and Block #sselect count(*), p1 filen, p2 blockn from v$ashwhere event='buffer busy waits'group by p1, p2, hash_value;

COUNT(*) FILEN BLOCKN---------- -------- -------- 1 11 90644 2 11 90651 3 11 98233 1 11 104767 3 11 113291 1 11 119842 1 11 119856 3 11 121632 1 11 126334

Pick a File and Block to find object

Find Objectcolumn segment_name format a30column segment_name format a30

select select owner,owner, segment_name, segment_name, segment_type, segment_type, block_id, blocks+block_id block_id, blocks+block_idfrom dba_extentsfrom dba_extentswhere file_id = 11where file_id = 11and 126334 between block_id AND block_id + blocks-1;and 126334 between block_id AND block_id + blocks-1;

OWNER SEGMENT_NAME SEGMENT_TY BLOCK_ID BLOCKS+BLOCK_IDOWNER SEGMENT_NAME SEGMENT_TY BLOCK_ID BLOCKS+BLOCK_ID---------- ------------------ ---------- ---------- ------------------------- ------------------ ---------- ---------- ---------------SYSTEM TOTO1 TABLE 125201 127249SYSTEM TOTO1 TABLE 125201 127249

Data block ( not header block )

COUNT(*) FILEN BLOCKN---------- -------- -------- 1 11 126334

126334

What SQL ?SQL> select count(*), sql_hash_valueSQL> select count(*), sql_hash_value2 from v$ash2 from v$ash3 where event='buffer busy waits'3 where event='buffer busy waits'4 group by hash_value;4 group by hash_value;

COUNT(*) SQL_HASH_VALUECOUNT(*) SQL_HASH_VALUE---------- ------------------------ -------------- 3423 558666863558666863

All the same SQL SQL_HASH_VALUE= 558666863

In version 10g+ use SQL_ID

SQL Statement ?select sql_textfrom v$sqltextwhere hash_value=558666863;

SQL_TEXT-----------------------------------------------------INSERT into toto1 values (:b1,lpad('a',1000,'a'))

• Insert statement

• Problem on a data block on

• table toto1

Solution: Freelists or ASSM

Sampling Summary

• Sampling v$session – finds the root of problems– otherwise impossible to diagnose

• Sample Data from v$session is called – Active Session History (ASH)– v$active_session_history

Diagnosis is still complex

But

done automatically by ADDM

OEM & ADDM Example

Database Home Page

Performance Page

Drilldowns

SQL Session

ADDM Page

Solutions

Four Tuning Paths

Path 1

Database Home Page

Performance Page

Drilldowns

SQL Session

ADDM Page

Solutions

Actions

Database Home Page

ADDM Details

Solution ASSMSolution ASSM

Solution 2: PartitioningSolution 2: Partitioning

Solution 3: Free ListsSolution 3: Free Lists

TablespaceTablespace

ObjectObject

Path 2

Database Home Page

Performance Page

Drilldowns

SQL Session

ADDM Page

Solutions

Actions

Database Home Page

ADDM Page

ADDM Details

EM Product Layout for Performance


Database Home Page

Performance Page

Drilldowns

SQL Session

ADDM Page

Solutions

Database Home Page

Database Performance Page

Lots of waitsLots of waits

ADDM ReportADDM Report

ADDM Page

ADDM Details

EM Product Layout for Performance

Database Home Page

Performance Page

Drilldowns

SQL Session

ADDM Page

Solutions

Database Performance Page

““click on click on the big the big stuffstuff””

Wait Drilldown : Concurrency

SQL Drilldown

Session Drilldown

Session : Wait History


Part II – Taking ADDM Advice• 10 users inserting concurrently into same table

insert into toto1 values (t,lpad('a',1000,'a'));insert into toto1 values (t,lpad('a',1000,'a'));

Test 1

Elapsed Time 14:44Elapsed Time 14:44

ADDM Details

• Use ASSM

• Use Freelists

• Partition Object

=> Trying out Freelists

Test 2


Was 14:44 – now slower! Was 14:44 – now slower!

ADDM Details

• Increase Log File Size to 2000M to hold at least 20 minutes of redo

Test 3

Elapsed 1:23Elapsed 1:2314:4414:44 14:5914:59


10x Improvement! 10x Improvement!

Final ADDM Details

Reduce the number of commits

=> Commit every 50 records instead of every record

Final

• Commit every 50 records instead of every record• Got elapsed down from

– 1:23

– 0:47

In Review

• By using– Free Lists

– Large Redo Log files

• Job Processing went from– Over 14 minutes to

– Just over 1 minute

• Thanks to ADDM and OEM !

Ideas for Today

ASHASH

SamplingSampling

WaitWaitss

AASAAS

Measuring Active Sessions

Query run often

Fast query run rarely

Slow query

User 1

User 2

User 3

Sampling Every Second

. . .

TimeTime

idleidle

Active Session

idleidle idleidle idleidleQuery 1 Query 2 Query 3

TimeTime

Active SessionActive Session

CPU IO Wait

idleidleidleidle idleidle idleidleQuery 1 Query 2 Query 3

TimeTime

Active SessionActive Session

CPU IO Wait

Work Work LatencyLatencyContentionContention

v$active_session_historyv$active_session_history

SESSION_ID NUMBER SESSION_SERIAL# NUMBER USER_ID NUMBER SERVICE_HASH NUMBER SESSION_TYPE VARCHAR2(10) PROGRAM VARCHAR2(64) MODULE VARCHAR2(48) ACTION VARCHAR2(32) CLIENT_ID VARCHAR2(64)

EVENT VARCHAR2(64) EVENT_ID NUMBER EVENT# NUMBER SEQ# NUMBER P1 NUMBER P2 NUMBER P3 NUMBER WAIT_TIME NUMBER TIME_WAITED NUMBER CURRENT_OBJ# NUMBER CURRENT_FILE# NUMBER CURRENT_BLOCK# NUMBER0

SQL_ID VARCHAR2(13) SQL_CHILD_NUMBER NUMBER SQL_PLAN_HASH_VALUE NUMBER SQL_OPCODE NUMBER QC_SESSION_ID NUMBER QC_INSTANCE_ID NUMBER

SAMPLE_ID NUMBER SAMPLE_TIME TIMESTAMP(3)

SESSION_STATE VARCHAR2(7) WAIT_TIME NUMBER

TIME_WAITED NUMBER

When

Session

SQL

Wait

State

Duration

Primary Fields of Primary Fields of ASHASH

SESSION_ID

EVENT

SQL_ID

SAMPLE_TIME Time

Session

SQL

Wait

SESSION_STATE State

Activity : Who, What, When, How

Who

What

When

How

Ideas for Today

ASHASH

SamplingSampling

WaitWaitss

AASAAS

Sampling is like taking Pictures

Session 1

Session 2

Session 3

Session 4

TIME

Graphical ASHGraphical ASH

Graph of User StatesGraph of User States

One Second GraphOne Second Graph

Ideas for Today

ASHASH

SamplingSampling

WaitWaitss

AASAAS

15 Second Averages15 Second Averages

Maximum CPU LineMaximum CPU Line

Idle UsersIdle Users

The Power The Power

ASH givesASH givesAASAAS


DB Home DB Home

PerformancePerformance

Top ActivityTop Activity

Based on Based on ASHASH

Based on TIMEBased on TIMEv$system_eventv$system_eventv$sysstatv$sysstat

AAS=db AAS=db time/elapsed timetime/elapsed time

AAS=count active AAS=count active users /samplesusers /samples

PerformancePerformance Top Activity Top Activity

2. OEM 10g : Top Activity2. OEM 10g : Top Activity

•Top Activity•Based on ASH•Enables Drilldowns

•Top SQL•Top Session

•Drill into a session• Stats• Raw waits• Open cursors• General info

•Drill into a SQL• Stats and text• Users executing• Explain plan• Tuning options

Top Activity : Based on ASHTop Activity : Based on ASH

Thanks Thanks

To To

ASHASH

Statistic Lag TimeStatistic Lag Time

Real TimeReal Time

Slight lagSlight lag

ASH

Wait Stats

OEM Perf PagesOEM Perf Pages

PerformancePerformance

SQLSQLSessionSession

Top ActivityTop Activity

SQLSQLSessionSession

13704/10/23

AAS from ASHAAS from ASH1. ASHRPT

– Based entirely on v$active_session_history

– @?/rdbms/admin/ashrpt.sql– Exec ASH_REPORT_TEXT/HTML

select * from tableselect * from table(dbms_workload_repository.ash_report_text((dbms_workload_repository.ash_report_text( (select dbid from v$database),(select dbid from v$database), 1,1, sysdate – 1/24, sysdate – 1/24, sysdate )) ;sysdate )) ;

ASHRPTASHRPTASH Report For TESTDB/testdbASH Report For TESTDB/testdb


------------ ----------- ------------ -------- ----------- --- ------------------------ ----------- ------------ -------- ----------- --- ------------



-------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ ------------------

22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%)











----------------------------------- --------------- ---------- --------------------------------------------- --------------- ---------- ----------





ASH Report For TESTDB/testdbASH Report For TESTDB/testdb


------------ ----------- ------------ -------- ----------- --- ------------------------ ----------- ------------ -------- ----------- --- ------------



-------- ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ ------------------ ------------------

22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%)











----------------------------------- --------------- ---------- --------------------------------------------- --------------- ---------- ----------





ASH RPTASH RPT

1) General info

2) Top User Events ***

3) Top Background Events

4) Top Event P1/P2/P3 Values

5) Top Service/Module

6) Top Client IDs

7) Top SQL Command Types

8) Top SQL Statements ***

1) General info

2) Top User Events ***

3) Top Background Events

4) Top Event P1/P2/P3 Values

5) Top Service/Module

6) Top Client IDs

7) Top SQL Command Types

8) Top SQL Statements ***

9) Top SQL using literals

10) Top Sessions ***

11) Top Blocking Sessions

12) Top Sessions running PQs

13) Top DB Objects

14) Top DB Files

15) Top Latches

16) Activity Over Time ***

9) Top SQL using literals

10) Top Sessions ***

11) Top Blocking Sessions

12) Top Sessions running PQs

13) Top DB Objects

14) Top DB Files

15) Top Latches

16) Activity Over Time ***

ASHRPT over TimeASHRPT over Time

• Waits over Time–Not in AAS

• Difficult but better

than nothing

Compare to …

14104/10/23

Custom ScriptsCustom Scripts• Hate Graphics ?

–Query v$active_session_history directly–Join to dba_hist_active_sess_history for week of data

• act.sql–Like top 5 timed events

• Aveact.sql–Charts with text AAS by hour (15 minute, minute , etc)

• Aveactn.sql–Ditto, with top 2 wait events per bucket

• Following Scripts Available on

http://ashmasters.com

Custom ScriptsCustom Scripts@actAnalysis Begin Time : 2007-07-24 11:04:48Analysis End Time : 2007-07-24 11:19:45Start time, mins ago: 15Request Duration : 15Collections : 528Data Values : 3327Elapsed Time: 15 mins

WAIT_EVENT CNT % Active Ave_Act_Sess

-------------------------------------- ---------- ------------

latch free 10 .3 .02log buffer space 13 .39 .02buffer busy waits 14 .42 .03db file scattered read 15 .45 .03library cache pin 78 2.34 .15log file sync 213 6.40 .40ON CPU 726 21.82 1.38enqueue 855 25.70 1.62db file sequential read 1399 42.05 2.65

------------

sum 6.30

@actAnalysis Begin Time : 2007-07-24 11:04:48Analysis End Time : 2007-07-24 11:19:45Start time, mins ago: 15Request Duration : 15Collections : 528Data Values : 3327Elapsed Time: 15 mins

WAIT_EVENT CNT % Active Ave_Act_Sess

-------------------------------------- ---------- ------------

latch free 10 .3 .02log buffer space 13 .39 .02buffer busy waits 14 .42 .03db file scattered read 15 .45 .03library cache pin 78 2.34 .15log file sync 213 6.40 .40ON CPU 726 21.82 1.38enqueue 855 25.70 1.62db file sequential read 1399 42.05 2.65

------------

sum 6.30

Custom ScriptsCustom Scripts

@aveactTM NPTS AVEACT GRAPH CPU WAITS---------------- ------ ------- ---------------------- ---- -----06-AUG 13:00:00 270 .33 +- 2 29 5906-AUG 14:00:00 1040 2.24 ++--------2--- 341 198406-AUG 15:00:00 623 6.67 ++++------2---------- 438 371806-AUG 16:00:00 1088 2.59 ++--------2---- 335 248606-AUG 17:00:00 1104 1.26 ++----- 2 349 104306-AUG 18:00:00 1093 1.38 +++---- 2 663 84206-AUG 19:00:00 1012 1.74 ++------- 2 373 138806-AUG 20:00:00 1131 .99 +---- 2 304 82006-AUG 21:00:00 1111 1.22 ++----- 2 344 101206-AUG 22:00:00 1010 1.66 ++------ 2 414 125906-AUG 23:00:00 1120 1.08 +---- 2 298 91307-AUG 00:00:00 1024 .83 +--- 2 273 57607-AUG 01:00:00 1006 1.74 ++------- 2 319 142807-AUG 02:00:00 1090 2.47 ++--------2---- 347 234507-AUG 03:00:00 687 6.59 +++-------2---------- 382 414207-AUG 04:00:00 1004 1.95 ++++++--- 2 1299 65907-AUG 05:00:00 1104 3.08 +++++-----2------ 1170 222607-AUG 06:00:00 1122 1.91 +++++++-- 2 1582 558

@aveactTM NPTS AVEACT GRAPH CPU WAITS---------------- ------ ------- ---------------------- ---- -----06-AUG 13:00:00 270 .33 +- 2 29 5906-AUG 14:00:00 1040 2.24 ++--------2--- 341 198406-AUG 15:00:00 623 6.67 ++++------2---------- 438 371806-AUG 16:00:00 1088 2.59 ++--------2---- 335 248606-AUG 17:00:00 1104 1.26 ++----- 2 349 104306-AUG 18:00:00 1093 1.38 +++---- 2 663 84206-AUG 19:00:00 1012 1.74 ++------- 2 373 138806-AUG 20:00:00 1131 .99 +---- 2 304 82006-AUG 21:00:00 1111 1.22 ++----- 2 344 101206-AUG 22:00:00 1010 1.66 ++------ 2 414 125906-AUG 23:00:00 1120 1.08 +---- 2 298 91307-AUG 00:00:00 1024 .83 +--- 2 273 57607-AUG 01:00:00 1006 1.74 ++------- 2 319 142807-AUG 02:00:00 1090 2.47 ++--------2---- 347 234507-AUG 03:00:00 687 6.59 +++-------2---------- 382 414207-AUG 04:00:00 1004 1.95 ++++++--- 2 1299 65907-AUG 05:00:00 1104 3.08 +++++-----2------ 1170 222607-AUG 06:00:00 1122 1.91 +++++++-- 2 1582 558

Aveact.sqlAveact.sql

Aveact.sqlAveact.sql

“-” = WAIT

“+” = CPU

which waits ? -> aveactn.sql

Aveactn.sqlAveactn.sql

Before Tuning Oracle, Tune the MachineMake sure the machine is healthy before tuning Oracle CPU => use run queue, < 2 * #CPU Memory => page out

VMSTAT

Summary

1.Machine - vmstat– Memory, CPU (we can see IO response in Oracle)

2.Database - AAS– Use wait interface and graphics

– Identify machine, application, database or SQL

3.SQL - VST– Indexes, stats, execution path

– Visual SQL Tuning

Bibliography

Refactoring SQL Applications - Stephane Faroult

Troubleshooting Oracle Performance - Christian Antognini

SQL Tuning - Dan Tow

Oracle Wait Interface - Kirtikumar Deshpande, K. Gopalakrishnan

Books

The Visual Display of

Quantitative Information– Eduard Tufte

For the Space Shuttle disaster, seehttp://williamwolff.org/wp-content/uploads/2013/01/tufte-challenger-1997.pdfhttp://www.onlineethics.org/cms/12709.aspx

DesignDesign

152

Using graphics for database

System Tuning

SQL Tuning

ASSM ?

Copyright 2006 Kyle HaileyCopyright 2006 Kyle Hailey

select FREELISTS, TABLESPACE_NAMEselect FREELISTS, TABLESPACE_NAMEfrom dba_tablesfrom dba_tableswhere table_name='TOTO1' and owner='SCOTT';where table_name='TOTO1' and owner='SCOTT';

FREELISTS TABLESPACE_NAMEFREELISTS TABLESPACE_NAME---------- ---------------------------------------- ------------------------------1 USERS1 USERS

Table only has 1 freelist

The table is in tablespace Users

Is it an ASSM tablespace ?

Sampling Output


select tablespace_name, SEGMENT_SPACE_MANAGEMENTfrom dba_tablespaces;

TABLESPACE_NAME SEGMEN------------------------------ ------SYSTEM MANUALUNDOTBS1 MANUALTEMP MANUALUSERS MANUALTOOLS AUTO

USERS tablespace is not in ASSM mode

Let’s add Freelists


select count(*), sid, serial select count(*), sid, serial from v$ash from v$ash where event='buffer busy waitswhere event='buffer busy waits‘‘group by sid, serial;group by sid, serial;

COUNT(*) SID SERIAL COUNT(*) SID SERIAL---------- ----- ------------------ ----- -------- 7 12 79 7 12 79 4 14 99 4 14 99 4 19 770 4 19 770 8 20 176 8 20 176

Max Concurrency is 4

Freelists >= 4

How many freelists?

OOUG - Oracle Performance Tuning with AAS

Technology

Transcript of OOUG - Oracle Performance Tuning with AAS