IO Waits Kyle Hailey [email protected] .

IO Waits

Kyle Hailey

[email protected]

http://perfvision.com/ftp/sb

#.2

Copyright 2006 Kyle Hailey

Waits Covered in this Section1. db file sequential read2. db file scattered read3. db file parallel read4. read by other session5. direct path read 6. direct path write7. direct path read temp8. direct path write temp9. direct path write (lob)10. file open11. file identify12. Local write wait13. data file init write14. sort segment request

Standard I/OStandard I/O

Direct I/ODirect I/O

Special CasesSpecial Cases

#.3


Buffer Cache

REDOREDO Lib Lib CacheCache

Buffer Buffer CacheCache

LocksLocks

NetworkNetwork

I/OI/O

#.4


IO Waits

Data FilesData Files

DBWRDBWR

Buffer Buffer CacheCache

User1User1

SGASGA

Library Library CacheCache

Log Log BufferBuffer

#.5


Standard Table/Index IO

1. db file sequential read Single block read

2. db file scattered read Multi block read

3. db file parallel read Non-contiguous multi block read

4. read by other session Wait for another session to do the io

#.6


db file sequential read

ShadowShadowProcessProcess

1) Search Buffer Cache for the block by rowid

2) Fails3) Reads of disk

• File• Block

Buffer CacheBuffer Cache The top wait Single Block Read

Data via Index and/or RowidRollback

Select * from emp where empno = 9999;Select * from emp where empno = 9999;

Note: “sequential” meansA sequence like a rowid

Index on emp.empno

#.7


db file scattered read

ShadowShadowProcessProcess 1) Search Buffer Cache for the

blocks

2) Fails3) Reads off disk

• File• Block• Multi Block Read Count

Buffer Cache

Multi Block ReadFull Table Scan Index Fast Full Scans

SQL> select * from all_objects;SQL> select * from all_objects;

Note: Scattered Means

Blocks are read and scattered throughout buffer cache

#.8


db file parallel read

Shadow

Process

1) Search Buffer Cache for the blocks

2) Fails

3) Reads that block off Disk

Buffer CacheBuffer Cache

Process issues multiple single block reads in parallel Documentation says only for recovery But seems to happen for normal read ops Async Call – wait for all reads to complete

Examples Not contiguous multi block read Index full or range scan Where values in Select * from emp where

Empno in (1,2,3,4,5,6);

Select * from emp where

Empno in (1,2,3,4,5,6);

#.9


Read by other Session

1. Block not found in cache2. Read block from disk3. Found other session already

reading block from disk4. Wait for the other session to finish

IO

S1S1


S2S2

Multiple sessions reading the same data that requires IO

Ex) to users doing a full table scan at the same time

#.10


Db file Sequential Read: Statspack

Top 5 Timed Events

~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time------------------------ ------------ ----------- --------db file scattered read 7,549 5 52.26CPU time 2 23.29db file sequential read 884 2 19.11control file sequential read 154 0 2.68control file parallel write 255 0 1.87

Top 5 Timed Events

~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time------------------------ ------------ ----------- --------db file scattered read 7,549 5 52.26CPU time 2 23.29db file sequential read 884 2 19.11control file sequential read 154 0 2.68control file parallel write 255 0 1.87

What SQL ?

What Object ?

#.11


Db file sequential read

#.12


db file scattered Read

#.13


Read by other session

#.14


IO – Further Investigationselect parameter1, parameter2, parameter3 from v$event_nameselect parameter1, parameter2, parameter3 from v$event_name

NAME P1 P2 P3NAME P1 P2 P3

----------------------- ----------- --------- ------------------------------- ----------- --------- --------

read by other session file# block# class#read by other session file# block# class#db file sequential read file# block# blocksdb file sequential read file# block# blocksdb file scattered read file# block# blocksdb file scattered read file# block# blocks

direct path read file number first dba block cntdirect path read file number first dba block cnt

direct path read temp file number first dba block cntdirect path read temp file number first dba block cnt

direct path write file number first dba block cntdirect path write file number first dba block cnt

direct path write temp file number first dba block cntdirect path write temp file number first dba block cntdb file parallel read files blocks requestsdb file parallel read files blocks requests

P1 and P2 are the same, file# , block# P1 and P2 are the same, file# , block#

We can use P1, P2, P3 from ASH to get more info We can use P1, P2, P3 from ASH to get more info

Exception:Exception:db file parallel read – p1,p2,p3 not usefuldb file parallel read – p1,p2,p3 not useful

#.15


ASH – sequential readsselectselect event,event, ash.p3,ash.p3, o.object_name objn,o.object_name objn, o.object_type otype,o.object_type otype, CURRENT_FILE# filen,CURRENT_FILE# filen, CURRENT_BLOCK# blockn,CURRENT_BLOCK# blockn, ash.SQL_IDash.SQL_IDfrom v$active_session_history ash,from v$active_session_history ash, all_objects oall_objects owhere event like 'db file sequential read'where event like 'db file sequential read' and o.object_id (+)= ash.CURRENT_OBJ#and o.object_id (+)= ash.CURRENT_OBJ#order by sample_time;order by sample_time;

EVENT P3 OBJN OTYPE FILEN BLOCKN SQL_IDEVENT P3 OBJN OTYPE FILEN BLOCKN SQL_ID

----------------------- -- ------------------------- ----- ----- ------ ------------------------------------ -- ------------------------- ----- ----- ------ -------------

db file sequential read 1 49890 MGMT_METRICS_1HOUR_ INDEX 3 41737db file sequential read 1 49890 MGMT_METRICS_1HOUR_ INDEX 3 41737db file sequential read 1 50908 MGMT_DB_SGA_ECM TABLE 3 28489 35at43xqj7bs0db file sequential read 1 50908 MGMT_DB_SGA_ECM TABLE 3 28489 35at43xqj7bs0db file sequential read 1 55303 TOTO TABLE 1 60218 7xftj55rvjw9sdb file sequential read 1 55303 TOTO TABLE 1 60218 7xftj55rvjw9s

#.16


ASH – scattered readsselectselect event,event, ash.p3,ash.p3, o.object_name objn,o.object_name objn, o.object_type otype,o.object_type otype, CURRENT_FILE# filen,CURRENT_FILE# filen, CURRENT_BLOCK# blockn,CURRENT_BLOCK# blockn, ash.SQL_IDash.SQL_IDfrom v$active_session_history ash,from v$active_session_history ash, all_objects oall_objects owhere event like 'db file scattered read'where event like 'db file scattered read' and o.object_id (+)= ash.CURRENT_OBJ#and o.object_id (+)= ash.CURRENT_OBJ#order by sample_time;order by sample_time;

EVENT P3 OBJN OTYPE FILEN BLOCKN SQL_IDEVENT P3 OBJN OTYPE FILEN BLOCKN SQL_ID

---------------------- -- ------------------------- ----- ----- ------ ----------------------------------- -- ------------------------- ----- ----- ------ -------------

db file scattered read 8 9078 WRH$_SYSMETRIC_HISTO TABLE 3 52363db file scattered read 8 9078 WRH$_SYSMETRIC_HISTO TABLE 3 52363db file scattered read 5 8781 WRI$_ALERT_HISTORY TABLE 3 2676db file scattered read 5 8781 WRI$_ALERT_HISTORY TABLE 3 2676db file scattered read 8 57 OBJAUTH$ TABLE 1 33993 3t2mk1prj24hudb file scattered read 8 57 OBJAUTH$ TABLE 1 33993 3t2mk1prj24hu

#.17

IO by TBS & Fileselect io.cnt,select io.cnt, round(io.cnt/(&v_minutes*60),2) aas, round(io.cnt/(&v_minutes*60),2) aas, io.event, io.event, io.p1 p1, io.p1 p1, f.tablespace_namef.tablespace_namefrom ( selectfrom ( select count(*) cnt,count(*) cnt, substr(event,0,25) event,substr(event,0,25) event, ash.p1 p1ash.p1 p1 from v$active_session_history ashfrom v$active_session_history ash where ( event like 'db file s%' or event like 'direct%' )where ( event like 'db file s%' or event like 'direct%' ) and sample_time > sysdate - &v_minutes/(60*24)and sample_time > sysdate - &v_minutes/(60*24) group by group by event , event , ash.p1ash.p1 ) io,) io, dba_data_files fdba_data_files fwhere where f.file_id = io.p1f.file_id = io.p1Order by io.cnt ;Order by io.cnt ;

CNT AAS EVENT P1 TABLESPACECNT AAS EVENT P1 TABLESPACE

---- --- ----------------------- --- -------------- --- ----------------------- --- ----------

1 .00 db file sequential read 1 SYSTEM1 .00 db file sequential read 1 SYSTEM 2 .00 db file sequential read 3 SYSAUX2 .00 db file sequential read 3 SYSAUX 38 .06 db file sequential read 6 SOE38 .06 db file sequential read 6 SOE 179 .30 db file sequential read 7 SOEINDEX179 .30 db file sequential read 7 SOEINDEX

#.18

IO by Objectselectselect

count(*) cnt,count(*) cnt, CURRENT_OBJ#||' '||o.object_name objn,CURRENT_OBJ#||' '||o.object_name objn, o.object_type otypeo.object_type otypefrom v$active_session_history ash,from v$active_session_history ash, all_objects oall_objects owhere ( event like 'db file s%' or event like 'direct%' )where ( event like 'db file s%' or event like 'direct%' ) and o.object_id (+)= ash.CURRENT_OBJ#and o.object_id (+)= ash.CURRENT_OBJ# and sample_time > sysdate - &1/(60*24)and sample_time > sysdate - &1/(60*24) and session_state='WAITING'and session_state='WAITING'group by group by CURRENT_OBJ#, o.object_name ,CURRENT_OBJ#, o.object_name , o.object_type o.object_type Order by count(*)Order by count(*)

CNT AAS OBJN OTYPECNT AAS OBJN OTYPE

---- ------- ------------------------- ---- ------- ------------------------- ------------------------------

79 .00 52949 ORDER_ITEMS TABLE 79 .00 52949 ORDER_ITEMS TABLE PARTITIONPARTITION 97 .00 -197 .00 -1 130 .00 53117 ORD_STATUS_IX INDEX130 .00 53117 ORD_STATUS_IX INDEX 498 .01 53120 CUST_EMAIL_IX INDEX498 .01 53120 CUST_EMAIL_IX INDEX 512 .01 0512 .01 01632 .03 53112 ITEM_ORDER_IX INDEX1632 .03 53112 ITEM_ORDER_IX INDEX

#.19IO by Object/tbs/waitSelect io.cnt cnt, Select io.cnt cnt,

io.aas aas,io.aas aas,

io.event event,io.event event,

substr(io.obj,1,20) obj, substr(io.obj,1,20) obj,

io.p1 p1,io.p1 p1,

f.tablespace_name tablespace_namef.tablespace_name tablespace_name

from ( selectfrom ( select

count(*) cnt,count(*) cnt,

round(count(*)/(&v_minutes*60),2) aas,round(count(*)/(&v_minutes*60),2) aas,

substr(event,0,15) event, substr(event,0,15) event,

nvl(o.object_name,decode(CURRENT_OBJ#,-1,0,CURRENT_OBJ#)) obj,nvl(o.object_name,decode(CURRENT_OBJ#,-1,0,CURRENT_OBJ#)) obj,

ash.p1, ash.p1,

o.object_type otypeo.object_type otype

from v$active_session_history ash,from v$active_session_history ash,

all_objects oall_objects o

where ( event like 'db file s%' or event like 'direct%' )where ( event like 'db file s%' or event like 'direct%' )

and o.object_id (+)= ash.CURRENT_OBJ#and o.object_id (+)= ash.CURRENT_OBJ#

and sample_time > sysdate - &v_minutes/(60*24)and sample_time > sysdate - &v_minutes/(60*24)

group by group by

substr(event,0,15) , substr(event,0,15) ,

CURRENT_OBJ#, o.object_name ,CURRENT_OBJ#, o.object_name ,

o.object_type ,o.object_type ,

ash.p1 ) io,ash.p1 ) io,

dba_data_files fdba_data_files f

where f.file_id = io.p1where f.file_id = io.p1

Order by io.cnt;Order by io.cnt;

CNT AAS EVENT OBJ P1 TABLESPACE_NAMECNT AAS EVENT OBJ P1 TABLESPACE_NAME

--- --- --------------- ----------------- --- ------------------ --- --------------- ----------------- --- ---------------

1 .02 db file sequent ORDER_ITEMS_PK 7 SOEINDEX1 .02 db file sequent ORDER_ITEMS_PK 7 SOEINDEX 1 .02 db file sequent 0 3 SYSAUX1 .02 db file sequent 0 3 SYSAUX 2 .03 db file scatter WAIT_OBJECTS 1 SYSTEM2 .03 db file scatter WAIT_OBJECTS 1 SYSTEM 2 .03 db file sequent ORDER_ITEMS 6 SOE2 .03 db file sequent ORDER_ITEMS 6 SOE 3 .05 db file sequent CUST_EMAIL_IX 7 SOEINDEX3 .05 db file sequent CUST_EMAIL_IX 7 SOEINDEX 4 .07 db file sequent CUST_LNAME_IX 7 SOEINDEX4 .07 db file sequent CUST_LNAME_IX 7 SOEINDEX 6 .10 db file sequent ORD_ORDER_DATE_IX 7 SOEINDEX6 .10 db file sequent ORD_ORDER_DATE_IX 7 SOEINDEX 13 .22 db file sequent ITEM_ORDER_IX 7 SOEINDEX13 .22 db file sequent ITEM_ORDER_IX 7 SOEINDEX 25 .42 db file sequent 0 2 UNDOTBS125 .42 db file sequent 0 2 UNDOTBS1

#.20IO by SQL selectselect

sum(cnt) over ( partition by io.sql_id order by sql_id ) tcnt,sum(cnt) over ( partition by io.sql_id order by sql_id ) tcnt, io.sql_id,io.sql_id, io.cnt cnt,io.cnt cnt, io.aas aas,io.aas aas, io.objn objn,io.objn objn, io.obj obj,io.obj obj, io.p1 p1,io.p1 p1, f.tablespace_name tablespace_namef.tablespace_name tablespace_namefrom from (( selectselect sql_id,sql_id, count(*) cnt,count(*) cnt, round(count(*)/(&v_minutes*60),2) aas,round(count(*)/(&v_minutes*60),2) aas, CURRENT_OBJ# objn,CURRENT_OBJ# objn, nvl(o.object_name,decode(CURRENT_OBJ#,-1,0,CURRENT_OBJ#)) obj,nvl(o.object_name,decode(CURRENT_OBJ#,-1,0,CURRENT_OBJ#)) obj, o.object_type otype,o.object_type otype, ash.p1ash.p1 from v$active_session_history ashfrom v$active_session_history ash ,all_objects o,all_objects o where ( event like 'db file s%' or event like 'direct%' )where ( event like 'db file s%' or event like 'direct%' ) and o.object_id (+)= ash.CURRENT_OBJ#and o.object_id (+)= ash.CURRENT_OBJ# and sample_time > sysdate - &v_minutes/(60*24)and sample_time > sysdate - &v_minutes/(60*24) group by group by CURRENT_OBJ#, CURRENT_OBJ#, o.object_name ,o.object_name , o.object_type ,o.object_type , ash.p1,ash.p1, sql_idsql_id) io,) io, dba_data_files fdba_data_files fwherewhere f.file_id = io.p1f.file_id = io.p1Order by tcnt, io.sql_id, io.cntOrder by tcnt, io.sql_id, io.cnt

TCNT SQL_ID CNT AAS OBJN OBJ P1 TABLESPAC

---- ------------- --- --- ----- --------------- -- ----------

30 0yas01u2p9ch4 1 .01 53113 ITEM_PRODUCT_IX 7 SOEINDEX 1 .01 53079 ORDER_ITEMS_UK 7 SOEINDEX 28 .16 53112 ITEM_ORDER_IX 7 SOEINDEX 58 6v6gm0fd1rgrz 4 .02 54533 WAIT_OBJECTS 1 SYSTEM 54 .30 0 0 2 UNDOTBS1

#.21


Missing Object IDs dba_extents – get object name

Dba_extents is notoriously slow Options

Create a table as select – use table recreate as object extents change

Catch blocks in x$bh when waits occur

select segment_name, segment_typefrom dba_extents where file_id = P1 and P2 between block_id and block_id + blocks – 1;

select segment_name, segment_typefrom dba_extents where file_id = P1 and P2 between block_id and block_id + blocks – 1;

#.22

Copy of dba_extentsEnter value for file: 3Enter value for file: 3Enter value for block: 6619Enter value for block: 6619OWNER SEGMENT_NAME SEGMENT_TYPEOWNER SEGMENT_NAME SEGMENT_TYPE

--------------- -------------------- --------------------------------- -------------------- ------------------

WMSYS LOG_TAB_PK INDEXWMSYS LOG_TAB_PK INDEX

Elapsed: 00:00:41.25Elapsed: 00:00:41.25

create table myextents as select * from dba_extentscreate table myextents as select * from dba_extents

Table created.Table created.

Elapsed: 00:01:25.73Elapsed: 00:01:25.73

CNT OWN SEGMENT_NAME SEGMENT_TYPECNT OWN SEGMENT_NAME SEGMENT_TYPE

--- ---- ---------------- ------------ --- ---- ---------------- ------------

11 SYS SMON_SCN_TO_TIME CLUSTER11 SYS SMON_SCN_TO_TIME CLUSTER 993 SYS _SYSSMU7$ TYPE2 UNDO993 SYS _SYSSMU7$ TYPE2 UNDO150 rows selected.150 rows selected.

Elapsed: 00:00:01.03Elapsed: 00:00:01.03

#.23


IO Solutions1. Check average read times per file

Should be between 5-20 ms Data in Statspack under “File IO Stats”

2. Check Cache buffer Hit ratio Check db_cache_advice 9i and higher Data in Statspack under “Buffer Pool Advisory” Want to optimize data caching

3. Tune High IO SQL

#.24


Step 1a: Ave Read TimeFile IO Stats DB/Inst:labsf03 Snaps: 1-2

Tablespace Filename------------------------ ----------------------------------------------------

Av Mx Av

Av Rd Rd Av Av Buffer BufWt

Reads Reads/s (ms) Bkt Blks/Rd Writes Writes/s Waits (ms)

---------- ---- ------- ----- --- ------- ------------ -------- ---------- --

SYSTEM /u01/app/oracle/oradata/labsf03/system01.dbf

445 15 0.4 16 1.0 1,157 39 2,744 93.3

USERS /u01/app/oracle/oradata/labsf03/users01.dbf

223 7 0.5 ### 1.0 9,725 324 4 100.0

File IO Stats DB/Inst:labsf03 Snaps: 1-2

Tablespace Filename------------------------ ----------------------------------------------------

Av Mx Av

Av Rd Rd Av Av Buffer BufWt

Reads Reads/s (ms) Bkt Blks/Rd Writes Writes/s Waits (ms)

---------- ---- ------- ----- --- ------- ------------ -------- ---------- --

SYSTEM /u01/app/oracle/oradata/labsf03/system01.dbf

445 15 0.4 16 1.0 1,157 39 2,744 93.3

USERS /u01/app/oracle/oradata/labsf03/users01.dbf

223 7 0.5 ### 1.0 9,725 324 4 100.0

IO should be under 20msIO should be under 20msFastest possible is around 7msFastest possible is around 7msFaster speeds are due to read cachingFaster speeds are due to read caching

#.25


Step 1c: Ave Read Time


select

to_char(begin_interval_time,'yyyy-mm-dd hh24:mi') snap_time,

file#, readtim/nullif(phyrds,0) avg_read_ms, phyrds

from

DBA_HIST_FILESTATXS f,

dba_hist_snapshot s

where f.snap_id=s.snap_id ;

select

to_char(begin_interval_time,'yyyy-mm-dd hh24:mi') snap_time,

file#, readtim/nullif(phyrds,0) avg_read_ms, phyrds

from

DBA_HIST_FILESTATXS f,

dba_hist_snapshot s

where f.snap_id=s.snap_id ;

SNAP_TIME FILE# AVG_READ_MS PHYRDS

---------------- ---------- ----------- ----

2008-01-05 12:00 9 36.67 392008-01-05 12:00 10 32.31 392008-01-05 13:00 1 11.63 1782242008-01-05 13:00 2 56.37 20142008-01-05 13:00 3 17.38 506682008-01-05 13:00 4 9.39 5653212008-01-05 13:00 5 38.78 412008-01-05 13:00 6 28.29 412008-01-05 13:00 7 27.44 392008-01-05 13:00 8 42.56 392008-01-05 13:00 9 36.67 392008-01-05 13:00 10 32.31 39

SNAP_TIME FILE# AVG_READ_MS PHYRDS

---------------- ---------- ----------- ----

2008-01-05 12:00 9 36.67 392008-01-05 12:00 10 32.31 392008-01-05 13:00 1 11.63 1782242008-01-05 13:00 2 56.37 20142008-01-05 13:00 3 17.38 506682008-01-05 13:00 4 9.39 5653212008-01-05 13:00 5 38.78 412008-01-05 13:00 6 28.29 412008-01-05 13:00 7 27.44 392008-01-05 13:00 8 42.56 392008-01-05 13:00 9 36.67 392008-01-05 13:00 10 32.31 39

#.26


Step 1c: Ave Read Time


select

to_char(begin_time,'yyyy-mm-dd hh24:mi') begin_time, file_id fid, average_read_time *10 avgrd_ms, average_write_time *10 avgwr_ms, physical_reads pr, physical_writes pwfrom V$FILEMETRIC_HISTORY forder by begin_time;

select

to_char(begin_time,'yyyy-mm-dd hh24:mi') begin_time, file_id fid, average_read_time *10 avgrd_ms, average_write_time *10 avgwr_ms, physical_reads pr, physical_writes pwfrom V$FILEMETRIC_HISTORY forder by begin_time;

BEGIN_TIME FID AVGRD_MS AVGWR_MS PR PW

---------------- ---- -------- -------- ---------- ----------

2008-01-30 12:25 1 5.2 2.5 100 24

2008-01-30 12:25 5 80.0 20.0 2 2

2008-01-30 12:25 2 36.7 2.0 3 218

2008-01-30 12:25 4 22.8 1.8 89 2754

2008-01-30 12:25 3 22.9 2.6 14 47

BEGIN_TIME FID AVGRD_MS AVGWR_MS PR PW

---------------- ---- -------- -------- ---------- ----------

2008-01-30 12:25 1 5.2 2.5 100 24

2008-01-30 12:25 5 80.0 20.0 2 2

2008-01-30 12:25 2 36.7 2.0 3 218

2008-01-30 12:25 4 22.8 1.8 89 2754

2008-01-30 12:25 3 22.9 2.6 14 47

#.27


Step 2 : Buffer Pool Advisory

Buffer Pool Advisory Size for Size Buffers for Read EstimatedP Est (M) Factor Estimate Factor Physical Reads--- -------- ------ ------------ ------ --------------D 56 .1 6,986 2.3 58,928D 112 .2 13,972 1.6 42,043D 224 .4 27,944 1.0 25,772D 336 .6 41,916 1.0 25,715D 448 .8 55,888 1.0 25,715D 596 1.0 74,351 1.0 25,715D 728 1.2 90,818 1.0 25,715D 840 1.4 104,790 1.0 25,715D 952 1.6 118,762 1.0 25,715D 1,064 1.8 132,734 1.0 25,715

Buffer Pool Advisory Size for Size Buffers for Read EstimatedP Est (M) Factor Estimate Factor Physical Reads--- -------- ------ ------------ ------ --------------D 56 .1 6,986 2.3 58,928D 112 .2 13,972 1.6 42,043D 224 .4 27,944 1.0 25,772D 336 .6 41,916 1.0 25,715D 448 .8 55,888 1.0 25,715D 596 1.0 74,351 1.0 25,715D 728 1.2 90,818 1.0 25,715D 840 1.4 104,790 1.0 25,715D 952 1.6 118,762 1.0 25,715D 1,064 1.8 132,734 1.0 25,715

#.28


Step 3: Find Top I/O SQL

select count(*),sql_id, event from v$active_session_historywhere event in ('db file sequential read', 'db file scattered read', 'db file parallel read')group by sql_id, eventorder by count(*);

select count(*),sql_id, event from v$active_session_historywhere event in ('db file sequential read', 'db file scattered read', 'db file parallel read')group by sql_id, eventorder by count(*);

COUNT(*) SQL_ID EVENT

--------- ------------- ------------------------

10 8hk7xvhua40va db file sequential read 335 3hatpjzrqvfn7 db file sequential read 343 0uuqgjq7k12nf db file sequential read 481 db file scattered read 549 75621g9y3xmvd db file sequential read 1158 db file sequential read 1311 0bzhqhhj9mpaa db file sequential read 1523 41zu158rqf4kf db file sequential read 1757 0yas01u2p9ch4 db file sequential read

COUNT(*) SQL_ID EVENT

--------- ------------- ------------------------

10 8hk7xvhua40va db file sequential read 335 3hatpjzrqvfn7 db file sequential read 343 0uuqgjq7k12nf db file sequential read 481 db file scattered read 549 75621g9y3xmvd db file sequential read 1158 db file sequential read 1311 0bzhqhhj9mpaa db file sequential read 1523 41zu158rqf4kf db file sequential read 1757 0yas01u2p9ch4 db file sequential read

#.29


Step 3: OEM find top SQL

#.30


IO Solutions

CheckFile IO response timesBuffer Cache Hit Ratio

Always Check SQL to Tune

#.31


IO Configuration Issues

Unix Buffer Cache Disk Spindles Raid 5

#.32


Machine Memory Machine Memory

Unix File Buffer Cache

SGA SGA Unix File Unix File CacheCache

If average IO read times are underIf average IO read times are under

7ms them probably coming from 7ms them probably coming from

Unix File CacheUnix File Cache

#.33


IO Histograms

IO at > 1ms comes IO at > 1ms comes

From Unix File CacheFrom Unix File Cache

#.34


IO Throughput IO Operations/Sec 150 IOPS/sec optimistic

Large Disks – Few IOPSLarge Disks – Few IOPS

Small Disks – Many IOPSSmall Disks – Many IOPS

#.35


Disk 2Gb vs 70GBSeagate Barracuda 4LP

Seagate Cheetah 73LP

Capacity 2.16GB 73.4GB

Rotation Speed 7200rpm 10000rpm

Rotational Delay(avg) 4.1ms 3ms

Time to read 32Kb 6ms 3ms

Seek Time (avg) 9.4ms 4.9

Total time for Single I/O 19.5ms 10.9ms

I/O per second (conservative)

51 92

IOPs/sec per 100GB 2550 126

#.36


Raid 5

http://www.miracleas.com/BAARF/BAARF2.html

B – Battle

A – Against

A – Any

R – Raid

F - Five

#.37


Direct I/O WAITSDirect I/O : This mechanism lets the client bypass the buffer cache forI/O intensive operations. The disk blocks are written into and read from process

private memory.

direct path read :Parallel Query

direct path writesqlldr loading/reading LOBsparallel DMLscreate table as select create index

direct path read temp , direct path write tempSorting

#.38


Direct IO ShadowShadow

ProcessProcess

Buffer CacheBuffer CachePGA PGA

Sort AreaSort AreaDirect PathDirect PathPQO ReadsPQO Reads

XX

#.39


direct path read

Parallel QuerySee v$px_session

_serial_direct_read = true_serial_direct_read = true

select parameter1, parameter2, parameter3 select parameter1, parameter2, parameter3 from v$event_namefrom v$event_namewhere name='direct path read';where name='direct path read';

PARAMETER1 PARAMETER2 PARAMETER3

---------- ---------- ----------

file number first dba block cntShadowShadow

ProcessProcess

Buffer CacheBuffer CachePGAPGAPQO

XX

ShadowShadow

ProcessProcess

Buffer CacheBuffer CachePGAPGAPQO

XX

#.40


Further Investigation ASH

SID QSID P3 OBJN OTYPE FN BLOCKN SQL_ID--- ---- -- ----- ------ -- ------ -------------149 152 8 TOTO TABLE 1 194072 4gp8tg0b2s722144 152 8 TOTO TABLE 1 192304 4gp8tg0b2s722 60 152 8 TOTO TABLE 1 190592 4gp8tg0b2s722 54 152 6 TOTO TABLE 1 201274 4gp8tg0b2s722149 152 8 TOTO TABLE 1 198552 4gp8tg0b2s722144 152 8 TOTO TABLE 1 200464 4gp8tg0b2s722 60 152 8 TOTO TABLE 1 196784 4gp8tg0b2s722

SID QSID P3 OBJN OTYPE FN BLOCKN SQL_ID--- ---- -- ----- ------ -- ------ -------------149 152 8 TOTO TABLE 1 194072 4gp8tg0b2s722144 152 8 TOTO TABLE 1 192304 4gp8tg0b2s722 60 152 8 TOTO TABLE 1 190592 4gp8tg0b2s722 54 152 6 TOTO TABLE 1 201274 4gp8tg0b2s722149 152 8 TOTO TABLE 1 198552 4gp8tg0b2s722144 152 8 TOTO TABLE 1 200464 4gp8tg0b2s722 60 152 8 TOTO TABLE 1 196784 4gp8tg0b2s722

select session_id sid, QC_SESSION_ID qsid, ash.p3, CURRENT_OBJ#||' '||o.object_name objn, o.object_type otype, CURRENT_FILE# fn, CURRENT_BLOCK# blockn, ash.SQL_IDfrom v$active_session_history ash, all_objects owhere event like 'direct path read' and o.object_id (+)= ash.CURRENT_OBJ#Order by sample_time;

select session_id sid, QC_SESSION_ID qsid, ash.p3, CURRENT_OBJ#||' '||o.object_name objn, o.object_type otype, CURRENT_FILE# fn, CURRENT_BLOCK# blockn, ash.SQL_IDfrom v$active_session_history ash, all_objects owhere event like 'direct path read' and o.object_id (+)= ash.CURRENT_OBJ#Order by sample_time;

#.41

select

ash.SQL_ID, QC_SESSION_ID qsid, count(*) cnt, count (distinct session_id) deg, nvl(o.object_name,to_char(CURRENT_OBJ#)) obj, o.object_type otype, decode(session_state, 'WAITING',event,'CPU') eventfrom v$active_session_history ash, all_objects owhere o.object_id (+)= ash.CURRENT_OBJ# and qc_session_id is not nullgroup by qc_session_id, sql_id, o.object_name, o.object_type, CURRENT_OBJ#, event, session_stateOrder by qc_session_id, sql_id

PQO - ASH

SQL_ID QSID CNT DEG OBJ OTYPE EVENTSQL_ID QSID CNT DEG OBJ OTYPE EVENT

------------- ---- ---- --- ------------ ----- ------------- ---- ---- --- ------------ ----- ----------------------------------------------------

6rvy3wj2u0h6u 123 42 4 WAIT_OBJECTS TABLE direct path read6rvy3wj2u0h6u 123 42 4 WAIT_OBJECTS TABLE direct path read 123 3 3 WAIT_OBJECTS TABLE CPU123 3 3 WAIT_OBJECTS TABLE CPU 123 96 4 0 db file sequential read123 96 4 0 db file sequential read 123 9 4 0 read by other session123 9 4 0 read by other session 123 2 2 0 CPU123 2 2 0 CPU7p3jt75phub2d 144 386 4 WAIT_OBJECTS TABLE PX Deq Credit: send blkd7p3jt75phub2d 144 386 4 WAIT_OBJECTS TABLE PX Deq Credit: send blkd 144 4 3 WAIT_OBJECTS TABLE PX qref latch144 4 3 WAIT_OBJECTS TABLE PX qref latch 144 37 1 WAIT_OBJECTS TABLE db file sequential read144 37 1 WAIT_OBJECTS TABLE db file sequential read 144 3 2 WAIT_OBJECTS TABLE direct path read144 3 2 WAIT_OBJECTS TABLE direct path read 144 70 1 WAIT_OBJECTS TABLE CPU144 70 1 WAIT_OBJECTS TABLE CPU 144 21 4 0 PX Deq Credit: send blkd144 21 4 0 PX Deq Credit: send blkd 144 12 4 0 db file sequential read144 12 4 0 db file sequential read

#.42


direct path write

ShadowShadow

ProcessProcess


PGAPGA

DBWRDBWR

DATADATA

ShadowShadow

ProcessProcess


PGAPGA

DBWRDBWR

DATADATA

Occurs when: insert /*+ APPEND */ sql*loader direct=yCreate table as select

select parameter1, parameter2, parameter3 select parameter1, parameter2, parameter3 from v$event_namefrom v$event_namewhere name='direct path write';where name='direct path write';

PARAMETER1 PARAMETER2 PARAMETER3PARAMETER1 PARAMETER2 PARAMETER3

---------- ---------- -------------------- ---------- ----------

file number first dba block cntfile number first dba block cnt

#.43


Direct Path Read TempDirect Path Write Temp

Sorting Write to Temp Read from Temp

ShadowShadow

ProcessProcess


PGAPGA

TEMPTEMP DATADATA

ShadowShadow

ProcessProcess


PGAPGA

TEMPTEMP DATADATA

select parameter1, parameter2, parameter3 select parameter1, parameter2, parameter3 from v$event_namefrom v$event_namewhere name in ('direct path write temp‘,where name in ('direct path write temp‘, ‘ ‘direct path read temp’);direct path read temp’);

PARAMETER1 PARAMETER2 PARAMETER3PARAMETER1 PARAMETER2 PARAMETER3

---------- ---------- -------------------- ---------- ----------

file number first dba block cntfile number first dba block cntfile number first dba block cntfile number first dba block cnt

#.44


Further investigation“real time“ can use Select

su.username, su.segtype, s.sql_id, su.blocksfrom v$sort_usage su, v$session sWhere s.saddr = su.session_addr and s.serial# = su.session_num;

Select su.username, su.segtype, s.sql_id, su.blocksfrom v$sort_usage su, v$session sWhere s.saddr = su.session_addr and s.serial# = su.session_num;

USERNAME SEGTYPE SQL_ID BLOCKSUSERNAME SEGTYPE SQL_ID BLOCKS

--------- ------- ------------- --------------- ------- ------------- ------

SYS SORT 3xp88dstadjs4 109440SYS SORT 3xp88dstadjs4 109440

NoteNote: SQL_ID : SQL_ID doesn’t seem filled doesn’t seem filled in for in for v$sort_usage.sql_idv$sort_usage.sql_id

#.46


file open Args

P1 fib P2 iov P3 0

Wait on an open file The time it takes to open the file Times should be order of a block read To tune, try pre-opening files Leave sessoins connected

High reconnection rates can exacerbate

Can try holding log files open (see script from Steve Adams)

#.47


file identify

The time it takes to identify a file so that it can be opened later.

Datafile should be identified just once Logfiles are identified every checkpoint

Reduce checkpoints Arguments

p1 fib p2 filename p3 opcode (1 test for create, 2 get details)

#.48


Data file init write

When autoextend is set and many extensions are happening

#.49


Local Write Wait

Truncating a table Wait for Data Cache to be cleared of all

blocks of truncated table Wait by shadow for DBWR

If a problem try GTT Reduce fast_start_mttr_target

#.50


Further Investigation

NAME P1 P2 P3

----------------- ----- ------ -----

local write wait file# block#

NAME P1 P2 P3

----------------- ----- ------ -----

local write wait file# block#

select * from v$event_namewhere name = ‘local write wait'

select ash.p1,ash.p2, ash.SQL_ID, count(*)from v$ash ashwhere event='local write wait'group byash.p1,ash.p2, ash.SQL_ID;

P1 P2 SQL_ID COUNT

--- -- ------------- -----

201 2 0q8k9xfmz0k2k 194201 5 0q8k9xfmz0k2k 34201 6 0q8k9xfmz0k2k 9201 4 0q8k9xfmz0k2k 10

#.51


Temporary File #’s

SQL> select file# from v$datafile; FILE#

----------

1 2 3 4 5 6 7 8SQL> select file# from v$tempfile; FILE#

----------

2 1

SQL> show parameters db_files

NAME VALUE

---------- -------------

db_files 200

Wait Temporary File#’s =

Db_files + file#

File# 201 = v$tempfile 1

#.52


Local Write Wait:extent allocation

CREATE TEMPORARY TABLESPACE "NEWTEMP" TEMPFILE '/d3/temp01.dbf' SIZE 100M AUTOEXTEND ON EXTENT MANAGEMENT LOCAL UNIFORM SIZE 64K;

select a.*, b.*

from dba_objects a, dba_objects b

order by a.owner

ALTER DATABASE DEFAULT TEMPORARY TABLESPACE newtemp;

#.53


Local Write Waits

#.54


Local Write Wait: Solutions

Truncating a tableReduce cache sizeUse GTT

Temporary Table Space Allocations Increase extent size

#.55


Summary I/O

Buffer Cache IOdb file sequential readdb file scattered readdb file parallel read

Tune I/O’s should be 5-15msCheck Buffer Cache AdvisoryTune SQL

#.56


Summary Direct I/O

direct path read : PQO direct path write : direct path operations direct path read temp: sorts direct path write temp: sorts direct path write (lob) : stored NOCACHE

#.57


IO Summary Other

sort segment request : SMON occupied file open : log files or session reconnects file identify : log files or session reconnects local write wait –

Temp extensions Truncates

data file init write – data file extension

IO Waits Kyle Hailey [email protected] .

Documents

Transcript of IO Waits Kyle Hailey [email protected] .