Tracing Parallel Execution (UKOUG 2006)

Tracing PX SlavesTracing PX Slaves

Doug BurnsDoug Burns

[email protected]@yahoo.com

http://oracledoug.comhttp://oracledoug.com

AgendaAgenda

IntroductionIntroduction Test EnvironmentTest Environment Tracing Parallel Tracing Parallel

ExecutionExecution Trace OutputTrace Output ConsolidationConsolidation Timing IssuesTiming Issues ConclusionConclusion

IntroductionIntroduction

Who (or what) am I ?Who (or what) am I ? ScottishScottish Predominantly a DBAPredominantly a DBA Training and ConsultancyTraining and Consultancy

Current AssignmentCurrent Assignment BSkyBBSkyB Very Cool Projects and Very Cool Projects and

HardwareHardware Less Cool Release Less Cool Release

ManagementManagement http://oracledoug.comhttp://oracledoug.com

BlogBlog


Previous Related PapersPrevious Related Papers Suck It Dry - Tuning Parallel ExecutionSuck It Dry - Tuning Parallel Execution How Many Slaves? – PX and the Magic of 2How Many Slaves? – PX and the Magic of 2 Available at Available at http://oracledoug.comhttp://oracledoug.com

ObjectivesObjectives Answer tracing questions that cropped upAnswer tracing questions that cropped up Show the detailShow the detail

Why Parallel Execution?Why Parallel Execution? Increasing Volumes of DataIncreasing Volumes of Data Increasing User ExpectationsIncreasing User Expectations More Powerful HardwareMore Powerful Hardware

Parallel Execution (PX) splits a Parallel Execution (PX) splits a single large task into multiple single large task into multiple smaller tasks which are handled smaller tasks which are handled by separate processes running by separate processes running concurrently.concurrently. Full Table ScansFull Table Scans SortsSorts Index Creation, Direct Path inserts etc …Index Creation, Direct Path inserts etc …


Test EnvironmentTest Environment

ExampleExample Small TableSmall Table

Easier to wade through trace filesEasier to wade through trace files Quick executionQuick execution Not representative of real applicationsNot representative of real applications Force PX using HintsForce PX using Hints

TEST_TABTEST_TAB 2,048,000 rows2,048,000 rows Average row length – 109 bytesAverage row length – 109 bytes 8Kb Blocks/ 32M Extents / 256 Mb Segment8Kb Blocks/ 32M Extents / 256 Mb Segment No IndexesNo Indexes


Oracle 9i Release 2Oracle 9i Release 2 The most common database at workThe most common database at work Tracing similar to earlier versionsTracing similar to earlier versions

Oracle 10g Release 2Oracle 10g Release 2 Principles are the samePrinciples are the same Improved tracing functionalityImproved tracing functionality

Single CPU / Single HDDSingle CPU / Single HDD TryingTrying to introduce wait events to introduce wait events


The QueryThe Query

SELECT /*+ parallel(tt1, 2) */ SELECT /*+ parallel(tt1, 2) */

MOD(tt1.pk_id, 2), COUNT(*)MOD(tt1.pk_id, 2), COUNT(*)

FROM test_tab tt1FROM test_tab tt1

GROUP BY MOD(tt1.pk_id,2)GROUP BY MOD(tt1.pk_id,2)

ORDER BY MOD(tt1.pk_id,2);ORDER BY MOD(tt1.pk_id,2);

Degree of Parallelism (DOP) = 2Degree of Parallelism (DOP) = 2 Two Sets of PX Slaves RequestedTwo Sets of PX Slaves Requested

One set for Full Table ScanOne set for Full Table Scan One set for Grouping OperationOne set for Grouping Operation

Tracing Parallel Tracing Parallel ExecutionExecution

Enable TracingEnable Tracing 10g10galter session set tracefile_identifier=‘doug_test';alter session set tracefile_identifier=‘doug_test';exec dbms_session.set_identifier(‘doug_test');exec dbms_session.set_identifier(‘doug_test');exec dbms_monitor.client_id_trace_enable(exec dbms_monitor.client_id_trace_enable(

client_id => ‘doug_test');client_id => ‘doug_test'); 9i9ialter session set tracefile_identifier=‘doug_test';alter session set tracefile_identifier=‘doug_test';alter session set events '10046 trace name context alter session set events '10046 trace name context

forever, level 8'; forever, level 8';

tracefile_identifier not as useful for PXtracefile_identifier not as useful for PX Only applies to the Query Co-ordinator (QC) processOnly applies to the Query Co-ordinator (QC) process Makes sense as the PX slave doesn’t ‘belong’ to the Makes sense as the PX slave doesn’t ‘belong’ to the

sessionsession


Run QueryRun Query Disable TracingDisable Tracing

10g10gexec dbms_monitor.client_id_trace_disable(exec dbms_monitor.client_id_trace_disable(

client_id => ‘doug_test');client_id => ‘doug_test'); 9i9ialter session set events '10046 trace name alter session set events '10046 trace name context off'; context off';

Several Trace files producedSeveral Trace files produced Remember – this is for the Remember – this is for the lowestlowest DOP of 2 DOP of 2


USER PROCESS

SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)FROM test_tab tt1GROUP BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2);

Slave 0

Sorting/Grouping 0

Slave 1

Sorting/Grouping 1

Slave 2

Reading first half of blocks

Slave 3

Reading second half of blocks

TEST_TAB

Query Coordinator

· Allocate sort ranges· Allocate FTS block

ranges· Collate data


user_dump_destuser_dump_desttest1020_ora_17239_test1020_ora_17239_doug_testdoug_test.trc.trc

background_dump_destbackground_dump_desttest1020_test1020_p000p000_17219.trc_17219.trc

test1020_test1020_p001p001_17221.trc_17221.trc



No mention of tracefile_identifierNo mention of tracefile_identifier Slave numbers could be differentSlave numbers could be different


IssuesIssues Difficult to get a consolidated view of the user Difficult to get a consolidated view of the user

actionaction

PX Slaves are shared background processesPX Slaves are shared background processes How to ensure that you’re looking at the correct slaves?How to ensure that you’re looking at the correct slaves? How to ensure that you don’t include unrelated activity?How to ensure that you don’t include unrelated activity?

9i - With a little difficulty and rm!9i - With a little difficulty and rm! 10g – dbms_monitor and trcsess10g – dbms_monitor and trcsess

Trace Output – 9iTrace Output – 9i Query Co-ordinator (QC)Query Co-ordinator (QC)

Unix process pid: 5790, image: Unix process pid: 5790, image: [email protected] (TNS V1-V3)[email protected] (TNS V1-V3)

*** SESSION ID:(8.283) 2006-09-12 10:54:33.245*** SESSION ID:(8.283) 2006-09-12 10:54:33.245APPNAME mod='SQL*Plus' mh=3669949024 act='' APPNAME mod='SQL*Plus' mh=3669949024 act=''

ah=4029777240ah=4029777240

Trace Output – 9iTrace Output – 9i Followed by tons of recursive callsFollowed by tons of recursive calls

Enabling Tracing, CBO work, etc.Enabling Tracing, CBO work, etc.

PARSING IN CURSOR #1 len=130 dep=0 uid=23 oct=3 lid=23 PARSING IN CURSOR #1 len=130 dep=0 uid=23 oct=3 lid=23 tim=1130912963021298 hv=3017045374 ad='959ffae8'tim=1130912963021298 hv=3017045374 ad='959ffae8'

SELECT /*+ parallel(tt1, 2) */SELECT /*+ parallel(tt1, 2) */MOD(tt1.pk_id, 2), COUNT(*)MOD(tt1.pk_id, 2), COUNT(*)

FROM test_tab1 tt1FROM test_tab1 tt1GROUP BY MOD(tt1.pk_id,2)GROUP BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)END OF STMTEND OF STMT

PARSE PARSE #1:c=10998,e=12643,p=0,cr=7,cu=3,mis=1,r=0,dep=0,og=4#1:c=10998,e=12643,p=0,cr=7,cu=3,mis=1,r=0,dep=0,og=4,tim=1130912963021287,tim=1130912963021287

WAIT #1: nam='enqueue' ela= 2998428 p1=1413677062 WAIT #1: nam='enqueue' ela= 2998428 p1=1413677062 p2=65544 p3=0p2=65544 p3=0

Trace Output – 9i (QC Trace Output – 9i (QC Contd)Contd)

Now to set up the slave setsNow to set up the slave setsWAIT #1: nam='rdbms ipc reply' ela= 122 p1=5 p2=21474836 WAIT #1: nam='rdbms ipc reply' ela= 122 p1=5 p2=21474836

p3=0p3=0WAIT #1: nam=WAIT #1: nam='process startup'process startup' ela= 13 p1=80 p2=1 p3=0' ela= 13 p1=80 p2=1 p3=0WAIT #1: nam='process startup' ela= 34572 p1=80 p2=1 p3=1WAIT #1: nam='process startup' ela= 34572 p1=80 p2=1 p3=1WAIT #1: nam='WAIT #1: nam='PX Deq: Join ACKPX Deq: Join ACK' ela= 1983 p1=268500992 ' ela= 1983 p1=268500992

p2=1 p3=0p2=1 p3=0WAIT #1: nam='PX Deq: Join ACK' ela= 1479 p1=268500993 WAIT #1: nam='PX Deq: Join ACK' ela= 1479 p1=268500993

p2=1 p3=0p2=1 p3=0WAIT #1: nam='process startup' ela= 19432 p1=80 p2=2 p3=0WAIT #1: nam='process startup' ela= 19432 p1=80 p2=2 p3=0WAIT #1: nam='process startup' ela= 26803 p1=80 p2=3 p3=0WAIT #1: nam='process startup' ela= 26803 p1=80 p2=3 p3=0WAIT #1: nam='PX Deq: Join ACK' ela= 2930 p1=268500994 WAIT #1: nam='PX Deq: Join ACK' ela= 2930 p1=268500994

p2=1 p3=0p2=1 p3=0WAIT #1: nam='PX Deq: Join ACK' ela= 1557 p1=268500995 WAIT #1: nam='PX Deq: Join ACK' ela= 1557 p1=268500995

p2=1 p3=0p2=1 p3=0


Set up queues and wait for slaves to parse Set up queues and wait for slaves to parse statementsstatements

WAIT #1: nam='WAIT #1: nam='PX qref latchPX qref latch' ela= 4 p1=1 p2=1 p3=-' ela= 4 p1=1 p2=1 p3=-18215416161821541616

WAIT #1: nam='PX qref latch' ela= 10385 p1=1 p2=2 p3=-WAIT #1: nam='PX qref latch' ela= 10385 p1=1 p2=2 p3=-18215416161821541616



WAIT #1: nam='WAIT #1: nam='PX Deq: Parse ReplyPX Deq: Parse Reply' ela= 3277 p1=200 ' ela= 3277 p1=200 p2=1 p3=0p2=1 p3=0

WAIT #1: nam='PX Deq: Parse Reply' ela= 1992474 p1=200 WAIT #1: nam='PX Deq: Parse Reply' ela= 1992474 p1=200 p2=1 p3=0p2=1 p3=0

WAIT #1: nam='PX Deq: Parse Reply' ela= 960495 p1=200 WAIT #1: nam='PX Deq: Parse Reply' ela= 960495 p1=200 p2=2 p3=0p2=2 p3=0


Start waiting for responses and take out Start waiting for responses and take out PX qref latch when communicatingPX qref latch when communicating

WAIT #1: nam='WAIT #1: nam='PX Deq: Execute ReplyPX Deq: Execute Reply' ela= 210184 p1=200 ' ela= 210184 p1=200 p2=1 p3=0p2=1 p3=0

WAIT #1: nam='PX Deq: Execute Reply' ela= 42955 p1=200 WAIT #1: nam='PX Deq: Execute Reply' ela= 42955 p1=200 p2=1 p3=0p2=1 p3=0

WAIT #1: nam='PX Deq: Execute Reply' ela= 1636370 WAIT #1: nam='PX Deq: Execute Reply' ela= 1636370 p1=200 p2=1 p3=0p1=200 p2=1 p3=0

WAIT #1: nam='WAIT #1: nam='PX qref latchPX qref latch' ela= 11128 p1=1 p2=1 p3=-' ela= 11128 p1=1 p2=1 p3=-18215430641821543064

WAIT #1: nam='PX Deq: Execute Reply' ela= 501343 p1=200 WAIT #1: nam='PX Deq: Execute Reply' ela= 501343 p1=200 p2=1 p3=0p2=1 p3=0

WAIT #1: nam='PX qref latchWAIT #1: nam='PX qref latch'' ela= 11011 p1=1 p2=1 p3=- ela= 11011 p1=1 p2=1 p3=-18215416161821541616

WAIT #1: nam='PX Deq: Execute Reply' ela= 1424051 WAIT #1: nam='PX Deq: Execute Reply' ela= 1424051 p1=200 p2=1 p3=0p1=200 p2=1 p3=0


Some wrap-up towards the endSome wrap-up towards the end

WAIT #1: nam='PX Deq: Signal ACK' ela= 2 p1=0 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 2 p1=0 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 20 p1=10 p2=2 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 20 p1=10 p2=2 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 98523 p1=10 p2=3 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 98523 p1=10 p2=3 p3=0FETCH FETCH

#1:c=1000,e=98953,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=11#1:c=1000,e=98953,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=113091303363795330913033637953

WAIT #1: nam='PX Deq: Signal ACK' ela= 2351 p1=200 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 2351 p1=200 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 20 p1=200 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 20 p1=200 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 200 p1=200 p2=2 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 200 p1=200 p2=2 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 1031 p1=200 p2=1 p3=0WAIT #1: nam='PX Deq: Signal ACK' ela= 1031 p1=200 p2=1 p3=0

Trace Output – 9iTrace Output – 9i Slaves 0 and 1 – Sort/GroupSlaves 0 and 1 – Sort/Group

Instance name: TEST92Instance name: TEST92Redo thread mounted by this instance: 1Redo thread mounted by this instance: 1Oracle process number: 9Oracle process number: 9Unix process pid: 5794, image: [email protected] Unix process pid: 5794, image: [email protected]

(P000)(P000)

*** SESSION ID:(13.34) 2006-09-12 10:55:19.741*** SESSION ID:(13.34) 2006-09-12 10:55:19.741<<snipped><<snipped>PARSING IN CURSOR #1 len=98 dep=0 uid=23 oct=3 lid=23 PARSING IN CURSOR #1 len=98 dep=0 uid=23 oct=3 lid=23

tim=1130913007567950 hv=4220356981 ad='959d6c34'tim=1130913007567950 hv=4220356981 ad='959d6c34'SELECT /*+ CIV_GB */ A1.C0,COUNT(SYS_OP_CSR(A1.C1,0)) SELECT /*+ CIV_GB */ A1.C0,COUNT(SYS_OP_CSR(A1.C1,0))

FROM :Q3000 A1 GROUP BY A1.C0 ORDER BY A1.C0FROM :Q3000 A1 GROUP BY A1.C0 ORDER BY A1.C0END OF STMTEND OF STMTPARSE PARSE


Trace Output – 9i (Sort Trace Output – 9i (Sort Contd)Contd)

Cycle round waiting for data to come Cycle round waiting for data to come back from other slave setback from other slave set

WAIT #1: nam='PX Deq: Execution Msg' ela= 3 p1=268566527 WAIT #1: nam='PX Deq: Execution Msg' ela= 3 p1=268566527 p2=1 p3=0p2=1 p3=0

WAIT #1: nam='PX Deq: Execution Msg' ela= 1992813 WAIT #1: nam='PX Deq: Execution Msg' ela= 1992813 p1=268566527 p2=2 p3=0p1=268566527 p2=2 p3=0




Trace Output – 9i (Sort Trace Output – 9i (Sort Contd)Contd)

EXEC EXEC #1:c=0,e=248,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=11309130334#1:c=0,e=248,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=113091303343553635536WAIT #1: nam='PX Deq: Table Q Sample' ela= 98589 p1=10 p2=1 WAIT #1: nam='PX Deq: Table Q Sample' ela= 98589 p1=10 p2=1

p3=0p3=0WAIT #1: nam='PX Deq: Table Q Sample' ela= 2042 p1=200 p2=2 WAIT #1: nam='PX Deq: Table Q Sample' ela= 2042 p1=200 p2=2

p3=0p3=0FETCH FETCH


WAIT #1: nam=WAIT #1: nam='PX Deq Credit: send blkd'PX Deq Credit: send blkd' ela= 511 p1=268566527 ' ela= 511 p1=268566527 p2=1 p3=0p2=1 p3=0

FETCH FETCH #1:c=0,e=33,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=11309130#1:c=0,e=33,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=113091303353770033537700



Trace Output – 9iTrace Output – 9i

Slaves 2 and 3 – FTSSlaves 2 and 3 – FTS

PARSING IN CURSOR #1 len=208 dep=0 uid=23 oct=3 lid=23 PARSING IN CURSOR #1 len=208 dep=0 uid=23 oct=3 lid=23 tim=1130913007579832 hv=721990200 ad='959d0330'tim=1130913007579832 hv=721990200 ad='959d0330'

SELECT /*+ PIV_GB */ MOD(A1.C0,2) SELECT /*+ PIV_GB */ MOD(A1.C0,2) C0,SYS_OP_MSR(COUNT(*)) C1 FROM (SELECT /*+ NO_EXPAND C0,SYS_OP_MSR(COUNT(*)) C1 FROM (SELECT /*+ NO_EXPAND ROWID(A2) */ A2."PK_ID" C0 FROM ROWID(A2) */ A2."PK_ID" C0 FROM "TESTUSER"."TEST_TAB1" PX_GRANULE(0, BLOCK_RANGE, "TESTUSER"."TEST_TAB1" PX_GRANULE(0, BLOCK_RANGE, DYNAMIC) A2) A1 GROUP BY MOD(A1.C0,2)DYNAMIC) A2) A1 GROUP BY MOD(A1.C0,2)

END OF STMTEND OF STMT

PARSE PARSE #1:c=9999,e=9226,p=0,cr=1,cu=0,mis=1,r=0,dep=0,og=4,t#1:c=9999,e=9226,p=0,cr=1,cu=0,mis=1,r=0,dep=0,og=4,tim=1130913007579821im=1130913007579821

Trace Output – 9i FTS Trace Output – 9i FTS (Contd)(Contd)

Go out and read data (with Direct Path Go out and read data (with Direct Path Reads)Reads)



EXEC EXEC #1:c=0,e=180,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=1130913010#1:c=0,e=180,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=1130913010567272567272

WAIT #1: nam=WAIT #1: nam='direct path read'direct path read' ela= 50 p1=5 p2=10 p3=128' ela= 50 p1=5 p2=10 p3=128WAIT #1: nam='direct path read' ela= 14 p1=5 p2=138 p3=128WAIT #1: nam='direct path read' ela= 14 p1=5 p2=138 p3=128WAIT #1: nam='direct path read' ela= 17 p1=5 p2=266 p3=128WAIT #1: nam='direct path read' ela= 17 p1=5 p2=266 p3=128

<snipped><snipped>

WAIT #1: nam='direct path read' ela= 32 p1=5 p2=522 p3=128WAIT #1: nam='direct path read' ela= 32 p1=5 p2=522 p3=128WAIT #1: nam='PX Deq: Execution Msg' ela= 11253 p1=268566527 WAIT #1: nam='PX Deq: Execution Msg' ela= 11253 p1=268566527

p2=1 p3=0p2=1 p3=0

Trace Output – 9i FTS Trace Output – 9i FTS (Contd)(Contd)

Wrap-upWrap-up


FETCH FETCH #1:c=11410266,e=22810016,p=16263,cr=16486,cu=0,mis=0,r=0,dep=#1:c=11410266,e=22810016,p=16263,cr=16486,cu=0,mis=0,r=0,dep=0,og=4,tim=11309130333773630,og=4,tim=1130913033377363


FETCH FETCH #1:c=0,e=86,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=1130913033#1:c=0,e=86,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=4,tim=1130913033436946436946

WAIT #1: nam='PX Deq: Execution Msg' ela= 20 p1=268566527 p2=1 WAIT #1: nam='PX Deq: Execution Msg' ela= 20 p1=268566527 p2=1 p3=0p3=0

WAIT #1: nam='PX Deq: Execution Msg' ela= 389 p1=268566527 p2=2 WAIT #1: nam='PX Deq: Execution Msg' ela= 389 p1=268566527 p2=2 p3=0p3=0



Trace Output – 10gTrace Output – 10g

Query Co-ordinator (QC)Query Co-ordinator (QC)

Unix process pid: Unix process pid: 1723917239, image: [email protected] (TNS , image: [email protected] (TNS V1-V3)V1-V3)

*** ACTION NAME:() 2006-08-20 10:59:33.605*** ACTION NAME:() 2006-08-20 10:59:33.605

*** MODULE NAME:(SQL*Plus) 2006-08-20 10:59:33.605*** MODULE NAME:(SQL*Plus) 2006-08-20 10:59:33.605

*** SERVICE NAME:(SYS$USERS) 2006-08-20 10:59:33.605*** SERVICE NAME:(SYS$USERS) 2006-08-20 10:59:33.605

*** CLIENT ID:(doug_test) 2006-08-20 10:59:33.605*** CLIENT ID:(doug_test) 2006-08-20 10:59:33.605

*** SESSION ID:(*** SESSION ID:(1095.51095.5) 2006-08-20 10:59:33.605) 2006-08-20 10:59:33.605

Followed by tons of recursive callsFollowed by tons of recursive calls Enabling Tracing, CBO work, etc.Enabling Tracing, CBO work, etc.

Trace Output – 10g (QC Trace Output – 10g (QC Contd)Contd)

Essentially the sameEssentially the same

PARSING IN CURSOR #4 len=129 dep=0 uid=27 oct=3 lid=27 PARSING IN CURSOR #4 len=129 dep=0 uid=27 oct=3 lid=27 tim=1128972630780520 hv=3411843401 ad='77e67d54'tim=1128972630780520 hv=3411843401 ad='77e67d54'

SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)

FROM test_tab1 tt1FROM test_tab1 tt1

GROUP BY MOD(tt1.pk_id,2)GROUP BY MOD(tt1.pk_id,2)

ORDER BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)

END OF STMTEND OF STMT

PARSE PARSE #4:c=267960,e=270534,p=20,cr=447,cu=3,mis=1,r=0,dep=0,og=1,tim=#4:c=267960,e=270534,p=20,cr=447,cu=3,mis=1,r=0,dep=0,og=1,tim=11289726307805081128972630780508

Trace Output – 10g (QC Trace Output – 10g (QC Contd)Contd)

Note the different timed event formatNote the different timed event format

WAIT #4: nam='PX Deq: Execute Reply' ela= 31 WAIT #4: nam='PX Deq: Execute Reply' ela= 31 sleeptime/senderid=200 passes=1 p3=0 obj#=10907 sleeptime/senderid=200 passes=1 p3=0 obj#=10907 tim=1129017680317433tim=1129017680317433


WAIT #4: nam='PX qref latch' ela= 57346 function=1 WAIT #4: nam='PX qref latch' ela= 57346 function=1 sleeptime=2042549608 qref=0 obj#=10907 tim=1129017680657526sleeptime=2042549608 qref=0 obj#=10907 tim=1129017680657526




Slaves 0 and 1 – Sort / GroupSlaves 0 and 1 – Sort / Group

Instance name: TEST1020Instance name: TEST1020

Redo thread mounted by this instance: 1Redo thread mounted by this instance: 1

Oracle process number: 12Oracle process number: 12

Unix process pid: Unix process pid: 1721917219, image: [email protected] , image: [email protected] ((P000P000))

*** 2006-08-20 10:59:33.938*** 2006-08-20 10:59:33.938


*** CLIENT ID:(doug_test) 2006-08-20 10:59:33.938*** CLIENT ID:(doug_test) 2006-08-20 10:59:33.938

*** SESSION ID:(1090.11) 2006-08-20 10:59:33.938*** SESSION ID:(1090.11) 2006-08-20 10:59:33.938

WAIT #0: nam='PX Deq: Msg Fragment' ela= 1611 WAIT #0: nam='PX Deq: Msg Fragment' ela= 1611 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 tim=1128972630798951tim=1128972630798951

Trace Output – 10g (Sort Trace Output – 10g (Sort contd)contd)

Change to a shared cursor modelChange to a shared cursor modelPARSING IN CURSOR #1 len=129 dep=1 uid=27 oct=3 lid=27 PARSING IN CURSOR #1 len=129 dep=1 uid=27 oct=3 lid=27

tim=1129017679619468 hv=3411843401 ad='77e66bd4'tim=1129017679619468 hv=3411843401 ad='77e66bd4'SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)FROM test_tab1 tt1FROM test_tab1 tt1GROUP BY MOD(tt1.pk_id,2)GROUP BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)END OF STMTEND OF STMTPARSE PARSE


WAIT #1: nam='PX qref latch' ela= 52088 function=1 WAIT #1: nam='PX qref latch' ela= 52088 function=1 sleeptime=2042551468 qref=0 obj#=-1 tim=1129017679672279sleeptime=2042551468 qref=0 obj#=-1 tim=1129017679672279

WAIT #1: nam='PX Deq: Execution Msg' ela= 1954088 WAIT #1: nam='PX Deq: Execution Msg' ela= 1954088 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 tim=1129017681626630tim=1129017681626630



Slaves 2 and 3 - FTSSlaves 2 and 3 - FTS

Instance name: TEST1020Instance name: TEST1020

Redo thread mounted by this instance: 1Redo thread mounted by this instance: 1

Oracle process number: 12Oracle process number: 12

Unix process pid: Unix process pid: 1721917219, image: [email protected] , image: [email protected] ((P000P000))

*** 2006-08-20 10:59:33.938*** 2006-08-20 10:59:33.938


*** CLIENT ID:(*** CLIENT ID:(doug_testdoug_test) 2006-08-20 10:59:33.938) 2006-08-20 10:59:33.938

*** SESSION ID:(1090.11) 2006-08-20 10:59:33.938*** SESSION ID:(1090.11) 2006-08-20 10:59:33.938

WAIT #0: nam='PX Deq: Msg Fragment' ela= 1611 WAIT #0: nam='PX Deq: Msg Fragment' ela= 1611 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 sleeptime/senderid=268566527 passes=1 p3=0 obj#=-1 tim=1128972630798951tim=1128972630798951

Trace Output – 10g (FTS Trace Output – 10g (FTS Contd)Contd)

PARSING IN CURSOR #1 len=129 dep=1 uid=27 oct=3 lid=27 PARSING IN CURSOR #1 len=129 dep=1 uid=27 oct=3 lid=27 tim=1129017679653431 hv=3411843401 ad='77e66bd4'tim=1129017679653431 hv=3411843401 ad='77e66bd4'

SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)SELECT /*+ parallel(tt1, 2) */ MOD(tt1.pk_id, 2), COUNT(*)FROM test_tab1 tt1FROM test_tab1 tt1GROUP BY MOD(tt1.pk_id,2)GROUP BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)ORDER BY MOD(tt1.pk_id,2)END OF STMTEND OF STMTPARSE PARSE


WAIT #1: nam='PX qref latch' ela= 18705 function=1 WAIT #1: nam='PX qref latch' ela= 18705 function=1 sleeptime=2042549980 qref=0 obj#=-1 tim=1129017679672803sleeptime=2042549980 qref=0 obj#=-1 tim=1129017679672803


WAIT #1: nam='direct path read' ela= 46 file number=6 first WAIT #1: nam='direct path read' ela= 46 file number=6 first dba=10 block cnt=118 obj#=10907 tim=1129017679688488dba=10 block cnt=118 obj#=10907 tim=1129017679688488


ConsolidationConsolidation trcsesstrcsess utility (10g) utility (10g)

Consolidates all trace files associated with a Consolidates all trace files associated with a specific specific session identifiersession identifier

Consolidated trace file looks like any standard Consolidated trace file looks like any standard trace filetrace file

Can be processed using Can be processed using tkproftkprof to produce to produce statement-level summary informationstatement-level summary information

trcsess output=output_file trcsess output=output_file clientid=session_identifier clientid=session_identifier /oracle/admin/TEST1020/udump/*.trc /oracle/admin/TEST1020/udump/*.trc /oracle/admin/TEST1020/bdump/*.trc/oracle/admin/TEST1020/bdump/*.trc

tkprof trcsess_output_file output_file tkprof trcsess_output_file output_file sort=prsela,fchela,exeelasort=prsela,fchela,exeela

Output is in cursor order, not strict time orderOutput is in cursor order, not strict time order

Timing IssuesTiming Issues Example TimingsExample Timings

Wall Clock elapsed timeWall Clock elapsed time 13.5613.56 seconds seconds

call count cpu elapsed call count cpu elapsed ------- ------ -------- ---------- ------- ------ -------- ---------- Parse Parse 55 0.01 0.02 0.01 0.02Execute Execute 55 12.98 51.57 12.98 51.57Fetch Fetch 22 0.01 12.81 0.01 12.81------- ------ -------- ----------------- ------ -------- ----------total 12 13.01 total 12 13.01 64.4164.41

Picking up the wrong trace files?Picking up the wrong trace files? trcsess bug?trcsess bug?

Timing IssuesTiming Issues

Wait Time also ‘too high’Wait Time also ‘too high’Event waited on Times Max. Wait Total WaitedEvent waited on Times Max. Wait Total Waited---------------------------------------- Waited ---------- ---------------------------------------------------- Waited ---------- ------------PX qref latch 31 0.09 0.76PX qref latch 31 0.09 0.76PX Deq: Execution Msg 49 1.95 PX Deq: Execution Msg 49 1.95 27.7627.76direct path read 281 0.00 0.00direct path read 281 0.00 0.00db file sequential read 2 0.00 0.00db file sequential read 2 0.00 0.00PX Deq: Parse Reply 2 0.00 0.00PX Deq: Parse Reply 2 0.00 0.00SQL*Net message to client 2 0.00 0.00SQL*Net message to client 2 0.00 0.00PX Deq: Execute Reply 39 0.92 PX Deq: Execute Reply 39 0.92 12.1512.15PX Deq: Table Q qref 1 0.00 0.00PX Deq: Table Q qref 1 0.00 0.00SQL*Net message from client 2 0.00 0.00SQL*Net message from client 2 0.00 0.00PX Deq: Signal ACK 6 0.00 0.01PX Deq: Signal ACK 6 0.00 0.01enq: PS - contention 1 0.00 0.00enq: PS - contention 1 0.00 0.00PX Deq: Table Q Sample 2 0.00 0.00PX Deq: Table Q Sample 2 0.00 0.00PX Deq Credit: send blkd 3 0.00 0.00PX Deq Credit: send blkd 3 0.00 0.00


This is a This is a consolidatedconsolidated trace file trace file Multiple concurrent processesMultiple concurrent processes Some running on multiple CPUsSome running on multiple CPUs

There is more CPU time available than wall clock timeThere is more CPU time available than wall clock time Some just waitingSome just waiting

Systems have an infinite capacity for waitingSystems have an infinite capacity for waiting Will vary, depending on the actual DOP Will vary, depending on the actual DOP

acquiredacquired The DOP acquired could varyThe DOP acquired could vary How many slaves are available in the pool?How many slaves are available in the pool? parallel_adaptive_multi_userparallel_adaptive_multi_user


Exactly the same job, but DOP 16Exactly the same job, but DOP 16 Wall Clock elapsed time = Wall Clock elapsed time = 16.6016.60 seconds seconds

Slightly higher than Slightly higher than 13.5613.56 seconds seconds

call count cpu elapsed call count cpu elapsed ------- ------ -------- ---------- ------- ------ -------- ---------- Parse Parse 3333 0.03 0.16 0.03 0.16Execute Execute 3333 13.42 430.24 13.42 430.24Fetch 2 0.02 12.28Fetch 2 0.02 12.28------- ------ -------- ----------------- ------ -------- ----------total 68 13.48 total 68 13.48 442.69442.69

MuchMuch higher elapsed time higher elapsed time Couple of seconds additional wait timeCouple of seconds additional wait time

ConclusionConclusion Tracing PX is a little more tricky that tracing Tracing PX is a little more tricky that tracing

serial processesserial processes A bit like trying to trace multiple users, or an A bit like trying to trace multiple users, or an

entire server?entire server? 10g dbms_monitor package and trcsess 10g dbms_monitor package and trcsess

utility helputility help Consolidated timings need to be interpreted Consolidated timings need to be interpreted

differentlydifferently More papers available at More papers available at http://oracledoug.comhttp://oracledoug.com Next presentation – How Many Slaves – Hall Next presentation – How Many Slaves – Hall

7b at 16:007b at 16:00

Tracing PX SlavesTracing PX Slaves

Doug BurnsDoug Burns

[email protected]@yahoo.com

http://oracledoug.comhttp://oracledoug.com

Tracing Parallel Execution (UKOUG 2006)

Technology

Transcript of Tracing Parallel Execution (UKOUG 2006)