Oracle Database 11g RAC Performance Tuning

8/10/2019 Oracle Database 11g RAC Performance Tuning

1/52

Instance Recovery in RAC

Learning Objective

After completing this topic, you should be able to

identify how instance recovery works in a RAC environment

1. Elements of RAC instance recovery

When tuning your system, it is important that you compare the CPU time with the wait

time of your system. Comparing CPU time with wait time helps to determine how much of

the response time is spent on useful work and how much on waiting for resources

potentially held by other processes.

As a general rule, the systems where CPU time is dominant usually need less tuning than

the ones where wait time is dominant. On the other hand, heavy CPU usage can be

caused by badly written !" statements.

Graphic

In this example graph, the X-axis is Wait time and the Y-axis is CPU time. The

application is scalable when the CPU and Wait time are eqal in proportion. When

the CPU time is more than the wait time, the application might need !"# tning.

When the wait time is more than the CPU time, it needs instance or $%C tning.In sch a case, no gain is achie&ed b' adding CPUs or nodes.

Spplement

!electing the lin( title opens the resorce in a new browser window.

Style consi!erations

#iew more information on the style considerations for Oracle $$g%atabase used

in this course.

Lanch "in!o"

Although the proportion of CPU time to wait time always tends to decrease as load on the

system increases, steep increases in wait time are a sign of contention and must be

addressed for good scalability.

Adding more CPUs to a node, or nodes to a cluster, would provide very limited benefit
http://dowindow%28%27../t2/misc/lwod_orac_a03_it_enust201.html')http://dowindow%28%27../t2/misc/lwod_orac_a03_it_enust201.html')


2/52

under contention. Conversely, a system where the proportion of CPU time does not

decrease significantly as load increases can scale better, and would most likely benefit

from adding CPUs or &eal Application Clusters or &AC instances if needed.

#ote

%tomatic Wor(load $epositor' or %W$ reports displa' CPU time together with

wait time in the Top ) Timed *&ents section, i+ the CPU time portion is among the

top +i&e e&ents.

Although there are specific tuning areas for &AC such as instance recovery and

interconnect traffic, you get most benefits by tuning your system like a single'instance

system. At least this must be your starting point. Obviously, if you have seriali(ation

issues in a single'instance environment, these may be e)acerbated with &AC.

*ou have basically the same tuning tools with &AC as with a single'instance system.

+owever, certain combinations of specific wait events and statistics are well'known &ACtuning cases.

*ou see some of those specific combinations, as well as the &AC'specific information

that you can get from the nterprise -anager performance pages and tatspack and

AW& reports. inally, you see the &AC'specific information that you can get from the

Automatic %atabase %iagnostic -onitor or A%%-.

When an instance fails and the failure is detected by another instance, the second

instance performs these recovery steps.

Graphic

$eco&er' time process is di&ided into two phases. irst phase contains steps

and o+ the process, sch as $emaster enqee resorces and $emaster cache

resorces. In this phase, 'o se in+ormation +or other caches and the #/!

reco&ers 0$1. The second phase contains steps 2, 3, and ), which are 4ild

reco&er' set, $esorce claim, and $oll +orward reco&er' set. In this phase, !/56

reco&ers the database and merges +ailed redo threads.

Remaster en$ee resorces

%uring the first phase of recovery, /lobal n0ueue ervices or / remasters the

en0ueues.

Remaster cache resorces

1he /lobal Cache ervices or /C remasters its resources. 1he /C processes

remaster only those resources that lose their masters. %uring this time, all /C resource

re0uests and write re0uests are temporarily suspended. +owever, transactions can


3/52

continue to modify data blocks as long as these transactions have already ac0uired the

necessary resources.

%il! recovery set

After en0ueues are reconfigured, one of the surviving instances can grab the 2nstance

&ecovery en0ueue. 1herefore, at the same time as /C resources are remastered,-O3 determines the set of blocks that need recovery. 1his set is called the recovery set.

4ecause, with Cache usion, an instance ships the contents of its blocks to the re0uesting

instance without writing the blocks to the disk, the on'disk version of the blocks may not

contain the changes that are made by either instance.

1his implies that -O3 needs to merge the content of all the online redo logs of each

failed instance to determine the recovery set. 1his is because one failed thread might

contain a hole in the redo that needs to be applied to a particular block. o redo threads of

failed instances cannot be applied serially. Also, redo threads of surviving instances are

not needed for recovery because -O3 could use past or current images of their

corresponding buffer caches.

Resorce claim

4uffer space for recovery is allocated and the resources that were identified in the previous

reading of the redo logs are claimed as recovery resources. 1his is done to avoid other

instances to access those resources.

Roll for"ar! recovery set

All resources re0uired for subse0uent processing have been ac0uired and the /lobal

&esource %irectory or /&% is now unfro(en. Any data blocks that are not in recovery can

now be accessed. 3ote that the system is already partially available. 1hen, assuming that

there are past images or current images of blocks to be recovered in other caches in thecluster database, the most recent image is the starting point of recovery for these

particular blocks.

2f neither the past image buffers nor the current buffer for a data block is in any of the

surviving instances5 caches, then -O3 performs a log merge of the failed instances.

-O3 recovers and writes each block identified in step 6, releasing the recovery

resources immediately after block recovery so that more blocks become available as

recovery proceeds.

After all blocks have been recovered and the recovery resources have been released, the

system is again fully available.

2n summary, the recovered database or the recovered portions of the database becomes

available earlier, and before the completion of the entire recovery se0uence. 1his makes

the system available sooner and it makes recovery more scalable.

#ote


4/52

The per+ormance o&erhead o+ a log merge is proportional to the nmber o+ +ailed

instances and to the si7e o+ the amont o+ redo written in the redo logs +or each

instance.

&estion

When an instance fails and the failure is detected by another instance, the second

instance performs a series of recovery steps. What is the first step performed by

the second instance7

Options'

$. 1he /&% is unfro(en

8. 1he / remasters the en0ueues

6. 4uffer space for recovery is allocated

9. 1he /lobal Cache ervices :/C; remasters its resources

Ans"er

Option 1:Incorrect. 1ring the +inal step in instance reco&er', all resorces

reqired +or sbseqent processing ha&e been acqired and the 0lobal $esorce

1irector' 80$19 is n+ro7en. %n' data bloc(s that are not in reco&er' can then be

accessed.

Option 2:Correct. 1ring the +irst step in instance reco&er', the 0lobal *nqee

!er&ices 80*!9 remasters the enqees. 5nce this step has been per+ormed, one

o+ the sr&i&ing instances can grab the Instance $eco&er' enqee, 0C!resorces are remastered, and !/56 determines the set o+ bloc(s that need

reco&er'.

Option 3:Incorrect. 1ring the +orth step per+ormed dring instance reco&er',

b++er space +or reco&er' is allocated and the resorces that were identi+ied in the

pre&ios reading o+ the redo logs are claimed as reco&er' resorces. This is done

to a&oid other instances accessing those resorces.

Option 4:Incorrect. 1ring the second step in instance reco&er', the 0lobal

Cache !er&ices 80C!9 remasters its resorces. The 0C! processes remaster

onl' those resorces that lose their masters. 1ring this time, all 0C! resorcereqests and write reqests are temporaril' sspended. :owe&er, transactions

can contine to modi+' data bloc(s as long as these transactions ha&e alread'

acqired the necessar' resorces.

Correct ans"er(s)'

8. 1he / remasters the en0ueues


5/52

1he e)ample illustrates the degree of database availability during each step of Oracle

instance recovery.

Graphic

The example +or 5racle instance reco&er' contains a database a&ailabilit' ranging

+rom 6one, Partial, and ll with increasing *lapsed time.

A

&eal Application Clusters is running on multiple nodes.

%

3ode failure is detected.

C

1he en0ueue part of the /&% is reconfigured< resource management is redistributed to the

surviving nodes. 1his operation occurs relatively 0uickly.

*

1he cache part of the /&% is reconfigured and -O3 reads the redo log of the failed

instance to identify the database blocks that it needs to recover.

E

-O3 issues the /&% re0uests to obtain all the database blocks it needs for recovery.

After the re0uests are complete, all other blocks are accessible.

+

1he Oracle server performs roll forward recovery. &edo logs of the failed threads areapplied to the database, and blocks are available right after their recovery is completed.

G

1he Oracle server performs rollback recovery. Undo blocks are applied to the database for

all uncommitted transactions.

,

2nstance recovery is complete and all data is accessible.

1he dotted steps represent the ones identified in the &AC and 2nstance or Crash

&ecovery process. 1he dashed line represents the blocks identified while remastering

cache resources in this process.

2n a single'instance environment, the instance startup combined with the crash recovery

time is controlled by the setting of the FAST_START_MTTR_TARGETinitiali(ation

parameter. *ou can set its value if you want incremental checkpointing to be more

aggressive than autotune checkpointing. +owever, this is at the e)pense of a much

higher 2=O overhead.


6/52

2n a &AC environment, including the startup time of the instance in this calculation is

useless because one of the surviving instances is doing the recovery.

2n a &AC environment, it is possible to monitor the estimated target :in seconds; for the

duration from the start of instance recovery to the time when /C% is open for lock

re0uests for blocks that are not needed for recovery. 1his estimation is published in the

V$INSTANCE_RECOVERYview through the ESTD_CLUSTER_AVAILABLE_TIMEcolumn.

4asically, you can monitor the time your cluster is fro(en during instance recovery

situations.

2n a &AC environment, the FAST_START_MTTR_TARGETinitiali(ation parameter is used

to bind the entire instance recovery time, assuming that it is instance recovery for single'

instance death.

#ote

I+ 'o reall' want to ha&e small instance reco&er' time b' setting

FAST_START_MTTR_TARGET, 'o can sa+el' ignore the alert log messages abot

raising its &ale.

+ere are some guidelines you can use to make sure that instance recovery in your &AC

environment is faster>

se parallel instance recovery

Use parallel instance recovery by setting RECOVERY_PARALLISM.

increase PARALLEL_EXECUTION_MESSAGE_SIZE

2ncrease PARALLEL_EXECUTION_MESSAGE_SIZE from its default of 8,$9? bytes to 9 @4

or ? @4. 1his should provide better recovery slave performance.

set PARALLEL_MIN_SERVERS

et PARALLEL_MIN_SERVERStoCPU_COUNT-1. 1his will prespawn recovery slaves at

startup time.

se asynchronos I-O an!

Using asynchronous 2=O is one of the most crucial factors in recovery time. 1he first'pass

log read uses asynchronous 2=O.

increase !efalt bffer cache si/e2nstance recovery uses B percent of the default buffer cache for recovery buffers. 2f this is

not enough, some of the steps of instance recovery will be done in several passes. *ou

should be able to identify such situations by looking at your alertlo!file. 2n that case,

you should increase the si(e of your default buffer cache.

Smmary


7/52

When the wait time of a system is more than the CPU time, it needs instance or &AC

tuning. pecific wait events, system and en0ueue statistics, nterprise -anager

performance pages, tatspack and AW& reports, and A%%- reports help in &AC specific

tuning.

1he recovery time in &AC and instance or crash recovery processes is divided into two

phases. 1he first phase includes remastering en0ueue and cache resources. 1he second

phase includes building recovery set, resource claim, and roll forward of recovery set.

%atabase availability ranges from none to full during the Oracle instance recovery

process. ollow simple but important guidelines to ensure fast instance recovery in your

&AC environment.

RAC 0ait Events an! Latencies

Learning Objective


recognize typical wait events and latencies for RAC

1. Cache fsion latencies an! "ait events

1he effect of accessing blocks in the global cache and maintaining cache coherency are

represented as follows>

the /lobal Cache ervices statistics for current and cr blocks such as gc current blocks received

and gc cr blocks received and

the /lobal Cache ervices wait events for gc current block 6'way, gc cr grant 8'way, and so on

1he response time for Cache usion transfers is determined by the messaging time and

processing time imposed by the physical interconnect components, the 2PC protocol, and

the /C protocol.

2t is not affected by disk input=output or 2=O factors other than occasional log writes. 1he

Cache usion protocol does not re0uire 2=O to data files in order to guarantee cache

coherency, and &AC inherently does not cause any more 2=O to disk than a nonclusteredinstance.

2n a &AC AW& report, the /lobal Cache and n0ueue ervices ' Workload

Characteristics table in the &AC tatistics section contains average times :latencies; for

some /lobal Cache ervices and /lobal n0ueue ervices operations.

1hose latencies should be monitored over time, and significant increases in their values


8/52

should be investigated. 1he table presents some typical values, based on empirical

observations. 1he factors that may cause variations to those latencies are as follows>

Graphic

The 0lobal Cache and *nqee !er&ices - Wor(load Characteristics table has

two colmns. The +irst colmn incldes ser&ices and the second colmn

incldes a &ale +or each ser&ice. The ser&ices in the +irst colmn are %&g

global enqee get time 8ms9, %&g global cache cr bloc( recei&e time 8ms9, %&g

global cache crrent bloc( recei&e time 8ms9, %&g global cache cr bloc( bild time

8ms9, %&g global cache cr bloc( send time 8ms9, 0lobal cache log +lshes +or cr

bloc(s ser&ed ;, %&g global cache cr bloc( +lsh time 8ms9, %&g global cache

crrent bloc( pin time 8ms9, %&g global cache crrent bloc( send time 8ms9, 0lobal

cache log +lshes +or crrent bloc(s ser&ed ;, and %&g global cache crrent bloc(

+lsh time 8ms9.

utili(ation of the 2PC protocol, user'mode 2PC protocols are faster, but only 1ru9Ds &%/ is

recommended for use

scheduling delays, when the system is under high CPU utili(ation, and

log flushes for current blocks served

Other &AC latencies in AW& reports are mostly derived from V$GES_STATISTICSand

may be useful for debugging purposes, but do not re0uire fre0uent monitoring.

Graphic

The table incldes +or colmns < %W$ $eport #atenc' 6ame, #ower 4ond,

T'pical, and Upper 4ond.The latenc' names are %&erage time to process cr

bloc( reqest, %&g global cache cr bloc( recei&e time 8ms9, %&erage time to

process crrent bloc( reqest, and %&g global cache crrent bloc( recei&e time

8ms9. The &ales +or #ower 4ond, T'pical, and Upper 4ond are pro&ided +or

each latenc'.

#ote

The time to process consistent read or C$ bloc( reqest in the cache corresponds

to

8"#l%t&e'(l#)*t&e')e+%t&e9, and the time to process crrent bloc(

reqest in the cache corresponds to 8,+t&e'(l#)*t&e')e+%t&e.

Analy(ing what sessions are waiting for is an important method to determine where time

is spent. 2n &AC, the wait time is attributed to an event that reflects the e)act outcome of


9/52

a re0uest.

or e)ample, when a session on an instance is looking for a block in the global cache, it

does not know whether it will receive the data cached by another instance or whether it

will receive a message to read from disk.

1he wait events for the global cache convey precise information and wait for global cache

blocks or messages. 1hey are mainly categori(ed by the following>

summari(ed in a broader category called Cluster Wait Class

temporarily represented by a placeholder event that is active while waiting for a block, and

attributed to precise events when the outcome of the re0uest is known

When it takes some time to ac0uire resources because of the total path length and

latency for re0uests, processes sleep to avoid spinning for indeterminate periods of time.

When the process decides to wait, it wakes up either after a specific timer value e)pires

:timeout; or when the event it is waiting for occurs and the process is posted. 1he wait

events are recorded and aggregated in the views.

1he first three are aggregations of wait times, timeouts, and the number of times waited

for a particular event, whereas the rest enable the monitoring of waiting sessions in real

time, including a history of recent events waited for.

Graphic

There are se&en &iews.

The =>!Y!T*/?*=*6T &iew displa's total waits +or an e&ent. The

=>!*!!I56?W%IT?C#%!! &iew waits +or a wait e&ent class b' a session. The

=>!*!!I56?*=*6T &iew waits +or an e&ent b' a session. The

=>%CTI=*?!*!!I56?:I!T5$Y &iew displa's acti&it' o+ recent acti&e sessions.

The =>!*!!I56?W%IT?:I!T5$Y &iew displa's last @ wait e&ents +or each

acti&e session. The =>!*!!I56?W%IT &iew displa's e&ents +or which acti&e

sessions are waiting. The =>!"#!T%T! &iews identi+' !"# statements impacted

b' interconnect latencies.

or most of the global cache wait events, the parameters include file number, block

number, the block class, and access mode dispositions, such as mode held and

re0uested.

1he wait times for events presented and aggregated in these views are very useful when

debugging response time performance issues. 3ote that the time waited is cumulative,

and that the event with the highest score is not necessarily a problem.


10/52

+owever, if the available CPU power cannot be ma)imi(ed, or response times for an

application are too high, the top wait events provide valuable performance diagnostics.

#ote

Use the CLUSTER_.AIT_TIMEcolmn in V$S/LSTATSto identi+' !"#

statements impacted b' interconnect latencies, or rn an %11/ report on the

corresponding %W$ snapshot.

&estion

Which wait event view contains a CLUSTER_.AIT_TIMEcolumn that can be used

to identify !" statements impacted by interconnect latencies7

Options'

$. V$S/LSTATS

8. V$SESSION_.AIT

6. V$SYSTEM_EVENT

9. V$SESSION_.AIT_0ISTORY

Ans"er

Option 1:Correct. Use the CLUSTER_.AIT_TIMEcolmn in V$S/LSTATSto

identi+' !"# statements impacted b' interconnect latencies, or rn an %11/

report on the corresponding %W$ snapshot.

Option 2:Incorrect. This wait e&ent &iew shows e&ents +or which acti&e sessions

are waiting.

Option 3:Incorrect. This wait e&ent &iew shows total waits +or an e&ent.

Option 4:Incorrect. This wait e&ent &iew shows last @ wait e&ents +or each acti&e

session.

Correct ans"er(s)'

$. V$S/LSTATS

. Cache "ait events an! en$ee "aits

1he main global cache wait events for Oracle %atabase $Bgare briefly described.


11/52

gc[current/cr][multil!c"]re#ue$t

1hese wait events are relevant only while a gc re0uest for a cr or current buffer is in

progress. 1hey act as placeholders until the re0uest completes.

gc[current/cr][%/&]'()*

A current or cr block is re0uested and received after two or three network hops. 1here0uest is processed immediately> the block is not busy or congested.

gc[current/cr]l!c"u$*

A current or cr block is re0uested and received, but is not sent immediately by "-

because some special condition that delayed the sending was found.

gc[current/cr]gr)nt%'()*

A current or cr block is re0uested and a grant message received. 1he grant is given

without any significant delays. 2f the block is not in its local cache, a current or cr grant is

followed by a disk read on the re0uesting instance.

gccurrentgr)ntu$*

A current block is re0uested and a grant message received. 1he busy hint implies that the

re0uest is blocked because others are ahead of it or cannot be handled immediately.

gc[current/cr][l!c"/gr)nt]c!nge$te,

A current or cr block is re0uested and a block or grant message received. 1he congested

hint implies that the re0uest spent more than $ ms in internal 0ueues.

gc[current/cr][-)ilure/retr*]

A block is re0uested and a failure status received or some other e)ceptional event has

occurred.

gcu--eru$*

2f the time between buffer accesses becomes less than the time the buffer is pinned in

memory, the buffer containing a block is said to become busy and as a result interested

users may have to wait for it to be unpinned.

or dynamic remastering, two events are of the most practical importance> !

re&a)terand ! 3#e)e. 1hey can be symptoms of the effect of remastering on the

running processes.

1he 4-5a6block re0uest shows what typically happens when the master instance

re0uests a block that is not cached locally. +ere it is supposed that the master instance iscalled SGA1, and SGA4contains the re0uested block. 1he scenario contains these steps>

step 1

SGA1sends a direct re0uest to SGA4. o SGA1waits on the ! #rre+t "lo7

re3#e)tevent.

step an!


12/52

When SGA4receives the re0uest, its local "/W& process may need to flush some

recovery information to its local redo log files. or e)ample, if the cached block is

fre0uently changed and the changes have not been logged yet, "- would have to ask

"/W& to flush the log before it can ship the block. 1his may add a delay to the serving of

the block and may show up in the re0uesting node as a busy wait.

step 2

1hen, SGA4sends the re0uested block to SGA1. When the block arrives in SGA1, the wait

event is complete and is reflected as ! #rre+t "lo7 4-5a6.

Using the notation RE time at re0uestor, .E wire time and transfer delay, and SE time at

server, the total time for a round'trip would be> R:send; F .:small msg; F S:process msg,

process block, send; F .:block; F R:receive block;.

1his is a modified scenario for a cluster with more than two nodes. 2t is very similar to the

4-5a6 4lock &e0uest. +owever, the master for this block is on a node that is different

from that of the re0uestor, and where the block is cached. 1hus, the re0uest must beforwarded.

1here is an additional delay for one message and the processing at the master node>

R:send; F .:small msg; F S:process msg, send; F .:small msg; F S:process msg,

process block, send; F .:block; F R:receive block;.

While a remote read is pending, any process on the re0uesting instance that is trying to

write or read the data cached in the buffer has to wait for a gc buffer busy. 1he buffer

remains globally busy until the block arrives.

2n this scenario, a grant message is sent by the master because the re0uested block isnot cached in any instance. 2f the local instance is the resource master, the grant happens

immediately. 2f not, the grant is always 8'way, regardless of the number of instances in

the cluster.

1he grant messages are small. or every block read from the disk, a grant has to be

received before the 2=O is initiated, which adds the latency of the grant round'trip to the

disk latency> R:send; F .:small msg; F S:process msg, send; F .:small msg; F R:receive

block;.

1he round'trip looks similar to a 8'way block round'trip, with the difference that the wire

time is determined by a small message, and the processing does not involve the buffercache.

An en0ueue wait is not &AC specific, but involves a global lock operation when &AC is

enabled.

-ost of the global re0uests for en0ueues are synchronous, and foreground processes

wait for them. 1herefore, contention on en0ueues in &AC is more visible than in single'


13/52

instance environments. 1he following are the en0ueue types for which most waits for

en0ueues occur.

34

1ransaction en0ueue< used for transaction demarcation and tracking.

35

1able or partition en0ueue< used to protect table definitions during %-" operations.

,0

+igh'water mark en0ueue< ac0uired to synchroni(e a new block operation.

S&

e0uence en0ueue< used to seriali(e incrementing of an Oracle se0uence number.

3A

n0ueue used mainly for transaction recovery as part of instance recovery.

6S

Undo segment en0ueue< mainly used by the Automatic Undo -anagement :AU-; feature.

2n the case of all the en0ueue types, the waits are synchronous and may constitute

serious seriali(ation points that can be e)acerbated in a &AC environment.

#ote

In 5racle 1atabase @g, the enqee wait e&ents speci+' the resorce name and

a reason +or the wait A +or example, TX *nqee index bloc( split. This ma(es

diagnostics o+ enqee waits easier.

Using system statistics based on V$SYSSTATenables characteri(ation of the database

activity based on averages. 2t is the basis for many metrics and ratios used in various

tools and methods, such as AW&, tatspack, and %atabase Control.

2n order to drill down to individual sessions or groups of sessions, V$SESSTATis useful

when the important session identifiers to monitor are known. 2ts usefulness is enhanced if

an application fills in the MODULEand ACTIONcolumns in V$SESSION.

V$SEGMENT_STATISTICSis useful for &AC because it also tracks the number of C&

and current blocks received by the obGect.

1he &AC'relevant statistics can be grouped into the following categories>

/lobal Cache ervice statistics such as gccrbloc(srecei&ed,gccrbloc(recei&etime, and so on

/lobal n0ueue ervice statistics such as globalenqeegetsand so on, and


14/52

tatistics for messages sent such as gcsmessagessentand gesmessagessent

V$EN/UEUE_STATISTICScan be 0ueried to determine which en0ueue has the highest

impact on database service times and eventually response times.

V$INSTANCE_CAC0E_TRANSFERindicates how many current and C& blocks per blockclass are received from each instance, including how many transfers incurred a delay.

Smmary

2n a &AC AW& report you monitor typical latencies of some /lobal Cache ervices and

/lobal n0ueue ervices operations for significant increases in their values. actors

causing variations include utili(ation of the 2PC protocol, scheduling delays, and log

flushes. Other &AC latencies in AW& reports are mostly derived from

V$GES_STATISTICSand may be useful for debugging purposes, but do not re0uire

fre0uent monitoring.

1here are many global cache wait events but for dynamic remastering, two events are of

the most practical importance> gc remaster and gc 0uiesce. 1o manage and monitor

database activities and sessions, you can use session and system statistics such as,

V$SYSSTAT, V$SEGMENT_STATISTICS, V$EN/UEUE_STATISTICS, and

V$INSTANCE_CAC0E_TRANSFER.

RAC 3ning 3ips an! Consi!erations

Learning Objectives


identify the most common RAC tuning tips

identify factors that affect tuning in a RAC environment

1. +actors affecting tning in RAC

2n any database system, &AC or single instance, the most significant performance gains

are usually obtained from traditional application'tuning techni0ues. 1he benefits of those

techni0ues are even more remarkable in a &AC database. 2n addition to traditional

application tuning, these are some of the techni0ues that are particularly important for

&AC.

Avoi! long fll7table scans

1ry to avoid long full'table scans to minimi(e /C re0uests. 1he overhead caused by the

global C& re0uests in this scenario is because of the fact that when 0ueries result in local


15/52

cache misses, an attempt is first made to find the data in another cache, based on the

assumption that the chance that another instance has cached the block is high.

Resi/e an! tne the bffer cache

+ash partitioning may help to reduce buffer busy contention by making buffer access

distribution patterns sparser, enabling more buffers to be available for concurrent access.6se ASS5

Automatic egment pace -anagement can provide instance affinity to table blocks.

Increase se$ence caches

2ncreasing se0uence caches improves instance affinity to inde) keys deriving their values

from se0uences. 1hat techni0ue may result in significant performance gains for multi'

instance insert'intensive applications.

Re!ce interinstance traffic "ith partitioning

&ange or list partitioning may be very effective in conGunction with data'dependent routing,

if the workload can be directed to modify a particular range of values from a particularinstance.

Avoi! nnecessary parsing

2n &AC, library cache and row cache operations are globally coordinated. o e)cessive

parsing means additional interconnect traffic. "ibrary cache locks are heavily used, in

particular by applications using P"=!" or Advanced !ueuing. "ibrary cache locks are

ac0uired in e)clusive mode whenever a package or procedure has to be recompiled.

5inimi/e loc8ing sage

4ecause transaction locks are globally coordinated, they also deserve special attention in

&AC. or e)ample, using tables instead of Oracle se0uences to generate uni0ue numbersis not recommended because it may cause severe contention even for a single instance

system.

Remove nselective in!e9es

2nde)es that are not selective do not improve 0uery performance, but can degrade %-"

performance. 2n &AC, unselective inde) blocks may be subGect to interinstance contention,

increasing the fre0uency of cache transfers for inde)es belonging to INSERT'intensive

tables.

Configre interconnect properly

Always verify that you use a private network for your interconnect, and that your private

network is configured properly. nsure that a network link is operating in full duple) mode.

nsure that your network interface and thernet switches support -1U si(e of H @4. 3ote

that a single /4 can scale up to ten thousand ?'@4 blocks per second before saturation.

&estion


16/52

Which common &AC tuning techni0ues may result in significant performance

gains for multi'instance, insert'intensive applications7

Options'

$. using hash partitioning

8. increasing se0uence caches

6. using range or list partitioning

9. avoiding long full'table scans

Ans"er

Option 1:Incorrect. :ash partitioning ma' help to redce b++er bs' contention

b' ma(ing b++er access distribtion patterns sparser, enabling more b++ers to be

a&ailable +or concrrent access.

Option 2:Correct. Increasing seqence caches impro&es instance a++init' to index

(e's deri&ing their &ales +rom seqences. That techniqe ma' reslt in signi+icant

per+ormance gains +or mlti-instance, insert-intensi&e applications.

Option 3:Incorrect. $ange or list partitioning ma' be &er' e++ecti&e in conBnction

with data-dependent roting, i+ the wor(load can be directed to modi+' a particlar

range o+ &ales +rom a particlar instance.

Option 4:Incorrect. Tr' to a&oid long +ll-table scans to minimi7e 0C! reqests.

The o&erhead cased b' the global C$ reqests in this scenario is de to the +act

that when qeries reslt in local cache misses, an attempt is +irst made to +ind the

data in another cache, based on the assmption that the chance that another

instance has cached the bloc( is high.

Correct ans"er(s)'

8. increasing se0uence caches

2n application systems where the loading or batch processing of data is a dominant

business function, there may be performance issues affecting response times because of

the high volume of data inserted into inde)es. %epending on the access fre0uency and

the number of processes concurrently inserting data, inde)es can become hot spots and

contention can be e)acerbated by these three factors.

irst, ordered, monotonically increasing key values in the inde) :right'growing trees;.

econd, fre0uent leaf block splits, and third, low tree depth which means all leaf block

access go through the root block. A leaf or branch block split can become an important

seriali(ation point if the particular leaf block or branch of the tree is concurrently

accessed.


17/52

1he tables sum up the most common symptoms associated with the splitting of inde)

blocks, listing wait events and statistics that are commonly elevated when inde) block

splits are prevalent. As a general recommendation, to alleviate the performance impact of

globally hot inde) blocks and leaf block splits, a more uniform, less skewed distribution of

the concurrency in the inde) tree should be the primary obGective.

1he primary obGective can be achieved by the following actions>

Graphic

There are two tables. The +irst table contains onl' one colmn named Wait e&ents.

The wait e&ents listed in this colmn are enq TX - indexcontention, gc b++er bs',

gc crrent bloc( bs', and gc crrent split.

The second table also contains onl' one colmn named as !'stem statistics. The

s'stem statistics listed in this colmn are #ea+ node splits, 4ranch node splits,

*xchange deadloc(s, gcs re+se xid, gcs ast xid, and !er&ice IT# waits.

global inde) hash partitioning

increasing the se0uence cache, if the key value is derived from a se0uence

use natural keys as opposed to surrogate keys, and

use reverse key inde)es

2nde)es with key values generated by se0uences tend to be subGect to leaf block

contention when the insert rate is high. 1hat is because the inde) leaf block holding the

highest key value is changed for every row inserted, as the values are monotonically

ascending. 2n &AC, this may lead to a high rate of current and C& blocks transferred

between nodes.

One of the simplest techni0ues that can be used to limit this overhead is to increase the

se0uence cache, if you are using Oracle se0uences. As the difference between se0uence

values generated by different instances increases, successive inde) block splits tend to

create instance affinity to inde) leaf blocks.

Graphic

%n index lea+ bloc( consists o+ )@@ rows. The bloc( has instance and instance .

Instance inserts &ales ...)@@@@ in $%C@. Instance inserts &ales

)@@@...@@@@@ in $%C@.

or e)ample, suppose that an inde) key value is generated by a CAC0E NOORDER

se0uence and each inde) leaf block can hold BB rows. 2f the se0uence cache is set to


18/52

BBBB, while instance $ inserts values $, 8, 6, and so on, instance 8 concurrently inserts

BBB$, BBB8, and so on. After some block splits, each instance writes to a different part

of the inde) tree.

1he ideal value for a se0uence cache has a dual purpose. 2t avoids interinstance leaf

inde) block contention and minimi(es possible gaps. One of the main variables to

consider is the insert rate> the higher it is, the higher the se0uence cache must be.

+owever, creating a simulation to evaluate the gains for a specific configuration is

recommended.

#ote

4' de+alt, the cache &ale is @. T'picall', @ is too small +or this example.

)cessive undo block shipment and contention for undo buffers usually happens when

inde) blocks containing active transactions from multiple instances are read fre0uently.

When a SELECTstatement needs to read a block with active transactions, it has to undo

the changes to create a C& version.

2f the active transactions in the block belong to more than one instance, there is a need to

combine local and remote undo information for the consistent read. %epending on the

amount of inde) blocks changed by multiple instances and the duration of the

transactions, undo block shipment may become a bottleneck.

Graphic

In this example, there is an Index with Instance and Instance . It also contains$%C@, $%C@, !0%, and !0%. $%C@ sends changes to Instance and

!0%. There are constant reads between Instance and !0% and between

!0% and $%C@. This reslts in %dditional interconnect tra++ic. The Undo bloc(s

in !0% and !0% compare Instance with $%C@ and $%C@ respecti&el' to

chec( i+ the C$ &ersion exists.

Usually this happens in applications that read recently inserted data very fre0uently, but

commit infre0uently. 1hese are the techni0ues that can be used to reduce such

situations.

horter transactions reduce the likelihood that any given inde) block in the cachecontains uncommitted data, thereby reducing the need to access undo information for

consistent read.

2ncreasing se0uence cache si(es can reduce interinstance concurrent access to inde)

leaf blocks. C& versions of inde) blocks modified by only one instance can be fabricated

without the need of remote undo information.


19/52

#ote

In $%C, the problem is exacerbated b' the +act that a sbset o+ the ndo

in+ormation has to be obtained +rom remote instances.

A certain combination of wait events and statistics presents itself in applications where

the insertion of data is a dominant business function and new blocks have to be allocated

fre0uently to a segment. 2f data is inserted at a high rate, new blocks may have to be

made available after unfruitful searches for free space.

1his has to happen while holding the +igh'Water -ark or +W- en0ueue. 1herefore, the

most common symptoms for this scenario include a high percentage of wait time for e+38

0. 9 o+te+to+and a high percentage of wait time for !#rre+t!ra+tevents.

Graphic

$%C@ and $%C@ ha&e hea&' inserts dring changes and reads with :W/.

:W/ enqee ma(es new bloc(s a&ailable +or +ree space.

1he former is a conse0uence of the seriali(ation of the +W- en0ueue, and the latter is

because of the fact that current access to the new data blocks that need formatting is

re0uired for the new block operation.

2n a &AC environment, the length of this space management operation is proportional to

the time it takes to ac0uire the +W- en0ueue and the time it takes to ac0uire global locks

for all the new blocks that need formatting. 1his time is small under normal circumstances

because there is never any access conflict for the new blocks.

1herefore, this scenario may be observed in applications with business functions

re0uiring a lot of data loading, and the main recommendation to alleviate the symptoms is

to define uniform and large e)tent si(es for the locally managed and automatic space

managed segments that are subGect to high'volume inserts.

2n data warehouse and data mart environments, it is not uncommon to see a lot of

TRUNCATEoperations.

1hese essentially happen on tables containing temporary data. 2n a &AC environment,

truncating tables concurrently from different instances does not scale well, especially if,you are also using direct read operations such as parallel 0ueries.

A truncate operation re0uires a cross'instance call to flush dirty blocks of the table that

may be spread across instances. 1his constitutes a point of seriali(ation. o, while the

first TRUNCATEcommand is processing, the second has to wait until the first one

completes.


20/52

Graphic

The example contains !0%and !0%. !0% consists o+ Table and Table.

!0% consists o+ Table with a 1irt' bloc( and Table. Trncate Table is

processing with !0% and Trncate Table with !0%. There is a Cross-instance

call between the two trncate commands. Chec(pointing CDPT connects !0%,!0%, and the Cross-instance call.

1here are different types of cross'instance calls. +owever, all use the same seriali(ation

mechanism.

or e)ample, the cache flush for a partitioned table with many partitions may add latency

to a corresponding parallel 0uery. 1his is because each cross'instance call is seriali(ed at

the cluster level, and one cross'instance call is needed for each partition at the start of

the parallel 0uery for direct read purposes.

&estion

2n a &AC environment, truncating tables concurrently from different instances does

not scale well, especially if, you are also using direct read operations such as

parallel 0ueries. What causes this issue7

Options'

$. undo information has to be obtained from remote instances

8. concurrent cross'instance calls are a point of seriali(ation

6. all leaf block access going through the root block leads to contention9. key values generated by se0uences are subGect to leaf block contention

Ans"er

Option 1:Incorrect. *xcessi&e ndo bloc( shipment and contention +or ndo

b++ers sall' happens when index bloc(s containing acti&e transactions +rom

mltiple instances are read +reqentl'. In $%C, the problem is exacerbated b' the

+act that a sbset o+ the ndo in+ormation has to be obtained +rom remote

instances.

Option 2:Correct. % trncate operation reqires a cross-instance call to +lsh dirt'

bloc(s o+ the table that ma' be spread across instances. This constittes a point

o+ seriali7ation. !o, while the +irst TRUNCATEcommand is processing, the second

has to wait ntil the +irst one completes. There are di++erent t'pes o+ cross-

instance calls. :owe&er, all se the same seriali7ation mechanism.

Option 3:Incorrect. 1epending on the access +reqenc' and the nmber o+

processes concrrentl' inserting data, indexes can become hot spots and


21/52

contention can be exacerbated b' low tree depth. That is all lea+ bloc( access

going throgh the root bloc(.

Option 4:Incorrect. Indexes with (e' &ales generated b' seqences tend to be

sbBect to lea+ bloc( contention when the insert rate is high. That is becase the

index lea+ bloc( holding the highest (e' &ale is changed +or e&er' row inserted,

as the &ales are monotonicall' ascending. In $%C, this ma' lead to a high rate o+

crrent and C$ bloc(s trans+erred between nodes.

Correct ans"er(s)'

8. concurrent cross'instance calls are a point of seriali(ation

Smmary

*ou should avoid long full'table scans, use A-, and increase se0uence caches during

&AC tuning. 2nde) 4lock Contention is worsened by increasing key values in the inde),

fre0uent leaf block splits and low tree depth.

*ou can use global inde) hash partitioning for a less skewed distribution of the

concurrency in the inde) tree. 2ncreasing se0uence cache, replacing surrogate with

natural keys, and using reverse key inde)es will also facilitate this concurrency. 4y

increasing the se0uence cache, you can limit the overhead caused by high rates of block

transfer in &AC.

1o handle new data that is fre0uently read but rarely committed, you can use shorter

transactions and increase se0uence cache si(e. 4e sure to define uniform and large

e)tent si(es while dealing with +igh'Water -ark en0ueues.

5onitoring RAC *atabase an! Clster :erformance

Learning Objective


recognize how to use the Cluster Database Performance pages to monitor RAC database and

cluster performance

1. 3he Clster *atabase :erformance page

4oth Oracle nterprise -anager %atabase Control and /rid Control are cluster'aware

and provide a central console to manage your cluster database.

rom the Cluster %atabase +ome page, you can perform the following functions>


22/52

vie" overall system stats

#iew the overall system status, such as the number of nodes in the cluster and their

current status so you do not have to access each individual database instance for details.

vie" alert messages

#iew alert messages aggregated across all the instances with lists for the source of eachalert message.

revie" isses

&eview issues that are affecting the entire cluster as well as those that are affecting

individual instances.

monitor clster cache coherency statistics

-onitor cluster cache coherency statistics to help you identify processing trends and

optimi(e performance for your Oracle &AC environment. Cache coherency statistics

measure how well the data in caches on multiple instances is synchroni(ed.

!etermine availability problems an!

%etermine if any of the services for the cluster database are having availability problems. A

service is determined to be a problem service if it is not running on all preferred instances,

if its response time thresholds are not met, and so on.

revie" alerts

&eview any outstanding Clusterware interconnect alerts.

&estion

Which two tasks can be performed directly from the Cluster %atabase +omepage7

Options'

$. #iew the overall system status

8. -onitor cluster cache coherency statistics

6. -onitor summary charts for cache coherency metrics for the cluster

9. #iew how much work the database is performing on behalf of the users or

applications

Ans"er

Option 1:Correct. rom the Clster 1atabase :ome page, 'o can &iew the

o&erall s'stem stats, sch as the nmber o+ nodes in the clster and their crrent

stats so that 'o do not ha&e to access each indi&idal database instance +or

details.


23/52

Option 2:Correct. rom the Clster 1atabase :ome page, 'o can monitor

clster cache coherenc' statistics to help 'o identi+' processing trends and

optimi7e per+ormance +or 'or 5racle $%C en&ironment. Cache coherenc'

statistics measre how well the data in caches on mltiple instances is

s'nchroni7ed.

Option 3:Incorrect. The Clster Cache Coherenc' page contains smmar' charts

+or cache coherenc' metrics +or the clster.

Option 4:Incorrect. The 1atabase Throghpt charts, on the Clster 1atabase

Per+ormance page, smmari7e an' resorce contention that appears in the

%&erage %cti&e !essions chart, and also show how mch wor( the database is

per+orming on behal+ o+ the sers or applications.

Correct ans"er(s)'

$. #iew the overall system status

8. -onitor cluster cache coherency statistics

1he Cluster %atabase Performance page provides a 0uick glimpse of the performance

statistics for a database. nterprise -anager accumulates data from each instance over

specified periods of time :known as collection-based data;. nterprise -anager also

provides current data from each instance :known as real-time data;.

tatistics are rolled up across all the instances in the cluster database. Using the links

ne)t to the charts, you can get more specific information and perform any of the following

tasks>

identify the causes of performance issues

decide whether resources need to be added or redistributed

tune your !" plan and schema for better optimi(ation, and

resolve performance issues

1his a partial view of the Cluster %atabase Performance page. *ou access this page by

clicking the :erformancetab on the Cluster %atabase +ome page.

Graphic

This page displa's graphs that pro&ide in+ormation on Clster :ost #oad %&erage,

0lobal Cache 4loc( %ccess #atenc', and %&erage %cti&e !essions.

1he Cluster +ost "oad Average chart on the Cluster %atabase Performance page shows

potential problems that are outside the database. 1he chart shows ma)imum, average,


24/52

and minimum load values for available nodes in the cluster for the previous hour.

2f the load average is higher than the average of the total number of CPUs across all the

hosts in the cluster, then too many processes are waiting for CPU resources. !"

statements that are not tuned often cause high CPU usage.

Compare the load average values with the values displayed for CPU Used in the Average

Active essions chart. 2f the session5s value is low and the load average value is high,

something else on the host I other than your database I is consuming the CPU.

*ou can click any of the load value labels for the Cluster +ost "oad Average chart to view

more detailed information about that load value.

or e)ample I if you click the label Average, the +osts> "oad Average page appears,

displaying charts that depict the average host load for up to four nodes in the cluster.

*ou can select whether the data is displayed in a summary chart :combining the data for

each node in one display; or in tile charts :where the data for each node is displayed in its

own chart;. *ou can click Cstomi/eto change the number of tile charts displayed in

each row or the method of ordering the tile charts.

1he /lobal Cache 4lock Access "atency chart shows the latency for each different type

of data block re0uests I current and consistent'read or C& blocks. 1hat is the elapsed

time it takes to locate and transfer consistent'read and current blocks between the buffer

caches.

*ou can click metric for the Global Cache %loc8 Access Latencychart to view more

detailed information about that type of cached block.

2f the /lobal Cache 4lock Access "atency chart shows high latencies :high elapsed

times;, the cause can be any of the following>

a high number of re0uests caused by !" statements that are not tuned

a large number of processes in the 0ueue waiting for the CPU, or scheduling delays, or

slow, busy, or faulty interconnects

When you e)perience slow, busy, or faulty interconnects, check your network connection

for dropped packets, retransmittals, or cyclic redundancy check or C&C errors.

Concurrent read and write activity on shared data in a cluster is a fre0uently occurring

activity. %epending on the service re0uirements, this activity does not usually cause

performance problems.

+owever, when global cache re0uests cause a performance problem, optimi(ing !"


25/52

plans and the schema to improve the rate at which data blocks are located in the local

buffer cache, and minimi(ing 2=O is a successful strategy for performance tuning.

2f the latency for consistent'read and current block re0uests reaches $B milliseconds, the

first step in resolving the problem should be to go to the Cluster Cache Coherency page

for more detailed information.

1he Average Active essions chart in the Cluster %atabase Performance page shows

potential problems inside the database.

Categories I called wait classes I show how much of the database is using a resource,

such as CPU or disk 2=O. Comparing CPU time to wait time helps determine how much of

the response time is consumed with useful work rather than waiting for resources that are

potentially held by other processes.

At the cluster database level, this chart shows the aggregate wait class statistics across

all the instances.

1he steps to be followed for a more detailed analysis are as follows>

click the clipboar! icon at the bottom of the chart to view the A%%- analysis for the database for

that time period

The clipboard icon is a chec( mar( present in the %&erage %cti&e !essions graph.

click the wait class legends beside the Average Active essions chart I to view instance'level

information stored in Active essions 4y 2nstance pages

There are 2 legends next to the %&erage %cti&e !essions chart < 5ther, Clster, "eeing,

6etwor(, %dministrati&e, Con+igration, Commit, %pplication, Concrrenc', !'stem IE5, User IE5,

!chedler, and CPU Used.

use the Wait Class action list on the Active essions 4y 2nstance page to view the different wait

classes

use the Cstomi/ebutton I and select the instances that are displayed because Active essions

4y 2nstance pages show the service times for up to four instances, and

use tile charts or combine the data into a single summary chart to view the data for the instances

separately

1he 1hroughput chart on the Performance page monitors the usage of various database

resources. 4y clicking the 3hroghpttab at the top of this chart, you can view the

%atabase 1hroughput chart.

Compare the peaks on the Average Active essions chart with those on the %atabase

1hroughput charts. 2f internal contention is high and throughput is low, consider tuning the

database.


26/52

1he %atabase 1hroughput charts summari(e any resource contention that appears in the

Average Active essions chart, and also show how much work the database is

performing on behalf of the users or applications. 1he Per econd view shows the

number of transactions compared to the number of logons, and :not shown here; the

amount of physical reads compared to the redo si(e per second.

1he Per 1ransaction view shows the amount of physical reads compared to the redo si(e

per transaction. "ogons is the number of users that are logged on to the database.

*ou can also obtain information at the instance level by clicking one of the legends to the

right of the charts to access the %atabase 1hroughput by 2nstance page. 1his page

shows the breakdown of the aggregated %atabase 1hroughput chart for up to four

instances. *ou can select the instances that are displayed.

*ou can drill down further on the %atabase 1hroughput by 2nstance page to see the

sessions of an instance that is consuming the greatest resources. Click an instance name

legend Gust under the chart to go to the 1op essions subpage of the 1op Consumers

page for that instance.

1he 1hroughput chart on the Performance page monitors the usage of various database

resources. 4y clicking the Instancestab at the top of this chart, you can view the Active

essions by 2nstance chart, which summari(es any resource contention that appears in

the Average Active essions chart. *ou can thus 0uickly determine how much of the

database work is being performed on each instance.

*ou can also obtain information at the instance level by clicking one of the legends to the

right of the chart to access the 1op essions page, where you can view real'time data

showing the sessions that consume the greatest system resources. 2n the graph, the

ora4instance after ?>8B P- consistently shows more active sessions than the ora1

instance.

. Other RAC !atabase an! performance pages

1he steps to access the Cluster Cache Coherency page are as follows>

click the :erformancetab on the Cluster %atabase +ome page

click Clster Cache Coherencyin the Additional -onitoring "inks section, and

click either of the legends of the /lobal Cache 4lock Access "atency chart

There are two legends next to the 0lobal Cache 4loc( %ccess #atenc' chart < %&erage Crrent

4loc( $ecei&e Time and %&erage C$ 4loc( $ecei&e Time.

1he Cluster Cache Coherency page contains summary charts for cache coherency

metrics for the cluster.


27/52

/lobal Cache 4lock Access "atency shows the total elapsed time, or latency, for a block

re0uest. Click one of the legends to the right of the chart to view the average time it takes

to receive data blocks for each block type :current or C&; by instance.

On the Average Current 4lock &eceive 1ime 4y 2nstance page, you can click an instance

legend under the chart to go to the 4lock 1ransfer for "ocal 2nstance page, where you can

identify which block classes :such as undo blocks, data blocks, and so on; are subGect to

intense global cache activity. 1his page displays the block classes that are being

transferred and the instances that are transferring most of the blocks.

Cache transfer indicates how many current and C& blocks for each block class were

received from remote instances, including how many transfers incurred a delay :busy; or

an une)pected longer delay :congested.;

1he /lobal Cache 4lock 1ransfer &ate section shows the total aggregated number of

blocks received by all instances in the cluster by way of an interconnect. Click one of the

legends to the right of the chart to go to the /C Current 4locks &eceived 4y 2nstance

page for that type of block.

Click an instance legend under the chart to go to the egment tatistics by 2nstance page

I where you can see which segments are causing cache contention.

1he /lobal Cache 4lock 1ransfers and Physical &eads :vs. "ogical &eads; section

shows the percentage of logical read operations that retrieved data from the buffer cache

of other instances by way of %irect -emory Access and from disk. 2t is essentially a

profile of how much work is performed in the local buffer cache rather than the portion of

remote references and physical reads :which both have higher latencies.;

Click one of the legends to the right of the chart to go to the J/lobal Cache 4lock

1ransfers and Physical &eads :vs. "ogical &eads; and KPhysical &eads vs. "ogical

&eads 4y 2nstanceJ pages. rom there, you can click an instance legend under the chart

to go to the Jegment tatistics by 2nstanceJ page, where you can see which segments

are causing cache contention.

&estion

Which summary chart on the Cluster Cache Coherency page shows the

percentage of logical read operations that retrieved data from the buffer cache of

other instances by way of %irect -emory Access and from disk7

Options'

$. /lobal Cache 4lock 1ransfer &ate

8. /lobal Cache 4lock Access "atency

6. /lobal Cache 4lock 1ransfers and Physical &eads


28/52

Ans"er

Option 1:Incorrect. This smmar' chart shows the total aggregated nmber o+

bloc(s recei&ed b' all instances in the clster b' wa' o+ an interconnect.

Option 2:Incorrect. This smmar' chart shows the total elapsed time, or latenc',

+or a bloc( reqest.

Option 3:Correct. This smmar' chart shows the percentage o+ logical read

operations that retrie&ed data +rom the b++er cache o+ other instances b' wa' o+

1irect /emor' %ccess and +rom dis(. It is essentiall' a pro+ile o+ how mch wor(

is per+ormed in the local b++er cache rather than the portion o+ remote re+erences

and ph'sical reads. 4oth o+ which ha&e higher latencies.

Correct ans"er(s)'

6. /lobal Cache 4lock 1ransfers and Physical &eads

1he Cluster 2nterconnects page is useful for monitoring the interconnect interfaces,

determining configuration issues, and identifying transfer rate'related issues, including

e)cess traffic. 1his page helps determine the load added by individual instances and

databases on the interconnect.

ometimes you can immediately identify interconnect delays that are due to applications

outside Oracle.

Graphic

Interconnects is one o+ the +i&e tabs o+ the Clster page. 5ther tabs are :ome,

Per+ormance, Targets, and Topolog'.

*ou can use the Cluster 2nterconnects page to perform the following tasks>

view all interfaces that are configured across the cluster

view statistics for the interfaces, such as absolute transfer rates and errors

determine the type of interfaces, such as private or public

determine whether the instance is using a public or private network

determine which database instance is currently using which interface, and

determine how much the instance is contributing to the transfer rate on the interface


29/52

1he Private 2nterconnect 1ransfer &ate :-4=ec; value shows a global view of the private

interconnect traffic, which is the estimated traffic on all the private networks in the cluster.

1he traffic is calculated as the summary of the input rate of all private interfaces that are

known to the cluster.

On the Cluster 2nterconnects page, you can access the +ardware %etails page, where

you obtain more information about all the network interfaces defined on each node of

your cluster.

imilarly, you can access the 1ransfer &ate metric page, which collects the internode

communication traffic of a cluster database instance. 1he critical and warning thresholds

of this metric are not set by default. *ou can set them according to the speed of your

cluster interconnects.

#ote

Yo can qer' the V$CLUSTER_INTERCONNECTS &iew +or in+ormation abot thepri&ate interconnect.

Use the %atabase "ocks page to determine if multiple instances are holding locks for the

same obGect. 1he page shows user locks, all database locks, or locks that are blocking

other users or applications. *ou can use this information to stop a session that is

unnecessarily locking an obGect.

1o access the %atabase "ocks page, you perform the following steps>

select :erformanceon the Cluster %atabase +ome page, and

Per+ormance is one o+ the eight a&ailable tabs on the Clster 1atabase :ome page.

click *atabase Loc8sin the Additional -onitoring "inks section at the bottom of the Performance

subpage

2. Snapshots of A0R in RAC

2n &AC environments, each AW& snapshot captures data from all active instances within

the cluster. 1he data for each snapshot set that is captured for all active instances is from

roughly the same point in time. 2n addition, the data for each instance is stored separately

and is identified with an instance identifier.

or e)ample, the "#((er_"#)6_5atstatistic shows the number of buffer waits on

each instance. 1he AW& does not store data that is aggregated from across the entire

cluster. 1hat is, the data is stored for each individual instance.

Graphic


30/52

The In-memor' statistics is connected to %W$ tables in !Y!%UX.

1he statistics snapshots generated by the AW& can be evaluated by producing reports

displaying summary data such as load and cluster profiles based on regular statistics and

wait events gathered on each instance.

1he AW& functions in a similar way as tatspack 1he difference is that the AW&

automatically collects and maintains performance statistics for problem detection and

self'tuning purposes. Unlike in tatspack, in the AW& there is only one )+a,)*ot_%

per snapshot across instances.

1he &AC'related statistics in an AW& report are organi(ed in different sections. A &AC

statistics section appears after the 1op 1imed vents containing

nmber of instances

1he number of instances open at the time of the begin snapshot and the end snapshot to

indicate instances Goined or left between the two snapshots.

Global Cache Loa! :rofile

/lobal Cache "oad Profile lists the number of blocks and messages that are sent and

received, as well as the number of fusion writes.

Global Cache Efficiency :ercentages

/lobal Cache fficiency Percentages indicates the percentage of buffer divided into

buffers received from the disk, local cache, and remote caches. 2deally, the percentage of

disk buffer access should be close to (ero.

GCS an! GES 0or8loa! Characteristics an!

/C and / Workload Characteristics give you an overview of the more important

numbers first. 4ecause the global en0ueue convert statistics have been consolidated with

the global en0ueue get statistics, the report prints only the average global en0ueue get

time. 1he round'trip times for C& and current block transfers follow, as well as the

individual sender'side statistics for C& and current blocks. 1he average log flush times are

computed by dividing the total log flush time by the number of actual log flushes. Also, the

report prints the percentage of blocks served that actually incurred a log flush.

GCS an! GES 5essaging Statistics

2n /C and / -essaging tatistics, the most important statistic is the average

message sent 0ueue time on ks)p, which gives a good indicator of how well the 2PCworks. Average numbers should be less than $ ms.

Additional &AC statistics are organi(ed in these sections>

the /lobal n0ueue tatistics section contains data e)tracted from V$GES_STATISTICS

the /lobal C& erved tats section contains data from V$CR_BLOC:_SERVER


31/52

the /lobal CU&&31 erved tats section contains data from V$CURRENT_BLOC:_SERVER,

and

the /lobal Cache 1ransfer tats section contains data from V$INSTANCE_CAC0E_TRANSFER

1he egment tatistics section also includes the /C 4uffer 4usy Waits, C& 4locks&eceived, and

CU& 4locks &eceived information for relevant segments.

&estion

Which &AC'related statistic section in an AW& report shows round'trip times for

C& and current block transfers7

Options'

$. /lobal Cache "oad Profile

8. /lobal Cache fficiency Percentages

6. /C and / -essaging tatistics

9. /C and / Workload Characteristics

Ans"er

Option 1:Incorrect. This section essentiall' lists the nmber o+ bloc(s and

messages that are sent and recei&ed, as well as the nmber o+ +sion writes.

Option 2:Incorrect. This section indicates the percentage o+ b++er di&ided intob++ers recei&ed +rom the dis(, local cache, and remote caches. Ideall', the

percentage o+ dis( b++er access shold be close to 7ero.

Option 3:Incorrect. The most important statistic in this section is the a;era!e

&e))a!e)e+t3#e#et&eo+7)


32/52

9. /C and / Workload Characteristics

Smmary

Oracle nterprise -anager %atabase Control and /rid Control are cluster'aware and

help to manage your cluster database. Using the Cluster %atabase Performance page,

you can identify the causes of performance issues, decide whether resources need to be

added or redistributed, tune !" plans, and resolve performance issues. 1his page

provides various charts that provide information on database performance. or e)ample,

the Average Active essions chart shows potential problems inside the database and the

1hroughput chart displays the usage of various database resources.

1he Cluster Cache Coherency page contains summary charts for cache coherency

metrics for the cluster. 1he Cluster 2nterconnects page monitors the interconnect

interfaces, determines configuration issues, and identifies transfer rate'related issues.

1he Private 2nterconnect 1ransfer &ate value contains a global view of the private

interconnect traffic. 1he %atabase "ocks page helps determine if multiple instances areholding locks for the same obGect.

2n &AC environments, each AW& snapshot captures data from all active instances within

the cluster for problem detection and self'tuning purposes. &AC'related statistics are

organi(ed in different sections in the AW& report.

Atomatic *atabase *iagnostic 5onitor for RAC

Learning Objectives


recognize how to discover RAC performance problems using ADDM and nterprise Manager

recognize how to fi! performance problems in RAC

1. +in!ing RAC performance isses "ith A**5

Oracle %atabase $$goffers an e)tension to the set of functionality that increases the

database5s manageability by offering clusterwide analysis of performance.

A special mode of Automatic %atabase %iagnostic -onitor or A%%- analy(es a &AC

database cluster and reports on issues that affect the entire cluster as well as on those

that affect individual instances. 1his mode is called database A%%- :as opposed to

instance A%%-, which already e)isted with Oracle %atabase $Bg;.

%atabase A%%- for &AC is not simply a report of reports. &ather, it has independent

analysis that is appropriate for &AC.


33/52

#ote

The 1atabase %11/ report is generated on the %W$ snapshot coordinator.

2n Oracle %atabase $$g, you can create a period analysis mode for A%%- that analy(es

the throughput performance for an entire cluster. When the advisor runs in this mode, it is

called database %11/. *ou can run the advisor for a single instance, which is e0uivalent

to Oracle %atabase $BgA%%- and is now called instance%11/.

%atabase A%%- has access to AW& data generated by all instances, thereby making the

analysis of global resources more accurate. 4oth database and instance A%%- run on

continuous time periods that can contain instance startup and shutdown.

2n the case of database A%%-, there may be several instances that are shut down or

started during the analysis period. *ou must, however, maintain the same database

version throughout the entire time period.

%atabase A%%- runs automatically after each snapshot is taken. 1he automatic instance

A%%- runs are the same as in Oracle %atabase $Bg. *ou can also perform analysis on a

subset of instances in the cluster. 1his is calledpartial anal'sis %11/.

2=O capacity finding :the 2=O system is overused; is a global finding because it concerns a

global resource affecting multiple instances. A local finding concerns a local resource or

issue that affects a single instance. or e)ample, a CPU'bound instance results in a local

finding about the CPU.

Although A%%- can be used during application development to test changes to either the

application, the database system, or the hosting machines, database A%%- is targeted at%4As.

&estion

Which mode of Automatic %atabase %iagnostic -onitor or A%%- analy(es a &AC

database cluster and reports on issues that affect the entire cluster as well as on

those that affect individual instances7

Options'

$. 2nstance A%%-

8. %atabase A%%-

6. Partial analysis A%%-

Ans"er


34/52

Option 1:Incorrect. Yo can rn the ad&isor +or a single instance, which is

eqi&alent to 5racle 1atabase @g %11/ and is now called instance %11/.

Option 2:Correct. % special mode o+ %11/ anal'7es a $%C database clster

and reports on isses that a++ect the entire clster as well as on those that a++ect

indi&idal instances. This mode is called database %11/ 8as opposed to instance

%11/, which alread' existed with 5racle 1atabase @g9. 1atabase %11/ +or

$%C is not simpl' a report o+ reports. $ather, it has independent anal'sis that is

appropriate +or $%C. The 1atabase %11/ report is generated on the %W$

snapshot coordinator.

Option 3:Incorrect. Yo can per+orm anal'sis on a sbset o+ instances in the

clster. This is called partial anal'sis %11/.

Correct ans"er(s)'

8. %atabase A%%-

&estion

Which are true statements about the database A%%- mode7

Options'

$. %atabase A%%- became available in Oracle %atabase $Bg

8. %atabase A%%- runs automatically after each snapshot is taken

6. %atabase A%%- has access to AW& data generated by all instances

9. %atabase A%%- performs analysis on a subset of instances in the cluster

Ans"er

Option 1:Incorrect. In 5racle 1atabase g, 'o can create a period anal'sis

mode +or %11/ that anal'7es the throghpt per+ormance +or an entire clster.

When the ad&isor rns in this mode, it is called database %11/.

Option 2:Correct. 1atabase %11/ rns atomaticall' a+ter each snapshot is

ta(en. The atomatic instance %11/ rns are the same as in 5racle 1atabase

@g.

Option 3:Correct. 1atabase %11/ has access to %W$ data generated b' all

instances, thereb' ma(ing the anal'sis o+ global resorces more accrate. 4oth

database and instance %11/ rn on continos time periods that can contain

instance startp and shtdown. In the case o+ database %11/, there ma' be

se&eral instances that are sht down or started dring the anal'sis period. Yo


35/52

mst, howe&er, maintain the same database &ersion throghot the entire time

period.

Option 4:Incorrect. Yo can per+orm anal'sis on a sbset o+ instances in the

clster. This is called partial anal'sis %11/.

Correct ans"er(s)'

8. %atabase A%%- runs automatically after each snapshot is taken

6. %atabase A%%- has access to AW& data generated by all instances

A%%- diagnoses the following in &AC>

latency problems in interconnect

congestion :identifying top instances affecting the entire cluster;

contention :buffer busy, top obGects, and more;

top consumers of multiblock re0uests

lost blocks

information about interconnect devices :warns about using PU4"2C interfaces;, and

throughput of devices I how much of it is used by Oracle and for what purpose :/C, locks, and

P!;

1he data sources of &AC that A%%- diagnoses are as follows>

wait events :especially Cluster class and buffer busy;

A+

instance cache transfer data, and

interconnect statistics :throughput, usage by component, pings;

A%%- analy(es the effects of &AC for both the entire database :%A1A4A analysismode; and for each instance :231A3C analysis mode.;

Oracle %atabase $$gnterprise -anager displays the A%%- analysis on the Cluster

%atabase home page.

On the Automatic %atabase %iagnostic -onitor or A%%- page, the %atabase Activity

chart plots the database activity during the A%%- analysis period. %atabase activity types


36/52

are defined in the legend based on its corresponding color in the chart. ach icon below

the chart represents a different A%%- task, which in turn corresponds to a pair of

individual Oracle %atabase snapshots saved in the Workload &epository.

2n the A%%- Performance Analysis section, the A%%- findings are listed in descending

order, from highest impact to least impact. or each finding, the Affected 2nstances

column displays the number :m of n; of instances affected. %rilling down further on the

findings takes you to the Performance indings %etail page.

Graphic

The table in the %11/ Per+ormance %nal'sis section is di&ided into +or colmns

named Impact 8;9, inding, %++ected Instances, and 5ccrrences 8last 3 hrs9.

1he 2nformational indings section lists the areas that do not have a performance impact

and are for informational purpose only.

1he Affected 2nstances chart shows how much each instance is impacted by these

findings. 1he display indicates the percentage impact for each instance.

Graphic

The %++ected Instances chart in the In+ormational indings section contains three

colmns < 6ame, Impact 8;9, and !tats.

. +i9ing RAC performance problems

1his demonstration shows you how to discover performance problems in your &AC

environment. 2n this demonstration, you identify performance issues using nterprise

-anagement performance pages as well as A%%- and you fi) those issues in three

different steps. At each step, you generate the same workload to make sure you are

making progress in your resolution.

*ou first generate a workload that uses a bad &AC application design. Using %atabase

Control, and connected as user *, navigate to the Performance page of your Cluster

%atabase.

1he steps to navigate to the Performance page are as follows>

click the :erformancetab on the Cluster %atabase +ome page and

on the Cluster %atabase Performance page, make sure Real 3ime' 1; Secon! Refreshis

selected from the #iew %ata drop'down list

Use P"=!" to create a new AW& snapshot.


37/52

Co!e

6=>at ?*o&e?orale?+o%e+(o @ )e% -+ 11,>

=>at ?*o&e?orale?+o%e+(o @ )e% -+ 44,>

DBNAME=>,) -e( @ !re, %"5_RDB @ !re, -; !re, @ !re, -;allo#t1 @ a57 ,r+t $ @ )e% )?1?? @ )e%

)?ora_%"5_??>

I1NAME=$DBNAMEH1H

I4NAME=$DBNAMEH4H

e

on the Performance page, click the *atabase Loc8slink in the Additional -onitoring "inks

section of the page

on the %atabase "ocks page, make sure that %loc8ing Loc8sis selected from the #iew drop'

down list, and

The 1atabase #oc(s page has a table with se&eral colmns. !ome o+ the colmns inclde !elect,

Username, !essions 4loc(ed, Instance 6ame, !ession I1, !erial 6mber, Process I1, !"# :ash

=ale, #oc( T'pe, /ode :eld, /ode $eqested, 5bBect T'pe, 5bBect 5wner, 5bBect 6ame, and

$5WI1. %ll colmns except the Username colmn are blan(. The Username colmn contains the

text < 6o loc(s o+ this t'pe crrentl' exist.

in this case no blocking locks currently e)ist in the database

While the scripts are still e)ecuting, look at the Average Active essions graph. 1hen drill

down to the Cluster wait class for the first node. *ou can see the following>

Graphic

The %&erage %cti&e !essions graph has its Y-axis contains &ales sch as @.@,

@.3, and @.F and its X-axis contains &ales sch as 3, @, and ).


38/52

"ook at the Average Active essions graph. 1hen drill down to the Other wait class.

Click the Clster *atabase locator link at the top of the page to return to the Cluster %atabase

Performance page.

rom there you can now see the Average Active essions graph. -ake sure that the #iew %atafield is set to Real 3ime' 1; Secon! Refresh.

Using the 1hroughput tabbed page graph underneath the Average Active essions graph, you

can see the transaction rate per second.

2n the Average Active essions graph, click the Otherlink on the right. 1his takes you to the

Active essions 4y 2nstance> Other page.

On the Active essions 4y 2nstance> Other page, you can see the number of active sessions for

the Other wait class.

The %cti&e !essions 4' Instance 5ther page has the !mmar' Chart graph that is similar to the

%&erage %cti&e !essions graph.

After the workload finishes, use P"=!" to create a new AW& snapshot.

Co!e

6=>at ?*o&e?orale?+o%e+(o @ )e% -+ 11,>

=>at ?*o&e?orale?+o%e+(o @ )e% -+ 44,>

DBNAME=>,) -e( @ !re, %"5_RDB @ !re, -; !re, @ !re, -;

allo#t1 @ a57 ,r+t $ @ )e% )?1?? @ )e%)?ora_%"5_??>

I1NAME=$DBNAMEH1H

I4NAME=$DBNAMEH4H

e


39/52

On the Cluster %atabase +ome page, click the A!visor Centrallink.

The %d&isor Central lin( is located in the $elated #in(s section o+ the page.

On the Advisor Central page, make sure that the Advisory 1ype field is set to All 3ypes, and that

the Advisor &uns field is set to Last Rn. Click Go.

The %d&isor Central page also contains the Tas( 6ame and !tats +ields.

2n the &esults table, select the latest A**5run corresponding to 2nstance All. 1hen click


40/52

DBNAME=>,) -e( @ !re, %"5_RDB @ !re, -; !re, @ !re, -;

allo#t1 @ a57 ,r+t $ @ )e% )?1?? @ )e%

)?ora_%"5_??>

I1NAME=$DBNAMEH1H

I4NAME=$DBNAMEH4H

e

"ook at the Average Active essions graph. %rill down to the System I-Owait class to view

ystem 2=O wait class details.

Click the Clster *atabaselocator link at the top of the page to return to the Cluster %atabase

Performance page.

On the 2=O tabbed page underneath the Average Active essions graph, you can see graphs

based on 2=O functions.

This example shows the IE5 /egab'tes per !econd b' IE5 nction graph.

Click the LG0Rlink of the 2=O &e0uests per econd by 2=O unction graph to see specific 2=O

&e0uests per econd 4y 2nstance or 2=O unction> "/W& graphs.

Click the IOdrop down list and select %ffer Cache Rea!sto view the graphs specific to

4uffer Cache &eads on the 2=O -egabytes per econd 4y 2nstance or 2=O unction>

4uffer Cache &eads page.

*ou can drill down on any of the functions for the 2=O -egabytes per econd by 2=O

unction and 2=O &e0uests per econd by 2=O unction pages, which are e)amples of Gust

some of the drill down functionality available in the nterprise -anager performance

pages.

After the workload finishes, use P"=!" to create a new AW& snapshot.


41/52

Co!e

6=>at ?*o&e?orale?+o%e+(o @ )e% -+ 11,>

=>at ?*o&e?orale?+o%e+(o @ )e% -+ 44,>

DBNAME=>,) -e( @ !re, %"5_RDB @ !re, -; !re, @ !re, -;allo#t1 @ a57 ,r+t $ @ )e% )?1?? @ )e%

)?ora_%"5_??>

I1NAME=$DBNAMEH1H

I4NAME=$DBNAMEH4H

e

On the Cluster %atabase +ome page, click the A!visor Centrallink.

The %d&isor Central lin( is located in the $elated #in(s section o+ the Clster 1atabase :ome

page.

On the Advisor Central page, make sure that the Advisory 1ype field is set to All 3ypesand that

the Advisor &uns field is set to Last Rn. Click Go.

The %d&isor' T'pe and %d&isor $ns +ields are located in the !earch section o+ the %d&isor Central

page.

2n the &esults table, select the latest A**5 run corresponding to 2nstance All. 1hen click


42/52

%atabase A%%- for &AC reports on issues that

affect the entire cluster and individual instances. When an advisor is run for a single

instance, it is called instance A%%-. %atabase A%%- runs automatically when taking

AW& snapshots. When analysis is performed on a subset of instances in the cluster, it is

called partial analysis A%%-. A%%- diagnoses data sources for &AC, which includes

Wait events and A+.

Performance problems in a &AC environment can be manually discovered by using the

nterprise -anager performance pages as well as A%%-.

Atomatic *atabase *iagnostic 5onitor for RAC

Learning Objectives


recognize how to discover RAC performance problems usi

Oracle Database 11g RAC Performance Tuning

Documents

Transcript of Oracle Database 11g RAC Performance Tuning