Realworld Rac PerfRealworld Rac Perf
-
Upload
gauravkumarsinha -
Category
Documents
-
view
249 -
download
2
Transcript of Realworld Rac PerfRealworld Rac Perf
Real Life RAC Performance Real Life RAC Performance TuningTuning
Arup NandaArup NandaLead DBALead DBA
Starwood HotelsStarwood Hotels
© Arup Nanda© Arup Nanda
Who am IWho am IOracle DBA for 13 years and countingOracle DBA for 13 years and countingWorking on OPS from 1999Working on OPS from 1999Speak at conferences, write papers, booksSpeak at conferences, write papers, books
© Arup Nanda© Arup Nanda
Why This SessionWhy This Session I get emails like this:I get emails like this:
We are facing performance issues in RAC. We are facing performance issues in RAC. What should I do next?What should I do next?
Real Life AdviceReal Life AdviceCommon Issues (with Wait Events)Common Issues (with Wait Events)Dispelling MythsDispelling MythsFormulate a Plan of AttackFormulate a Plan of AttackReal Life Case StudyReal Life Case Study
proligence.com/downloads.htmlproligence.com/downloads.html
© Arup Nanda© Arup Nanda
Our RAC ImplementationOur RAC ImplementationOracle 10g RAC in March 2004Oracle 10g RAC in March 2004
Itanium Platform running HP/UXItanium Platform running HP/UXOracle 10.1.0.2Oracle 10.1.0.2
Result: Failed Result: Failed Second Attempt: Dec 2004 – 10.1.0.3Second Attempt: Dec 2004 – 10.1.0.3Result: Failed Again Result: Failed Again Third Attempt: March 2005 – 10.1.0.4Third Attempt: March 2005 – 10.1.0.4Result: Success! Result: Success!
© Arup Nanda© Arup Nanda
ChallengesChallengesTechnologyTechnology
Lone rangerLone rangerA lot of “mystery” and disconnected “facts”!A lot of “mystery” and disconnected “facts”!
PeoplePeopleBuilding a team that could not only deliver; but Building a team that could not only deliver; but
also sustain the delivered partsalso sustain the delivered partsEach day we learned something newEach day we learned something new
In today’s session: real performance In today’s session: real performance issues we faced and how we resolved issues we faced and how we resolved them, along with wait events.them, along with wait events.
© Arup Nanda© Arup Nanda
Why “RAC” Performance?Why “RAC” Performance?All tuning concepts in single instance All tuning concepts in single instance
applied to RAC as wellapplied to RAC as wellRAC has other complexitiesRAC has other complexities
More than 1 buffer cacheMore than 1 buffer cacheMultiple caches – library cache, row cacheMultiple caches – library cache, row cache InterconnectInterconnectPingingPingingGlobal LockingGlobal Locking
© Arup Nanda© Arup Nanda
Why RAC Perf TuningWhy RAC Perf TuningWe want to make sure we identify the right We want to make sure we identify the right
problem and go after it problem and go after it ……. not just . not just aa problem problem
© Arup Nanda© Arup Nanda
Switch
VIP
Service
Listener
Instance
ASM
Clusterware
Op Sys
VIP
Service
Listener
Instance
ASM
Clusterware
Op Sys
Public Interface
Cache
Cache Fusion
OCR Voting
SwitchInterconnect
Storage
Node1 Node2
Lock Manager
© Arup Nanda© Arup Nanda
Areas of Concern in RACAreas of Concern in RACMore than 1 buffer cacheMore than 1 buffer cacheMultiple caches – library cache, row cacheMultiple caches – library cache, row cache InterconnectInterconnectGlobal LockingGlobal Locking
© Arup Nanda© Arup Nanda
Cache IssuesCache IssuesTwo Caches, requires synchronizationTwo Caches, requires synchronizationWhat that means:What that means:
A changed block in one instance, when A changed block in one instance, when requested by another, should be sent across requested by another, should be sent across via a “bridge”via a “bridge”
This bridge is the InterconnectThis bridge is the Interconnect
© Arup Nanda© Arup Nanda
Interconnect PerformanceInterconnect Performance Interconnect must be on a private LANInterconnect must be on a private LANPort aggregation to increase throughputPort aggregation to increase throughput
APA on HPUXAPA on HPUX If using Gigabit over Ethernet, use Jumbo If using Gigabit over Ethernet, use Jumbo
FramesFrames
© Arup Nanda© Arup Nanda
Checking Interconnect UsedChecking Interconnect Used Identify the interconnect usedIdentify the interconnect used$ oifcfg getif$ oifcfg getiflan902 172.17.1.0 global lan902 172.17.1.0 global cluster_interconnectcluster_interconnect
lan901 10.28.188.0 global publiclan901 10.28.188.0 global public
Is lan902 the bonded interface? If not, Is lan902 the bonded interface? If not, then set itthen set it
$ oifcfg setif …$ oifcfg setif …
© Arup Nanda© Arup Nanda
Pop QuizPop Quiz If I have a very fast interconnect, I can If I have a very fast interconnect, I can
perform the same work in multiple node perform the same work in multiple node RAC as a single server with faster CPUs. RAC as a single server with faster CPUs. True/False?True/False?
Since cache fusion is now write-write, a Since cache fusion is now write-write, a fast interconnect will compensate for a fast interconnect will compensate for a slower IO subsystem. True/False?slower IO subsystem. True/False?
© Arup Nanda© Arup Nanda
Cache Coherence TimesCache Coherence TimesThe time is a sum of time for:The time is a sum of time for:
Finding the block in the cacheFinding the block in the cache Identifying the masterIdentifying the masterGet the block in the interconnectGet the block in the interconnectTransfer speed of the interconnectTransfer speed of the interconnectLatency of the interconnectLatency of the interconnectReceive the block by the remote instanceReceive the block by the remote instanceCreate the consistent image for the userCreate the consistent image for the user
CPU
CPU
Interconnect
© Arup Nanda© Arup Nanda
So it all boils down to:So it all boils down to: Block Access CostBlock Access Cost
more blocks -> more the timemore blocks -> more the time Parallel QueryParallel Query
Lock Management CostLock Management Cost More coordination -> more timeMore coordination -> more time Implicit Cache Checks – Sequence NumbersImplicit Cache Checks – Sequence Numbers
Interconnect CostInterconnect Cost LatencyLatency SpeedSpeed more data to transfer -> more the timemore data to transfer -> more the time
© Arup Nanda© Arup Nanda
Hard LessonsHard Lessons In RAC, problem symptoms may not In RAC, problem symptoms may not
indicate the correct problem!indicate the correct problem!Example:Example:
When the CPU is too busy to receive or send When the CPU is too busy to receive or send packets via UDP, the packets fails and the packets via UDP, the packets fails and the Clusterware thinks the node is down and Clusterware thinks the node is down and evicts it.evicts it.
© Arup Nanda© Arup Nanda
OS TroubleshootingOS TroubleshootingOS utilities to troubleshoot CPU issuesOS utilities to troubleshoot CPU issues
toptopglanceglance
OS Utilities to troubleshoot process OS Utilities to troubleshoot process issues:issues: trusstrussstracestracedbxdbxpstackpstack
© Arup Nanda© Arup Nanda
Reducing LatencyReducing Latency A factor of technologyA factor of technology TCP is the most latentTCP is the most latent UDP is better (over Ethernet)UDP is better (over Ethernet) Proprietary protocols are usually betterProprietary protocols are usually better
HyperFabric by HPHyperFabric by HP Reliable Datagram (RDP) Reliable Datagram (RDP) Direct Memory ChannelDirect Memory Channel
InfinibandInfiniband UDP over InfinibandUDP over Infiniband RDP over InfinibandRDP over Infiniband
© Arup Nanda© Arup Nanda
Start with AWRStart with AWR
© Arup Nanda© Arup Nanda
gc current|cr grant 2-waygc current|cr grant 2-way
Instance 1
Instance 2
Session
Database
LMS LGWR
Log Buffer
LMS
gc current block requestgc current grant 2-way
© Arup Nanda© Arup Nanda
gc current|cr block 2-waygc current|cr block 2-way
Instance 1
Instance 2
Session
Database
LMS LGWR
Log Buffer
LMS
gc current block request log file syncgc current block 2-way
© Arup Nanda© Arup Nanda
gc current|cr block 3-waygc current|cr block 3-way
Instance 1
Instance 2
Instance 3
Session
Requestor
Master
Holder
gc current block 3-way
© Arup Nanda© Arup Nanda
RAC related StatsRAC related Stats
© Arup Nanda© Arup Nanda
RAC Stats contd.RAC Stats contd.
© Arup Nanda© Arup Nanda
© Arup Nanda© Arup Nanda
Other GC Block WaitsOther GC Block Waitsgc current/cr block lostgc current/cr block lost
Lost blocks due to Interconnect or CPULost blocks due to Interconnect or CPUgc curent/cr block busygc curent/cr block busy
The consistent read request was delayed, The consistent read request was delayed, most likely an I/O bottleneckmost likely an I/O bottleneck
gc current/cr block congestedgc current/cr block congestedLong run queues and/or paging due to Long run queues and/or paging due to
memory deficiency.memory deficiency.
© Arup Nanda© Arup Nanda
Hung or Slow?Hung or Slow?Check Check V$SESSIONV$SESSION for for WAIT_TIMEWAIT_TIME
If 0, then it’s not waiting; it’s hungIf 0, then it’s not waiting; it’s hungWhen hung:When hung:
Take a systemstate dump from all nodesTake a systemstate dump from all nodesWait some timeWait some timeTake another systemstate dumpTake another systemstate dumpCheck change in values. If unchanged, then Check change in values. If unchanged, then
system is hungsystem is hung
© Arup Nanda© Arup Nanda
Chart a PlanChart a PlanRule out the obviousRule out the obviousStart with AWR ReportStart with AWR ReportStart with Top-5 WaitsStart with Top-5 WaitsSee if they have any significant waitsSee if they have any significant waits… … especially RAC relatedespecially RAC relatedGo on to RAC StatisticsGo on to RAC StatisticsBase your solution based on the wait Base your solution based on the wait
eventevent
© Arup Nanda© Arup Nanda
Rule out the obviousRule out the obvious Is interconnect private?Is interconnect private? Is interconnect on UDP?Is interconnect on UDP?Do you see high CPU?Do you see high CPU?Do you see a lot of IO bottleneck?Do you see a lot of IO bottleneck?How about memory?How about memory?Are the apps spread over evenly?Are the apps spread over evenly?Do you see lost blocks?Do you see lost blocks?
© Arup Nanda© Arup Nanda
Make Simple FixesMake Simple FixesStrongly consider RAID 0+1Strongly consider RAID 0+1Highest possible number of I/O pathsHighest possible number of I/O pathsUse fastest interconnect possibleUse fastest interconnect possibleUse private collision free domain for I/CUse private collision free domain for I/CCache and NOORDER sequencesCache and NOORDER sequences
© Arup Nanda© Arup Nanda
Enterprise ManagerEnterprise Manager
© Arup Nanda© Arup Nanda
Buffer BusyBuffer BusyCauseCause
Instance wants to bring something from disk Instance wants to bring something from disk to the buffer cacheto the buffer cache
Delay, due to space not availableDelay, due to space not availableDelay, b’coz the source buffer is not readyDelay, b’coz the source buffer is not readyDelay, I/O is slowDelay, I/O is slowDelay, b’coz redo log is being flushedDelay, b’coz redo log is being flushed
In summaryIn summaryLog buffer flush -> gc buffer busyLog buffer flush -> gc buffer busy
© Arup Nanda© Arup Nanda
Parallel QueryParallel QueryOne major issue in RAC is parallel query One major issue in RAC is parallel query
that goes across many nodesthat goes across many nodes
Instance 1 Instance 2QC
Slave Slave Slave Slave
ViaInterconnect
© Arup Nanda© Arup Nanda
Restricting PQRestricting PQDefine Instance GroupsDefine Instance Groups
Specify in init.oraSpecify in init.oraprodb1.instance_groups='pqgroup1'prodb1.instance_groups='pqgroup1'prodb2.instance_groups='pqgroup2' prodb2.instance_groups='pqgroup2'
Specify Instance Groups in SessionSpecify Instance Groups in SessionSQL> alter session set SQL> alter session set parallel_instance_group = parallel_instance_group = 'pqgroup1'; 'pqgroup1';
© Arup Nanda© Arup Nanda
Forcing PQ on both NodesForcing PQ on both NodesDefine a common Instance GroupDefine a common Instance Group
prodb1.instance_groups='pqgroup1‘prodb1.instance_groups='pqgroup1‘prodb1.instance_groups=‘pq2nodes'prodb1.instance_groups=‘pq2nodes'prodb2.instance_groups='pqgroup2'prodb2.instance_groups='pqgroup2'prodb2.instance_groups='pq2nodes' prodb2.instance_groups='pq2nodes'
Specify Instance Groups in SessionSpecify Instance Groups in SessionSQL> alter session set SQL> alter session set parallel_instance_group = parallel_instance_group = 'pq2nodes'; 'pq2nodes';
© Arup Nanda© Arup Nanda
Vital Cache Fusion ViewsVital Cache Fusion Viewsgv$cache_transfer:gv$cache_transfer: Monitor blocks Monitor blocks
transferred by objecttransferred by objectgv$class_cache_transfer:gv$class_cache_transfer: Monitor Monitor
block transfer by classblock transfer by classgv$file_cache_transfer:gv$file_cache_transfer: Monitor Monitor
the blocks transferred per file the blocks transferred per file gv$temp_cache_transfer:gv$temp_cache_transfer: Monitor Monitor
the transfer of temporary tablespace the transfer of temporary tablespace blocksblocks
© Arup Nanda© Arup Nanda
““Hot” TablesHot” Tables Tables, e.g. Rate PlansTables, e.g. Rate Plans
SmallSmall Compact blocksCompact blocks High updatesHigh updates High readsHigh reads
SymptomsSymptoms gc buffer busy waitsgc buffer busy waits
SolutionSolution Less rows per blockLess rows per block High PCTFREE, INITRANS, High PCTFREE, INITRANS, ALTER TABLE … MINIMIZE ALTER TABLE … MINIMIZE
RECORDS_PER_BLOCKRECORDS_PER_BLOCK
© Arup Nanda© Arup Nanda
Hot SequencesHot SequencesSymptoms:Symptoms:
High waits on Sequence Number latchHigh waits on Sequence Number latchHigh waits on SEQ$ tableHigh waits on SEQ$ table
Solution:Solution: Increase the cacheIncrease the cacheMake it NOORDERMake it NOORDER
Especially AUDSESS$ sequence in SYS, Especially AUDSESS$ sequence in SYS, used in Auditingused in Auditing
© Arup Nanda© Arup Nanda
Read Only? Say So.Read Only? Say So.Reading table data from other instances Reading table data from other instances
create “gc *” contentionscreate “gc *” contentionsSuggestion:Suggestion:
Move Read Only tables to a single tablespaceMove Read Only tables to a single tablespaceMake this tablespace Read OnlyMake this tablespace Read Only
SQL> alter tablespace ROD read SQL> alter tablespace ROD read only;only;
© Arup Nanda© Arup Nanda
PartitioningPartitioningPartitioning creates several segments for Partitioning creates several segments for
the same table (or index) the same table (or index) => more resources => more resources => less contention=> less contention
© Arup Nanda© Arup Nanda
Monotonically Increasing IndexMonotonically Increasing IndexProblem:Problem:
““Reservation ID”, a sequence generated keyReservation ID”, a sequence generated key Index is heavy on one sideIndex is heavy on one side
SymptomsSymptomsBuffer busy waitsBuffer busy waits Index block spiltsIndex block spilts
Solutions:Solutions:Reverse key indexesReverse key indexesHash partitioned index (even if the table is not Hash partitioned index (even if the table is not
partitioned) 10gR2partitioned) 10gR2
© Arup Nanda© Arup Nanda
Library CacheLibrary Cache In RAC, Library Cache is globalIn RAC, Library Cache is globalSo, parsing cost is worse than non-RACSo, parsing cost is worse than non-RACSolutions:Solutions:
Minimize table alters, drops, creates, Minimize table alters, drops, creates, truncatestruncates
Use PL/SQL stored programs, not unnamed Use PL/SQL stored programs, not unnamed blocksblocks
© Arup Nanda© Arup Nanda
Log FilesLog Files In 10g R2, the log files are in a single location:In 10g R2, the log files are in a single location: $CRS_HOME/log/<Host>/…$CRS_HOME/log/<Host>/…
racgracg crsdcrsd cssdcssd evmdevmd clientclient cssd/oclsmoncssd/oclsmon
$ORACLE_HOME/racg/dump$ORACLE_HOME/racg/dump
© Arup Nanda© Arup Nanda
Case StudyCase Study
© Arup Nanda© Arup Nanda
DiagnosisDiagnosisifconfig -aifconfig -a shows no congestion or shows no congestion or
dropped packetsdropped packetsTop shows 1% idle time on node 2Top shows 1% idle time on node 2Top processesTop processes
LMS and LMDLMS and LMDAnd, several Netbackup processesAnd, several Netbackup processes
© Arup Nanda© Arup Nanda
Further DiagnosisFurther Diagnosis SQL:SQL:
select * from v$instance_cache_transferselect * from v$instance_cache_transferwhere class = 'data block'where class = 'data block'and instance = 1;and instance = 1;
Output:Output: INSTANCE CLASS CR_BLOCK CR_BUSYINSTANCE CLASS CR_BLOCK CR_BUSY---------- ------------------ ---------- -------------------- ------------------ ---------- ----------CR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTEDCR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTED------------ ------------- ------------ ----------------------------- ------------- ------------ ----------------- 1 data block 162478682 50971491 data block 162478682 5097149 477721 347917908 2950144 16320267477721 347917908 2950144 16320267
After sometime:After sometime: INSTANCE CLASS CR_BLOCK CR_BUSYINSTANCE CLASS CR_BLOCK CR_BUSY---------- ------------------ ---------- -------------------- ------------------ ---------- ----------CR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTEDCR_CONGESTED CURRENT_BLOCK CURRENT_BUSY CURRENT_CONGESTED------------ ------------- ------------ ----------------------------- ------------- ------------ ----------------- 1 data block 162480580 50971851 data block 162480580 5097185 477722 347923719 2950376 16320269477722 347923719 2950376 16320269
See increases
© Arup Nanda© Arup Nanda
Diagnosis:Diagnosis:CPU starvation by LMS/D processes caused CPU starvation by LMS/D processes caused
GC waits.GC waits.Solution:Solution:
Killed the Netbackup processesKilled the Netbackup processesLMD and LMS got the CPULMD and LMS got the CPU
© Arup Nanda© Arup Nanda
Increasing Interconnect SpeedIncreasing Interconnect Speed Faster HardwareFaster Hardware
Gigabit Ethernet; not FastGigabit Ethernet; not Fast Infiniband, even if IP over IBInfiniband, even if IP over IB
NIC settingsNIC settings Duplex ModeDuplex Mode Highest Top Bit RateHighest Top Bit Rate
TCP SettingsTCP Settings Flow Control SettingsFlow Control Settings Network Interrupts for CPUNetwork Interrupts for CPU Socket Receive BufferSocket Receive Buffer
LAN PlanningLAN Planning Private LANsPrivate LANs Collision DomainsCollision Domains
© Arup Nanda© Arup Nanda
High Speed InterconnectsHigh Speed InterconnectsOracle will support RDS over InfinibandOracle will support RDS over Infinibandhttp://oss.oracle.com/projects/rds/http://oss.oracle.com/projects/rds/On 10 Gig Ethernet as wellOn 10 Gig Ethernet as well
© Arup Nanda© Arup Nanda
In summary: PlanningIn summary: PlanningAdequate CPU, Network, MemoryAdequate CPU, Network, MemorySequences – cache, noorderSequences – cache, noorderTablespaces read onlyTablespaces read onlyUn-compact small hot tablesUn-compact small hot tablesKeep undo and redo on fastest disksKeep undo and redo on fastest disksAvoid full table scans of large tablesAvoid full table scans of large tablesAvoid DDLs and unnamed PL/SQL blocksAvoid DDLs and unnamed PL/SQL blocks
© Arup Nanda© Arup Nanda
In summary: DiagnosisIn summary: DiagnosisStart with AWRStart with AWR Identify symptoms and assign causesIdentify symptoms and assign causesDon’t get fooled by “gc” waits as Don’t get fooled by “gc” waits as
interconnect issuesinterconnect issuesFind the correlation between “dropped” Find the correlation between “dropped”
packets in network, CPU issues from sar packets in network, CPU issues from sar and “gc buffer lost” in sysstat reports.and “gc buffer lost” in sysstat reports.
© Arup Nanda© Arup Nanda
Thank You!Thank You!Download from:Download from:
proligence.com/downloads.htmlproligence.com/downloads.html