Microsoft PowerPoint - Managing Performance for RAC

Managing Performance for Oracle Real Application

Clustering (RAC)

Presented Wednesday, October 27, 2004

Performance Management of Oracle Real Application Clusters (RAC)

Agenda

• RAC – Why, Who, What• RAC Performance Tuning

• What’s Not on the Agenda…– How to Set up your RAC environment– Cluster Guard, Failover, Recovery


Why Real Application Clusers (RAC)

Why RAC? (Or Why clusters?)• Availability

– Redundency for critical applications and data

• Performance– Access / Processing Intensive Applications


A Brief History of Oracle Clusters

• Oracle Parallel Server (OPS)– Not well received– An initial “solution” for data warehouses

• Real Application Clusters (RAC)– OLTP +– Business Intelligence (OLAP)– Gaining Acceptance– The key to Oracle Grid Initiative


RAC Adoption


What is a RAC System?

• Many instances of Oracle running on many nodes• …and sharing a single physical database• All instances having common data and control files• Each instance has individual log files and undo

segments• All instances can simultaneously execute transactions

against the single physical database• Cache synchronization using cluster interconnect


Basic Challenges of RAC Management

More of everythingMore InstancesMore Machines

More Complexity

More of everything except Experience?


RAC in Pictures…

Users UsersOracle Server

Node 2Node 1Cluster

InterconnectInstance 1Instance 1 Instance 2Instance 2


RAC in Pictures (2)

Data files andControl files

InstanceA

SMON

DBW R

PMON

CKPT

LGW R

ARCH

SystemMonitorProcess

DatabaseW riter Process

ProcessMonitorProcess

CheckpointProcess

Log writerProcess

ArchiverProcess

DatabaseBufferCache

LibraryCache

DataDictionary

Cache Redo LogBuffer

SGA

Shared Pool

Java Pool Large Pool Miscellaneous

PGAPrivate

SQLArea

Cursors

SQLArea

InstanceB

SMON

DBW R

PMON

CKPTLGW R

ARCH

SystemMonitorProcess

DatabaseWriter

Process

ProcessMonitorProcess

CheckpointProcess

Log writerProcess

ArchiverProcess

DatabaseBufferCache

LibraryCache

DataDictionary

Cache Redo LogBuffer

SGA

Shared Pool

Java Pool Large Pool Miscellaneous

PGAPrivate

SQLArea

Cursors

SQLArea

Redo LogFiles Redo Log

Files


RAC in Pictures (3)

ora--db2ora-db1

InstanceRAC1

ClusterManager

Shared Disk Driver

Comm.Layer

IPCInstance

RAC2

ClusterManager

Shared Disk Driver

Comm.Layer

IPC

Cache Fusion

Cluster Interconnect(Heartbeat, Membership)

Shared DiskSubsystem

Listeners ListenersMonitors Monitors


RAC in Pictures (4)

SGASGA SGASGA

LGWRLGWR DBWRDBWR GCSGCS GCSGCS DBWRDBWR LGWRLGWR

Cache Fusion

Redologs Database Files

Shared storage subsystem

Redologs


Cluster Manager (CM)

• Is a distributed kernel component that monitors whether cluster members can communicate with each other

• Enforces rules of cluster membership• Forms a cluster, adds members to a cluster and removes members

from a cluster• Tracks which members in a cluster are active• Maintains a cluster membership list that is consistent on all cluster

members• Provides timely notification of membership changes• Detects and handles possible cluster partitions

Oracle World 2003, Paris | Performance Tuning RAC | Murali Vallath | Slide: 12


Oracle Cache Fusion Management

•Global Cache Service (GCS)The process that implements Cache Fusion. It maintains the block mode for blocks in the global role. It is responsible for block transfers between instances. The Global Cache Service employs various background processes such as the Global Cache Service Processes (LMSn) and Global Enqueue Service Daemon (LMD).

•Global Cache Service Processes (LMSn)The processes that handle remote Global Cache Service messages. Real Application Clusters provides for up to 10 Global Cache Service Processes. The number of LMSn varies depending on the amount of messaging traffic amount the nodes in the cluster. The LMSn handle the acquisition interrupt and blocking interrupt requests from an instance for Global Cache Service resources. For cross-instance consistent read requests, LMSn creates a consistent read version of the block and sends it to the requesting instance. LMSn also controls the flow of messages to and from remote instances.



• Global Enqueue Service (GES)This service coordinates local and global enqueues.

• Global Enqueue Service Daemon (LMD)The resource agent process that manages Global Enqueue Service (GES)

resource requests. The LMD process also handles deadlock detection GES requests. Remote resource requests are requests originating from another instance.

• Global Enqueue Service Monitor (LMON)The process that monitors the entire cluster to manage global resources.

LMON manages instance and process expirations and the associatedrecovery for the GCS and GES . In particular, LMON handles the part of recovery associated with global resources.



• Global Resource DirectoryThe data structures associated with global resources. It is distributed

across all instances in a cluster.

• Global Services Daemon (GSD)A component that receives requests from SRVCTL to execute

administrative job tasks, such as startup or shutdown. The command is executed locally on each node, and the results are sent back to SRVCTL.


Tuning RAC and its Applications

• Tuning a RAC based Application• Tuning the RAC Environment

– Problem Diagnostics– Tuning the single instance– Tuning the cluster


Application Tuning: General

• Application Tuning is the key to most performance issues

• Testing prior to deployment:– Tune the individual SQL statements– Unit test in as representative environment as

possible


Coding Acts for Performance - 1

• Use Bind Variables– Hard coded values rarely advocated anywhere– Reduced parsing because the forms are shared– Know your data however…– DBAs – how to capture the bind variable at run

time?


Coding Acts for Performance - 2

• Parallel Queries CAN be helpful– Know your data! Know your environment! Know

your application!– For queries engaging a full table scan– Best for those complex “What if” types of

Warehousing queries– OPQ is a team effort, the DBA has to enable the

DB to take advantage– Use on a per statement basis via hints– Not an excuse to create too many joins


Oracle Parallel Hints

• How to force parallelism– Use the parallel hint to use multiple CPUs on

multiple instances– SELECT /*+ PARALLEL(employees, 2,4) */ COUNT(*)

FROM employees – Here we have a 2 Degree of Parallelism on 4

different machines within our cluster– You can force parallelism for specific tables or a

session, etc.• REMEMBER: Oracle will default to force a query

to run on a single instance


Application Driving the Architecture…

• A Decision Support application can get by with Gigabit ethernets usually

• The more parallel queries you have means more cross node traffic– Infiniband, fibre channel, high throughput

ethernet


Tuning RAC: The Instance(s)

• Tuning the Single Instance– Instance Tuning– Database Tuning– O/S Tuning


Tools & Utilities

• Explain (Execution Plan)

• STATSPACK

• Oracle Enterprise Manager (OEM)

• Oracle’s Wait Interface

• Oracle’s Event Interface

• Quest Tools– Database Analysis– Spotlight on Oracle– Performance Analysis ( Version 5.0)

Oracle Real Application Cluster | Murali Vallath | Slide: 23


Execution Plan

• Autotrace– SQL> SET AUTOTRACE TRACEONLY;– SQL> SET TIMING ON;

• SQL TRACE– SQL_TRACE=TRUE

• TKPROF– Usage: tkprof tracefile outputfile [explain= ][table= ][print= ] [insert= ][sys= ][sort= ]

© Summersky Enterprises LLC | Oracle Real Application Cluster |Murali Vallath | Slide: 24


Instance Tuning: Memory Management

• Memory Tuning– Each instance working with its own SGA / PGA

• Shared / Synchronized Pool Sections of the SGA– Library Cache, Dictionary Cache, Buffers

• PGA remains dedicated to each instance

– Important to tune according to your application / RAC usage in mind

– Important to tune the balance of SGA / PGA sizing holistically


Instance Tuning

• TIMED_STATISTICS

• TIMED_OS_STATISTICS

• STATISTICS_LEVEL



Oracle’s Event Interface

• Event 10046– Level 1 - Basic SQL TRACE– Level 4 – Collect Bind Variables – Level 8 - Collect Wait Events– Level 12 – Provides all the above levels



Oracle’s Wait Interface

• GV$SYSTEM_EVENT• GV$SESSION_EVENT• GV$SESSION_WAIT• GV$EVENT_NAME• GV$SYSSTAT• GV$SESSTAT



Consistent Gets

• Physical I/O (PIO)

• Logical I/O (LIO)– buffer gets– consistent gets– db-block gets– buffer pinned count



Important Parameters

• CURSOR_SHARING

• SESSION_CACHED_CURSORS



Tuning Parallel Operators

• PARALLEL_MIN_SERVERS

• PARALLEL_MAX_SERVERS

• PARALLEL_AUTOMATIC_TUNING

• PARALLEL_THREADS_PER_CPU

| Oracle Real Application Cluster |Murali Vallath | Slide: 31


Instance Tuning

• SHARED_POOL• DB_CACHE_SIZE• DB_CACHE_ADVICE• DB_KEEP_CACHE_SIZE• DB_RECYCLE_CACHE_SIZE• PGA_AGGREGATE_TARGET• SGA_MAX_SIZE



Instance Tuning

• DB_BLOCK_SIZE

• DB_FILE_MULTIBLOCK_READ_COUNT

Oracle World 2003, Paris | Performance Tuning RAC | Murali Vallath | Slide: 33


Database Tuning

• V$SEGEMENT_STATISTICS

• V$SEGSTAT

• V$RESOURCE_LIMIT

You can also get segment level statistics from STATSPAK with a level 7 snapshot.

© Summersky Enterprises LLC | Oracle Real Application Cluster || Murali Vallath | Slide: 34


Oracle Wait Interface

• buffer busy waits– This event occurs when more than one process is trying to

access the same buffer in the buffer cache• db file sequential read

– This wait indicates that the query requests a sequential access on the table to retrieve rows and is waiting for the read to complete

• db file scattered read

• log file Sync– This wait is encountered when a user session commits (or

rollbacks), and the session’s redo information must be flushed to the redo log file by the LGWR.


Tuning the Cluster

Cluster Level Tuning

Oracle Real Application Cluster | Murali Vallath | Slide: 36Oracle Real Application Cluster || Murali Vallath | Slide: 36


Tuning the Cluster Interconnect

• ORADEBUG

• SQL TRACE

• STATSPACK– Level 5 - SQL Statements– Level 6 - SQL Plans and Plan Usage– Level 7 - Segment Level Statistics– Level 10 – Parent Child Latches



Cache Fusion - Interconnect – Why Tuning?

• When users on one instance request data held in cache on anotherinstance, the holding instance transfers the data across a cluster interconnect – avoids any writes and reads to disk

• The performance of the cluster interconnects is crucial to the performance of the RAC cluster and to the movement of cached data between instances– Performance is measured by determining the average time a

block takes to reach the requested node i.e., from the moment that a block was requested by an instance to the moment the requesting instance received the block.



SQL>ORADEBUG SETMYPIDSQL>ORADEBUG IPCSQL>EXIT

SSKGXPT 0x3671e28 flags SSKGXPT_READPENDING info for network 0

socket no 9 IP 142.23.153.1 UDP 59084sflags SSKGXPT_WRITESSKGXPT_UPinfo for network 1socket no 0 IP 0.0.0.0 UDP 0sflags SSKGXPT_DOWN

context timestamp 0x4402dno ports



Tuning the Cluster Interconnect:

SELECT INST_ID,NAME,FILE#,CLASS#,MAX(XNC)

FROM GV$CACHE_TRANSFERGROUP BY INST_ID,

NAME,FILE#,CLASS#

INST_ID NAME FILE# CLASS# MAX(XNC) ------ ---------- ---------- ---------- ----------

1 IDL_UB2$ 1 4 231 1 PK_USPRL 4 1 47 1 PK_COMP 4 1 39 1 COMPANY 171 1 2849




SELECT FILE#,BLOCK#,CLASS#,STATUS,XNC

FROM GV$CACHE_TRANSFERWHERE NAME = 'COMPANY'AND FILE# = 171

FILE# BLOCK# CLASS# STAT XNC ------ ---------------- ---------- ---------- ----------

171 898 1 XCUR 1321 171 1945 1 XCUR 27 171 1976 1 XCUR 19 171 2039 1 XCUR 849

Oracle Real Application Cluster |Murali Vallath | Slide: 41


COMP_ID NAME ----------------- ------------------------

3949 ARGUS CONSULTANTS 3952 CATAMARAN INC. 3957 PARIYAR BROTHERS INC.3961 DIGITAL BROADCASTING INC.

SELECT COMP_ID, COMP_NAME

FROM COMPANYWHERE DBMS_ROWID.ROWID_BLOCK_NUMBER(ROWID) = 898/

| Oracle Real Application Cluster Murali Vallath | Slide: 42



Wait Events – Session

Global Cache Service Waits– Global cache busy– Buffer busy due to global cache– Global cache CR request

SELECT INST_ID, EVENT, p1 FILE_NUMBER, p2 BLOCK_NUMBER, WAIT_TIME

FROM GV$SESSION_WAIT WHERE EVENT IN (‘global cache cr request',

'global cache busy','buffer busy due to global cache')

| Oracle Real Application Cluster |Murali Vallath | Slide: 43


Wait Events - Session

SID SQL_ADDRESS EVENT P1 P2 P3 WT FO---- ---------------- ------------------------ --- ------ ------------- ---- --

32 0000000442174720 db file sequential read 18 25804 1 0 NO32 00000004421739C0 db file sequential read 18 11786 1 0 NO32 0000000443AB2528 global cache null to x 18 25804 17917873152 0 NO36 0000000442178608 db file sequential read 18 24480 1 0 NO36 0000000442178608 global cache s to x 18 1164 17934759808 0 NO39 00000004421781C8 global cache open x 18 28712 17968379616 0 NO45 00000004421760F8 db file sequential read 18 16482 1 0 NO50 00000004448FA0C8 db file sequential read 18 11787 1 0 NO77 0000000443D27A68 global cache null to x 18 19651 16978156672 0 NO

SID SQL_ADDRESS EVENT P1 P2 P3 WT FO---- ---------------- ------------------------ --- ------ ------------- ---- --

90 00000004448FE370 db file sequential read 18 15422 1 0 NO99 0000000443D7DDD8 global cache null to x 18 11888 17045147456 0 NO

106 0000000444901728 global cache s to x 18 7053 17397452288 0 NO109 00000004448FE370 db file sequential read 18 17213 1 0 NO117 0000000444901728 global cache null to s 18 7053 17397452288 0 NO120 00000004448F9728 db file sequential read 18 11891 1 0 NO125 00000004448F8FA0 db file sequential read 18 11864 1 0 NO125 00000004448FE370 global cache cr request 18 23689 17028513088 0 NO



Global cache CR Request Latency

COLUMN "AVG CR BLOCK RECEIVE TIME(ms)" FORMAT 9999999.9SELECT b1.INST_ID,

b2.VALUE "GCS CR BLOCKS RECEIVED",b1.VALUE "GCS CR BLOCK RECEIVE TIME",

((b1.VALUE / b2.VALUE)*10) "AVG CR BLOCK RECEIVE TIME(ms)"FROM GV$SYSSTAT b1,

GV$SYSSTAT b2WHERE b1.NAME = 'global cache cr block receive time' AND b2.NAME = 'global cache cr blocks received' AND b1.INST_ID = b2.INST_ID;

INST_ID BLKS RECVD BLK RECV TIME AVG RECV TIME(ms)------- ---------- -------------- ----------------

1 2758 12394 443.782 1346 1457 10.8

2 rows selected.

HIGH



Global Cache Service Request Latency

1. Average time to build consistent read block– Global cache cr block time / global cache cr block served

2. Average time to wait for log flush– Global cache cr block flush time / global cache cr blocks

served3. Average time to send completed block

– Global cache cr block send time / global cache cr blocks served

4. Average LMS service time– Average latency (4 = 1 – 2 - 3)



Global cache cr request

Reasons for this wait event – Data Blocks Modified frequently on all instances– Cache miss on block requests– LMS cannot keep up with block requests– Latency on the interconnect– Full table scans _LM_DLMD_PROCS

Time waited for a CR version of a block and cannot find it in its local cache and hence has made a request against a remote instance for a block

Oracle Real Application Cluster Murali Vallath | Slide: 47


Analyze Global Enqueue Statistics

• Collect global statistical data by querying V$SYSSTAT• Calculate average global enqueue time• Calculate average global lock convert time• Identify types of resources causing problems• Further analyze with V$LIBRARYCACHE and V$ROWCAHCE views



Collect Global Enqueue Statistics

SELECT NAME,

VALUE,

FROM V$SYSSTAT

WHERE NAME LIKE '%global lock%';



Calculate Average Global Enqueue Time

Average global enqueue time, in milliseconds

10 * (global lock get time) /

(global lock sync gets +

global lock async gets)



Calculate Average Global Lock Convert Time

Average global lock convert time, in milliseconds

10 * (global lock convert time) /

(global lock sync converts +

global lock async converts)

If average is larger than 20 to 30 milliseconds indicates concernand requires further investigation



Further Analysis with

V$LIBRARYCACHE -SELECT NAMESPACE,

DLM_LOCK_REQUESTS,

DLM_PIN_REQUESTS,

DLM_PIN_RELEASES,

DLM_INVALIDATION_REQUESTS,

DLM_INVALIDATIONS

FROM V$LIBRARYCACHE;


V$ROWCACHE -

SELECT PARAMETER,

DLM_REQUESTS,

DLM_CONFLICTS,

DLM_RELEASES

FROM V$ROWCACHE;


Other Waits Events

• Global cache null to x and Global cache null to s– Block was transferred to another instance and requested again

• Global cache open x and Global cache open s– Initial access to a particular data block by an instance

• Buffer busy due to global cache– Counter to ‘buffer busy wait’ however occurs at the global cache

level



Additional Views to Monitor

• V$LOCK_ACTIVITY• V$CLASS_CACHE_TRANSFER• V$CACHE_TRANSFER• V$GES_CONVERT_LOCAL• V$GES_CONVERT_REMOTE• V$SYSTEM_EVENT



Monitoring Contention

• RACdiag.sql

• STATSPACK

• Oracle Enterprise Manager (OEM)

• Spotlight on Oracle

Metalink Note# 135714.1


Spotlight on Oracle


Database Analysis



RAC Interconnect CacheRAC Interconnect CacheTransfer LatencyTransfer Latency

• Doesn’t appear when non-RAC• Measures interconnect latency per node

Performance Analysis



Performance Analysis: RAC views in Quest Central 5.0


Breaking out Cluster views into per Instance Views


Intra-Instance Activity

Thanks for Listening!

www.quest.com

Microsoft PowerPoint - Managing Performance for RAC

Technology

Transcript of Microsoft PowerPoint - Managing Performance for RAC