RAC Performance Monitoring and Diagnosis using Oracle Enterprise Manager

Kai YuSenior System Engineer

Dell Oracle Solutions Engineering

• Kai Yu• Senior System Engineer, Dell Oracle

Solutions Engineering Lab– 15 years Oracle DBA and Solutions Engineering– Specialized in Oracle RAC, Oracle EBS and OVM– Oracle ACE Director– Oracle Technology article author and frequent

presenter at OOW and Collaborate conferences– IOUG Oracle RAC SIG President– Owner of IOUG RAC Tuning BootCamp

About Author

2

Agenda• Database Performance Monitoring & Diagnosis Tools• RAC Performance Monitoring and Diagnosis • Monitoring & Diagnosis with Enterprise Manager• Examples of RAC Performance Monitoring & Diagnosis • QA

Database Performance Monitoring & Diagnosis

• Database Performance Statistics Gathering:– Oracle generates cumulative performance statistics – Dynamic Performance Views: v$views, gv$views:– Automatic Workload Repository: AWR snapshot of statistics on

period of times.– Active Session history samples current state of all active sessions

• Database Performance tuning Tools.– Oracle Database 11g : clustered or single node database– Oracle Diagnostic Pack:

Automatic Workload Repository(AWR): AWR snapshot, AWR report Automatic Database Diagnostic Monitor (ADDM) Active Session History (ASH)

– Enterprise Manager: Database Control and Grid control

RAC Performance Monitoring and Diagnosis– Multiple Levels: clusterware, cluster database, Instances– All single node database performance statistics apply to RAC– RAC specific performance statistics: Cluster cache coherency, Global Cache Block Access Latency: Interconnect RAC Wait events: cluster waits:– Performance Monitoring & Diagnosis tools for RAC AWR captures data from all active instances of RAC ADDM presents data in cluster wide perspective ASH reports statistics for all active sessions of all active instances Enterprise manager is RAC aware

Monitoring & Diagnosis with Enterprise Manager

• Enterprise manager provides:– Web based integrated management solutions, preferred method– Integrated AWR, ADDM, ASH: execution and viewing results.– Monitoring RAC performance on Cluster, Database &Instances– Automatic Monitoring vs Real Time Monitoring– Performance monitoring and diagnosis: – Proactive performance monitoring and tuning vs Reactive tuning.– Monitoring performance over period of time vs diagnose short-

lived performance issue.

• Cluster Database Home: Cluster status & alerts, ADDM findings/analysis

Monitoring & Diagnosis with Enterprise Manager

Monitoring & Diagnosis with Enterprise Manager

• Cluster Database Performance Page:Host Load, Global Cache Block Access Latency, Average Active Sessions,throughput

• Throughput, I/O, etc and additional monitoring links

Monitoring & Diagnosis with Enterprise Manager

• Cluster Cache Coherency Page:

Monitoring & Diagnosis with Enterprise Manager

• Cluster Instance: Instance status & alerts, ADDM findings/analysis

Monitoring & Diagnosis with Enterprise Manager

• Cluster Database Instance Performance Page: real time & historical statistics & ADDM and ASH

Monitoring & Diagnosis with Enterprise Manager

• Cluster Home: Cluster status & alerts, health and workload the cluster

Monitoring & Diagnosis with Enterprise Manager

– Cluster Performance: CPU, Memory, Disk I/O of hosts

Monitoring & Diagnosis with Enterprise Manager

Interconnect performancesMonitoring & Diagnosis with Enterprise Manager

Example of RAC Performance Monitoring & Diagnosis

• Environment– 8 node RAC database testdb1– Workload: run PL/SQL script update.sql. Loop for 200000 times:

select rows of customer table(most copy in other node)update rows to establish the master copy in local node

. Insert into customer table using sequence valueend loopworkload.sh: executes update.sql on all 8 instances at same time

• Goal– Monitor real time performance; diagnose performance issue using

historical data – Examples of using ADDM, ASH, AWR snapshots in EM

Examples of RAC Performance Monitoring & Diagnosis

• First Run:

Examples of RAC Performance Monitoring & Diagnosis

• Run ADDM

Examples of RAC Performance Monitoring & Diagnosis

• High CPU usage

• High Cluster

Wait

Examples of RAC Performance Monitoring & Diagnosis

• ADDM Tuning Recommendation

• Implementation: Create an Index on customer table

Examples of RAC Performance Monitoring & Diagnosis

• Second Run

Examples of RAC Performance Monitoring & Diagnosis

• Real Time Monitoring:

• ADDM Findings

Examples of RAC Performance Monitoring & Diagnosis

• ADDM Tuning Recommendations

Examples of RAC Performance Monitoring & Diagnosis

• ADDM Tuning Recommendations

Examples of RAC Performance Monitoring & Diagnosis

• Third Run: implementing recommendations: alter system set gcs_server_processes = 4 scope = spfile;recreate the sequence by setting cache value 9000

Example of RAC Performance Monitoring & Diagnosis

• Performance Comparisons of three Runs:– Throughputs: transactions/sec

1st: 30-60

2nd: 750-800

3rd : 7000

Examples of RAC Performance Monitoring & Diagnosis

• Performance Comparisons of three runs:– Time to complete the tests (mins)Instance#

• Summary:– Use EM to monitor and diagnose the RAC database performance– Identified the root cause of major waiting time and recommended

the tuning solution: improve the performance more than 10times each– SQL and database objects tuning can reduce cluster wait time and

CPU time.

Instance# 1st Run 2nd Run 3 Run1 213 7 1

2 373 10 1

3 222 10 2

4 260 13 2

5 313 9 2

6 376 12 2

7 370 12 1

8 385 11 2

QA

Thank you Please complete the survey Speaker: Kai Yu, kai_yu@dell.comTitle: RAC Performance Monitoring and Diagnosis

using Enterprise Manager, #Q354, Visit my Oracle blog:

http://kyuoracleblog.wordpress.com/