SkillBuilders Presents Kyle Hailey ( ... buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free...

SkillBuilders PresentsKyle Hailey (www.perfvision.com)Oracle Performance Tuning

Average Session Load(ASL)

The Golden Metric ?

Who is Kyle Hailey1990 Oracle

90 Oracle Unix support92 Ported v6 onto Dec Ultrix93 Oracle France94 Gold Support Specialist95 Performance Expert at Digital Europe98 Kernel Development California

2000 Dot.Com2001 Quest 2002 Oracle OEM 10g2005 Embarcadero2006 Independent

When not being being a Geek writing for the Oaktable etc- photographer- Just Married and new home owner in Portland- and wonder how I missed the dot.com millions

In this SessionThe Holy Grail of Performance : ASL

1. ASL : stethoscope for Database HealthTap is a metric that taps directly into the heart of your DB

Heart stopped - Hung?Beating very slow - Idle ?Beating fast - Heavily Loaded ?

2. ASL ComponentsCPUWaitsTime series

3. Maximum CPU as YardstickTo measure ASL against

OEM 10g ties it all together

Goal of this Presentation

Simplify the data and empower the DBA

My goal is to cut out a lot of the unnecessary information and extraneous data andhone in on the essential

When to actHow hard to act

What’s the Database Doing?

Your manager calls, it’s 2am, do you know where your database is?

BlockedRunning okIdle

I want a quick and easy way to see what the database is doing

Checking the Database

How do *you* check the Database Health?Routine Exams?Statspack?

1300 lines of dataWhich linesOnce and hour

Automated Alerts?What do you set alerts on?What if no alerts fire ? Are you ok?Do alerts really tell you what’s happening?

What’s the Database Doing?!

Whip out the stethoscope

8Welcome to …The Cult of ASL

Once you’ve been initiated …

there is no going back

For those of you who are already members, welcome back

Average Session Load (ASL)

The Golden Metric Powerful MultidimensionalIndispensable

Average load of sessions on the database measured in number of sessions

Core of 10g Performance Monitoring

Performance Page

Average Session Load (ASL)

Average Active Sessions (AAS?)OEM 10g Graphs

Session LoadI often refer to it this way

Centi-seconds per second (or secs/sec)In the dark ages, before OEM 10g, waits were often measured as Centi-secs per sec

ASL – Average Session LoadAverage

over 15 secs in OEM 10gRepresents Load Active Sessions Only

Active sessions put load the databaseInactive Sessions don’t put load

Except for memory usage

ASL : sources

1. Manuallyv$system_eventv$sysstat

2. Statspack some calculations 3. OEM 10g4. ASHRPT

ASL Calculation

ASL = DB TIME / Elapsed Time

DB Time (DBT) = sum over all sessions of time spent in a call. A call could be

executing SQLbackground work

DBWR writing blocksLGWR writing redo

DB Time

DB Time (DBT) = sum active session time

DB TIME (10g) =

DB TIME =

select value from v$sysstat where name = ‘DB time’;select value from v$sysstat where name = ‘DB time’;‘DB time’

Select sum(time_waited) from v$system_event where event not in ( ... idle events …);+Select value from v$sysstat where name = ‘CPU used by this session’;

1. ASL ManuallyData

10g, easy DBTime = V$sysstat ‘DB time’ 9i ( 7 to 11 actually), DBTime =

V$system_eventSum wait timesNon-idle waits

V$sysstat CPU used by this session

ASL= DBTime/elapsed timeProduces

Session time / elapsed timeSession cent-secs/secSession secs/ secAvg Session Load (ASL)

2. Statspack: Calculating ASL

Look forTop 5 Timed EventsElapsed Time

Statspack: Cheat Sheet

InstallConnect as SYSDBA@?/rdbms/admin/spcreate.sql

RunExec statspack.snap;

Generate Reports@?/rdbms/admin/spreport.sql

Statspack Output

Trusty statspack reportElapased TimeCheck “Top 5 Timed Events”

Start at line 52 of Start at line 52 of about 1300!about 1300!

21Statspack Elapsed Time

STATSPACK report for

DB Name DB Id Instance Inst Num Release RAC Host------- ----------- -------- -------- ---------- --- -------LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr

Snap Id Snap Time Sessions Curs/Sess--------- ------------------ -------- ---------

Begin Snap: 1 03-Apr-06 12:34:06 18 5.6End Snap: 2 03-Apr-06 12:34:36 18 4.8Elapsed: 1.00 (mins)

STATSPACK report for

DB Name DB Id Instance Inst Num Release RAC Host------- ----------- -------- -------- ---------- --- -------LABSF03 1420044432 labsf03 1 10.1.0.2.0 NO labsfr

Snap Id Snap Time Sessions Curs/Sess--------- ------------------ -------- ---------

Begin Snap: 1 03-Apr-06 12:34:06 18 5.6End Snap: 2 03-Apr-06 12:34:36 18 4.8Elapsed: 1.00 (mins)

Statspack: Top 5 Timed Events

Look at Top 5 Timed Events

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call Time--------------------- --------- --------- -------buffer busy waits 2,748 250 78.72CPU time 32 10.16free buffer waits 1,588 15 4.63write complete waits 10 8 2.51log buffer space 306 5 1.51

Top 5 Timed EventsTop 5 Timed Events~~~~~~~~~~~~~~~~~~ % Total~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call TimeEvent Waits Time (s) Call Time------------------------------------------ ------------------ ------------------ --------------buffer busy waits 2,748 250 78.72buffer busy waits 2,748 250 78.72CPU time 32 10.16CPU time 32 10.16free buffer waits 1,588 15 4.63free buffer waits 1,588 15 4.63write complete waits 10 8 2.51write complete waits 10 8 2.51log buffer space 306 5 1.51log buffer space 306 5 1.51

Example

CPU + WAITSCPU = 32WAITS = 250 + 15 + 8 + 5

= 278 secsElapsed Time

= 60 secs(32 + 278) secs / 60 secs= 5.1 Average Session Load

Top 5 Timed Events

Event Time (s) ----------------- -----buffer busy waits 250CPU time 32free buffer waits 15write complete waits 8log buffer space 5

Top 5 Timed EventsTop 5 Timed Events

Event Time (s) Event Time (s) ---------------------------------- ----------buffer busy waits 250buffer busy waits 250CPU time 32CPU time 32free buffer waits 15free buffer waits 15write complete waits 8write complete waits 8log buffer spacelog buffer space 55

3. ASL in OEM DB Home Page

4. ASHRPT

?@/rdbms/admin/ashrpt.sql

Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)Avg Active

Event Class % Activity Sessions--------------------------- ----------- ---------- ----------CPU + Wait for CPU CPU 67.98 8.42enq: TX - row lock co Application 23.98 2.97buffer busy waits Concurrency 4.66 0.58latch: cache buffers chai Concurrency 2.26 0.28

Top User EventsTop User Events DB/Inst: TESTDB/DB/Inst: TESTDB/testdbtestdb (Apr 21 12:00 to 12:05)(Apr 21 12:00 to 12:05)

AvgAvg ActiveActive

EventEvent ClassClass % Activity% Activity SessionsSessions

------------------------------------------------------ ---------------------- -------------------- --------------------

CPU + Wait for CPUCPU + Wait for CPU CPUCPU 67.9867.98 8.428.42

enqenq: TX : TX -- row lock corow lock co ApplicationApplication 23.9823.98 2.972.97

buffer busy waitsbuffer busy waits ConcurrencyConcurrency 4.664.66 0.580.58

latch: cache buffers latch: cache buffers chaichai ConcurrencyConcurrency 2.262.26 0.280.28

Got ASL, Now What ?

Need one more item: CPU Count

# of CPUs available on SystemShared with other applications ? difficultOn dual & quad cores, lower the CPU countRepresents max active sessions doing work

A value above CPU represents some bottleneck

CPU count, OEM DB Home Page

CPU Count, OEM Perf Page

CPU CountCPU Count

Available CPU – init.ora

Statspack 10g shows # of CPUSStatspack 9i # of CPUs missing:

SQLPLUS> show parameters cpu_count

NAME VALUE------------------ ----------cpu_count 2

SQLPLUS> show parameters cpu_count

NAME VALUE------------------ ----------cpu_count 2

4. ASHRPTASH Report For TESTDB/testdbDB Name DB Id Instance Inst Num Release RAC Host------------ ----------- ------------ -------- ----------- --- ------------TESTDB 2371570538 testdb 1 10.2.0.1.0 NO sdbe604aCPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size---- ------------------ ------------------ ------------------ ------------------

2 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%)Analysis Begin Time: 21-Apr-06 12:00:01

Analysis End Time: 21-Apr-06 12:05:01Elapsed Time: 5.0 (mins)Sample Count: 3,716

Average Session Load: 12.39Avg. Active Session per CPU: 6.19

Report Target: None specifiedTop User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05)

Avg ActiveEvent Event Class % Activity Sessions----------------------------------- --------------- ---------- ----------CPU + Wait for CPU CPU 67.98 8.42enq: TX - row lock contention Application 23.98 2.97buffer busy waits Concurrency 4.66 0.58latch: cache buffers chains Concurrency 2.26 0.28

ASH Report For TESTDB/ASH Report For TESTDB/testdbtestdb

DB NameDB Name DB IdDB Id InstanceInstance Inst Num ReleaseInst Num Release RAC HostRAC Host

------------------------ ---------------------- ------------------------ ---------------- ---------------------- ------ ------------------------

TESTDBTESTDB 2371570538 2371570538 testdbtestdb 1 10.2.0.1.01 10.2.0.1.0 NONO sdbe604asdbe604a

CPUsCPUs SGA SizeSGA Size Buffer CacheBuffer Cache Shared PoolShared Pool ASH Buffer SizeASH Buffer Size

-------- ------------------------------------ ------------------------------------ ------------------------------------ ------------------------------------

22 1,000M (100%)1,000M (100%) 468M (46.8%)468M (46.8%) 112M (11.2%)112M (11.2%) 4.0M (0.4%)4.0M (0.4%)

Analysis Begin Time:Analysis Begin Time: 2121--AprApr--06 12:00:0106 12:00:01

Analysis End Time:Analysis End Time: 2121--AprApr--06 12:05:0106 12:05:01

Elapsed Time:Elapsed Time: 5.0 (5.0 (minsmins))

Sample Count:Sample Count: 3,7163,716

Average Session Load:Average Session Load: 12.3912.39

Avg. Active Session per CPU:Avg. Active Session per CPU: 6.196.19

Report Target:Report Target: None specifiedNone specified

Top User EventsTop User Events DB/Inst: TESTDB/DB/Inst: TESTDB/testdbtestdb (Apr 21 12:00 to 12:05)(Apr 21 12:00 to 12:05)

AvgAvg ActiveActive

EventEvent EventEvent ClassClass % Activity% Activity SessionsSessions

---------------------------------------------------------------------- ------------------------------ -------------------- --------------------

CPU + Wait for CPUCPU + Wait for CPU CPUCPU 67.9867.98 8.428.42

enqenq: TX : TX -- row lock contentionrow lock contention ApplicationApplication 23.9823.98 2.972.97

buffer busy waitsbuffer busy waits ConcurrencyConcurrency 4.664.66 0.580.58

latch: cache buffers chainslatch: cache buffers chains ConcurrencyConcurrency 2.262.26 0.280.28

ASL FormulasUse CPU count as yardstick:

< 1 Database is not blockedASL ~= 0

Database basically idleASL < # of CPUs

CPU availableDatabase is probably not blocked

• ASL > # of CPUsCould have performance problemsASL >> # of CPUS

There is a bottleneck

Available CPU vs ASL

ASL = 5.1ASL = 5.1

# of CPU = 2# of CPU = 2

ASL far above available CPU => problem

Going Farther with ASL

ASL can tell you a lotBut it’s components tell you much moreTo go farther need the components of ASL

CPU How many CPUs (max CPU available)

WaitWhich waits

Value over time

EM DB Home Page

35OEM 10g Perf PagesDB Home DB Home

PerformancePerformance

ASL Point in TimeASL Point in Time

ASL over TimeASL over Time

Performance Page

ASL – Performance Page

OEM 10g - ASL

ASL is the top of the curveASL is the top of the curve

33 1.91.9 22

OEM 10g - CPU

CPU UsageCPU Usage

OEM 10g - Waits

WaitsWaits

OEM 10g – CPU vs Wait

WaitsWaitsCPUCPU

OEM 10g – Max CPU

Maximum Possible CPUMaximum Possible CPU

OEM 10g Zoom-In

CPU + WAIT

Available CPU

OEM 10g – get to work !

Looks Looks OK OK

But …But …

Relax Relax Get to Work!Get to Work!

Alternative ASL Calculations

ASL = DB TIME / Elapsed Time

But there is another wayBut there is another way ……

ASL based on ASHASH - Active Session History

v$active_session_historyFormula

Count active sessions over 15 secondsDivided by # of samples in 15 seconds= Average Session Load

A statistical approximation, but surprisingly closeASH data source empowers drilldown into the causes of bottlenecks

2. ASL from ASH

776655443322

ASHASH

t0 t0 tntn

t0t0--nn

active sessions(ti)nnΣΣ00

ASL =ASL = // nn

Two Sources comparison1. v$system_event & v$sysstat

IndirectAccurateLags (especially CPU)Limits analysis

2. v$active_session_historyDirectReal timeApproximationAllows drilldowns

Top ActivityTop Activity

Same Chart butSame Chart but

calculated differently calculated differently

1. ASL – Performance Page

v$v$sysstatsysstat

DB Home DB Home

Top ActivityTop Activityv$system_eventv$system_event

ASL = ∆DBT / ASL = ∆DBT / ∆Time∆Time

2. ASL – Top Activity Page

v$active_session_historyv$active_session_history

active sessions(ti)nnΣΣ00

ASL =ASL =

DB Home DB Home

Samples VS Counters

CountersCounters

SamplesSamples

Slight LagsSlight Lags

v$system_eventv$system_event

v$active_session_historyv$active_session_history

54The Power ASH gives ASLDB Home DB Home

ASHASH

ASH in OEM

Top Activity gives Top Activity gives more informationmore information

Top Activity : Based on ASH

missingmissing

Thanks Thanks To To ASHASH

Top Activity : ASH Dimensions

58ASL – %Session Time Issue

Shown in % DB TimeShown in % DB TimeMissing % Session TimeMissing % Session Time

Top Activity: ASH Sessions

Many Users Active Many Users Active

On Performance Page, no way to tell how many usersOn Performance Page, no way to tell how many usersBut Top Activity Page fixes thatBut Top Activity Page fixes that

Top Activity: ASH Sessions

Two Users ActiveTwo Users Active

SQLSQLSessionSession

62OEM 10g Perf PagesTop ActivityTop Activity

SQLSQLSessionSession

63Session : ASH Activity

64SQL : ASH Activity

Getting the Most out of ASLNeed to know the System’s ProfileWhat your application is like?

Data WarehouseOLTPTypical load

Once you get to know it you can see anomaliesIs ASL near 0 when it should be higherIs that Data Warehouse query running normal

Do you know what it looks like?Is there an unusual bottleneck

Knowing your DB Profile

When to tune?

General rules of ThumbWaits >> CPUCPU > Max CPU

Waits > CPU

CPU > Max CPU

Idle Database

Value of proving the database is Idle It’s the Databases Fault

How many times do you hear that?Database Idle

No load on databaseDatabase “performance” is fine

Under utilized

Problem lies elsewhereSaved me time and stress many times

Idle Database – Perf Page

Idle Database – Top Activity

Harnessing ASLStatspack

lackingOEM 10g

Best tool10g (or 11) onlyNeed licences

Third Party ToolsSome good productsGenerally as expensive as OEM

Ashmon – “OEM-lite”FreeAny version with S-ASHLimited analysis

Statspack

Free and any versionNeeds numerical massagingLacking detailed SQL and Session info

OEM 10gClearest, easiest, most powerful

Overview Load chart - ASL

Breakdown –Top SQL and SessionOr Service, Module, Action, Objects, Files

Drill downsSQL Session

PlusADDM – Automatic Database Diagnostics MonitorSQL Advisor – SQL tuning advice

DRAWBACK – 10g (or 11) and need license

SessionSession SQLSQL

Alternatives

Third Party toolsQuest – Performance AnalyzerSymantec – I3Confio – Ignite

ASHMON a free “OEM-lite”

http://perfvision.com/ashmon.phpWindows application Monitors Oracle 10g or 11Show “Top Activity” page from OEM

Note: Can be used on any version of Oracle with performance repository available on http://perfvision.com/sash.php

79CPU > Max CPU

Radical Change in Profile

Going Further – Wait Analysis

If ASL identifies Wait BottlenecksUse OEM 10g

Automatic Diagnostic Database Monitor (ADDM)Oracle Wait Interface

by R Shee and K Deshpande Wait Interface Class

Detail explanation, analysis and solutions to the top 30 wait events (98% of all waits encountered)

Where Can You Learn More

Oracle 10g Advanced Performance Tuning Class

With Kyle Hailey and SkillBuilders2 Day ClassOffsite or onsiteRegistration

www.skillbuilders.comGary Belke at 1-888-803-5607

Oracle 10g Advanced Performance Tuning Class

Learn new 10g performance and diagnostic toolsUnderstand the 10g Enterprise Manager Performance page ASL, AWR, ADDMAdvanced level of detail

Collection, Aggregation, Display, Analyze

Top 30 wait eventsASHConquer your database woes!

In summary

ASL is simple and PowerfulASL’s components are even more powerful

CPUWAITValue over Time

Use # of CPUs as a yardstick Know your application load profile to see anomaliesMonitor ASL with OEM 10g, Ashmon or third party tools

SkillBuilders Presents Kyle Hailey ( ... buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free...

Documents

Transcript of SkillBuilders Presents Kyle Hailey ( ... buffer busy waits 2,748 250 78.72 CPU time 32 10.16 free...

Redo Waits Kyle Hailey Kyle.hailey@hotmail.com. #.2 Copyright 2006 Kyle Hailey Redo REDO Lib Cache Buffer Cache Locks Network I/O.

Tom Waits - Beautiful

Antologia Tom Waits

BOOK Tom Waits - Anthology

“True Love Waits” Campaign

Tom Waits - 2002

Tom Waits Swordfishtrombone

Tom Waits - Alice

Waits Tom - Anthology

What’s So Special About Waits in Real Application · Web viewThe remote instance’s Lock Manager Service (LMS) processes will be accessing the global buffer cache, and the DBWR

05 WAITS Intro

Waits Cadillac Bass

Tom Waits Blood Maoney

Tom Waits Anthology - bs-gss.ru Waits Anthology_BOOGIEWOOGIE.RU.pdf · BOOGIEWOOGIE.RU. Title: Tom Waits Anthology Created Date: 1/16/2009 4:28:34 PM

A Man Waits

Waits Defined Kyle Hailey . #.2 Copyright 2006 Kyle Hailey Top 36 Foreground Waits 19. 19.write complete waits 20. 20.library cache.

Vienna waits for you

Thomas Alan "Tom" Waits

Tom Waits Anthlogy

Canadian Emergency Department Triage and Acuity Scale - … · Waits for Routine Care Waits for Emergency Department Care Waits for Acute Care Waits for Specialized Care Waits in