A Holistic Approach To Performance Tuning Oracle Applications Release 11 and 11i

A Holistic Approach ToPerformance TuningOracle Applications Release 11 and 11i

Andy Tremayne Applications Performance Group

Agenda

The Methodology– Problem Definition– Project Documentation Summary (PDS)– Benchmark Information

Tuning The Technology Stacks– The Client– The Middle Tier– The Database– The Server– The SQL Access Paths– The Network

Tuning Applications

The Methodology

Problem definition methods– Necessarily abstract and complex– Deployment requires specialist skills

This methodology is– Very simple, fast and generic– Based on best practice and real world approach

The systematic approach– Focuses investigative attention and the tuning effort– Enables early identification of resources needed– You do not need to be an expert

Three Key Areas

1. Accurately define the problem– Define the problems and their characteristics– Key to identifying the source of performance issues – Focuses tuning effort

2. Define and agree performance target levels – To support the critical business functions– To provide the definition of success

3. Understand the system performance factors

W HAT?

W HERE?

PRIORITY

W HEN?

EXTENT?

– Characterise the problem– What is the nature of the problem?– Ensure you fully understand– “Slow database” is too ambiguous

The Problem Definition Stages

W HAT?

W HERE?

PRIORITY

W HEN?

EXTENT?

– Specify the locations– Use converse questions– List differences between them– Compare data routes, middle tiers...


W HAT?

W HERE?

PRIORITY

W HEN?

EXTENT?

– Note the times of day / dates– Link to events e.g. business cycle– Identify origin of the problem– Identify if a problem when isolated


W HAT?

W HERE?

PRIORITY

W HEN?

EXTENT?

– How many users are affected?– How much functionality? – How many applications?– Look for problem commonality


W HAT?

W HERE?

PRIORITY

W HEN?

EXTENT?

–Define the relative importance–When does it need to be fixed by?–Is there a viable workaround?–Consider the impact to the business


W HAT?

W HERE?

PRIORITY

W HEN?

EXTENT?


–Define the problem quickly–Define the problem accurately–Identify the appropriate resources–Solve the problem quickly

The “When” Stage Checklist

When does the problem occur? – Identify an underlying trend. Correlate with

system events and changes…

Ask the questions:– When does the problem occur? – Has it always been slow? – Did it change suddenly? – Is it slow at all the time? – Is it slow when the system is heavily used? – Is it getting slower?

The “When” Stage Flowchart

Identify system conditionswhen under performing

Identify the times / daysit underperforms

Identify when the problemwas first observed andwhen it was not a problem

Is itgettingslower?

Has italways been

slow?

Didit change

suddenly?

Identify the date/time

Yes

Is itslow all the

time?

Is it slowwhen the system

is heavilyused?

YesPossible Causes:Data volume, no. of users, networktraffic or system load related

No

Possible Causes:System Load

Yes

Possible Causes:System changes and/or events.Investigate using the customer'ssystem change record

Possible Causes:Indeterminate

WHEN- do the problems occur?

No No

Yes

Yes

No

When does theproblem occur?

No

Project Documentation Summary (PDS)

The PDS – Structures and summarizes the salient performance

indicators– Spans the entire system– Identifies areas to monitor– Ensures that the investigative focus remains impartial

Project Documentation Summary

Documents– Oracle and Customer Contacts– Problem Definition– System Environment– Oracle and Applications Environments– Web/Forms Server Environments– Client and Network– Process Information

ProjectProjectDocumentationDocumentation

SummarySummary

It should only take 3 hours to complete

PDS-Summary

PDS-Process Information

The PDS provides a discussion document– It breaks down technical and political barriers – It helps build a business case and justify change

When will your next emergency occur?What is your strategy?

Agenda



Tuning Applications

Release 11i Browser Benchmarks GX100, 500MHz Celeron 128MB JInitiator 1.1.8.16

Win 95/98 SE Win NT SP5 Win 2000 SP1 Win XPIE 5.5

Net 4.7

IE 6

IE 5.5

Net 4.7

IE 6

IE 5.5

Net 4.7

IE 6

IE 5.5

Net 4.7

IE 6

Applications Startup (Open) 28 23 19 20 20 18 20 19 19 19 20(PO) Purchase Order W/B (Open) 4.5 4 3.8 3.5 4 3.5 4 5 3.7 3.5 3(AP Invoice Entry W/B (Open) 8.7 7 8 8.5 7 6.7 7 8 6.8 7 6.8SA Users(Open) 3.5 3 3 3.2 3 3 3 4 3 3 3.6Concurrent Requests (Find All) 2 1.7 2 1.8 1.6 1.4 1.8 2 1.5 1.6 1.5 IE 6 appears consistently best (except 1 or 2 results)

– All results for Windows NT, 2000 and XP are within 3% – Always perform your own tests!

Other benchmarks have shownWindows NT is faster with 300MHz and higher

PC Speed vs Latency Benchmark

LatencyLatency 6ms 6ms 300ms300ms1400ms1400ms

133MHz Win 95 48MB133MHz Win 95 48MB

233MHz Win 95 48MB233MHz Win 95 48MB

300MHz Win NT 128MB300MHz Win NT 128MB

400MHz Win NT 128MB400MHz Win NT 128MB

66 s66 s 67.7 s67.7 s 80 s80 s

30 s30 s 36.5 s36.5 s 53 s53 s

25.5 s25.5 s 29.4 s29.4 s 35 s35 s

21.4 s21.4 s 26.5 s26.5 s 35 s35 s

CPU speed compensates for high-latency situations

– A little dated now– JInitiator provides better times– Timings are very comparable with the browser benchmark

Floating menu bars – Frequently ‘polled’ wasting usually 10% CPU– 300MHz becomes a 266MHz ….– Keyboard shortcuts are a quicker, easier alternative

Generic PC Tuning - Still very common!

Windows screen savers– Use up to 8MB of memory

Video– Up-to-date drivers can provide 20% improvement– Using 256 colors saves 0.5 - 0.75 sec opening a form

Paper contains a complete tuning checklist

Benchmark and compare with targets – If targets are achieved– Move the PC away from the server, 1 hop at a time

Database Server

Application Server(s)

Switch

LAN

Tuned Client

What Next?

Agenda



Tuning Applications

Middle Tier Profiles

Middle Tier Server profile– Memory is greater concern than CPU

Database Server profile– I/O and CPU are more of a concern than memory

Separate machines – Scalability– Simpler profile management

Keeping multiple forms open on a poorly tuned system only exacerbates memory problems, slowing response times even further

Java Virtual Machine Questions

How many JVMs and how many users per JVM?– Between 20 and 50 “active” users per JVM

Hundreds of concurrent users Depends on CPU speed, Applications mix, the users

– Perform a saturation test for your particular Applications mix and business

How much Memory?– -ms128m and -mx256m ……up to 400Mb– Set soft limit to 80% of the hard limit– set "ulimit -n 1024" for the Apache and JServ processes

Optimizing Apache

Current default for Apache is HTTP 1.1 – “keep alive” session feature in 1.1– Messages are sent between the browser and the server

Performance boost for Internet Explorer web users– Force Apache to use HTTP v1.0 – 26 to 5 seconds when opening some dialog boxes– In the httpd.conf file:

– BrowserMatch "MSIE 5\.[^;]*; " nokeepalive downgrade-1.0 force-response-1.0

Always test and verify changes yourself

Client, Network, DB or Forms Server Timings Most Time

Performance Collector (Forms 6i)– Specify “record=performance”

Analyze output using f60parse.pl– Four main screens

FRD/Performance Collector

Agenda



Tuning Applications

Getting Things Right for 11i

8KB block size is strongly recommended for 8i/11i– Don’t use 4KB unless constrained by platform!– Increase performance by

40% for batch programs and large data-sort queries 4-5% for light data-sort queries

Remove extraneous settings and eventsfrom the database initialization files!

Statspack - Introduction

Introduced in 8.1.6– Captures performance for new DB features– Summary page– Improved drill down– Captures high-load and some literal SQL– Backports available (unsupported)

“Diagnosing Performance With Statspack” papers – http://technet.oracle.com/deploy/performance/content.htm

– Results are stored in tables– Each snap is Identified by a Snapshot_id

Statspack - With Applications

Always use the 8.1.7 version– Space management and report

Use special versions that include the module name– No module name denotes custom code– MetaLink Note No. 153507.1 – Use Snap Level 5

Oracle 9i– New Snap Level 6 collects execution plans

Use modified parameters with Applications One hour is the best unit of work

Statspack – Load ProfileLoad Profile Per Second Per Transaction

Redo size: 234,110.14 15,400.80Logical reads: 194,977.77 12,826.50Block changes 1,413.80 93.01

Physical reads: 3,546.96 233.33Physical writes: 84.18 5.54

User calls: 492.08 32.37Parses: 207.88 13.68

Hard parses: 4.66 0.31Sorts: 293.56 19.31

Logons: 1.17 0.08Executes: 1,300.36 85.54

Transactions: 15.20

Statspack – High Load SQL

Buffer Gets Executions Gets per Exec % Total Hash Value Module--------------- ------------ -------------- ------- ------------ ------ 35,087,907 43 815,997.8 5.0 2683031174 WSHRDPAKSELECT MSI.SEGMENT1 c_item_flex , det . inventory_item_id c_inv_item_id , msi . description c_item_description , det . customer_item_id c_customer_item_id , det . source_header_number c_so_number , det . source_line_number c_so_line_number , det . cust_po_number c_po_number , sum ( round ( nvl ( det . requested_quanti

Module Name

NEVER ANALYZE the SYS schema!!!

Gathering Statistics

Release 11i - only ever use Gather Schema Statistics– 10% sample

Prior to Release 11i– Analyze using an adequate sample size

Improves batch performance by around 40%, especially in AR, PA and concurrent processing.

– Inadequate sample sizes may reduce performance by up to 40%.

Do pin– A Core set of SYS and FND packages– A core set for the products you are using– Any large (> 50KB) constantly aged-out packages

Size and number of executions in V$db_object_cache

Monitor X$KSMLRU– The no. of objects (KSMLRNUM) – Displaced packages and PL/SQL

Package Pinning Strategy

Do not pin– Small infrequently used packages– Those used by long-running batch processes or reports

Literal SQL

Literal SQL – Select …. From …. Where Order No = 123456

Literal SQL is fine for batch processing – Statements are run infrequently– Enables the CBO to accurately cost the use of an index

When statements use large amounts of shared memory e.g. 1MB. 100 statements = 100MB

Literal SQL cannot be shared – It severely limits OLTP scalability and throughput– Concern is with large data sets e.g. order_no – Fix by converting literal SQL to use bind variables

Agenda



Tuning Applications

Consider using raw devices only if disk I/O is the only remaining performance bottleneck and cannot be resolved.

BUT – Database export and import– Eradicates row chaining; rebuilds and balances indexes

Raw Partitions

Raw devices are required for OPS and RAC– Generally much more difficult to administer

In theory, raw device buffering improves disk I/O– Some conversions from UNIX to raw devices – Have improved performance by 10%-15%

Stripe Size (simplified)

Oracle

Operating System

64KB Stripe Size

64KB

64KB

1 x 64KBDisk Read

Benefit depends on amount of I/O. Use Statspack I/O figures.

Measuring Disk I/O

When using disk arrays– Operating system utilities are limited– Sometimes need specialist software

Only the Production instanceshould be running on the Production server

Instead use the Oracle figures– This is the time Oracle sees for an I/O – FileStat figures in the UtlEstat or StatsPack report– <20ms read <30ms write (max!) non striped– <10ms on a striped disk array

Agenda



Tuning Applications

A Stuck Performance Issue?

Always keep the raw trace file!

Does the total = real world time?+

Mapping A User Session

Sign–On Auditing - set at User level or higher–Records each time a person signs on to an application

Applications help screen contains: –AUDSID (maps to V$SESSION.AUDSID)select U.USER_NAMEfrom FND_USER U, FND_LOGINS L, V$SESSION Vwhere U.USER_ID = L.USER_IDand L.SPID = V.PROCESSand V.AUDSID = &1;

USER_NAME -----------------------------------OPERATIONS

System: Initialization SQL Statement – custom profile option'ALTER SESSION SET EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 4 '||''''

LevelLevel Included InformationIncluded Information

null or 1null or 1 Same as standard SQL Trace functionality Same as standard SQL Trace functionality 44 As level 1 + bind variable informationAs level 1 + bind variable information88 As level 1 + session wait information As level 1 + session wait information

1212 As level 1 + bind variable and session wait informationAs level 1 + bind variable and session wait information

Using Event 10046

Enable Trace for a forms session or report, within a stored procedure or in a concurrent program

– Set timed_statistics = true– Set event 10046 for more advanced troubleshooting

Agenda



Tuning Applications

Latency– Delay between the instant a request is made for data

and the transfer starts– Influenced by both device and link latency

Bandwidth and Latency

Bandwidth – Amount of data that the network can transfer – Size of the pipe/how many packets at once

Create a detailed network diagram including:– The location and number of all Oracle users– Every device from the client to the database server– The bandwidth and latency of all links and devices

Use a full size packet: ping -l1472 -n50 <host>

Understanding the Network

To identify a performance problem– You need only basic knowledge

Note! TraceRoute shows the optimal route - which is not necessarily the actual route.

Bandwidth

0 2 4 6 8

HeavyNCA-11NCA-11ii

10 SC10 SC

Normal

Low

Heavy

Normal

Low

4 - 84 - 8

4 - 64 - 6

2 - 42 - 4

2 - 32 - 3

2 - 62 - 6

0 - 20 - 2

Typ

e of

Use

r

Kilobits Per Second

Avg. 4.8

Avg. 2.4

Avg. 1.2

Isn’t Bandwidth Enough?

Mission-critical applications suffer while less important traffic can dominate your network

– Large downloads from corporate and external web sites– Email synchronization– Downloading large email attachments– MP3 uploads and downloads– RealPlayer and other streaming traffic– Global Single Instance

Some customers have doubled link capacity only to find that Application performance problems persist!

WAN links continue to be problematic because of their cost and relatively low bandwidth

Quality Of Service (QoS)

QoS - A set of features (and tools)– Usually implemented in routers– Classifies traffic enabling differentiated service levels

Four Main Policies– All concerned with congestion management

Packet Classification Packet Shaping Rate Limiting Priority Queuing

Classification (Coloring)& Rate Limiting Engine

Traffic Queues

Shaping and Rate Limiting

Engine

High

Normal

Low

Medium

Queue Bypass Transmission Queue

Input Queue

QoS Traffic Control

Mission CriticalApplicationsEmail

Other TrafficInternet Browsing

Shaped Traffic

Mission CriticalApplicationsEmail

Other TrafficInternet Browsing

Uncontrolled Traffic

WAN/Internet

Network Traffic Shaping

Agenda



Tuning Applications

Concurrent Manager Tuning

Many now have up to 100,000 requests– Check your top 10 SQL statements– Run purge - not just fnd_concurrent_requests!

If you are tight on CPU– 50% of tuning is in the business– A strategy will mean you only need 20/25 managers– Watch the sleep times

Separate the log and out directories to reduce contention.There are several other recommendations in the paper.

Enhancing The Concurrent Manager

Paper 164085.1– Moving sensitive report files to secure directories– Compressing reports and distributing them during off-

peak periods over Wide Area Network links– Automatically faxing reports and orders – Converting documents to PDF

With and without Adobe Automatically printing from UNIX

– Automatically emailing reports or documents– Automatically archiving selected requests before purging

Enhancing The Concurrent Manager

Embedding SQL in UNIX Scripts

unixvar=`sqlplus -s apps/test @<<! set termout off set feedback off set pagesize 0 select R.REQUEST_ID from FND_CONCURRENT_PROGRAMS_VL P, FND_CONCURRENT_REQUESTS Rwhere P.CONCURRENT_PROGRAM_ID = R.CONCURRENT_PROGRAM_ID and P.USER_CONCURRENT_PROGRAM_NAME = 'Active Users' and R.PHASE_CODE = 'C' and R.STATUS_CODE = 'C'; exit!` echo $unixvar ….

Reads down to the !Suppresses

SQL “overlay”

SuppressesSQL login Replace With

$FCP_LOGIN

Use for loop

Clever Queries

Operator Description Comment= xyz Equals xyz xyz can be a number, word, or

date enclosed in single quotes

!= xyz Not Equal to xyz As Above

< xyz Less than xyz As Above

> xyz Greater Than xyz As Above

like xyz Similar to xyz like ‘xyz’ may contain the _ wildcard or %

Between x and y Between x and y a and b may be numbers, words or dates

in (x,y,z,...) exists in list As Above

is null is empty for example printed_date is null

AR1020 ...

Clever Queries

Clever Queries

Clever Queries

Clever queries may be extended – Using :a, :b, :c …….

– :a like ‘%’ and PRINTED_DATE is null– :a like ‘%’ and POSTED_FLAG = ‘Y’– :a like ‘%’ and ATTRIBUTE1 = ‘Special Item’– :a like ‘%’ and VENDOR_ID in

(select VENDOR_ID from PO_VENDORS where HOLD_FLAG = 'Y')

Integrating with form folders is a very powerful techniqueReduces the need for customizations

In Summary…

The holistic approach – Provides a simple, fast approach– Key to identifying the source of performance issues – Focuses tuning effort

Throughout the tuning exercise – Make a change and measure the affect – Investigate every area – Manage the server load at all times

Although tuning is a science it also involves common sense and sometimes, a little ingenuity.

Questions– Catch me!

For More Information

Holistic Paper– 84 pages: 69565.1– PDS: 165300.1– Concurrent Paper: 164085.1

Tuning Handbook:– ISBN 0-07-212549-7

A Holistic Approach To Performance Tuning Oracle Applications Release 11 and 11i

Documents

Transcript of A Holistic Approach To Performance Tuning Oracle Applications Release 11 and 11i