Holistic Tuning in Oracle

8/9/2019 Holistic Tuning in Oracle

1/12

Holistic Tuning in OracleBy Mike Ault, Oracle Guru, TMS Inc.

Introduction

What does the term Holistic really mean? From the online Webster Dictionary:

Holistic:

relating to or concerned with complete systems rather than with the analysis of, treatment of, ordissection into parts

Given the above definition of Holistic, this paper will deal with looking at the entire database whenperforming a tuning exercise, not just the immediate statement or other item that seems to be theproblem.

This Holistic view of tuning means that we must consider the total effects of design, structure,

applications, and physical systems on tuning and treat each as a part of the whole rather than piecingthem out into individual areas.

Holistic View of Oracle

Oracle is like an onion; each layer rests on the one underneath. If any layer is rotten then the entirestructure is in danger. Likewise, a change to any layer may propagate effects up and down through otherlayers. Lets look at an example.

Given: Statement X is generating excessive physical IOPS when it executesLocal Fix: Add indexesImmediate local effect: Statement X runs faster with fewer IOPS

Holistic effects: Statements Y and Z switch from proper indexes for them to indexes added forstatement XHolistic Result: X runs faster, but Y and Z run slower, overall result: Slower system even though thetuning effort on X was a success!

Now, this is not to say that all local actions are going to have a negative effect. Usually a switch to anew index is because the optimizer has looked at statistics and determined the index would be betterbefore using it. However, we all know this may not always work as designed!

Of course everyone generally agrees that caching is a good thing. The more we can cache data forOracle to use, the faster the application will run. However, this comes at a cost. The cost of caching is

that now instead of handing off IO to an external IO processor, IO has to be handled by the CPU usinglogical IO. Now, logical IO is several orders of magnitude faster than physical IO, except if it introducesso many additional CPU cycles that the system becomes CPU bound.

The converse is also true; without enough caching or disk IO bandwidth, adding memory or CPU to asystem that is IO bound may not help at all. So again we have to truly understand the holistic view totake the proper course of action.


2/12

Local Actions that Have a Negative Holistic Effect

In my job with TMS I am tasked with doing system evaluations using statspack and AWR reports todetermine if a client really needs a new IO subsystem or if other issues are causing their systemperformance issues. Believe me, it does us no good as a company to have someone say they bought oursolution and it didnt help their problem!

It is amazing how many people still suffer from basic, local problems that affect the holisticperformance of the entire database. Lets look at a few of these local issues that have a global effect.

Structure Layer

The structure layer consists of the internal physical structures that make up the database. In thebeginning it was just tables and indexes; now of course we have several types of tables, several types ofindexes, materialized views, LOBs, objects, and many other structures to be concerned about.

Improperly Designed Tables

Even though processors have gotten more powerful and disks and storage arrays more capable, you canstill defeat them with improper table design. Generally, improper table designs fall into two broad

categories: insufficiently normalized and over normalized designs.Insufficiently normalized tables result in repeating values within a table. Insufficient table design usuallyproduces excessive IO, excessive sorts, and excessive use of DISTINCT.

Over-normalized tables usually result in too many joins, excessive sorts caused by Cartesian productsfrom insufficient WHERE clauses, and overly complex SQL.

In most cases Oracle will not produce an efficient execution plan for over 6 tables unless you have adata warehouse and utilize a star join path. The undocumented parameter_optimizer_max_permutations sets the number of path determinations to 2000. Since the number of

permutations is based on the factorial of the number of tables in the join, you reach 2000 between 6tables (6 n!=720) and 7 tables (7 n!=5040) . Place the most important (i.e. the ones that reduce the resultset the most) tables first in the FROM clause if you have over 6 tables as (at least last time I reviewedthis) the tables are evaluated from left to right when it is going through possible paths.

Use of flex fields can also be a show stopper for performance. Flex fields are fields whose contents aredetermined by the context of previous rows. In one case I was called on to help with creating a datawarehouse from an application that was based on Oracle applications. In this case they had used theflex fields in Oracle applications to specify that a single column was either a number, a date, or acharacter field, depending on other values in the row. In actuality there where several different fields inthe same row used the same way. In order to resolve one entry into the data warehouse we had to

recursively join this table to itself seven times.

Use of improperly defined fields can also cause performance and storage problems, for example using alarge CHAR based field for small length character values. CHAR reserves whatever space is specifiedregardless of the length of the input data. This requires extra programming for comparison semanticsand wastes storage space.


3/12

Insufficient Indexes

Insufficient indexing results in full table scans, excessive sorting, and excessive CPU usage. Indexes,when used properly are probably one of the most performance enhancing features of Oracle.

Over-Indexing

Over indexing results in excessive times for DML operations such as insert, update, and delete because

the amount of IOPS required to perform each action is multiplied several fold with the addition of eachindex.

Use of an Incorrect Index

Using the wrong type of index can result in either the index not being used at all, or being usedinefficiently. For example, using a single bitmap index will usually result in poorer performance thanusing the equivalent B-tree index. Using a regular index when a function-based index is required willresult in a full table scan. Another problem with indexes is improper use such as using a singleconcatenated index rather than single indexes. Single indexes can sometimes provide a greateradvantage than a single composite by allowing a more diverse group of SQL statements to utilize thesingle indexes, either as single indexes or by using two or more of the indexes. There are many other

issues of this type; be sure to research the proper index to use in each situation.

Code Actions

Generally, code actions will be restricted in this paper to SQL related issues. However, each code base(JAVA, C, C+, etc) probably has a list of general problems when related to Oracle as well.

No Bind Variables

It is hard to believe that as long as Oracle has been supporting bind variables and as often as tuningexperts write, suggest, cajole, and otherwise belabor the bind variable usage issue, we still see this. Iwould venture to say that at least 50% of the reports I see have bind variable issues. Why are bindvariables so important?

Bind variables allow a statement to be easily reused. Without a bind variable each use of an otherwiseidentical statement (identical except for literals) results in a new version of the statement in the sharedpool. Lack of bind variables means the optimizer has to re-evaluate the statement each time it is issued,causing recursive SQL.

Recursive SQL generates CPU cycles, IOPS, and logical IO. Generally, if I look at a statspack or AWRand see a large shared pool (sometimes larger than the total db cache area), high recursive SQL, andhigh use of shared pool memory with low reuse for statements with greater than 1 execution, I canguarantee that I will see few bind variables in the SQL sections of the report.

Cache Sizes

Begin End

Buffer Cache: 640M 592MStd Block Size: 8K

Shared Pool Size: 1,168M 1,216MLog Buffer: 14,384K


4/12

Load Profile

% Blocks changed per Read: 0.35Recursive Call %: 33.43

Rollback per transaction %: 2.02Rows per Sort: 5.45

Of course a quick bandaid for improper bind variable use is to set the initialization parameter

CURSOR_SHARING to SIMILAR or FORCE, but the ultimate fix is to correct the bad coding andteach good bind variable usage going forward.

Use of Distinct

Usually the use of DISTINCT is caused by either database design (non-normalized data structures) orpoor SQL design (insufficient WHERE clause). Use of DISTINCT can indicate the programmer got theresults he expected, except there were duplicates, so a quick check of the SQL reference showed theDISTINCT keyword eliminates duplicates. Viola! the problem was solvedWRONG! What seems towork on a 100 row test database will cause problems in a 1 million row real database.Example of over use of DISTINCT:

SQL ordered by Elapsed Time DB/Inst: test/test2 Snaps: 29864-29865-> Resources reported for PL/SQL code includes the resources used by all SQL

statements called by the code.-> % Total DB Time is the Elapsed Time of the SQL statement divided

into the Total Database Time multiplied by 100

Elapsed CPU Elap per % TotalTime (s) Time (s) Executions Exec (s) DB Time SQL Id

---------- ---------- ------------ ---------- ------- -------------414 25 7,382 0.1 30.5 g7a19b1wmsvgq

SELECT DISTINCT

323 34 4,048 0.1 23.8 50gk22jk74jvy

SELECT

183 24 1,666 0.1 13.5 c1ysvka1vfyfdSELECT

117 9 4,809 0.0 8.6 c9humn05ydj4qSELECT DISTINCT

60 3 678 0.1 4.4 5c4kar29763mvSELECT DISTINCT

30 1 1 30.2 2.2 2f33rza4xwrk7SELECT

15 1 351 0.0 1.1 b6pdf4vxgtgr7SELECT DISTINCT

So in the above example of the top 7 statements for elapsed time, 4 used a SELECT DISTINCT.

When I see large numbers of sorts and large numbers of sort rows I will usually see excessive use ofDISTINCT as well. Of course without database ERD or other design documents I cant tell if the distinctis because of issues with design or SQL, unless I want to go in and analyze the detailed SQL and do aWHERE clause evaluation. Usually a detailed SQL examination is beyond the scope of my evaluations.


5/12

Remember, if there are X tables in a query then at a minimum you need X-1 join conditions in the whereclause, each referencing a different table.

IO Subsystem Level

The IO subsystem includes routers, hubs, HBAs, disk arrays, switches, and anything else that provides

persistent storage for the database.

Low Capacity IO Subsystem

If there is a low capacity IO subsystem I normally see excessive read and write times when there is notreally a high load on the system in terms of needed IOPS. Generally a disk will respond in 5 ms or less ifit is not overloaded. As a disk gets loaded down with requests, its response time increases 2-5milliseconds for each additional simultaneous response it is expected to make. This queuing of requestsagainst disks is usually caused by several factors.

An example of an overloaded disk subsystem:

Tablespace IO Stats DB/Inst: TEST/test Snaps: 21560-21658-> ordered by IOs (Reads + Writes) desc

Tablespace------------------------------

Av Av Av Av Buffer Av BufReads Reads/s Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)

-------------- ------- ------ ------- ------------ -------- ---------- ------TEST2_SSA_ZIP_AD

16,897,918 48 13.7 1.0 15,433,132 44 27 7.4TEST2_SSA_ST_AK

16,700,222 47 14.0 1.0 15,314,651 43 67 5.2TEST2_SSA_ST_SN

16,573,311 47 11.5 1.0 15,322,214 43 798 2.7

TEST2_SSA_ST_HN16,306,046 46 14.2 1.0 14,916,357 42 0 0.0

TEST2_SSN01_ID16,087,914 46 13.1 1.0 14,781,609 42 2 15.0

TEST2_PKT18,200,932 52 9.6 1.0 11,313,171 32 0 0.0

Notice that all of the disk read times are well over the 5 millisecond latency time expected and so thereare buffer waits. Buffer waits are usually an indication of excessive disk write times.

Of course the first factor is an overloaded disk. Most disk experts suggest that for maximum IOPS andlowest latency a striping model should be used that only utilizes a maximum of 30-50% of each disk. In

fact many say only 20-30% of a disk should be used if you want to get maximum performance. Thispractice is called short-stroking the disk. Of course this means that you must buy at least 3-4 times theamount of disks than you need for storage capacity. Back when disks were small and you boughthundreds for just normal databases, this problem was less apparent because you had many disk headssearching for your data at the same time. Now with capacities of hundreds of gigabytes and droppingspindle counts we see performance issues.

Capacity is a two-sided coin; you must not only consider the amount of storage you need, you must alsoconsider the number of peak IOPS needed to give you the performance you need at peak load. A normal


6/12

disk drive, regardless of storage capacity, generally will not deliver more than 200 IOPS of randomactivity. Peak load IOPS are more than just a number of actions required from the disk farm; you mustalso categorize the IOPS based on needed simultaneous access to the same disk block.

This IOPS and needed simultaneous access to data blocks has resulted in some vendors actually usinganywhere from 30-50 times the storage capacity needed for tests such as the TPC-H to get the neededIOPS and simultaneous access needs fulfilled.

In most cases if you meet the needed IOPS and simultaneous access requirements for a database, youwill exceed the needed storage capacity.

The advent of relatively inexpensive SSD technology (when measured in terms of $/IOPS) has mitigatedthis IOPS/storage capacity issue. By eliminating rotations and lateral latency and nearly eliminating thesimultaneous block access limitations, you can truly buy just the capacity you need and still get moreIOPS that you will most likely ever need (most SSDs provide in excess of 80,000 IOPS.)

Improperly configured IO Subsystem

One thing that can drive IO waits on a properly sized IO subsystem (with disks) is if that subsystem isimproperly configured. The system or storage administrator must understand that the Oracle system mayhave only a few write processes, but may have many read processes because each user process orparallel user process is responsible for its own reading from the IO subsystem. This plethora of readprocesses leads to the situation where in an improperly configured disk system a single process doing alarge read, such as a full index or table scan, will block many other users.

This diagram shows what happens when a full table scan is initiated on an improperly configured diskarray:


7/12

By aligning the expected physical IO size to the underlying stripe width per disk the situation shownabove can be mitigated or eliminated, as is shown in the next figure:

Of course, the actual disk operations are more complex than shown, especially with hundreds of users,but the general concept is valid; you want as few a number of disks as possible servicing a single readfor concurrency, while you want as many as possible for speed of access. It is a tough balancing act for


8/12

the IO subsystem due to the nature of physical disk drives and how the data is actually retrieved fromthe drives. Note that SSD-type systems usually eliminate blocking of this type.

The storage manager must understand how Oracle accesses the IO subsystem and how to best optimizethat IO subsystem for Oracle; one size doesnt fit all!

Memory Layer

The memory layer is the actual area where the database caches, pools, and other memory structuresreside.

Insufficient Memory Resources

Generally speaking, insufficient memory resource shows up as excessive physical reads. Usually thefirst indication in a properly designed system will be excessive db file sequential reads. If a system isusing automatic memory management it may also show as numerous deferred actions or excessiveswapping of memory between the shared pool components and the database cache areas.

Example of memory starvation indication:

Load Profile Per Second Per Transaction~~~~~~~~~~~~ --------------- ---------------

Physical reads: 5,789.69 10.86Physical writes: 1,073.95 2.01

Top 5 Timed Events Avg %Total~~~~~~~~~~~~~~~~~~ wait CallEvent Waits Time (s) (ms) Time----------------------------------------- ------------ ----------- ------ ------db file sequential read 16,555,520 119,937 7 71.3CPU time 30,420 18.1

-------------------------------------------------------------Another indication is when the cache analysis section of the AWR or Statspack indicates a large amountof change in physical reads from increasing the cache size:

Buffer Pool Advisory-> Only rows with estimated physical reads >0 are displayed-> ordered by Block Size, Buffers For Estimate

EstPhys

Size for Size Buffers for Read EstimatedP Est (M) Factor Estimate Factor Physical Reads--- -------- ------ ---------------- ------ ------------------

D 2,016 .9 63,756 1.0 25,481,605D 2,240 1.0 70,840 1.0 24,857,599D 2,320 1.0 73,370 1.0 24,642,801D 2,464 1.1 77,924 1.0 24,183,584D 2,688 1.2 85,008 0.9 22,843,987D 2,912 1.3 92,092 0.8 19,788,845D 3,136 1.4 99,176 0.7 16,978,075D 3,360 1.4 106,260 0.6 15,417,232D 3,584 1.5 113,344 0.6 14,522,845D 3,808 1.6 120,428 0.6 13,701,710D 4,032 1.7 127,512 0.5 13,237,721


9/12

D 4,256 1.8 134,596 0.5 12,826,268D 4,480 1.9 141,680 0.5 12,447,012

-------------------------------------------------------------

As you can see in the example shown, doubling the cache size (if possible) would result in at least a50% reduction in physical reads.

Correcting insufficient memory resources may be as easy as increasing the settings for AMM or the

floor values for the various components in the Oracle memory. However, if the system memory has beenexhausted then buying more memory and inserting it in the system is indicated. Many times this may notbe enough.

Access to memory and the back path to the IO subsystem can also be stressed if there is too muchmemory. This can be helped by switching to a RAC based system, thus spreading the memory and IOpathways amongst several servers. In a RAC environment excessive interconnect traffic can be a signthat the individual servers have insufficient memory.

Improper Memory Configuration

Within the Oracle memory area there are multiple caches and pools. The wary DBA should be utilizing

proper settings for the shared pool database cache, keep, recycle, and multiple block size caches.Additional settings for the large pool, Java pool, and streams pool should be used as needed in aparticular environment. Even when AMM is used, floor values should be set for these parameters toensure minimum levels of performance are met.

When using SGA_TARGET and SGA_MAX_SIZE or MEMORY_TARGET andMEMORY_MAX_SIZE, be sure they arent set equal to each other, because this leads to memorythrashing as the internal memory pools and caches are forced to release and reclaim memory, leading todeferred actions. I usually recommend that at least 1-2 gigabytes be between either set of settings. Ofcourse, as you gain operational experience for your systems needs this gap can be reduced, orincreased. The dynamic performance view, v$sga_resize_ops and the AWR resize operations section

can be utilized to see if you need to rethink the floor settings for specific parameters.

A typical v$sga_resize_ops query:

selectCOMPONENT, OPER_TYPE, OPER_MODE,INITIAL_SIZE, TARGET_SIZE, FINAL_SIZE,STATUS,to_char(START_TIME,'mmdd hh24:mi') start_time,to_char(END_TIME,'mmdd hh24:mi') end_timefrom V$SGA_RESIZE_OPS order by start_time

/


10/12

Example Output:Date: 01/31/08 Page: 1Time: 05:19 AM Component Resize Operations SYSTEM

suprac1 database

COMPONENT Oper OPER_MODE INITIAL TARGET FINAL STATUS START_TIME END_TIME-------------------- ------ --------- --------- --------- --------- --------- ------------ ----------shared pool SHRINK DEFERRED 130023424 125829120 125829120 COMPLETE 0128 01:00 0128 01:00

DEFAULT buffer cache GROW DEFERRED 29360128 33554432 33554432 COMPLETE 0128 01:00 0128 01:00DEFAULT buffer cache SHRINK DEFERRED 33554432 29360128 29360128 COMPLETE 0128 01:01 0128 01:01shared pool GROW DEFERRED 125829120 130023424 130023424 COMPLETE 0128 01:01 0128 01:01DEFAULT buffer cache SHRINK DEFERRED 29360128 25165824 25165824 COMPLETE 0128 01:02 0128 01:02shared pool GROW DEFERRED 130023424 134217728 134217728 COMPLETE 0128 01:02 0128 01:02shared pool SHRINK DEFERRED 134217728 130023424 130023424 COMPLETE 0128 01:09 0128 01:09DEFAULT buffer cache GROW DEFERRED 25165824 29360128 29360128 COMPLETE 0128 01:09 0128 01:09DEFAULT buffer cache SHRINK DEFERRED 29360128 25165824 25165824 COMPLETE 0128 01:15 0128 01:15shared pool GROW DEFERRED 130023424 134217728 134217728 COMPLETE 0128 01:15 0128 01:15shared pool SHRINK DEFERRED 134217728 130023424 130023424 COMPLETE 0128 01:49 0128 01:49DEFAULT buffer cache GROW DEFERRED 25165824 29360128 29360128 COMPLETE 0128 01:49 0128 01:49shared pool SHRINK DEFERRED 130023424 125829120 125829120 COMPLETE 0128 02:00 0128 02:00DEFAULT buffer cache GROW DEFERRED 29360128 33554432 33554432 COMPLETE 0128 02:00 0128 02:00DEFAULT buffer cache SHRINK DEFERRED 33554432 29360128 29360128 COMPLETE 0128 02:02 0128 02:02shared pool GROW DEFERRED 125829120 130023424 130023424 COMPLETE 0128 02:02 0128 02:02

In some databases, notably data warehouse (DWH) and decision support (DSS), it may be impossible toprovide enough cache to support all operations. However, in many cases the developers or the DBAs orboth may not be aware of all the options Oracle provides. Many times when reviewing the AWR orstatspack for DWH systems I see they are not utilizing star joins. Star joins will reduce 30 minutequeries to 30 seconds when utilized correctly.

In the case where you must have db file scattered reads (full table or index scans), you must make surethe IO subsystem is properly sized to handle the load. As was said in the section on physical IO, to getadequate performance from disk based systems it may be necessary to over buy the number of disksneeded to the tune of 30-40 or more times the number of disks needed for storage capacity alone to get

the needed number of IOPS. The major cause of the need for all of these drives is the contention andqueuing that result from too-few spindles. By utilizing low latency IO subsystems that dont block, suchas SSDs, you dont need to over buy storage capacity to meet IOPS needs.

In the case where is it not economical or feasible to add memory to a server or servers (RAC) thenreplacing some of the high latency disks with low latency IO subsystems such as large cache or SSDs isa method to get higher performance from your system.

CPU Layer

The CPU layer is where all of the processing occurs within a database. Of course now with theintroduction of extremely distributed processing as in the Exadata systems, this line is blurring because

some of the processing is actually being done at the storage level. In the future this may even blur moreas some companies are offering CPUs built into memory chips (8 CPUs on a single gigabyte memorychip).

Insufficient CPU Resources

I have heard it said that tuning usually pushes the log jam from one part of the information stream toanother; I have found this to be true. Once we tune the code, fix the IO subsystem, and fix the memoryissues we may have with our system, we usually end up with higher CPU utilization. Higher CPU


11/12

utilization in itself is not a bad thing, unless it leads to over utilization or overloading of the systemCPUs.When examining an AWR or Statspack report, I look at the balance between Busy, Idle, and IO Wait onthe CPUs in the system before making a recommendation as to how much an SSD will improveperformance. For example, if the percent busy is 30, the percent idle 20, and the IO wait percent 50, theneven if we reduce the IO wait to less than 5% we can still only reclaim 20 percent of performance losses.In order to get the maximum bang for the performance buck for any improvement to the IO subsystemabove 20 percent we would need to either tune to reduce CPU usage, add more CPUs, or replace theCPUs with ones that can sustain a higher level of operations.

Middleware or Application Server Layer

The middleware/application server layer may be were application logic is processed and may also act asa caching area for frequently used data.

Bad SQL Code Generators

Middleware and software configurations can be a real problem. Many times the local staff have no

control over the code that is delivered, especially with third party applications. The biggest problemsusually have to do with code generators that develop ad-hoc queries against the database. Too oftenthese code generators (actually, their developers) have no clue how to write optimal SQL for Oracle,they dont use proper joins, overuse outside joins, use distincts when they arent needed, and fail to usebind variables.

Improperly Configured Middleware/Software

Another common issue is these applications fail to allow for array processing when fetching results fromOracle. For example, SQLNet uses a default array size of 10; JAVA uses 15. If your result set is in thehundreds or thousands of values and you leave these settings at their default, you just got hundreds ofSQLNet roundtrips with the addition of the latency of the network, twice for each roundtrip, added to

your response time.

In one example, a company called me in to examine their application. It was taking 30 seconds to get aresponse and of course Oracle was getting the blame. Using the v$sqltext and v$sqlarea views I was ableto isolate the key SQL for specific forms and reports, then I used the Quest Benchmark Factory tomodel the application and was able to demonstrate that in over 90 percent of the scenarios they coulddescribe accurately the response at the database layer was less than 1 second for all queries. Furtherinvestigation showed the Cognos report layer was not using bind variables and was using the defaultSQLPlus arraysize of 10. This resulted in excessive recursive SQL and hundreds of SQLNet roundtripson a network where some areas were remote (India, South America, Europe) and have anywhere from100-500 millisecond network ping times.

Always test assuming data volumes will be hundreds or thousands of times your test set, unless you arelucky enough to test with full data sets. Always monitor the effects of all layers upon the others.

Summary

So what have we learned? In order to be holistically tuned, a transaction (insert, update, delete, or select)utilizes the minimum number of resources (Physical IO, logical IO, memory, and CPU) possible. Thismeans that the database must be properly designed, the IO subsystem adequate to handle both thestorage capacity and the IOPS needed with the proper latency, and the CPU and memory resources are


12/12

enough to handle the needed load. Once you start thinking holistically about performance and how theentire system, not a single statement, is tuned, you are on the way to better, longer lasting performance.

Holistic Tuning in Oracle

Documents

Transcript of Holistic Tuning in Oracle