Project MegaGrid: Performance Collection and Analysis in ... MegaGrid... · Project MEGAGRID:...

Project MegaGrid: Performance Collection and Analysis in Large Clusters An Oracle, Dell, EMC, Intel Joint White Paper December 2004

Project MEGAGRID: Performance Collection and Analysis in Large Clusters


Introduction ....................................................................................................... 3 Grid Infrastucture ............................................................................................. 4

Service Level Objectives and Thresholds.................................................. 7 Response Time ......................................................................................... 7 Throughput ............................................................................................... 7

Monitoring ..................................................................................................... 7 Performance Statistics and Metrics ............................................................ 8

CPU Consumption................................................................................... 8 I/O Volume/Throughput/Latencies.................................................... 8 Memory Consumption ............................................................................ 8 Network Throughput and Latency........................................................ 8 Execution and Commit Rate .................................................................. 9 Sessions and Connections between Mid-Tier and DB Servers ......... 9

Business Level ............................................................................................... 9 Transaction Throughput and Response Time ..................................... 9 Users or Jobs........................................................................................... 10

Database Alerts ........................................................................................... 11 Setting Service Thresholds.................................................................... 12

Tuning............................................................................................................... 13 Workload Classes ............................................................................................ 14

AWR/ADDM ............................................................................................. 14 AWR Snapshots...................................................................................... 15 Service Statistics...................................................................................... 15 EMC ControlCenter Performance Manager ...................................... 17 EMC Navisphere Manager ................................................................... 18 Storage Performance Management...................................................... 18

ADDM ......................................................................................................... 19 Advisories..................................................................................................... 20

Conclusion........................................................................................................ 21



INTRODUCTION To successfully manage the performance of an Enterprise Grid Computing environment, which will provide high quality service to applications and optimally utilize resources a thorough knowledge of the Grid environment as well as various components that make up the Grid is essential. Through project MegaGrid, Oracle, DELL, EMC and Intel invested together to jointly design, test and document enterprise Grid Computing best practices. These infrastructures are architected, configured, and validated in a Joint Development Center located in the Oracle Global IT Data Center in Austin, TX.

At the core, a Grid provides database services to the applications. It combines various resources like servers, storage, network, operating system, clusterware and database software. This integration needs to be implemented in a way that these resources can be provisioned on demand based on the service-level priorities of the applications.

The MegaGrid project environment is composed of standard high-volume servers such as the Dell PowerEdge 1750 or 1850 servers using Xeon processors, Dell PowerEdge 7250 servers using Itanium 2, SAN and NAS storage components by EMC, Oracle 10g RAC and ASM, iAS and various additional components to manage a grid. Two different clusters were built: a 24-node cluster with Dell PowerEdge 1750’s and a 12-node cluster with Dell PowerEdge 7250’s.

After the initial deployment it is necessary to develop a set of practical guidelines to capture baseline performance data, analyze differences in performance and drill down the different components of a Grid environment.

In this white paper we provide detailed practical guidance in deploying end-to-end performance collection and analysis.

Oracle Database 10g provides the features necessary to implement a database service in the Grid, such as

• Grid workload management software • Standard clusterware for commodity hardware • Automatic self-tuning and self-managing hardware • Automatic storage management in the Grid


• Global workload repository It can therefore leverage server and storage technologies such as those provided by Dell, Intel and EMC to provision application services in an optimal and high-performing way.

A Grid-computing environment created from these building blocks constitutes the basis for a more open and immediate approach to performance management. The capacity of a cluster can be extended on-demand and additional resources can be added incrementally. A global, application-service oriented view monitors the performance goals of an application and provides indications of additional capacity requirements. A flexible infrastructure allows for easy addition and/or provisioning of capacity to the application service that requires it, in time and at low cost.

To illustrate the approach to performance monitoring described here, two different application models will be used:

• A network provisioning system known as BMS • Several modules of the Oracle Application E-Business suite

GRID INFRASTUCTURE The basic concept of a Grid is to provide application, server and storage services. To achieve this goal it has to integrate different applications, different storage types and different server models and architectures.

To make that architecture happen certain requirements must be fulfilled:

• Universal connectivity

• Central monitoring

• Easy management of complexity

Universal connectivity is one of the foundations the Grid infrastructure is based on. It must be possible to provision storage when and where it is needed. It is obvious that to implement this architecture universal connectivity is needed.

Central monitoring and easy management is also one of the basic principles to make this environment manageable. The old way of management is not sufficient and a more automated approach is required.

We emphasize throughout this paper the usage of Oracle Enterprise Manager (OEM)/Grid Control and the automated database advisories.

The following figure shows these principles and the dependencies in more detail. On the top there are the application services that the Grid environment provides. Several of the application servers are providing each of the services. Some of the application servers are providing more than one application service. The application servers connect to the database.


The database defines the corresponding services inside the database. This makes a correlation between the business functions and the database operations possible. To have the flexibility to react to changes in load, the database grid has spare nodes available which can be assigned to database services as needed.

Figure 1: Grid Infrastructure

The database service is provisioning storage from Oracle Automatic Storage Management (ASM) as needed. The ASM Grid virtualizes the different types of storage devices that are available. Depending on the requirements different storage types are assigned.

A Grid infrastructure facilitates the managing of workloads based on business policies by dynamic resource sharing, continuous availability, standardization and complete automation.

• Dynamic resource sharing ensures that applications receive the resources that they need, based on priority.

• Continuous availability ensures that data and processing resources are available to applications when needed in the Grid.

• Standardization uses architecture-neutral hardware and software that makes all nodes and storage accessible.

CLARiiON CX NS600 DMX 1000

FC ATA

SAN SAN NAS

ASM

Payroll

Storage

Virtualization

Server Grid

User Applications

Self- Ser vice

Inventory Route Report

Inventory Status

DB Clusters App Clusters

Disk


• Complete automation maintains the quality of service for applications based on business policy.

With Oracle Database 10g, services define application workloads in a Grid environment. Each service represents an application with workload attributes, such as service level thresholds and a priority. Services enable the workloads to be measured and managed efficiently to deliver capacity on demand. Having the ability to deploy and re-deploy different services based on business polices allows businesses to run smoothly and efficiently. The Database Resource Manager allocates resources to these services based on their priority.

From the point of view of performance management, services

• Align a business model with lower-level, physical implementation, abstractions visible in the software layers of the application server and database tiers

• Represent workload class with similar performance requirements • “Insulate” performance critical tasks and services from the effects of

“perturbation” caused by other workloads executing on the same machines and therefore minimize variability

• Constitute an efficient approach for performance management and monitoring

This screenshot from OEM shows the “Cluster Top Consumers” display. The service module and action breakdown is very helpful in performance management. You are also able to drill-down to specific statistics.

Figure 2: Cluster Database Top Consumers


Service Level Objectives and Thresholds It is recommended to group services into classes with similar performance or resource requirements. The mappings of services to nodes should take the specific objectives into account. Two different service levels can be distinguished: response time and throughput.

Response Time

Critical on-line application services are usually aimed at minimizing the response times. If response times are

• Are CPU-bound, provision on the servers with the fastest CPU’s. Take resource usage of CPU per transaction into account.

• Are I/O-bound, provision on the storage subsystems with the best I/O performance characteristics

Additionally, response time can be improved by

• Separating the current workload from other services by redirection to spare computers if the resource utilization is critical

• Using low cost components at less than full utilization, without degradation of service times under load

Throughput

Batch jobs typically need to achieve high throughput and a reduction of the run length; the objective is to maximize the resource usage such as server CPU capacity and I/O capacity; services for batch processing are run off-peak and may steal resources from other systems; spare capacity in a cluster/Grid can be re-routed temporarily to provide more capacity.

These jobs are often very predictable and usually not subject to fluctuations and variability and their capacity requirements change slowly and can be predicted via trend/regression analysis; averages based on normal distributions; and the adjustments are usually seasonal and long-term

Oracle Database 10g Automatic Workload Repository (AWR) captures snapshots of performance data in a repository that can be used as a basis to determine requirements and adjustments.

Monitoring

In a Grid environment automatically monitoring the response time and the throughput of services is one of the most important tasks to maintain SLOs. The performance metrics and statistics can be divided into several sections. The statistics can be divided into two classes: system-level and component-level statistics. System-level statistics refers to the overall utilized capacity of a machine when running a certain application and the performance achieved. Component-


level statistics refers to the utilization and performance of a part of that system: e.g. the storage subsystem or the network.

Performance Statistics and Metrics The best way to monitor these statistics or to collect these statistic values is described in the parentheticals below.

CPU Consumption

Required to determine the server components, cluster topology and to estimate the additional CPU required for cluster coherency

• CPU utilization or consumption of a machine • Per user/per transaction

I/O Volume/Throughput/Latencies

Required to size the I/O capacity of a cluster of low-cost, high-performance servers, determine the number of nodes required to support a given I/O load and the choice of storage systems (Use the iostat command and additionally the tablespace and file I/O stats section of an AWR report)

• I/O rates in operations/sec and MB/sec • I/O sizes for reads and writes • I/O latencies for random I/O and sequential I/O • Read/write ratio

Memory Consumption

Required to decide what server architecture to choose, whether larger caches and 64-bit architectures should be preferred over 32-bit architectures; ultimately whether smaller, distributed caches are viable or may increase overhead

• PGA advisories (Check the “PGA Aggr Summary” and “PGA Aggr Target Stats” section of an AWR report. Also the underlying views V$PGASTAT, V$PGA_TARGET_ADVICE can be queried)

• SGA and buffer cache sizing and advisories: importance of right-sizing the buffer cache, especially when moving to a “distributed” cache supported by multiple instances on small machines (Check the “Buffer Pool Statistics” and “Buffer Pool Advisory” sections of an AWR report)

Network Throughput and Latency

• If interconnect information is available, use Oracle level statistics, otherwise estimate based on probabilities: e.g., x number of cache fusion requests per service/transaction etc. (Check the output of the netstat command and the “Global Cache Load Profile” section of an AWR report)

• Message flow rate and volume from application server to database server, SQL*Net traffic (Check the “bytes received via SQL*Net from client” and “bytes received via SQL*Net to client” statistics)


• Message flow from client program to application server

Execution and Commit Rate

Database level statistics that provide a profile of the workload from the database server perspective

• Ratio of Queries and DML, rate of execution (“Executes” and “User calls” statistic in AWR report)

Sessions and Connections between Mid-Tier and DB Servers

A large numbers of connections and frequent, on-demand spawning/forking of processes may be an additional load and cause overhead. On machines with a smaller number of CPUs, a smaller, controlled number of connections are recommended and connection pools should be used for efficiency.

Business Level

Transaction Throughput and Response Time

The MODULE and ACTION columns of the views V$SQLAREA and V$SQL enables us to correlate database statistics with business transactions. The MODULE and ACTION columns have to be endorsed by the corresponding application (e.g., a module from the Oracle E-Business suite).

Hint: If your applications are not yet setting the MODULE and ACTION fields, it is highly advisable to add that functionality. It makes performance diagnosis a lot easier, especially in Grid environments.

With Oracle Database 10g this information can be set with

• DBMS_APPLICATION_INFO PL/SQL package,

• OCIAttrSet OCI API call or

• With the setEndToEndMetrics method of the connection object in JDBC.

In JDBC the following code snippet can be used to set the MODULE and ACTION information:

String[] metrics = new

String[OracleConnection.END_TO_END_STATE_INDEX_MAX];

metrics[OracleConnection.END_TO_END_MODULE_INDEX] ="myModule";

metrics[OracleConnection.END_TO_END_ACTION_INDEX] ="myAction";

OracleConnection conn = ds.getConnection();

conn.setEndToEndMetrics(metrics, 0);


The function has been implemented with performance in mind - no extra SQL*Net message is sent. The information is transmitted with the next SQL*Net message.

In OCI the MODULE and ACTION information can be set with the OCIAttrSet function. The following code snippet shows an example:

OCIAttrSet(session, OCI_HTYPE_SESSION, (dvoid *)"mymodule", (ub4)strlen("mymodule"), OCI_ATTR_MODULE, error_handle); OCIAttrSet(session, OCI_HTYPE_SESSION, (dvoid *)"myaction", (ub4)strlen("myaction"), OCI_ATTR_ACTION, error_handle);

Also the OCI OCIAttrSet function does not send any additional SQL*Net messages. The PL/SQL implementation sends one additional SQL*Net message to the database server. The following example shows how the PL/SQL interface can be used.

DBMS_APPLICATION_INFO.SET_MODULE ( module_name => ‘mymodule’ action_name => ‘myaction’);

Hint: Currently there is no direct way to correlate a service with a SQL statement. It is advised to specify the module name in such a way that the name of the service can be identified from the module name. The service name is only part of the view V$SESSION.

Performance statistics about a specific service can be determined from the view V$SERVICE_STATS. The following SQL statement shows an example (output shortened).

SQL> select STAT_NAME, VALUE from V$SERVICE_STATS WHERE

SERVICE_NAME='ORDER ENTRY'

STAT_NAME VALUE

user calls 468355

DB time 8.1494E+10

DB CPU 2.8813E+10

cluster wait time 940141

concurrency wait time 190040

application wait time 107

user I/O wait time 3278953

Additionally CPU statistics for each service can be obtained with the view V$SERVICEMETRIC.

Users or Jobs

In addition to monitoring services there is also the possibility of getting performance metrics for a single user or job. The facility is also called end-to-end monitoring. This makes performance statistics collection even for multi-tier environments possible. The facility can be enabled with the DBMS_MONITOR


PL/SQL package or by using the corresponding OEM screens. By using the DBMS_MONITOR.CLIENT_ID_STAT_ENABLE procedure the statistics collection for one specific client or user is enabled globally. The statistics can be queried with the view V$CLIENT_STATS. The following SQL code shows how to enable client statistics.

execute DBMS_MONITOR.CLIENT_ID_STAT_ENABLE(‘<client id>’)

It is important that the client id that is selected as the parameter to the procedure is unique across all the sessions in the cluster (e.g. employee id). The other way to enable statistics for clients is based on Service, Module and Action. By using the procedure

execute DBMS_MONITOR.SERV_MOD_ACT_STAT_ENABLE

(<service name>, <module name>, <action name>);

client statistics can be captured for a service, a service and a module or down to the action level.

Additionally client tracing can be enabled. The following SQL code execute DBMS_MONITOR.CLIENT_ID_TRACE_ENABLE

(client_id => <client-id>);

enables client tracing. The functionality works across the cluster. It is the first time client tracing can be globally enabled for a three-tier environment. To check which clients have tracing enabled, the view DBA_ENABLED_TRACES can be queried. After the trace files are created the files can be analyzed with the trcsess utility. The advantage of this utility is that it can consolidate many trace files into one trace file. This is a very important functionality as potentially more than one server process and cluster node might run a service.

Hint: It is advisable to add the functionality to applications to turn on end-to-end statistics and tracing. This will make performance statistics collection based on specific users or jobs a lot easier.

Hint: It is important to choose a descriptive name for the client identifier. It is best to use a combination of the current user and service.

Database Alerts Server-Generated Alerts are messages that carry information about some event or state of an object notifying you of an impending problem, plus an optional suggestion for corrective action. The suggested action is a description of what the receiving user may perform to resolve the reported problem. What distinguishes them from traditional alerts is the fact that the database itself generates them in the most efficient and timely way possible by accessing the SGA directly. Database alerts also check that service level agreements are met. The alerts are created automatically; no intervention of the user is necessary.


The user can adjust the service level attributes. Alerts are automatically created when a problem occurs or performance statistics do not match expected metrics, such as the following:

• CPU time per call • Response time per call

Setting Service Thresholds

Thresholds are commonly set after you have observed your working systems over a period of time and decided on appropriate values to suit your business service quality requirements for each metric. The Oracle 10g alert infrastructure provides PL/SQL interfaces to manipulate their values.

The thresholds can be set with the procedure SET_THRESHOLD that is part of the PL/SQL package DBMS_SERVER_ALERT. The procedure allows setting two different levels: a warning level and a critical level. For services two thresholds can be set: CPU time per call and elapsed time per call. This example sets the alert for the elapsed time per call. The warning value is 0.5 sec. And the critical value is 0.75 sec. The values are specified in microseconds.

execute DBMS_SERVER_ALERT.SET_THRESHOLD(-

METRICS_ID => dbms_server_alert.elapsed_time_per_call,

WARNING_OPERATOR => dbms_server_alert.operator_ge,

WARNING_VALUE => '500000',

CRITICAL_OPERATOR => dbms_server_alert.operator_ge,

CRITICAL_VALUE => '750000',

OBSERVATION_PERIOD => 15,

CONSECUTIVE_OCCURRENCES => 3,

INSTANCE_NAME => 'I0n',

OBJECT_TYPE => dbms_server_alert.object_type_service,

OBJECT_NAME => 'ORDER_ENTRY');

Thresholds can also be set with OEM. The following screenshot shows that it is a lot easier to set service level thresholds with OEM.


Figure 3: Define Service Level Thresholds using Enterprise Manager Grid Control

This value must be set on each database instance running this service. The current active thresholds of service can be retrieved from the view DBA_THRESHOLDS. The view DBA_OUTSTANDING_ALERTS shows the outstanding alerts. But it is much easier to use the OEM screen to get this information.

Figure 4: Database Alerts

In summary by setting the correct thresholds for the CPU/Response time values for the corresponding services, performance management can be much simplified a lot. For a DBA it is now sufficient to only monitor the corresponding OEM screens and to react on reduced performance alerts before users start complaining.

TUNING With Oracle Database 10g a new methodology for tuning is introduced: instead of basing tuning on session and SQL, tuning is now based on Service and SQL. This has the advantage that each business process can be implemented as a database service and each of these workloads can in turn be measured independently and automatically.


It also makes the resource consumption of business transaction measurable and additionally the workloads can be prioritized depending on how important they are.

In a Grid environment it has to be stressed that the self-tuning capabilities are very important. Integrating all the components of a Grid makes it necessary to have a high level of automation. This high level of automation is supported by the various advisories in the database.

WORKLOAD CLASSES To make the analysis of performance statistics easier the statistics have been grouped into classes. These distinct groups classify different database activities to allow easier determination of resource consumption. Examples for the most important wait classes are:

• Application,

• Concurrency,

• I/O,

• Network,

• Cluster and

• Configuration.

At the database level there are several tools available to support you in this task: AWR, alerts and several different tuning advisories. Oracle Enterprise Manager also uses the workload classes.

AWR/ADDM AWR and Automatic Diagnostic Database Monitor (ADDM) are two tools that help in monitoring and analyzing the performance of a database Service. AWR can be described as a “Statspack built into the database kernel” AWR automatically collects performance statistics and metrics in a workload repository inside the database. A snapshot is collected every hour. Each snapshot is collected on all the nodes at the same time and has the same identifier.

Hint: Each snapshot has the same identifier. This makes correlating snapshots of different nodes easier.

Hint: The snapshot has to be collected on only one cluster node. The snapshots of all the other cluster nodes are also collected. This makes performance management much easier.

The snapshots are created in the tablespace SYSAUX. The snapshots are kept for seven days. Afterwards they are purged with the data of more current snapshots.


Hint: It is advisable to increase the size of the SYSAUX tablespace if you intend to keep the snapshots for more than seven days.

Hint: If you add more database instances to a database, it is recommended to add more space to the SYSAUX tablespace. Additionally it is a good best practice to export AWR statistics before they get purged.

AWR Snapshots

Additional snapshots can be created with the procedure DBMS_WORKLOAD_REPOSITORY.CREATE_SNAPSHOT. Reports of the statistics are created with the SQL script awrrpt.sql. The created reports are calculated in terms of response time.

Generally snapshots are deleted after 7 days. It is possible to collect snapshots that do not get deleted. These snapshots are called baselines. Baselines can be collected with the CREATE_BASELINE function provided by the PL/SQL package DBMS_WORKLOAD_REPOSITORY.

Hint: After the initial installation of the cluster and the application it is advisable to create a baseline. The baseline will never get deleted and is extremely helpful in determining differences in performance. After updating the application or database the same procedure is recommended.

Service Statistics

The AWR report includes a section on service-level statistics. In the service statistics section it shows the Database time and CPU statistics in addition to the physical and logical reads per second. The following section of an AWR report shows this information Service Statistics DB/Inst: CR64/cr646 Snaps: 535-536 -> ordered by DB Time -> us - microsecond - 1000000th of a second Physical Logical Service Name DB Time (s) DB CPU (s) Reads Reads --------------------- ------------ ------------ ---------- ---------- OE 2,677.3 693.3 3,081,767 3,763,519 SYS$USERS 6.9 3.4 1,370 64,724 SYS$BACKGROUND 0.0 0.0 611 1,304,542 OEXDB 0.0 0.0 0 0 -------------------------------------------------------------

This already shows which service is consuming most of the resources. It can also be determined whether most of the time is spent waiting or consuming CPU. If most


of the time is spent waiting, the service wait class statistics section can be used to drill-down further into the details. This shown in the next example:

Wait time Breakdown for Service OE (Order Entry) Service Name ---------------------------------------------------------------- User I/O User I/O Concurcy Concurcy Admin Admin Network Total Wts Wt Time Total Wts Wt Time Total Wts Wt Time Total Wts --------- --------- --------- --------- --------- --------- --------- OE 1592396 168442 18201 7766 0 0 5 -----------------------------------------------------------

The above report shows that most of the wait time is spent in User I/O. If we want to improve performance, the User I/O part has to be reduced. To analyze the User I/O wait time, the next step is to analyze the performance screens of ASM in OEM. We have to figure out if the reduction in performance can also be seen at that level. OEM offers several screens to monitor the ASM disk performance. The best starting point is to use the “Disk Group I/O Cumulative” performance page. The page shows the overall throughput, response times, number of I/O operations, and errors, if any. From the data and the graphs in this screen we can determine which disk group is responsible for the degradation.

From there we can drill-down to the disk group to get a more detailed picture.


Figure 5: Automatic Storage Management Disk Group I/O Performance

With the EMC performance tools ControlCenter, Navisphere and Celera Monitor the performance of the EMC storage platform can be further investigated.

The following sections outline the necessary steps after both the Oracle database statistics and the ASM monitoring screen of OEM indicated high response times for I/O requests. As outlined earlier, the Oracle Database 10g is equipped with various performance monitoring tools, including AWR (Automatic Workload Repository) and ADDM (Automatic Database Diagnostic Monitor). The outputs and reports from such tools can be used to identify the "hot spots" from the Oracle Database perspective (ASM disk members, data files, and table spaces). To correlate the information from the Oracle side with the storage-side information, the identification of the proper volumes on which the "hot spots" are stored is essential.

Once the "hot-spot" proper volumes are identified using the Oracle-provided tools, the performance can be checked using the EMC-provided tools as described below.

EMC ControlCenter Performance Manager

Performance Manager (formerly Workload Analyzer), a storage-resource monitoring and reporting application within the EMC ControlCenter family, collects, correlates, and graphically presents performance information for your end-to-end storage infrastructure—simplifying storage and system administration.


Figure 6: EMC ControlCenter Performance Manager The advantage of the toolset is that it provides a complete view of your SAN environment. It shows the mapping of host, switch and backend storage like CLARiiON and Symmetrix. Additionally it enables us to accomplish easy problem identification, performance tuning, and performance capacity planning.

EMC Navisphere Manager

Navisphere Manager, a WEB based, out-of-band (over IP network) management tool, is the primary facility for managing CLARiiON storage systems.

Through Navisphere Manager, storage usage statistics can be enabled. When usage statistics are enabled, usage counters are kept for the different storage system components, including the Storage Processors (SPs), LUNs and physical disk drives. Counters are updated at a user configurable frequency, with 60 seconds being the lowest period between updates. Utilization over a time period is calculated as running averages over the time period.

Storage Performance Management

Provided that there is no hardware-related problem, a storage performance problem typically is caused by application software. Often the grid service invokes many I/O operations, commonly writes, to a single or very few number of volumes. Under such circumstances, the following methods can be considered to resolve the problem:

• Additional storage provisioning with Oracle Database 10g with Automatic Storage Management

• Application-based load balancing • Storage reconfiguration and data migration


• Storage system migration

The first method is to improve the performance of the "hot-spot" storage volume itself by reconfiguring the volume, if applicable. This is to take advantage of the different performance characteristics of the RAID configuration options.

The second option, application-based load balancing, is dependent on the design of the schema as well as how the business logic is constructed.

The data migration can be performed using the Oracle Database 10g Automatic Storage Management (ASM) to migrate the data seamlessly while the service is online. Alternatively EMC SAN Copy can be used. However, storage-based migration typically requires that the application usage of the data be temporarily suspended.

Secondly, if the volume configuration is optimal and the configuration itself could not be the cause of the performance problem, it might be due to the lack of horsepower of the storage system itself. In this situation, storage system migration may be the better option.

Finally, if the problem is not attributed to application design (such as schema and business logic) but to the database layout, simply adding more storage as new disk members for ASM might remedy the situation by taking advantage of ASM's online data redistribution capability.

ADDM ADDM (Automatic Database Diagnostic Monitor) is a new tool in Oracle Database 10g and is based on AWR. By analyzing AWR snapshots it creates findings. Based on these findings it generates recommendations. The recommendations are not implemented automatically; it is still up to the user to decide which recommendation to implement. Either OEM or the DBMS_ADVISOR PL/SQL package can invoke ADDM.

Hint: It is recommended to use OEM to create ADDM reports.

Figure 7: ADDM Report Created by OEM

The following is the detailed ADDM recommendation from the above example.


RECOMMENDATION 5: SQL Tuning, 15% benefit (415 seconds)

ACTION: Run SQL Tuning Advisor on the SQL statement with SQL_ID

"767ax55645bqn".

RELEVANT OBJECT: SQL statement with SQL_ID 767ax55645bqn and

PLAN_HASH 199940852

SELECT DISTINCT CAM_node.NodeID AS NextNodeID FROM Node

Advisories Advisors are server centric modules that provide recommendations, within their realm, to improve resource utilization and performance for a particular database sub-component. Advisors provide the DBA with the necessary context sensitive information, related to the problems that are encountered. They complement the alert mechanism. The advisors may reference historical data in addition to current in-memory data in order to produce their recommendations. They are especially powerful because they provide vital tuning information that cannot be obtained any other way.

Performance Advisors such as SQL Tuning, SQL Access, Memory, Undo, and Segment are essential in identifying system bottlenecks and getting tuning advice for specific areas, on probable resolution paths.

The SQL advisory helps in tuning SQL statements. The difference from SQL optimizer is, that the SQL Tuning advisory has more time to determine a better execution plan. The following example continues the example from the ADDM section and shows how the SQL advisory can be used to optimize the SQL statement:

DECLARE

task_name VARCHAR2(30);

BEGIN

task_name := DBMS_SQLTUNE.CREATE_TUNING_TASK(

sql_id => 'ckmqpx4bdyngm',

plan_hash_value => '',

scope => DBMS_SQLTUNE.SCOPE_COMPREHENSIVE,

time_limit => 3600,

task_name => 'SQL STATEMENT 1',

description => NULL);

DBMS_SQLTUNE.EXECUTE_TUNING_TASK( task_name => task_name);

END;

With the above statement a new task for the tuning advisor is created and started. The important input parameters are the TASK_NAME and the SQL_ID of the statement to be optimized. The SQL_ID can be retrieved from V$SQL or an AWR/ADDM report. The result of the optimization can be retrieved with the following process:


SET LONG 10000 SET LONGCHUNKSIZE 10000 SET LINESIZE 100 SELECT DBMS_SQLTUNE.REPORT_TUNING_TASK('SQL STATEMENT 1') FROM DUAL;

The result is the new execution plan, if a better plan was found. By using the SQL statement advisor for the long running SQL statements, reduction in response times of more than 30% were achieved. The above process was presented to show the detailed process. The same can also be achieved with OEM; OEM is the recommended interface to run SQL Advisor.

CONCLUSION Once an Enterprise Grid Computing infrastructure is deployed, end-to-end performance management is critical to ensure the infrastructure continues to support application service level objectives. This requires a high degree of automation. This automation can be achieved by using the described tools AWR, OEM and others. Oracle Database 10g’s built-in server-generated alerts and Enterprise Manager’s propagation framework along with its browser-based interface provide the foundation to manage systems performance problems and database maintenance tasks. The various self-managing initiatives provided by Oracle Database 10g, assist in enabling automation of Oracle systems, helping to reduce manual intervention, lower costs and provide better quality of service.

By carefully planning and designing performance requirements and matching the various components to those performance requirements, we can effectively deploy a Grid Solution to achieve high quality of service levels. Through Project MegaGrid, Oracle, EMC, Dell, and Intel have pre-tested and validated the optimal combinations of technologies to design an Enterprise Grid environment where performance can be efficiently monitored, analyzed and tuned.

White Paper Title: Project MEGAGRID Performance Collection and Analysis in Large Clusters December 2004 Author: Sftean Roesch (Oracle) Contributing Authors: Michael Zoll (Oracle), Srinivasan Subramaniam (Oracle), Akira Hangai (EMC) Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA 94065 U.S.A. Worldwide Inquiries: Phone: +1.650.506.7000 Fax: +1.650.506.7200 www.oracle.com Copyright © 2003, Oracle. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL

INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

© Copyright 2004 Dell Inc. All rights reserved. Reproduction in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell. November 2004 Dell and PowerEdge are trademarks of Dell Inc. .Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others.

Copyright © 2004, EMC Corporation. All rights reserved. EMC is a registered trademark of EMC Corporation.

Portions copyrighted © 2004, Intel Corporation. All rights reserved. Intel, the Intel logo, Intel Xeon and Itanium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries

Project MegaGrid: Performance Collection and Analysis in ... MegaGrid... · Project MEGAGRID:...

Documents

Transcript of Project MegaGrid: Performance Collection and Analysis in ... MegaGrid... · Project MEGAGRID:...