FCM Performance Tuning

51
Oracle’s Hyperion Financial Close Management Performance Tuning Guide 1 Oracle ® Hyperion Financial Close Management September 2012 Performance Tuning Guide Contents Introduction .......................................................................................................................... 3 Financial Close Management Performance ............................................................................ 4 How to Use this Guide ......................................................................................................... 4 Performance Terminology ................................................................................................... 4 Workload Types .................................................................................................................. 5 Database Considerations ..................................................................................................... 5 Oracle and SQL Server Database Optimization ...........................................................................6 Oracle Enterprise Manager (for monitoring the Oracle Database) .................................................6 Tuning WebLogic Application Server Java Virtual Machine .................................................. 7 CPU Recommendations ...........................................................................................................7 Memory Tuning......................................................................................................................7 Monitoring the Java Virtual Machine (JVM) Memory ....................................................................8 JVM Monitoring - Sun JConsole ................................................................................................8 JVM Monitoring - Oracle Enterprise Manager ..............................................................................8 JVM Monitoring - JRocket Mission Control ..................................................................................9 Tuning SOA ......................................................................................................................... 9 Windows Operating System Memory Profile ........................................................................ 9 Tuning Operating Systems Parameters (for 32-bit Windows) ........................................... 10 PAE startup switch ............................................................................................................... 10 3GB startup switch ............................................................................................................... 10 Using Oracle JRockit Mission Control Monitoring ............................................................... 11 Enabling Mission Control Monitoring Support ........................................................................... 11 Enabling Flight Recorder Monitoring Support ........................................................................... 11 Connecting Mission Control to a WebLogic Server Instance ........................................................ 13 Recording Virtual Machine Metrics .......................................................................................... 14 Benchmarking Configuration ............................................................................................. 15 Environment Detail - Hardware .............................................................................................. 15 Environment Detail - Software ............................................................................................... 16 Environment Detail - Tuning .................................................................................................. 16 Close Manager Performance ................................................................................................ 17 Tuning SOA for CM ............................................................................................................ 17 Tuning Data Sources ............................................................................................................ 18 Benchmark Results for CM - Manual Testing...................................................................... 19

description

FCM Performance Tuning

Transcript of FCM Performance Tuning

Page 1: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 1

Oracle® Hyperion Financial Close Management

September 2012

Performance Tuning Guide

Contents

Introduction .......................................................................................................................... 3

Financial Close Management Performance ............................................................................ 4

How to Use this Guide ......................................................................................................... 4

Performance Terminology ................................................................................................... 4

Workload Types .................................................................................................................. 5

Database Considerations ..................................................................................................... 5

Oracle and SQL Server Database Optimization ........................................................................... 6

Oracle Enterprise Manager (for monitoring the Oracle Database) ................................................. 6

Tuning WebLogic Application Server Java Virtual Machine .................................................. 7

CPU Recommendations ........................................................................................................... 7

Memory Tuning ...................................................................................................................... 7

Monitoring the Java Virtual Machine (JVM) Memory .................................................................... 8

JVM Monitoring - Sun JConsole ................................................................................................ 8

JVM Monitoring - Oracle Enterprise Manager .............................................................................. 8

JVM Monitoring - JRocket Mission Control .................................................................................. 9

Tuning SOA ......................................................................................................................... 9

Windows Operating System Memory Profile ........................................................................ 9

Tuning Operating Systems Parameters (for 32-bit Windows) ........................................... 10

PAE startup switch ............................................................................................................... 10

3GB startup switch ............................................................................................................... 10

Using Oracle JRockit Mission Control Monitoring ............................................................... 11

Enabling Mission Control Monitoring Support ........................................................................... 11

Enabling Flight Recorder Monitoring Support ........................................................................... 11

Connecting Mission Control to a WebLogic Server Instance ........................................................ 13

Recording Virtual Machine Metrics .......................................................................................... 14

Benchmarking Configuration ............................................................................................. 15

Environment Detail - Hardware .............................................................................................. 15

Environment Detail - Software ............................................................................................... 16

Environment Detail - Tuning .................................................................................................. 16

Close Manager Performance ................................................................................................ 17

Tuning SOA for CM ............................................................................................................ 17

Tuning Data Sources ............................................................................................................ 18

Benchmark Results for CM - Manual Testing...................................................................... 19

Page 2: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 2

Test Detail .......................................................................................................................... 19

Results ............................................................................................................................... 19

Other Metrics....................................................................................................................... 22

Benchmark Results for CM - Multi-user Testing ................................................................. 24

Test Detail .......................................................................................................................... 24

Results ............................................................................................................................... 25

Other Metrics....................................................................................................................... 31

Account Reconciliation Manager Performance ..................................................................... 32

An Example of a Heavy Load ARM Configuration ............................................................... 32

Tuning Parameters – Oracle DB ............................................................................................. 32

Tuning Parameters – JVMs .................................................................................................... 33

Tuning ................................................................................................................................ 34

Results ............................................................................................................................... 34

Benchmark Results for ARM - Manual Testing ................................................................... 35

Test Detail .......................................................................................................................... 35

Results ............................................................................................................................... 35

Other Metrics....................................................................................................................... 36

Benchmark Results for ARM - Multi-user Testing .............................................................. 37

Test Detail .......................................................................................................................... 37

Results ............................................................................................................................... 39

Other Metrics....................................................................................................................... 49

Summary ............................................................................................................................. 50

Using Other EPM Products ................................................................................................. 50

Separating Out Products ................................................................................................... 50

Finally ............................................................................................................................... 50

Page 3: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 3

Introduction

This document is written for people who install and configure the components in an EPM/BI environment, and people who monitor and tune the performance of the components in an EPM/BI environment.

This document focuses on Oracle Hyperion Financial Close Management (FCM).

It is assumed that readers understand the administration and performance tuning fundamentals; for example, server hardware, web servers, java application servers, databases. Readers should also be familiar with the how the Financial Close Management application is used in their environment. It is also assumed that the reader understands the roles of the different components within an EPM environment needed for each EPM Products being used. Components you often find within an FCM based EPM environment include Oracle HTTP Server (OHS), WebLogic, Shared Services, ERPi and SOA.

Financial Close Management as of 11.1.2.2 contains two distinct feature-sets: Close Manager (CM) and Account Reconciliation Manager (ARM). From a performance and tuning perspective, these feature-sets are regarded as integrated entities, but have different performance tuning requirements.

This document is divided into three main sections:

1. Financial Close Management Performance - Covers performance issues common to both feature-sets of FCM

2. Close Manager Performance - Covers issues mostly unique to CM 3. Account Reconciliation Manager Performance - Covers issues mostly unique to ARM

At the end of each section, benchmarking details and results are included to give the reader an idea of typical end-user performance and system resource usage. The “Account Reconciliation Manager Performance” section also contains a set of results for ARM used to generate the tuning recommendations.

As a result of the different tuning requirements for CM versus ARM, there are some tuning recommendations which vary slightly between the CM and ARM sections in this document.

The information presented in this guide is a clear indicative on tuning a system to support large loads.

For example, to look for a tuning recommendation to run a FCM schedule with 2000 tasks, you would look for the examples given in this guide along with the hardware information. From this information you should arrive at a close estimate on the things that you need for your environment. Start with making those changes to the system. Also see the Oracle Enterprise Performance System Installation and Configuration Troubleshooting Guide.

Performance tuning is an exercise that is influenced by a large number of external factors and the limitations and constraints of different tech-stacks in use.

At the end of this document there is a summary section which applies to both CM and ARM.

Page 4: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 4

Financial Close Management Performance

How to Use this Guide

The information presented in this document, including the benchmark results, are intended as a guide, and do not guarantee a given performance level.

Actual implementations and environments will vary widely based on business requirements, the user’s FCM data set, network topology, and hardware usage. Therefore, users must consider how to adapt these guidelines to their own implementations.

All tuning information stated in this guide is only for orientation; every modification has to be tested and its impact should be monitored and analyzed. All test results and performance numbers are only intended as examples to illustrate tuning concepts.

Before implementing any of the tuning settings, it is recommended to carry out end to end performance testing to obtain baseline performance data for the default configurations, make incremental changes to the tuning settings and then collect performance data. Otherwise it may impair system performance.

It is recommended that you have capacity planning, pre-go-live performance testing and ongoing performance monitoring strategies in place to assure that your implementation will meet your expected user requirements both initially and throughout its operational lifetime. This document will assist you in doing this, but does not negate the need for these processes to be in place.

Caution: Improper settings and configurations may prevent Financial Close Management from working.

Information about system requirements and supported platforms for all EPM System products is available in spreadsheet format in the Oracle Enterprise Performance Management System Certification Matrix. This matrix is posted on the Oracle Fusion Middleware Supported System Configurations page on the Oracle Technology Network (OTN):

http://www.oracle.com/technetwork/middleware/ias/downloads/fusion-certification-100350.html

Performance Terminology

This guide uses the following performance terms:

Scalability - The system's ability to perform within specification under increasing user load, data load and hardware expansion.

Latency - The time between the issuing of a request and the time when the work actually begins on the request.

Think time - The time a real user pauses to think between actions.

Resource utilization - A consumption metric, for example, the percent of CPU usage.

Response time - A time metric, for example round-trip time it takes the server to deliver a Web page.

Throughput - A rate metric (requests per unit of time), for example, requests per second, bits per second. For example, if an application can handle 20 customer requests simultaneously and each request takes one second to process, this site has a potential throughput of 20 requests per second.

Understanding Key Performance Drivers - A factor that dictates performance is called a key performance driver. Knowing how the drivers behave in combination further enhances your ability to

Page 5: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 5

deploy Oracle Hyperion EPM System optimally, based on the unique requirements of each deployment.

Hardware Capacity - Factors such as number of servers, quantity and speed of processors, available RAM, network speed.

Technical Platforms Tuning - Fine tuning other third party software required for installing and running Oracle Hyperion EPM products; for example: relational databases, Java application servers, Web servers, Server / Client Operating System and browsers.

Business Application Design - Application design is an important factor in system performance, (for example, structure, size, and use of product features in designing application databases, reports, schedules, profiles, templates, copy-to-period, data-load.)

Business Process Usage - Activities carried out by users in the normal flow of your business cycle. Business process usage has three components:

User Activity - Activities available to users for data load or data entry, database processing (for example, consolidations, copy, clear), and reporting and analysis

Rate of User Activity - A number of transactions executed by one user per one hour

User Concurrency - Number of users for each activity being carried out simultaneously

Workload Types

FCM has several types of workloads to monitor:

There is the pure interactive workload of a user performing something in the FCM Web user interface (for example, within EPM Workspace) that results in activity on the servers until the point the users gets a confirmation and/or result in Workspace. During this time the user normally waits for the reply before continuing and no background processing continues after the user gets the reply.

Background tasks are run (automatically or trigged by the user via Workspace or from an external event in another product). Background tasks may not directly interact with users via Workspace, although they may check progress from within Workspace indirectly. We can also regard some long running Workspace foreground tasks as background tasks, such as the Import & Export functions. Background tasks often involve large sets of data.

Some tasks are a mix of the previous two. Changing the status of some objects within FCM will just “ask” SOA to schedule the work; therefore this is an interactive task that triggers a background task.

From a performance and capacity planning point of view, these different types of workloads have different tuning objectives. Background tasks tend to be “batch”, similar to the way all available resource are used until the work is complete – their driver being the volume and complexity of the data being processed. Performance delays with interactive or foreground tasks are more noticeable to users, and tend to have a lower resource impact per user on the system.

Database Considerations

During installation the installation engineer will typically configure different databases (SQL Server) or schemas (Oracle) for different EPM products and components. For example (XXX_SOAINFRA, XXX_MDS & XXX_ORASDPM) are normally used for SOA in an Oracle DB.

In case the database is also used for hosting schemas of other products, you must take these schemas into considerations when changing parameters so that the change will not affect the functioning of other products. Most DBAs configure the DB to use a dedicated instance for all EPM

Page 6: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 6

products in the one environment, and use separate databases or schemas within the instance for each of the EPM products and components.

For good performance, including the ability to cope with large or complex datasets, the Database Administrator (DBA) engineer must optimize the database instance for CM and ARM.

Optimization includes:

Placing the database files on suitable disks

Setting database parameters (init.ora / spfile in the case of Oracle)

Running routine maintenance tasks

Oracle and SQL Server Database Optimization

The administrator performs routine database optimization to preserve optimal performance after operations in which large amounts of data are entered into the system or copied within the system. In the case of an Oracle DB, this optimization normally equates to running “dbstats” against the relevant schema.

In Close Manager, optimize after:

A template with a large number of tasks has been deployed as a schedule

A large number of tasks have been imported into a template or a schedule

In Account Reconciliation Manager, optimize after:

A data load from ERPI has been performed

A large number of transactions have been imported

A large number of reconciliation profiles have been copied to periods

To optimize Financial Close Management tables, use an available SQL connection tool and connect using the Financial Close Management schema user and execute:

Gathering schema statistics using Oracle:

where 'FCC' is replaced with the schema name appropriate for the ARM schema

Gathering schema statistics using SQL Server:

where is the name of the database that has the ARM.FCM schema.

Oracle Enterprise Manager (for monitoring the Oracle Database)

The Oracle Enterprise Manager database console can be used to analyze the performance of the Oracle database. It provides alerts automatically when poor performance is detected via the Automatic Database Diagnostic Monitor. The database console is launched via the URL:

Page 7: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 7

The ADM Performance Analysis section displays current alerts and the Performance tab contains charts illustrating database performance.

Tuning WebLogic Application Server Java Virtual Machine

The WebLogic Application Server is bundled with the JRockit Java Virtual Machine which provides optimal performance for the Financial Close Management and ARM applications. The JRockit JVM requires the least configuration because it is self-tuning based upon available resources and current workload. The JRockit JVM also supports extensive performance monitoring and incident diagnosis using the JRockit Mission Control utility.

The WebLogic Application Server also supports use of the Sun Java Virtual Machine and this JVM supports performance monitoring via the bundled JConsole utility as well as general monitoring using the Mission Control utility. Both Java virtual machines can be monitored using the Oracle Enterprise Manager.

CPU Recommendations

For best performance under moderate to heavy application usage by a large user base, ideally each WebLogic Java virtual machine process would have a CPU core available for its use. This guideline should be kept in mind when planning the topology of deployments. For instance, a server with 4 CPU cores could support the Foundation Services, SOA, and ARM WebLogic managed servers adequately. If other EPM products, for example, HFM, are part of the installation, 8 CPU cores would be preferable, as services (for example, the product specific WebLogic managed servers, Microsoft IIS, the HFM data source engine) are part of the environment and can be heavy on CPU usage.

Both the JRockit and Sun Java virtual machines will automatically utilize available CPU power, so no tuning is necessary by the administrator.

If “hyperthreading” is an available option for the server’s CPUs, it should be enabled via the BIOS setup. Although check with each EPM product’s documentation, as some EPM products which perform many floating-point operations may be negatively affected.

Memory Tuning

The goal of Java virtual machine memory tuning is to find a balance between resource consumption and optimal performance under moderately heavy usage. It will also prevent instability that has the potential to arise when a virtual machine must contend with unusually heavy concurrent usage. When a virtual machine is faced with tight memory constraints, more CPU time will be spent recovering usable memory and overall performance will degrade. Conversely, drastically over-allocating memory resources can also have a slight negative performance impact so choosing reasonable constraints is important.

The JVM associated with the ARM can accommodate a user base of 175 concurrent active users with a JVM memory allocation of 4GB. A small user base of 10 to 15 concurrent active users can be supported with a JVM memory allocation of only 650M.

The VM memory allocation is primarily controlled by two settings, the initial memory allocation and the maximum memory allocation, specified by the XMS and XMX JVM startup parameters. The values for these parameters come from the following files:

Page 8: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 8

For best performance, it is recommended that both the initial memory allocation value and maximum memory allocation value are set to the same amount. This reduces memory fragmentation and it bypasses part of the VM memory manager processing which dynamically requests memory from the operating system as usage increases.

The Sun Java virtual machine has an additional memory parameter, , which needs to

be set to a sufficient value to accommodate memory-intensive operations. The HIT installer provides an appropriate default value, but if out-of-memory errors are experienced, this value can be increased in the and (or ) files.

Monitoring the Java Virtual Machine (JVM) Memory

Several tools exist which allow the administrator to monitor the performance of a Java virtual machine and to observe how much CPU time is being spent reclaiming memory. All of the following tools have memory usage charts as well as counters which indicate how often memory reclamation occurred and how much time was spent reclaiming memory. Java virtual machine memory is subdivided into two main pools: a pool for short-lived and “young” objects known as the nursery, and a pool for long-lived, permanent, or “old” objects known as old space. Each pool has a “garbage collector” housekeeping process which reclaims memory periodically and this process consumes some CPU power. Ideally garbage collections should be infrequent and fast and the following tools allow the administrator to determine if the garbage collection is running too often or requiring too much CPU.

JVM Monitoring - Sun JConsole

The utility Windows executable or Linux script is located in:

JVM Monitoring - Oracle Enterprise Manager

The Oracle Enterprise Manager (OEM) console requires that the WebLogic Admin server is running. The EM console is accessed via the URL

Navigate to , right click on FinancialClose0 and select Performance Summary

Page 9: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 9

The default display includes a memory usage chart. Click “Show Metric Palette” to add additional metrics. Recommended metrics to observe JVM performance are:

Typically large chunks of the nursery memory will be consumed and freed very often while the old space memory, if the JVM is tuned properly, will remain fairly consistent, unless a large number of users are logging in or logging out of the system and sessions are being created or eliminated.

JVM Monitoring - JRocket Mission Control

Far more in-depth JVM monitoring can be performed if you are using the default JRocket JVM. You can use the Oracle JRocket Mission Control. This can also be used if you are using the Sun JVM, but the “flight recorder” option will not be available. To see more on this refer to the “Using Oracle JRockit Mission Control Monitoring” information at the end of this section.

Tuning SOA

How each feature set uses SOA is different, so the intended use of Financial Close Management (CM or ARM) will influence the tuning process.

High load with CM tends to have a bigger impact on SOA than the Financial Close Management JVM, whereas higher impact on ARM tends to only impact the Financial Close Management JVM. This impact is in the form of memory use and CPU use.

The main item to tune for SOA is the maximum number of connections each data-source can have open in its connection pool. Monitoring and tuning of these connections is via the WebLogic Console.

Although ARM does not use SOA as much as CM, ARM still uses many JDBC / data-source connections.

Also, make sure to monitor the total number of sessions that your database can accept so that none of the data-sources run short of connections / sessions to connect to the database.

Refer to the link below for the standard database tuning recommendations for the SOA server:

http://docs.oracle.com/cd/E23943_01/doc.1111/e17364/tuning.htm

Windows Operating System Memory Profile

The Windows Server operating system allows itself to be tuned:

With memory allocation favoring running programs

With memory allocation favoring the disk cache

For the purposes of running the WebLogic application server or a database service, it is preferable to provide more memory by tuning the operating system for memory allocation for running programs.

Page 10: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 10

To adjust the operating system:

1. Open the Control Panel, then open the System item, and then select the Advanced tab. 2. In the Performance section, click Setting to invoke the Performance Options dialog. 3. Select the Advanced tab and in the Memory Usage section, select Programs.

Tuning Operating Systems Parameters (for 32-bit Windows)

The most useful operating system tuning that you can perform for the ARM applications is to free up memory for use by the Java virtual machines and the database services. On 32-bit servers this is critical as processes are limited to 4G of virtual memory each (for all threads).

64-bit operating systems do not have this 4GB process limit.

To provide EPM application services the maximum amount of memory, use the following means:

PAE startup switch

The physical address extension startup switch enables 32-bit Windows Servers to fully access all of the 4G of memory and it allows enterprise editions of 32-bit Windows Servers to utilize more than 4G of memory, if the server hardware supports it. Many versions and service packs of Windows Server have the switch enabled by default.

To enable the switch:

1. Open Windows Explorer and select Tools, and then Folder Options. 2. Select the View tab, and verify that you can view hidden and operating system files in

Windows Explorer. The hidden operating system file resides in the root directory of the boot drive.

3. Edit the file with a Windows notepad or a text editor. 4. Add the switch to the boot configurations listed under the operating systems section.

3GB startup switch

The 3GB startup switch enables 32-bit Windows Servers to provide more memory to programs and services by relinquishing memory normally reserved for use by the operating system. Again this is most critical for 4G servers because by default 2G are reserved for programs and services and 2G are reserved for the OS. This default limits the amount of memory that can be specified for a JVM virtual machine: the memory pools and VM core cannot exceed 2G. The 3GB switch raises that constraint and this is necessary if an extremely large user community, such as 400 concurrent users accessing a 4G 32-bit server.

To enable the switch:

1. Open Windows Explorer and select Tools, and then Folder Options. 2. Select the View tab, and verify that you can view hidden and operating system files in

Windows Explorer. The hidden operating system file resides in the root directory of the boot drive.

3. Edit the file with a Windows notepad or a text editor. 4. Add the switch to the boot configurations listed under the operating systems section.

5. The 3GB switch can be further customized by adding the switch which allows the

administrator to precisely specify how much memory is relinquished from the operating system memory. For example, the file might be modified with the value which

allows the operating system 102M more than the 3GB switch alone.

Page 11: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 11

A side effect of using the 3GB switch is a reduction in the operating system’s Free System Page table. This number of free system page table entries should not go below 7,000 and an ideal number is 24,000. The number of free page table entries can be viewed using the Performance Monitor tool. Add the Memory counter Free System Page Table Entries and make note of the values for minimum and average. You can then use the switch to achieve a desirable amount.

Using Oracle JRockit Mission Control Monitoring

The JRockit Mission Control tool provides support for in-depth monitoring of the JRockit Java virtual machine’s environment and its performance as well as providing the ability for product support and developers to analyze runtime behaviors and diagnose failures. It complements the Enterprise Manager because it provides out of the box ability to record metrics over a period of time and to preserve those metrics in a file which can be forwarded to support and then to development. This provides an alternative to the Oracle Grid Control which is a significantly larger installation with much greater requirements.

Oracle JRockit Downloads:

Enabling Mission Control Monitoring Support

By default, the server nodes are installed without support for Mission Control monitoring enabled. Server startup scripts must be modified in order for the Mission Control tool discover and connect to JRockit virtual machine instances. An extra initialization parameter is added to the arguments which are passed to the virtual machine executable.

To add the extra initialization parameter:

1. Locate the startup script for non-Windows

servers) The file is typically in a location such as:

where [Weblogic Domain] is the directory of the domain which you want to monitor. 2. Locate the line with the following text:

3. Add the following argument to the end of the line, separated by a space:

Enabling Flight Recorder Monitoring Support

To enable Flight Recorder Monitoring support:

By default, the server nodes are installed with support for JRockit flight recorder monitoring disabled.

To enable the support:

1. Modify Server startup scripts for the Mission Control tool to activate flight recorder monitoring.

2. Remove the initialization parameter from server startup scripts.

3. Locate the startup script ( for non-Windows servers). 4. The file is typically in a location; for example:

where [Weblogic Domain] is the directory of the monitored domain. 5. Make a backup copy of the file.

Page 12: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 12

6. Make the following modifications: For , remove the following block:

7. Locate the startup script setCustomParamsFinancialClose.bat (setCustomParamsFinancialClose.sh for non-Windows servers) The file is in the location:

8. Make a backup copy of the file and locate and remove the text: .

Page 13: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 13

Connecting Mission Control to a WebLogic Server Instance

In the Mission Control application, a list of discovered WebLogic server nodes is displayed:

If it is unclear which server node is the desired node, select a node, choose Start Console, and chose the Runtime category. In the System Properties panel, locate the property.

Page 14: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 14

Recording Virtual Machine Metrics

Select the pertinent WebLogic server node, right-click and select Start Flight Recording.

You can select from a predefined profiling template and enable or disable specific metrics based on the type of analysis being performed. Once recording is enabled, the recording will be displayed in the Flight Recorder Control panel of the Mission Control interface. Recordings can also be opened thru the File menu.

Page 15: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 15

Developers can benefit from the recording’s code profiling that identifies heavily utilized classes or from the event profiling which quantifies time spent; for example, by garbage collection, blocked threads, code compilation.

Benchmarking Configuration

During product development tests are run to check the performance of Financial Close Management. Although these tests are designed to provide feedback to the Development Team, this information can be used as a rough “benchmark” guide by customers and partners implementing FCM.

Later in this document there are two “Benchmark Results” sections showing the results of these tests for CM and ARM respectively. The tests were performed on the same test environment. Details of this environment are covered here.

Environment Detail - Hardware

Two servers were used for the testing, excluding the server used to drive the multi-user test (workload injector) – for example, this is a “single-node” FCM environment.

Server environments:

pegbvps06:

o Dell PowerEdge 1950 v3 o 2 x quad-core Intel Xeon L5420 2.5GHz o 16GB of memory o 250GB local SATA disk o Gbit NIC o Windows 2008 R2 64bit Enterprise

pegbvps12:

o HP DL180 o 2 x quad-core Intel Xeon E5430 2.66GHz o 8GB of memory o Hardware RAID controller with 12 x 450GB SAS disks o Gbit NIC o Windows 2003 R2 64bit Enterprise o Oracle 11gR2 database o Database files all stored on RAID0+1 LUN

Page 16: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 16

Environment Detail - Software

All software installed on pegbvps06; for example, a “single-node” configuration.

EPM PS2 (11.1.2.2) GA release

ARM component upgraded to PSU300 pre-GA release (build#1613)

Essential Financial Close Management components installed: o Financial Close Management (ARM and CM feature-sets) o Foundation (Hyperion Shared Services + Workspace) o SOA o WebLogic o The embedded Web server was used (not IIS or OHS) o ERPi/ODI was not installed.

Data entry to Financial Close Management was entered directly to load the Financial Close Management database tables.

Environment Detail - Tuning

The following tuning was performed:

Oracle database:

In some earlier testing of ARM for PSU300, it was necessary to increase the number of OPEN_CURSORS in Oracle from the default of 300 to 6000 for large concurrent user volumes. However, this is less of an issue with the later (and so GA release) versions of PSU300.

WebLogic data source connection pool maximums (set higher than likely to be needed in a real production environment):

accountreconciliation_datasource = 300

EDNDataSource = 200

EDNLocalTxDataSource = 200

EPMSystemRegistry = 300

mds-owsm = 200

mds-soa = 500

OraSDPMDataSource = 500

SOADataSource = 500

SOALocalTxDataSource = 500

In Financial Close Management

In setSOADomainEnv.cmd:

Page 17: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 17

Close Manager Performance

This section contains information specific to Close Manager. It needs to be read in conjunction with the Financial Close Management Performance section.

Tuning SOA for CM

This section describes the SOA tuning that was tested under the following conditions:

Financial Close Management was configured on a clean machine, on a single node installation

The default configuration settings were used for Oracle DB, WebLogic server SOA

After executing multiple rounds of iterations with increasing numbers of Financial Close Management tasks that start at the same time, various parameters that might affect the performance have been carefully studied and documented below. The different values indicated below should be considered as directives and should work for most of the loads. However, the hardware that you configure Financial Close Management (single node or multimode) can change these parameters.

The following parameters can be used as a starting point in tuning the Financial Close Management system to achieve maximum productivity.

Most of the tuning parameters indicated in this document are taken from the ‘Oracle FMW Developers guide for Oracle Application Integration Architecture Foundation Pack’

http://docs.oracle.com/cd/E23943_01/doc.1111/e17364/tuning.htm

Multiple tests have been made with different numbers of Financial Close Management tasks that start at the same time (300, 500 and 1000) and it has been observed that the default setting that comes along with configuration of the SOA server for the first time would handle the load. The only changes needed are changes to the data sources connection pool sizes.

Page 18: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 18

Tuning Data Sources

The recommended number of connections for the various data sources that are seen after completely configuring Financial Close Management are listed in the table below. Use the below numbers in the ‘Maximum Capacity’ field for each data source. With the following numbers, Financial Close Management could successfully start 300 Financial Close Management tasks at the same time.

Make sure to increase the total number of sessions that your database can accept so that none of the data sources run short of sessions to connect to the database. The number of sessions that a database should make available should be greater than the Sum of (Number of connections specified for each data source times the number of managed servers that a data source is targeted to).

Data Source Default Numbers Suggested Numbers

Initial Capacity

Maximum Capacity

Initial Capacity

Maximum Capacity

EDNDataSource 0 20 0 20

EDNLocalTxDataSource 0 20 0 20

EPMSystemRegistry 0 150 0 150

financialclose_datasource 1 30 1 30

mds-owsm 0 15 0 15

mds-soa 0 50 0 50

OraSDPMDataSource 0 200 0 20

SOADataSource 0 50 0 100

SOALocalTxDataSource 0 50 0 50

Page 19: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 19

Benchmark Results for CM - Manual Testing

Test Detail

1000 tasks were imported (from CSV file) into a new empty schedule. This created a simple single-level 1000 task schedule, all of the tasks being overdue. The schedule was then opened so that all the 1000 tasks would be opened at the same time. This open puts a strain on the server, as both the SOA and Financial Close Management JVMs and the back-end database server, try to cope with the immediate demand.

Results

Task Step Time (minutes : seconds)

import tasks 0:20.2 open schedule 0:25.5 processing1 1:18.3 processing2 1:26.0 “Open schedule" ends when the UI "opening schedule" dialogue closes.

processing1-The initial single-threaded background processing by Financial Close Management and SOA.

processing2-The multi-threaded background processing.

After processing2, the tasks are all marked as open, before this they will have a status of pending.

Page 20: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 20

Key:

Black = total CPU, max 97.802 over all 8 cores

Green = CPU for Financial Close Management JVM (10th scale), max 281.810 (i.e. almost 3 cores)

Blue = CPU for SOA JVM (10th scale), max 572.872

Red = CPU for Foundation JVM (full scale), max 17.880

Yellow = CPU for WL JVM (full scale), max 45.057

Dotted = system-wide run queue, max 35

The above listing clearly shows the high CPU use of this task.

This starts at 13:58:20 (the "processing1" stage).

A little over a minute later the work becomes multi-threaded (using all the available CPU resource available on the server - the "processing2" stage).

This work for processing2 is performed by SOA (in blue) and Financial Close Management (in green) and even some WL (in yellow).

Note: Be aware of the different scales when reading this graph.

Page 21: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 21

The graph below shows the CPU and Heap use for the SOA JVM:

Page 22: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 22

The graph below shows the CPU and Heap use for the Financial Close Management JVM:

Other Metrics

After a number of multi-user tests and the above test was run a number of times, the servers key memory metrics were as follows:

Free memory = 3961 MB (physical memory is 16GB). Committed bytes = 12589 MB (this is the server's overall VM use)

Page 23: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 23

The following screenshot shows the state of the WebLogic data sources after the test finished. Some of the values are the result of earlier system use, but it gives an idea of how high values do not get even when doing a heavy background task:

Page 24: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 24

The following screenshot shows the composites and instances in SOA that resulted from a 1000 task schedule being created and opened:

Benchmark Results for CM - Multi-user Testing

Test Detail

LoadRunner was used to simulate 100 concurrent web users (vusers) accessing the server. All vusers were running the same LR script.

The script executes the following tasks for each vuser:

1. Open browser and access EPM Workspace home page.

Page 25: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 25

2. Log on to Workspace. 3. Open CM application, bringing up task list. 4. Open a task (each vuser uses a different task). 5. Click Add Comment. 6. Enter comment text. 7. Click OK to save the comment. 8. Close task dialogue, returning to task list. 9. Log off, returning to Workspace home page. 10. Close browser (but do not clear browser cache). 11. Wait for between 60 and 90 seconds randomly - this is called pacing. 12. Repeat the above task sequence from the start.

Each vuser pauses between 1 and 3 seconds randomly to simulate the pauses real users would have - this is called think-time.

Vusers were introduced to the system in two batches of 50 with a 10 minute delay between the two batches. Both batches start a new vuser at the rate of one per 20 seconds. Each vuser starts the above sequence of tasks as soon as it is started. Once all 100 vusers are on and running, the test continues for one hour, with all 100 vusers cycling through the task list repeatedly. After the hour the ramp-down starts, stopping each vuser at a rate of 5 per 30 seconds.

Results

Scenario Name: cm_scale100.lrs Duration: 1 hour, 53 minutes and 57 seconds. Maximum Running Vusers: 100 Total Throughput (bytes): 2,988,414,356 Average Throughput (bytes/s): 437,030 Total Hits: 645,059 Average Hits per Second: 94.334 Transactions: Total Passed: 42,152 Total Failed: 0 Total Stopped: 0 Transaction Name Minimum Average Maximum Std. Dev. 90% Pass Fail Stop

a01_001_home 0.015 0.04 0.484 0.027 0.043 5,269 0 0

a01_002_logon 0 0.033 0.738 0.024 0.043 5,269 0 0

a01_003_cm 0.453 0.534 1.266 0.095 0.693 5,269 0 0

a01_004_opentask 0.246 0.313 1.188 0.072 0.334 5,269 0 0

a01_005_addcom 0.078 0.136 0.828 0.062 0.203 5,269 0 0

a01_006_com_ok 0.031 0.081 0.859 0.063 0.16 5,269 0 0

a01_007_close 0.063 0.215 1.219 0.228 0.554 5,269 0 0

a01_008_logoff 0.016 0.047 0.747 0.037 0.053 5,269 0 0

Page 26: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 26

The following graph shows how the users were introduced during the test run:

The following graph shows how the transaction response times changed over time.

Note: There is no significant increase in response times as more users were added.

Page 27: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 27

The following graph shows how the transaction response times changed over time:

Note: There is no significant increase in response times as more users were added.

Page 28: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 28

The following graph shows the impact of the test on selected server metrics. It is seen that the only process making any real use of the CPU is the Financial Close Management JVM. It peeks at well over 300%, which indicates it is running multi-threaded:

This graph also shows that CPU increases broadly inline with the number of concurrent users on the system.

Key:

red = CPU for Foundation JVM blue = CPU for SOA JVM green = CPU for Financial Close Management JVM (at half scale when presented on graph area) yellow = CPU for WL JVM black = Total CPU over all 8 cores orange = Total CPU over all 8 cores on DB server Be aware of scaling and the impact of granularity when reading the above graph; for example, many of the points are averages, and the true maximums (even for a millisecond) are as shown in the table under the graph.

You can see the server's memory usage from the metric list at the bottom of the above graph:

Free memory ~= 3845 MB (physical memory is 16GB). Committed bytes ~= 13160 MB (this is the server's overall VM use).

Page 29: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 29

The following graph shows the CPU and Heap use for the SOA JVM:

Page 30: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 30

The following graph shows the CPU and Heap use for the Financial Close Management JVM:

Page 31: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 31

Other Metrics

The test created 1514 comment records.

The following graph shows the WebLogic data source use after the test finished.

Note: The Active Connections High Count of 35 for financialclose_datasource / FinancialClose0 JVM.

Page 32: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 32

Account Reconciliation Manager Performance

This section contains information specific to ARM. It needs to be read in conjunction with the Financial Close Management Performance section.

An Example of a Heavy Load ARM Configuration

Hardware and Software Configuration used for testing:

Machine # CPU Cores

Speed Server Model /

Processor Type

RAM OS Tier

Host 1 8 2.66 GHz HP DL180 G5 inc

h/w RAID 0+1 disks

Intel Xeon E5430

4 GB 2003 R2 x64 DB Server

Host 2 4 2.6GHz Dell SC1435

AMD dual-core

8 GB 2003 R2 x64 Foundation*

FCM (CM and ARM) #

Host 3 4 2.6 GHz Dell SC1435

AMD dual-core

16 GB 2003 R2 x64 WebLogic and SOA

* Foundation includes Shared Services and Workspace

# Using embedded web server

Tuning Parameters – Oracle DB

The following are the formulas that Oracle recommends for setting the initialization parameters.

Using an available SQL connection tool, connect using an account with administrative privileges and execute:

For example, to support 500 simultaneous sessions, set the initialization parameters:

Run the following commands:

Page 33: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 33

where is the maximum number of concurrent users multiplied by 50 plus 200 more cursors to

handle background processing and SOA.

For example, for 200 concurrent users, the number would be 10200

Restart the database service.

It might be simpler to connect to the database and run the commands by hand rather than updating a script file, copying it to the server, running SQL Plus, and executing the script.

To get the current value of these settings, run the following query:

Tuning Parameters – WebLogic

Tuning Parameters – JVMs

In Financial Close Management: :

In :

Page 34: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 34

Tuning

The above tuning settings are set high to make good use of the available resources on the servers. For most tasks many could be set lower with no impact on performance.

The above WebLogic settings were maximum values used for testing ARM to 125 concurrent users. We recommend you start with these values for a heavy-load 125 user system, then lower them having looked at the WebLogic Console's Services / Data Sources (Monitoring tab) screen's peak values after a period of mixed and varied heavy use.

The most important tuning parameter is to run the following (via SQL+/Dev) on the Oracle DB:

where “system” is the name of the user/schema owner of the Financial Close Management ARM database objects.

Run this parameter initially after you load the bulk data, just before running the ARM “copy to period” task.

Note: If you do not run the parameter initially, the reconciliations can take days to become open – especially when you copy 5,000+ profiles to a period for automatic reconciliations.

You must also run this stats process at regular intervals as part of ongoing system maintenance.

Results

This configuration was able to support the following workload:

50 users repeatedly accessing the BI Dashboard. 50 users repeatedly adding and deleting transactions against reconciliations. 25 users repeatedly submitting and rejecting reconciliations. 1 to 3 second pause between mouse clicks. 60 to 90 second delay between each repeated cycle per user. CPU on Host2 ran up to 90%, but was hardly used on the other servers.

Note: The provision of CPU capacity on Host3 is to allow for Financial Close Management CM use, it’s not expected to be used for this type of ARM workload.

About 3.5GB of the allocated 4GB heap was used by the Financial Close Management JVM on Host2.

Page 35: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 35

Benchmark Results for ARM - Manual Testing

Test Detail

A set of 5000 profiles was created as a CSV file. 50,750 matching data records were also created as a SQL file.

A new period was created, the profiles were loaded into the system, the data was loaded in to the ARM_BALANCES table directly (via SQL+). At this point the Oracle "dbstats" procedure was run. If you don't do this the copy to period will be longer, and the auto-reconciliation can take a day instead of seconds!

The period is opened and the copy-to-period button is clicked. The auto-reconciliation is triggered automatically at the end of the copy to period. The auto-reconciliation time is taken from the ARM log file.

Results

Task Step Time (minutes : seconds)

load profiles 0:48.9 load data 1:01.6 run db stats 0:17.5 copy to period 1:03.5 auto-recon. 0:08.0

Page 36: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 36

The graph below shows the impact of the test on selected server metrics. No other process had any significant CPU use, but together they add up to take the overall CPU from just over one of 8 cores (12.5%) used by Financial Close Management to 23% at peak for all processes on the server. You can see from this that the copy-to-period task is single-threaded:

Key:

Black = Total CPU, max 23.440 over all 8 cores

Green = CPU for Financial Close Management JVM (full scale), max 100.937

Other Metrics

After the above test is run, the servers key memory metrics were as follows:

Free memory = 9896 MB (physical memory is 16GB). Committed bytes = 7094 MB (this is the server's overall VM use).

Page 37: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 37

Benchmark Results for ARM - Multi-user Testing

Test Detail

LoadRunner was used to simulate 175 concurrent Web users (vusers) accessing the server. Each vuser ran one of 6 LR scripts:

Script Max. Users Description

arm01 54 BI dashboard view arm02 27 BI dashboard create and remove widgets arm03 28 transaction list view arm04 22 Add and delete transaction from reconciliation arm05 28 Add and delete comment from reconciliation arm06 16 Submit and reject reconciliation

The arm01 script executes the following tasks for each allocated vuser:

1. Open browser and access EPM Workspace home page. 2. Log on to Workspace. 3. Open ARM, bringing up existing BI dashboard. 4. The dashboard contains 4 widgets, showing a range of information for the scale05 period. 5. Log off, returning to Workspace home page. 6. Close browser (but do not clear browser cache). 7. Wait for between 60 and 90 seconds randomly - this is called pacing. 8. Repeat above task sequence from start.

The arm02 script executes the following tasks for each allocated vuser:

1. Open browser and access EPM Workspace home page. 2. Log on to Workspace. 3. Open ARM, bringing up existing BI blank dashboard. 4. Add status widget. 5. Add aging widget. 6. Add reconciliation list widget. 7. Add transaction list widget. 8. Delete each widget, in reverse order to how they were added. 9. Log off, returning to Workspace home page. 10. Close browser (but do not clear browser cache). 11. Wait for between 60 and 90 seconds randomly - this is called pacing. 12. Repeat above task sequence from start.

The arm03 script executes the following tasks for each allocated vuser:

1. Open browser and access EPM Workspace home page. 2. Log on to Workspace. 3. Open ARM, bringing up existing transaction list.

(lists transactions for scale05 period)

4. Refresh transaction list. 5. Log off, returning to Workspace home page. 6. Close browser (but do not clear browser cache). 7. Wait for between 60 and 90 seconds randomly - this is called pacing. 8. Repeat above task sequence from start.

The arm04 script executes the following tasks for each allocated vuser:

Page 38: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 38

1. Open browser and access EPM Workspace home page 2. Log on to Workspace 3. Open ARM, bringing up existing reconciliation list

(lists reconciliations for scale04 period)

4. Open a reconciliation (each vuser uses a different reconciliation). 5. Click on the Explained Balance tab. 6. Click on Add Transaction. 7. Fill in comment and value fields (date is left as default). 8. Click Save. 9. Close the reconciliation dialogue, returning to the reconciliation list. 10. Open the same reconciliation. 11. Click on the Explained Balance tab. 12. Select the transaction previously added (should be the 1st and only one). 13. Click Delete Transaction. 14. Click Yes on the dialogue to confirm deletion. 15. Close the reconciliation dialogue, returning to the reconciliation list. 16. Log off, returning to Workspace home page. 17. Close browser (but do not clear browser cache). 18. Wait for between 60 and 90 seconds randomly - this is called pacing. 19. Repeat above task sequence from start.

The arm05 script executes the following tasks for each allocated vuser:

1. Open browser and access EPM Workspace home page. 2. Log on to Workspace. 3. Open ARM, bringing up existing reconciliation list.

(lists reconciliations for scale05 period)

4. Open a reconciliation (each vuser uses a different reconciliation). 5. Click Add Comment. 6. Fill in comment field. 7. Click OK to save the comment. 8. Close the reconciliation dialogue, returning to the reconciliation list. 9. Open the same reconciliation. 10. Select the comment previously added (the first and only one). 11. Click Delete Comment. 12. Close the reconciliation dialogue, returning to the reconciliation list. 13. Log off, returning to Workspace home page. 14. Close the browser (but do not clear the browser cache). 15. Wait for between 60 and 90 seconds randomly - this is called pacing. 16. Repeat above task sequence from start

The arm06 script executes the following tasks for each allocated vuser:

1. Open browser and access EPM Workspace home page. 2. Log on to Workspace. 3. Open ARM, bringing up existing reconciliation list.

(lists reconciliations for scale06 period)

4. Open a reconciliation (each vuser uses a different reconciliation). 5. Click Submit. 6. Click Yes to submit, returning to the reconciliation list. 7. Log off, returning to Workspace home page. 8. Log on to Workspace.

Page 39: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 39

9. Open ARM, bringing up existing reconciliation list. (lists reconciliations for scale06 period)

10. Click Reject, returning to the reconciliation list. 11. Log off, returning to Workspace home page. 12. Close the browser (but do not clear browser cache). 13. Wait for between 60 and 90 seconds randomly - this is called pacing. 14. Repeat above task sequence from start.

Each vuser pauses between 1 and 3 seconds randomly to simulate the pauses real users would have - this is called think-time.

The 175 vusers were introduced to the system using the following ramp-up and ramp-down schedule:

1. Start 50 vusers - 1 per 20 seconds 2. Run for 10 minutes at 50 users 3. Start 50 vusers - 1 per 20 seconds 4. Run for 10 minutes at 100 users 5. Start 25 vusers - 1 per 20 seconds 6. Run for one hour at 125 users 7. Start 25 vusers - 1 per 20 seconds 8. Run for 10 minutes at 150 users 9. Start 25 vusers - 1 per 20 seconds 10. Run for one hour at 175 users 11. Ramp-down - stop all users - 5 per 30 seconds

Each vuser starts to execute its sequence of tasks (depending on the script it has been assigned) as soon as it has been started, continuing to cycle around its list of tasks as dictated by its LR script until the ramp-down phase.

Results

Scenario Name: scale175.lrs Duration: 3 hours and 45 minutes. Maximum Running Vusers: 175 Total Throughput (bytes): 14,444,780,207 Average Throughput (bytes/s): 1,069,904 Total Hits: 1,996,024 Average Hits per Second: 147.843 Transactions: Total Passed: 133,979 Total Failed: 100 Total Stopped: 0

Page 40: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 40

Transaction Name Minimum Average Maximum Std. Dev. 90% Pass Fail Stop

a01_001_home 0.016 0.035 0.656 0.022 0.031 5,900 0 0

a01_002_logon 0 0.023 0.938 0.032 0.033 5,900 0 0

a01_003_armdb 0.078 0.15 22.266 0.303 0.219 5,900 0 0

a01_004_logoff 0.016 0.041 0.719 0.042 0.051 5,900 0 0

a02_001_home 0.016 0.036 0.875 0.028 0.031 2,280 0 0

a02_002_logon 0 0.026 0.5 0.025 0.032 2,280 0 0

a02_003_arm 0.063 0.103 0.891 0.048 0.11 2,278 2 0

a02_004_addstatus 0.125 0.157 1.594 0.063 0.172 2,278 0 0

a02_005_addaging 0.094 0.124 1.204 0.051 0.141 2,278 0 0

a02_006_addreconlist 0.094 0.132 0.672 0.042 0.141 2,278 0 0

a02_007_addtxlist 0.109 0.179 0.828 0.047 0.207 2,278 0 0

a02_008_deltxlist 0.094 0.18 5.174 0.156 0.211 2,278 0 0

a02_009_delreconlist 0.094 0.115 1.015 0.049 0.13 2,278 0 0

a02_010_delaging 0.047 0.071 0.563 0.032 0.082 2,278 0 0

a02_011_delstatus 0 0.015 0.375 0.022 0.022 2,278 0 0

a02_012_logoff 0.016 0.038 1.157 0.043 0.041 2,278 0 0

a03_001_home 0.016 0.036 0.688 0.027 0.031 2,937 0 0

a03_002_logon 0 0.024 0.594 0.025 0.033 2,937 0 0

a03_003_arm 0.203 0.251 2.673 0.081 0.297 2,935 2 0

a03_004_refresh 0.188 0.584 1.579 0.08 0.672 2,935 0 0

a03_005_logoff 0.063 0.082 1.125 0.044 0.1 2,935 0 0

a04_001_home 0.016 0.037 0.656 0.031 0.047 1,626 0 0

a04_002_logon 0 0.026 0.391 0.018 0.032 1,626 0 0

a04_003_arm 0.485 0.592 1.751 0.096 0.672 1,600 26 0

a04_004_recon 0.641 0.704 1.329 0.068 0.782 1,600 0 0

a04_005_expbal 0.125 0.151 4.15 0.11 0.172 1,600 0 0

a04_006_addtx 0.219 0.295 1.063 0.064 0.36 1,600 0 0

a04_007_save 0.063 0.178 1.079 0.049 0.208 1,600 0 0

a04_008_close 0.344 0.744 5.768 0.153 0.828 1,600 0 0

a04_009_recon 0.328 0.381 5.205 0.131 0.438 1,600 0 0

a04_010_expbal 0.063 0.146 0.668 0.042 0.16 1,600 0 0

a04_011_selecttx 0.063 0.141 0.797 0.043 0.163 1,600 0 0

a04_012_delete 0.063 0.123 2.285 0.068 0.141 1,600 0 0

a04_013_deleteyes 0.109 0.172 1.188 0.05 0.188 1,600 0 0

a04_014_close 0.344 0.731 1.391 0.065 0.813 1,600 0 0

a04_015_logoff 0.047 0.072 0.828 0.042 0.079 1,600 0 0

a05_001_home 0.016 0.037 0.719 0.032 0.047 2,336 0 0

a05_002_logon 0 0.024 0.563 0.024 0.031 2,336 0 0

Page 41: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 41

a05_003_arm 0.485 0.582 2.126 0.1 0.656 2,307 29 0

a05_004_recon 0.625 0.723 17.485 0.358 0.797 2,307 0 0

a05_005_addcom 0.156 0.207 4.866 0.114 0.234 2,307 0 0

a05_006_ok 0.063 0.099 4.642 0.136 0.101 2,307 0 0

a05_007_close 0.391 0.447 1.094 0.06 0.516 2,307 0 0

a05_008_recon 0.344 0.387 1.313 0.06 0.453 2,307 0 0

a05_009_selcom 0.047 0.08 0.578 0.035 0.083 2,307 0 0

a05_010_delcom 0.063 0.089 0.911 0.039 0.096 2,307 0 0

a05_011_close 0.391 0.445 1.125 0.057 0.517 2,307 0 0

a05_012_logoff 0.047 0.07 0.563 0.032 0.079 2,307 0 0

a06_001_home 0.016 0.036 0.938 0.036 0.041 1,295 0 0

a06_002_logon 0 0.024 0.578 0.024 0.032 1,295 0 0

a06_003_arm 0.485 0.581 1.913 0.105 0.672 1,280 15 0

a06_004_recon 0.625 0.705 1.579 0.078 0.782 1,280 0 0

a06_005_submit 0.094 0.11 0.704 0.044 0.125 1,280 0 0

a06_006_submityes 0.078 0.462 1.454 0.097 0.552 1,280 0 0

a06_007_logoff 0.047 0.071 0.594 0.037 0.08 1,280 0 0

a06_008_home 0 0.001 0.109 0.005 0 1,280 0 0

a06_009_logon 0 0.018 0.453 0.017 0.022 1,280 0 0

a06_010_arm 0.485 0.575 1.282 0.08 0.656 1,254 26 0

a06_011_recon 0.625 0.71 2.048 0.079 0.797 1,254 0 0

a06_012_reject 0.125 0.485 0.938 0.068 0.563 1,254 0 0

a06_013_logoff 0.047 0.07 1.125 0.044 0.082 1,254 0 0

The following graph confirms how the users were introduced during the test run:

Page 42: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 42

The following graph shows how the transaction response times changed over time for the vusers running the arm01 script. Note there's no significant increase in response times as more users were added. Initial very high response times are due to the impact of caching (or lack of for the first iterations) - especially at the browser level:

Page 43: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 43

The following graphs show the same for the other 5 scripts:

Page 44: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 44

Page 45: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 45

Page 46: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 46

Page 47: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 47

Page 48: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 48

The following graph shows the impact of the test on selected server metrics. It can be seen that the only process making any real use of the CPU is the FINANCIAL CLOSE MANAGEMENT JVM. It peeks at 587%, which confirms it is running multi-threaded.

This graph also shows that CPU increases broadly inline with the number of concurrent users on the system.

Key:

red = CPU for Foundation JVM blue = CPU for SOA JVM green = CPU for FINANCIAL CLOSE MANAGEMENT JVM (at 1/5th scale when presented on graph area) yellow = CPU for WL JVM black = total CPU over all 8 cores orange = total CPU over all 8 cores on DB server Be aware of scaling and the impact of granularity when reading the above graph; for example, many of the points are averages, and the true maximums (even for a millisecond) are as shown in the table under the graph.

Also, from the metric list at the bottom of the above graph can be seen the server's memory usage: Free memory ~= 6085 MB (physical memory is 16GB).

Committed bytes ~= 10887 MB (this is the server's overall VM use).

Page 49: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 49

Other Metrics

The following graph shows the distribution of failed transactions over the duration of the test run. The total number of failed transactions is within acceptable limits at 100 out of 133,979. These failures are where the LR script sees a response from the server it was not expecting:

These failed transactions are currently being investigated. They appear to be related to an internal locking issue. In all cases a real user would just need to log out and in again before continuing, with it being very unlikely they would see the same error again for a long while. These failures only occur under repetitive concurrent multi-user load.

Page 50: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 50

Summary

Like many products, real-world FCM implementations need performance tuning and ongoing monitoring. Most situations also require some form of capacity planning before the initial implementation and when any major change in workload or configuration is planned.

Financial Close Management, with the hardware configuration detailed in “Benchmarking Configuration” earlier, comfortably copes with 100 concurrent Close Manager users and 175 concurrent ARM users. At 35% concurrency 175 users would be a user base of 500. Looking at the CPU and memory, both CM and ARM users could not use the system at the same time – (this scenario will be tested shortly). The only change may be the need to increase the 4GB heap allocated to the Financial Close Management JVM.

Using Other EPM Products

A real production system for Financial Close Management would also likely include HFM, a data load product like ERPi/ODI, and possibly Financial Reports.

Looking at the CPU load, it should be possible to run these levels of users (175 ARM, 100 CM) on this same server configuration as used for the benchmark tests with these other products installed. It is likely that more memory would be required though.

From a CPU point of view: ARM is heavy on the Financial Close Management JVM, CM mostly uses the Financial Close Management JVM, but also makes a lot of use of the SOA JVM for background processing (especially when opening tasks and schedules). Note, HFM is also a heavy user of CPU when it is running consolidations.

From a memory point of view: both ARM and CM need a lot of heap allocating in the Financial Close Management JVM. As CM also makes a lot of use of SOA, and SOA uses a lot of memory; therefore you also need to allow a good deal of heap for SOA when using CM. HFM also uses a lot of memory, depending on the application design.

Separating Out Products

It is possible to separate SOA from Financial Close Management in a multi-node configuration. It’s also possible to put the Foundation JVM on a different server. While it’s possible to separate SOA from WL Admin Server, it requires a lot of manual configuration during installation.

Therefore, looking at the CPU and memory metrics from these tests, unless the concurrent user volumes are going to be a lot higher, it would be advisable to stay with a single-node configuration for these Financial Close Management and related components. It would be easier and more beneficial for other products, especially HFM server ( ), to be installed on other servers if

the one server was thought to be too small.

Finally

Tuning is key for larger data or user volumes. It’s especially important for “dbstats” to be run on the Oracle DB after large data loads. The system is not good at letting you know that it is running slow because of poor database response times; therefore it can be hard to know if the DB instance has been allocated enough resource to meet your workload’s requirements.

Please be aware that any testing is limited to the exact test scenario – sometimes small changes in use or configuration can have a big impact. Take these results as a guide, and always do testing of your own system and workload pattern and volumes to confirm it will be fit for purpose.

Page 51: FCM Performance Tuning

Oracle’s Hyperion Financial Close Management Performance Tuning Guide 51

Copyright © 2012, Oracle and / or its affiliates. All rights reserved. http://www.oracle.com