HPE Reference Configuration comparing Oracle 11g … Reference Configuration comparing Oracle 11g...

HPE Reference Configuration comparing Oracle 11g and 12c performance on HPE ProLiant DL580 Gen9 and HPE 3PAR StoreServ 7450c

Technical white paper


Contents Executive summary .................................................................................................................................................................................................................................................................................................................................3 Introduction ....................................................................................................................................................................................................................................................................................................................................................3 Solution overview ..................................................................................................................................................................................................................................................................................................................................... 4 Solution components ............................................................................................................................................................................................................................................................................................................................ 4 HPE ProLiant DL580 Gen9 server ........................................................................................................................................................................................................................................................................................... 5

Commanding performance and scalability ................................................................................................................................................................................................................................................................. 6 Rock-solid availability and reliability ................................................................................................................................................................................................................................................................................ 6

HPE 3PAR StoreServ 7450c storage all-flash ................................................................................................................................................................................................................................................................ 6 Flash-optimized architecture featuring a Mesh-Active design ............................................................................................................................................................................................................... 6 Fine-grained virtualization and system-wide striping...................................................................................................................................................................................................................................... 7 Unique technologies extend your flash investments ....................................................................................................................................................................................................................................... 7 Databases ................................................................................................................................................................................................................................................................................................................................................. 7

HPE Application Tuner Express (HPE-ATX) ................................................................................................................................................................................................................................................................. 7 Software ............................................................................................................................................................................................................................................................................................................................................................ 8 Best practices and configuration guidance for the Oracle Database solution ................................................................................................................................................................................. 8 Capacity and sizing ..............................................................................................................................................................................................................................................................................................................................10

Workload description ...................................................................................................................................................................................................................................................................................................................10 Workload results ................................................................................................................................................................................................................................................................................................................................ 11 Analysis and recommendations ......................................................................................................................................................................................................................................................................................... 13

Summary ........................................................................................................................................................................................................................................................................................................................................................ 13 Implementing a proof-of-concept ........................................................................................................................................................................................................................................................................................... 13 Appendix A: Bill of materials ........................................................................................................................................................................................................................................................................................................ 14 Appendix B: Oracle configuration parameters for Oracle 12cR1 .................................................................................................................................................................................................................. 15 Appendix C: Oracle configuration parameters for Oracle 11gR2 ................................................................................................................................................................................................................. 16 Appendix D: Linux kernel configuration ........................................................................................................................................................................................................................................................................... 16 Appendix E: Multipath configuration ................................................................................................................................................................................................................................................................................... 17 Appendix F: HPE 3PAR StoreServ 7450c udev rules ............................................................................................................................................................................................................................................ 17 Appendix G: Udev rules .................................................................................................................................................................................................................................................................................................................... 18 Appendix H: Oracle tuned profile – ktune.sh ................................................................................................................................................................................................................................................................ 18 Appendix I: Oracle tuned profile – ktune.sysconfig ................................................................................................................................................................................................................................................ 18 Appendix J: Oracle tuned profile – sysctl.ktune ......................................................................................................................................................................................................................................................... 19 Appendix K: Oracle tuned profile – tuned.conf .......................................................................................................................................................................................................................................................... 20 Appendix L: Script for giving a higher priority to database and log writer processes ........................................................................................................................................................... 20 Appendix M: Script to start the Oracle Database in concert with HPE-ATX ................................................................................................................................................................................. 20 Resources and additional links .................................................................................................................................................................................................................................................................................................. 21


Executive summary The demands of database implementations continue to escalate. Faster transaction processing speeds, scalable capacity, and increased flexibility are required to meet the needs of today’s business. At the same time, Oracle continues to introduce new versions of their database software with new features that can have a positive business impact.

When deploying newer versions of the Oracle Database with an enhanced feature set, customers need to be cognizant of the impact on performance and to ensure the hardware infrastructure is able to meet the business throughput requirements.

Many customers will view an Oracle Database version upgrade as an inflection point upon which they will upgrade their infrastructure with the goal of improving performance and reducing TCO, accomplished by moving to newer hardware to replace aging or unsupported equipment, as well as moving from proprietary operating systems to Linux®.

HPE is committed to delivering customers a more thorough utilization of their servers. HPE has created a software product, named HPE Application Tuner Express (HPE-ATX) that will help customers do just that, by reducing the latency introduced by accessing memory not on the local processor. HPE-ATX homes processes and memory, such that less memory is located remotely to the process’ core, reducing the time spent fetching memory. This has the added benefit of requiring fewer cores to perform the same amount of work relative to the amount of work without HPE-ATX managed by older generation servers.

This Reference Configuration is intended to provide customers with the expected performance implications associated with deploying Oracle 12c vis-à-vis Oracle 11gR2 so they can plan to provide the appropriate level of performance and continue to meet SLAs that may be a requirement for the business enterprise.

Target audience: This Hewlett Packard Enterprise Reference Configuration (RC) is designed for IT professionals who use, program, manage, or administer large databases that require high performance. Specifically, this information is intended for those who evaluate, recommend, or design new and existing IT high performance architectures. Additionally, CIOs may be interested in this document as an aid to guide their organizations to determine when to implement the new features associated with Oracle 12c.

This RC describes testing completed in April 2016.

Document purpose: The purpose of this document is to describe a Reference Configuration that organizations can utilize to plan for their deployment of Oracle databases when upgrading from 11g to 12c.

Introduction Information Technology departments are under pressure to add value to the business, improve existing infrastructure, enable growth and reduce overhead. As ISVs introduce software updates, the impact of the application changes on the infrastructure needs to be understood. With the release of Oracle 12c, Oracle has added Pluggable and Container databases, Data Redaction, Top N Query and Fetch, Adaptive Query Optimization, RMAN enhancements, support for multiple indexes on a single column, online migration of a table partition, temporary UNDO and in-database archiving, along with other features. However, based on our testing, Oracle 12c Enterprise Edition consumes more system resources for an identical workload when run on 11g Enterprise Edition. Customers upgrading from Oracle 11g to Oracle 12c need to understand the performance ramifications associated with this upgrade in order to successfully plan for and implement the upgrade without negatively impacting business outcomes.

This Reference Configuration is one in a series of papers that describe the architecture and performance of the HPE ProLiant DL580 Gen9 server and HPE 3PAR StoreServ 7450c storage all-flash array (AFA). This RC focuses on comparing the transaction database performance when running Oracle 11g as compared to Oracle 12c with the exact same infrastructure. The solution was built on the HPE ProLiant DL580 Gen9 server and the HPE 3PAR StoreServ 7450c1 AFA with Red Hat® Enterprise Linux (RHEL) 6.6 and the Oracle database. The Oracle and Red Hat installations are standard configurations except where explicitly stated in the reference configuration.

Database workloads are acknowledged to be extremely resource intensive. The large number of I/O operations issued and the network activity to serve remote clients place significant challenges on the platform. The high consumption of CPU and memory resources leaves little room for inefficiency. This coupled with the additional system resources required by Oracle 12c should cause organizations upgrading from Oracle 11gR2

1 Although we tested with HPE 3PAR StoreServ 7450c, the performance benefits seen comparing Oracle 11g and 12c with an HPE 3PAR StoreServ 7450c would similarly apply using

an HPE 3PAR StoreServ 8450 array.


to Oracle 12c to carefully analyze if their current equipment can handle the additional load. If a customer’s current servers or storage are unable to meet the additional demand, it is a good idea to plan to upgrade the hardware infrastructure at the same time as the database version update is undertaken, to avoid two disruptive upgrade cycles.

Solution overview This RC outlines the architecture and expected performance you can expect when upgrading the Oracle Database software from version 11gR2 to 12cR1 running on Red Hat Enterprise Linux (RHEL) version 6.6.

Solution components The solution was made up of a 16-core HPE ProLiant DL580 Gen9 and an HPE 3PAR StoreServ 7450c AFA. The HPE ProLiant DL580 Gen9 was chosen because it is a great mid-range offering that allows for expandability in processors, memory and interface cards. The HPE 3PAR StoreServ 7450c AFA was chosen to reduce the latency associated with spinning media storage systems, allowing greater utilization of the server. The server, storage and external clients were all joined together using a pair of HPE SN6000B Fibre Channel switches.

The HPE ProLiant DL580 Gen9 server used in this comparison is based on an architecture balanced between compute capacity, memory capacity and speed, and PCIe expansion slots.

The HPE ProLiant DL580 Gen9 has the following configuration:

• 4 X Intel® Xeon® E7-8893 v3 4-core 3.2GHz processors (these processors are “license optimized” having fewer cores and greater clock speed)

• 96 X 32GB PC4-2133P SmartMemory LRDIMMs (total 3TB of memory)

• 4 X 8Gb Dual-port PCIe Fibre Channel HBA

• 2 X 400GB 12Gb Write Intensive SSDs (for OS and boot)

The HPE 3PAR StoreServ 7450c was configured as follows:

• 1 X HPE 3PAR StoreServ 7450c all-flash array

– 4-node

– 192GB of cache

– 6 X expansion shelves

– 80 X 480GB MLC SSD

– 8 X 8Gb Fibre Channel ports – of which 4 were used


Figure 1 provides a pictorial view of the environment on which the tests were run

12

110 23

12

110 23

3PARStoreServ

7450

3PARStoreServ

7450SAS SAS SAS SAS

10K 900GB 10K 900GB 10K 900GB 10K 900GB

SAS SAS SAS SAS10K 900GB 10K 900GB 10K 900GB 10K 900GB



SAS10K 900GB

SAS10K 900GB

SAS10K 900GB

SAS10K 900GB

12

110 23

12

110 23

3PARStoreServ

7450

3PARStoreServ

7450SAS SAS SAS SAS

10K 900GB 10K 900GB 10K 900GB 10K 900GB




SAS10K 900GB

SAS10K 900GB

SAS10K 900GB

SAS10K 900GB

12

110 23

12

110 23

3PARStoreServ

7450

3PARStoreServ

7450SAS SAS SAS SAS

10K 900GB 10K 900GB 10K 900GB 10K 900GB




SAS10K 900GB

SAS10K 900GB

SAS10K 900GB

SAS10K 900GB

12

110 23

12

110 23

3PARStoreServ

7450

3PARStoreServ

7450SAS SAS SAS SAS

10K 900GB 10K 900GB 10K 900GB 10K 900GB




SAS10K 900GB

SAS10K 900GB

SAS10K 900GB

SAS10K 900GB

• 3PAR StoreServ 7450c All Flash Arrayo 4-nodeo 192GB of cacheo 6 x Expansion Shelveso 8 x 8Gb FC ports – of which 4 were usedo 80 x 480GB MLC SSD

• 8 x 6GB RAID-10 LUNs for Oracle software• 8 x 52GB RAID-10 LUNs for OS swap space• 8 x 500 GB RAID-10 LUNs for Tablespaces• 8 x 128GB RAID-5 LUNs for Redo Logs

StoreServ 7450c Controller Nodes

StoreServ 7450c Expansion Drive Shelves

1

2

25

26

23

24

15

18

17

162 x HPE SN6000B Switches

8 x 8Gb Fibre Channel

HPE DL580 Gen9 • 4 x E7-8893 v3 4-core 3.2GHz Processors• 3TB Memory• 2 x 400GB 12Gb SAS Write Intensive SSDs

(OS)• 4 X 8Gb Dual Port PCI-e Fibre Channel HBA

UID

SID

54321

109876

1 2 3 4

ProLiantDL580Gen9

Figure 1. Pictorial view of HPE ProLiant DL580 Gen9 and HPE 3PAR StoreServ 7450c environment

HPE ProLiant DL580 Gen9 server

Figure 2. HPE ProLiant DL580 Gen9 server

The HPE ProLiant DL580 Gen9 server is HPE's four socket (4S) enterprise standard x86 server offering, commanding performance, rock-solid reliability and availability, and compelling consolidation and virtualization efficiencies.

Supporting Intel Xeon E7-4800/8800 v3 processors, the HPE ProLiant DL580 Gen9 offers enhanced processor performance, up to 6TB of memory, greater I/O bandwidth (9 PCIe Gen3.0 slots) than our HPE ProLiant DL580 G7, and 12Gb/s SAS bandwidth. The HPE ProLiant DL580 Gen9 has security and data protection features for system resiliency that your business can depend on. All making it ideal for mission-critical enterprise, business intelligence, and database applications.


Commanding performance and scalability The HPE ProLiant DL580 Gen9 server provides up to 39%2 more performance and 20%1 more cores with Intel Xeon E7-4800/8800 v3 processors than the previous generation of processors.

HPE SmartMemory prevents data loss and downtime with enhanced error handling. The HPE ProLiant DL580 Gen9 has support for up to 6TB DDR4 maximum capacity (96 DIMM slots) with speeds up to 1866MHz with up to 16%3 performance gain, making the HPE ProLiant DL580 Gen9 ideal for large-scale in-memory database computing.

The HPE ProLiant DL580 Gen9 comes standard with nine full length, full height PCIe 3.0 slots allowing a choice of interface cards to connect to your SAN or network fabric so you can adapt to changing business needs.

Rock-solid availability and reliability The HPE ProLiant DL580 Gen9 server provides increased system availability and less need for service with advanced error recovery, error diagnosis, and built-in redundancy, all features of the HPE ProLiant DL580 server Comprehensive Fault Management and Diagnostics.

Increased security is available with four new levels of Secure Boot with Unified Extensible Firmware Interface (UEFI) mode.

HPE 3PAR StoreServ 7450c storage all-flash

Figure 3. HPE 3PAR StoreServ 7450c storage

IT has never been more important to doing business, which means that storage infrastructure must be simpler, smarter, faster, more flexible, and more business aligned than ever. The world is moving rapidly, where success is defined by how quickly your business can turn ideas into value.

With a flexible, flash-optimized architecture, HPE 3PAR StoreServ storage provides the only primary storage architecture you’ll ever need. Regardless of whether you are a midsize enterprise experiencing rapid growth, a large enterprise looking to support IT as a Service (ITaaS), or a global service provider building a hybrid or private cloud, HPE 3PAR StoreServ storage features a modern architecture to support better business outcomes. A range of models brings Tier-1 data services to the midrange, delivering all-flash array performance for the cost of a spinning disk array, and providing mission-critical resiliency and quality of service (QoS).

Flash-optimized architecture featuring a Mesh-Active design HPE 3PAR StoreServ storage features a Mesh-Active design based on a unique system of controller interconnects. This flash-optimized architecture combines the benefits of monolithic and modular architectures while eliminating price premiums, scaling complexities, and the performance bottlenecks of legacy storage designs.

Unlike legacy Active-Active controller architectures, the HPE 3PAR Mesh-Active design allows each volume to be active on every controller in the system. This delivers robust, load-balanced performance and greater headroom for cost-effective scalability.

2 Intel internal measurements. Up to 39% top-bin performance increase based on OLTP Warehouse Oracle 11gR2 database workload comparing 4x Intel Xeon processor E7-8890 v3

(18C, 2.5GHz) with 2TB to similarly configured E7-4890 v2 (15C, 2.8GHz) with 1TB. 20% more cores comparing E7-8800/4800 (15 cores) v2 versus v3 (18 cores). January 2015. 3 Based on HPE internal calculations. Based on comparing the difference of DDR4 DIMMs of 1866 versus 1600MHz on Gen8 to Gen9 HPE servers, May 2015.


A high-speed, full-mesh interconnection joins multiple storage controllers to form a cache-coherent, flash-optimized Mesh-Active cluster that is ideal for low-latency, high-performance, internode communication. Purpose-built HPE 3PAR Gen4 ASICs in each node connect all controllers via dedicated, high-bandwidth, low-latency links and spread I/O workloads widely across the array using direct memory access (DMA) to reduce latency times.

Fine-grained virtualization and system-wide striping The HPE 3PAR Architecture uses three levels of storage virtualization to drive up capacity utilization and accelerate performance. This fine-grained approach to storage virtualization:

• Divides each physical disk into granular allocation units that can be independently assigned and dynamically reassigned to different logical disks to create virtual volumes

• Enables mixed RAID levels on the same physical drive

• Supports flash and other nonvolatile memory types

Logical disks are the virtualization layer in which QoS parameters are applied (availability level, drive media type, RAID level, etc.). This enables sub-LUN tiering and system-wide striping of data, increasing capacity utilization and performance levels. Fine-grained virtualization combined with system-wide striping drives uniform I/O patterns by spreading wear evenly and system-wide. System-wide sparing also helps guard against performance degradation if there is a media failure by enabling faster, “many-to-many” rebuilds.

Unique technologies extend your flash investments HPE innovations around flash not only help bring down the cost of flash media, but HPE 3PAR Gen4 ASICs within each node also provide an efficient, silicon-based, zero-detection mechanism that “thins” your storage and extends your flash media investments. These ASICs power inline deduplication for data compaction that removes allocated but unused space without impacting your production workloads—extending the life of flash-based media by avoiding unnecessary writes. The unique Adaptive Read and Write feature also serves to extend the life of flash drives by automatically matching host I/O size for reads and writes.

In addition, while other architectures generally reserve entire drives as spares, the HPE 3PAR Architecture reserves spare chunklets within each drive. Sparing policies are adjusted automatically and on the fly to avoid using flash for sparing, thus lengthening media lifespan and helping to drive down performance costs. A five-year warranty on all HPE 3PAR StoreServ flash drives protects your storage investments.

Databases Database performance and availability are so critical that many organizations apply generous capacity and management resources to maintain needed service levels. HPE 3PAR StoreServ storage removes these inefficiencies. For example, with HPE 3PAR Thin Persistence software and the new Oracle ASM Storage Reclamation Utility (ASRU), your Oracle databases stay thin by automatically reclaiming stranded database capacity. HPE also offers cost-effective Oracle-aware snapshot technologies.

HPE Application Tuner Express (HPE-ATX) HPE Application Tuner Express (HPE-ATX) is a utility for HPE Linux customers to enable their applications to achieve maximum performance while running on larger x86 servers. With HPE-ATX application execution is aligned with the data in memory resulting in increased performance even on servers with four sockets.

Since HPE-ATX runs alongside your applications no changes in your applications are required to realize the performance benefits from HPE-ATX.

These performance gains are possible as many x86 applications in use today were designed for older 2 and 4-socket systems. No consideration for scaling these applications onto larger socket systems was designed in, leading to significant application performance issues on large systems.

HPE-ATX helps applications run much more efficiently and perform better in larger system configurations.

HPE-ATX has a number of configuration policies. They are:

• Round Robin Tree – Use a round robin distribution for processes. Include the root process/thread and all of its descendants.

• Round Robin Flat – Use a round robin distribution for processes. Include the root process/thread and only its direct descendants.


• Fill First Tree – Completely allocate all cores and threads within a physical processor then move on to the next processor. Include the root process/thread and all of its descendants.

• Fill First Flat – Completely allocate all cores and threads within a physical processor then move on to the next processor. Include the root process/thread and only its direct descendants.

• Pack – The root process/thread and all of its descendants will be launched on the same NUMA node/processor.

HPE-ATX is fully supported by HPE and can be downloaded from HPE Software Depot. As HPE develops additional performance software, various packaging options will be provided.4

Software The following software was utilized during the testing of the HPE ProLiant DL580 Gen9 server.

• Red Hat Enterprise Linux version 6.6

• Oracle 11g Enterprise Edition

• Oracle 12c Enterprise Edition

• HPE-ATX v1.0.0

Best practices and configuration guidance for the Oracle Database solution To optimize the configuration for Oracle, the following changes were made to the hardware, firmware and software.

HPE ProLiant DL580 Gen9 BIOS • Hyper-Threading —Enabled

• Intel Turbo Boost—Enabled

• HPE Power Profile—Maximum Performance

RHEL configuration • Create udev rules to set the following device options for the SSD LUNs and required settings for the Oracle volumes (per values in Appendix F

and G).

– Set the sysfs “rotational” value for SSD disks to 0.

– Set the sysfs value rq_affinity to 2 for each device. Request completions all occurring on core 0 caused a bottleneck. Setting rq_affinity to a value of 2 resolved this problem.

– Set I/O scheduler to deadline. There is a special tuned profile for Oracle. It is contained in Appendices G though J.

– Set permissions and ownership for Oracle volumes.

• SSD loading—Load SSDs in groups of four per enclosure at a minimum.

• Volume size—Virtual volumes should all be the same size and SSD type for each Oracle ASM group.

• Multiple paths to maintain high availability while also maximizing performance and minimizing latencies. Use recommended multipath parameters (see Appendix E).

4 By default, starting from Oracle 11g onward, Oracle NUMA is disabled. Oracle recommends testing when enabling this feature or before deploying in production. See Oracle support

note 864633.1 and also the following: Oracle NUMA (Non Uniform Memory Architecture) support can be used with large SNMP multiprocessor environments with NUMA hardware. When enabled, Oracle NUMA support facilitates efficient use of underlying NUMA hardware and may improve database performance. Oracle NUMA support needs the right combination of hardware, operating system and Oracle version. Starting with 11.2.01, Oracle NUMA support is disabled by default. This note covers how Oracle NUMA support can be enabled via configuration for Oracle Server Version 11.2. For more information about Oracle NUMA usage recommendations and Oracle NUMA support in previous releases, please consult Note :759656.1. Care should be taken before enabling NUMA support. Contact your hardware vendor for recommendations and information on your system and operating system NUMA capabilities.

https://h20392.www2.hpe.com/portal/swdepot/displayProductInfo.do?productNumber=HPE-ATX


HPE 3PAR space allocation • 8 X 500GB RAID-10 LUNs for the database, tablespaces, indexes and Undo tablespace

• 8 X 128GB RAID-5 LUNs for the redo log space

• 8 X 52GB RAID-10 LUNs for swap space

• 8 X 6GB RAID-10 LUNs for Oracle software

The following is a pictorial representation of how storage on the HPE 3PAR StoreServ 7450c was laid out and how it was delivered to the HPE ProLiant DL580.

Oracle ASM

Oracle DatabaseRed Hat

Enterprise Linux

XFS File System

Swapper

8 X 6GB LUNsOracle

8 X 52GB LUNsSwap Space

8 X128GB LUNsOracle Redo

8 X500GB LUNsOracle

Tablespaces

Figure 4. Logical mapping between HPE 3PAR StoreServ 7450c and the HPE ProLiant DL580 Gen9 server

Oracle configuration best practices For specific Oracle database parameter settings, see Appendix B and C.

• Disable automatic memory management if applicable.

• Set buffer cache memory size large enough per your implementation to avoid physical reads.

• Create three large redo log file spaces of 256GB each to minimize log file switching and reduce log file waits.

• Create an undo tablespace of 200GB.


HPE-ATX best practices • Start the Oracle Listener process using the Round Robin Flat policy.

• Start the Oracle Database processes using the Round Robin Tree policy5.

Capacity and sizing The results of the testing indicate that at lower CPU utilization levels, a customer can expect between a 5% and a 10% degradation in performance when moving from Oracle 11g to Oracle 12c. However, utilizing the HPE-ATX utility there is an increase in performance and a noticeable difference occurs at higher levels of CPU utilization. Regardless of the level of impact this degradation has, this is the performance delta that needs to be planned and accounted for before an upgrade is executed.

The results also indicate that when the database server is highly overstressed, Oracle 12c delivered more throughput than Oracle 11g. HPE does not believe that many customers will fit in this category.

Workload description The Oracle workload was tested using HammerDB, an open-source tool. The tool implements an OLTP-type workload (60 percent read and 40 percent write) with small I/O sizes of a random nature. The transaction results have been normalized and are used to compare test configurations. Other metrics measured during the workload come from the operating system and/or standard Oracle Automatic Workload Repository (AWR) statistics reports.

The OLTP test, performed on a 500GB database, was both highly CPU and moderately I/O intensive. The environment was tuned for maximum user transactions. After the database was tuned, the transactions were recorded at different connection levels. Because customer workloads vary so much in characteristics, the measurement was made with a focus on maximum transactions.

Oracle Enterprise Database versions 11.2.0.1 and 12.1.0.1 were used in this test configuration.

The Fibre Channel attached HPE 3PAR StoreServ 7450c storage was configured with 8 x RAID-10 LUNs for the indexes and tablespaces, 8 x RAID-5 LUNs for the redo logs and the undo tablespace.

We used several different Oracle connection counts for our tests. The results of various user count tests can be seen in the following graphs.

5 The command to launch the Oracle Database using HPE-ATX is included in Appendix M.


Workload results We tested Oracle 11gR2 and compared those results with Oracle 12cR1.

The following graph shows the results from the tests. The “Number of Oracle Connections” is the count of users that drove the test. Each user had no think time and as a result, represented thousands of actual users. The transaction rate results have been normalized such that the 25 connection result using Oracle 11gR2 was set to 100%. All other results are relative to the initial result. The numbers at the top of the bar chart are the percentages of additional performance realized by using HPE-ATX with Oracle 12c. With HPE-ATX we can see up to a 9.8% performance increase. Additionally, the number of connections required for 12c to overtake 11g in throughput is reduced from 175 to 150, which is a reduction of almost 20%.

As you can see from the graph, Oracle 11gR2 provided better throughput numbers than did Oracle 12cR1 for up to 150 connections without HPE-ATX, and up to 125 connections with HPE-ATX. Beyond that point, Oracle 12c reacted better to the over commitment of CPU resources.

The following graph visually represents CPU utilization plotted against the relative performance of each connection count when running Oracle 11gR2.

7.5%

3.6%-2.1% 4.6% 5.4% 8.4% 4.5% 9.8%

020406080

100120140160180

25 50 75 100 125 150 175 200

Perc

enta

ge

Number of Oracle Connections

Results comparing 11g with 12c and 12c with HPE-ATX

Oracle 11g Oracle 12c Oracle 12c with ATX

42

7385 90 93 93 93 91

0102030405060708090100

020406080

100120140160180

25 50 75 100 125 150 175 200

CPU

Util

izat

ion

Rel

ativ

e O

racl

e Pe

rcen

tage


Performance results with Oracle 11g

Oracle 11g relative perf CPU utilization


The following graph visually represents CPU utilization plotted against the relative performance of each connection count when running Oracle 12cR1. As you can see, when compared to the above 11g graph, the CPU utilization never decreased. HPE believes this to be because we can have multiple Log Writers on Oracle 12cR1 which removed the Log Writers from being the bottleneck, which allowed more transactions to complete and more CPU to be consumed.

The following graph visually represents CPU utilization plotted against the relative performance of each connection count when running Oracle 12cR1 with HPE-ATX. As you can see, when compared to the above 12c without HPE-ATX graph, the CPU utilization was always less with HPE-ATX than that of the same workload and number of connections when run without HPE-ATX.

You can see in the graphs, all workloads show a tail off of throughput, but with 11g, there is also a tail off of the amount of CPU being utilized at the highest user count level. HPE believes this can be attributed to only having one Log Writer process available in 11g. HPE did increase the priority (see Appendix L) of both the Log Writer as well as the DB Writer, so they wouldn’t be starved for resources. Additionally, when utilizing HPE-ATX, the tail off was quite a bit less pronounced, which is a result of better utilization of system resources.

43

71

90 87 91 91 94 95

0102030405060708090100

020406080

100120140160180

25 50 75 100 125 150 175 200

CPU

Util

izat

ion

Rel

ativ

e O

racl

e Pe

rcen

tage


Performance results Oracle 12c

Oracle 12c relative perf CPU utilization

41

6574

84 86 90 92 94

0102030405060708090100

020406080

100120140160180

25 50 75 100 125 150 175 200

CPU

Util

izat

ion

Rel

ativ

e O

racl

e Pe

rcen

tage


Performance results Oracle 12c with HPE-ATX

Oracle 12c relative perf CPU utilization


The following graph represents the transactional latency incurred during the test run. Note that the latency is lower for Oracle 11gR2 than it is for Oracle 12cR1 up until the higher user connection counts when Oracle 12cR1 handles the over consumption of CPU resources better. You can also see the HPE-ATX latency is quite a bit less than the 12c only result.

Analysis and recommendations As you can see from the charts above, under less than the most dramatic over commitment of CPU resources, Oracle 12c reduces transactional throughput in the 5% to 10% range. Additional processor cores would alleviate this negative impact. HPE recommends that a customer look at their present environment and determine if the performance head room exists to accommodate the additional processing power required by Oracle 12c vis-à-vis Oracle 11g.

HPE-ATX alleviates some of the performance penalty incurred when moving from 11g to 12c and may, in some instances result in better performance by 12c than that of 11g. HPE recommends the utilization of HPE-ATX to assist with the better utilization of your HPE ProLiant DL580, four-socket server.

Summary Moving from one software version to a newer software version is never an easy decision. The new features offered by the new release is enticing. Additionally, software vendors discontinue support for older versions over time. Oracle ended Premier Support of 11g in January 2015, however they have waived the cost of Extended Support until May 31, 2017. After this date, they still support this version until December 2020 via Extended Support, which comes with an additional cost.

Because of this and the enticing new features offered in Oracle 12c, customers need to start planning for their upgrade to Oracle 12c.

The performance implications may be an issue for customers, and as a result HPE recommends using the upgrade as an inflection point to also upgrade the computing equipment associated with delivering Oracle Database services to your enterprise.

Implementing a proof-of-concept As a matter of best practice for all deployments, HPE recommends implementing a proof-of-concept using a test environment that matches as closely as possible the planned production environment. In this way, appropriate performance and scalability characterizations can be obtained. For help with a proof-of-concept, contact an HPE Services representative (hpe.com/us/en/services/consulting.html) or your HPE partner.

0

2

4

6

8

10

12

14

25 50 75 100 126 150 175 200

Milli

seco

nds


Transaction Latency

Oracle 11gR2 Oracle 12cR1 Oracle 12cR1 with HPE-ATX

http://www.hpe.com/us/en/services/consulting.html


Appendix A: Bill of materials

Note Part numbers are at time of testing and subject to change. The bill of materials does not include complete support options or other rack and power requirements. If you have questions regarding ordering, please consult with your HPE Reseller or HPE Sales Representative for more details. hpe.com/us/en/services/consulting.html

Table 1. Bill of materials for the HPE ProLiant DL580 Gen9 used for this testing

QTY PART NUMBER DESCRIPTION

1 H6J66A HPE 42U 600x1075mm Advanced Shock Rack

1 H6J66A 001 HPE Factory Express Base Racking Service

1 793161-B21 HPE DL580 Gen9 CTO Svr

1 788335-L21 HPE DL580 Gen9 E7-8893 v3 FIO 1P Kit

3 788335-B21 HPE DL580 Gen9 E7-8893 v3 1P Kit

8 788360-B21 HPE DL580 Gen9 12 DIMMs Memory Cartridge

96 726722-B21 HPE 32GB 4Rx4 PC4-2133P-L Kit

2 802582-B21 HPE 400GB 12G SAS WI 2.5in SC SSD

1 779800-B21 HPE Ethernet 10G 2P 546FLR-SFP+ FIO Adptr

2 758836-B21 HPE 2GB FIO Flash Backed Write Cache

4 AJ763B HPE 82E 8Gb Dual-port PCIe FC HBA

1 656364-B21 HPE 1200W CS Plat PL HtPlg Pwr Supply Kit

1 P8B31A HPE OV w/o iLO 3yr 24x7 FIO Phys 1 LTU

1 BD505A HPE iLO Adv incl 3yr TS U 1-Svr Lic

1 H1K92A3 HPE 3Y Proactive Care 24x7 Service

1 H1K92A3 R2M HPE iLO Advanced Non Blade - 3yr Support

1 H1K92A3 SVP HPE One View w/o Ilo Supp

1 H1K92A3 YAY HPE ProLiant DL580 Gen9 Support

1 H6J85A HPE Rack Hardware Kit

1 H5M57A HPE Basic 4.9kVA/60320/C13/WW PDU

1 AF593A HPE 3.6m C19 Nema L6-20P NA/JP Pwr Crd

1 BW932A HPE 600mm Rack Stabilizer Kit

1 BW930A HPE Air Flow Optimization Kit

1 BW906A HPE 42U 1075mm Side Panel Kit

1 120672-B21 HPE Rack Ballast Kit



Table 1b. Bill of materials for the HPE 3PAR StoreServ 7450c used for this testing

QTY PART NUMBER DESCRIPTION

1 BW904A HPE 642 1075mm Shock Intelligent Rack

1 E7X93A HPE 3PAR StoreServ 7450c 4N St Cent Base

20 E7W54B HPE M6710 480GB 6G SAS 2.5in MLC 5yr SSD

1 BC914B HPE 3PAR 7450c Reporting Suite LTU

1 BC890B HPE 3PAR 7450c OS Suite Base LTU

80 BC891A HPE 3PAR 7450c OS Suite Drive LTU

2 QK753B HPE SN6000B 16Gb 48/24 FC Switch

48 QK724A HPE B-series 16Gb SFP+SW XCVR

6 QR490A HPE M6710 2.5in 2U SAS Drive Enclosure

60 E7W54B HPE M6710 480GB 6G SAS 2.5in MLC 5yr SSD

1 QR516B HPE 3PAR 7000 Service Processor

1 TK808A HPE Rack Front Door Cover Kit

48 QK735A HPE Premier Flex LC/LC OM4 2f 15m Cbl

8 QK734A HPE Premier Flex LC/LC OM4 2f 5m Cbl

4 H5M58A HPE 4.9kVA 208V 20out NA/JP bPDU

1 BW932A HPE 600mm Rack Stabilizer Kit

1 BW906A HPE 42U 1075mm Side Panel Kit

1 BD362A HPE 3PAR StoreServ Mgmt/Core SW Media

1 BD363A HPE 3PAR OS Suite Media

1 BD365A HPE 3PAR Service Processor SW Media

1 BD373A HPE 3PAR Reporting Suite Media

1 TC472A HPE Intelligent Inft Analyzer SW v2 LTU

Appendix B: Oracle configuration parameters for Oracle 12cR1 *.open_cursors=3000 *.processes=3000 db_cache_size=127488M shared_pool_size=16G *.db_cache_size=751082405888 *.java_pool_size=3758096384 *.large_pool_size=536870912 *.shared_pool_size=19327352832 *.streams_pool_size=20M *.sga_target=0


Appendix C: Oracle configuration parameters for Oracle 11gR2 *.open_cursors=3000 *.processes=3000 oradl580.__pga_aggregate_target=0 db_cache_size=127488M shared_pool_size=16G *.db_cache_size=751082405888 *.java_pool_size=3758096384 *.large_pool_size=536870912 *.shared_pool_size=19327352832 *.streams_pool_size=20M

Appendix D: Linux kernel configuration fs.file-max = 6815744 kernel.sem = 250 32000 100 128 kernel.shmmni = 4096 kernel.shmall = 1073741824 kernel.shmmax = 4398046511104 kernel.panic_on_oops = 1 net.core.rmem_default = 262144 net.core.wmem_default = 262144 fs.aio-max-nr = 1048576 net.ipv4.ip_local_port_range is 9000 65500 net.ipv4.ip_local_port_range = 9000 65500 vm.nr_hugepages = 77571 net.core.rmem_max = 134217728 net.core.wmem_max = 134217728 net.ipv4.tcp_rmem = 4096 87380 134217728 net.ipv4.tcp_wmem = 4096 65536 134217728 net.core.netdev_max_backlog = 300000

Note The vm.nr_hugepages is dependent on the amount of memory installed in the server, meaning there needs to be enough memory to satisfy the setting of hugepages. We had 3TB in the HPE ProLiant DL580 Gen9, so we set it to 77571 which is based on the Oracle requirement for shared memory.


Appendix E: Multipath configuration The following multipath parameters were included in the /etc/multipath.conf file, which are the recommended settings for RHEL 6 and HPE 3PAR Persona 2 (ALUA).

defaults { polling_interval 10

user_friendly_names yes find_multipaths yes } devices {

device { vendor "3PARdata"

product "VV" path_grouping_policy group_by_prio path_selector "round-robin 0" path_checker tur features "0" hardware_handler "1 alua" prio alua

failback immediate rr_weight uniform no_path_retry 18 rr_min_io_rq 1 detect_prio yes

} } blacklist { devnode "^(ram|zram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*" }

Appendix F: HPE 3PAR StoreServ 7450c udev rules For HPE 3PAR StoreServ 7450c storage configurations, a udev rules file /etc/udev/rules.d/12-dm-permission.rules was created to set the required ownership of the Oracle ASM LUNs:

ENV{DM_NAME}=="mpathr", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathy", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathx", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathw", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathv", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathu", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpatht", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpaths", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathz", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathaa", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathab", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathac", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathad", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathae", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathaf", OWNER:="oracle", GROUP:="oinstall", MODE:="660" ENV{DM_NAME}=="mpathag", OWNER:="oracle", GROUP:="oinstall", MODE:="660"


Appendix G: Udev rules ACTION=="add|change", KERNEL=="dm-*", PROGRAM="/bin/bash -c 'cat /sys/block/$name/slaves/*/device/vendor | grep 3PARdata'", ATTR{queue/rotational}="0", ATTR {queue/scheduler}="noop", ATTR{queue/rq_affinity}="2", ATTR{queue/nomerges}="1", ATTR{queue/nr_requests}="128"

Appendix H: Oracle tuned profile – ktune.sh The following is what’s contained in the file /etc/tune-profiles/oracle/ktune.sh.

#!/bin/sh # Oracle ktune script # based on the enterprise-storage profile . /etc/tune-profiles/functions start() { set_cpu_governor performance set_transparent_hugepages never disable_disk_barriers multiply_disk_readahead 4 return 0 } stop() { restore_cpu_governor restore_transparent_hugepages enable_disk_barriers restore_disk_readahead return 0 }

Appendix I: Oracle tuned profile – ktune.sysconfig The following is what’s contained in the file /etc/tune-profiles/oracle/ktune.sysconfig.

# Oracle ktune service configuration # based on the enterprise storage profile # This is the ktune sysctl file. You can comment this out to prevent ktune # from applying its sysctl settings. SYSCTL="/etc/sysctl.ktune" # Use *.conf files in the ktune configuration directory /etc/ktune.d. # Value: yes|no, default: yes # It is useful if you want to load settings from additional files. Set this to # no if you to prevent ktune from using these additional files. USE_KTUNE_D="yes" # This is the custom sysctl configuration file. Any settings in this file will # be applied before the ktune settings, thus may be overridden by ktune. SYSCTL_PRE="/etc/sysctl.d/* /etc/sysctl.conf" # This is the custom sysctl configuration file. Any settings in this file will # be applied after the ktune settings, overriding them #SYSCTL_POST="/etc/sysctl.d/* /etc/sysctl.conf" # This is the I/O scheduler ktune will use. This will *not* override anything # explicitly set on the kernel command line, nor will it change the scheduler # for any block device that is using a non-default scheduler when ktune starts.


# You should probably leave this on "deadline", but "as", "cfq", and "noop" are # also legal values. Comment this out to prevent ktune from changing I/O # scheduler settings. ELEVATOR="deadline" # These are the devices, that should be tuned with the ELEVATOR ELEVATOR_TUNE_DEVS="/sys/block/{sd,cciss,dm-,vd}*/queue/scheduler"

Appendix J: Oracle tuned profile – sysctl.ktune The following is what’s contained in the file /etc/tune-profiles/oracle/sysctl.ktune.

# Oracle ktune sysctl settings # based on the enterprise-storage profile # ktune sysctl settings for rhel6 servers with enterprise-class # storage, maximizing i/o throughput # # Minimal preemption granularity for CPU-bound tasks: # (default: 1 msec# (1 + ilog(ncpus)), units: nanoseconds) kernel.sched_min_granularity_ns = 10000000 # SCHED_OTHER wake-up granularity. # (default: 1 msec# (1 + ilog(ncpus)), units: nanoseconds) # # This option delays the preemption effects of decoupled workloads # and reduces their over-scheduling. Synchronous workloads will still # have immediate wakeup/sleep latencies. kernel.sched_wakeup_granularity_ns = 15000000 # If a workload mostly uses anonymous memory and it hits this limit, the entire # working set is buffered for I/O, and any more write buffering would require # swapping, so it's time to throttle writes until I/O can catch up. Workloads # that mostly use file mappings may be able to use even higher values. # # Oracle specific tunings vm.swappiness = 1 vm.dirty_background_ratio = 3 vm.dirty_ratio = 80 vm.dirty_expire_centisecs = 500 vm.dirty_writeback_centisecs = 100 kernel.shmmax = 4398046511104 kernel.shmall = 1073741824 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 fs.file-max = 6815744 fs.aio-max-nr = 1048576 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576 kernel.panic_on_oops = 1


Appendix K: Oracle tuned profile – tuned.conf The following is what’s contained in the file /etc/tune-profiles/oracle/tuned.conf.

# Oracle tuned profile # based on the enterprise-storage profile [main] [DiskMonitor] enabled=False [DiskTuning] enabled=False [NetMonitor] enabled=False [NetTuning] enabled=False [CPUMonitor] enabled=False [CPUTuning] enabled=False

Appendix L: Script for giving a higher priority to database and log writer processes #!/bin/bash for i in ` ps -ef|grep ora_lg|grep -v ASM|grep -v grep|cut -c9-15` do renice -1 ${i} done for i in ` ps -ef|grep ora_dbw|grep -v ASM|grep -v grep|cut -c9-15` do renice -1 ${i} done

Appendix M: Script to start the Oracle Database in concert with HPE-ATX #!/bin/bash hpe-atx -p rr_flat -l listener.log lsnrctl start hpe-atx -p rr_tree -l atx.log srvctl start database -db orcl12cdl580


Sign up for updates

Rate this document © Copyright 2016 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without notice. The only warranties for HPE products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HPE shall not be liable for technical or editorial errors or omissions contained herein.

Oracle is a registered trademark of Oracle and/or its affiliates. Intel and Xeon are trademarks of Intel Corporation in the U.S. and other countries. Red Hat is a registered trademark of Red Hat, Inc. in the United States and other countries. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

4AA6-5961ENW, June 2016, Rev. 1

Resources and additional links HPE & Oracle Alliance http://h22168.www2.hpe.com/us/en/partners/oracle/

HPE & Oracle Alliance Technical page http://h17007.www1.hpe.com/us/en/enterprise/converged-infrastructure/info-library/index.aspx?app=oracle

HPE ProLiant hpe.com/servers/proliant

HPE ProLiant DL580 Gen9 hpe.com/servers/dl580

HPE 3PAR hpe.com/storage/3par

HPE 3PAR StoreServ 8450 http://www8.hp.com/us/en/products/disk-storage/product-detail.html?oid=8737847

HPE Storage Solutions for Oracle http://www8.hp.com/us/en/products/data-storage/data-storage-products.html?compURI=1226406#tab=6

HPE Application Tuner Express https://h20392.www2.hpe.com/portal/swdepot/displayProductInfo.do?productNumber=HPE-ATX

HPE Reference Architectures hpe.com/info/ra

HPE Technology Consulting Services hpe.com/us/en/services/consulting.html

To help us improve our documents, please provide feedback at hpe.com/contact/feedback.

http://www.hpe.com/info/getupdated

http://www.facebook.com/sharer.php?u=https://www.hpe.com/h20195/V2/GetDocument.aspx?docname=4AA6-5961ENW

http://twitter.com/home/?status=HPE%20Reference%20Configuration%20comparing%20Oracle%2011g%20and%2012c%20performance%20on%20HPE%20ProLiant%20DL580%20Gen9%20and%20HPE%203PAR%20StoreServ%207450c+@+https://www.hpe.com/h20195/V2/GetDocument.aspx?docname=4AA6-5961ENW

http://www.linkedin.com/shareArticle?mini=true&ro=true&url=https://www.hpe.com/h20195/V2/GetDocument.aspx?docname=4AA6-5961ENW&title=HPE%20Reference%20Configuration%20comparing%20Oracle%2011g%20and%2012c%20performance%20on%20HPE%20ProLiant%20DL580%20Gen9%20and%20HPE%203PAR%20StoreServ%207450c+&armin=armin

https://hpresearch.az1.qualtrics.com/SE/?SID=SV_6EuQKOr2Ku1CUzH&Pubnumber=4AA6-5961ENW

http://h22168.www2.hpe.com/us/en/partners/oracle/

http://h17007.www1.hpe.com/us/en/enterprise/converged-infrastructure/info-library/index.aspx?app=oracle

http://www.hpe.com/servers/proliant

http://www.hpe.com/servers/dl580

http://www.hpe.com/storage/3par

http://www8.hp.com/us/en/products/disk-storage/product-detail.html?oid=8737847

http://www8.hp.com/us/en/products/data-storage/data-storage-products.html?compURI=1226406#tab=6

https://h20392.www2.hpe.com/portal/swdepot/displayProductInfo.do?productNumber=HPE-ATX

http://www.hpe.com/info/ra


http://www.hpe.com/contact/feedback

HPE Reference Configuration comparing Oracle 11g … Reference Configuration comparing Oracle 11g...

Documents

Transcript of HPE Reference Configuration comparing Oracle 11g … Reference Configuration comparing Oracle 11g...