WHITE PAPER Oracle Single-Instance Database Benchmark and ... · ORACLE INGLE-INSTANCE DATABASE...

WHITE PAPER

Oracle Single-Instance Database Benchmark and Tuning Practices

2© 2019 Datrium, Inc. All rights reserved Datrium | 385 Moffett Park Dr. Sunnyvale, CA 94089 | 844-478-8349 | www.Datrium.com

ORACLE SINGLE-INSTANCE DATABASE BENCHMARK & TUNING PRACTICESWHITE PAPER

Contents

Introduction 3

Terminology 4

Datrium DVX 5

Compute Node 5

Data Node 5

Oracle Single-Instance Database Benchmark Setup 6VM and OS Configuration 6

Hardware Configuration 6

2-Hour Oracle Burn-In Sustained Workload 6

6-Hour Oracle Burn-In with 10-minute Snapshots 7Know more about Datrium Native Snapshots 8

Encryption with Near-Zero Performance Impact 8

Oracle Databases on Datrium Tuning Practices 10

Conclusion 11

Appendix 12



IntroductionAs IT transforms to deliver with more agility into the On-Demand Economy, it has to change. There are too many products to manage data, and differences increase in multiple clouds. This keeps administrative load too high, limits workload portability, and makes governance across sites nearly impossible. Datrium Automatrix is the first autonomous data platform to converge the five essential data services - primary storage, backup, encryption, mobility and DR - with a consistent operating experience across multiple clouds. It eliminates product sprawl, workload immobility, and siloed data management to remove the roadblocks to business agility, while optimizing IT investment and mitigating risk.

The first steps to IT Transformation are to automate, simplify, and converge to lower cost. DVX was the first technology to use split provisioning to keep IO speed local to host configurations. It uses a separate and scaleable persistence pool to deliver zero storage administration. It also includes NoBackup™ data protection with policies at the workgroup level to give users instant RTO. With its unique approach to security, DVX can encrypt data in use, in flight, and at rest, while still supporting always-on deduplication, compression and erasure coding.

The Datrium DVX system provides an ideal virtualization platform for building a modern, state of the practice data center solution for database application services. Starting as small as a single virtualization host with a single database and scaling and clustering the instances as needed to meet the demands of business growth is easily accomplished with the capabilities of the Datrium platform.

This document demonstrate Datrium DVX resiliency and performance for Oracle workloads. This document also provides guidelines for deploying Oracle single-instance databases on Datrium DVX. The recommendations in this guide are specific to Datrium, and the examples and considerations provide guidance, but do not represent strict design requirements.

A successful deployment of Oracle on Datrium is not significantly different from deploying Oracle on physical servers. DBAs can leverage their current skill set while delivering the benefits associated with virtualization and Datrium’s enterprise features such as encryption, data reduction, snapshotting and replication.

Datrium is the perfect platform for running mission-critical Oracle databases.

Oracle solutions on Datrium starts with a single server and a single datanode and scale up to 10 data nodes and 128 servers as part of a single management domain, and a single namespace.

With Datrium, Oracle databases are always stored on Flash devices on the host where the Oracle virtual machine is in execution, and a copy of the data is protected on nodes dedicated to persistently storing data, aka datanodes. That means all read operations remains local to the host improving performance and reducing latency, while only write operations are sent over to datanodes, considerably decreasing read operation latencies.

What you will learn:

• Datrium provides consistent high-performance for Oracle databases

• Datrium protect Oracle databases with zero-impact native snapshots and replication

• Datrium native encryption provides near-zero performance degradation for Oracle databases

• Tuning practices for Oracle databases running on Datrium



TerminologySome of the terminology used in this document is covered here and in more detail throughout the document.

• DVX - The combination of at least one Compute Node and at least one Data Node configured together to provide an on-prem platform for running VMware vSphere hypervisor

• Compute Node – Servers that run the VMware vSphere hypervisor and the Datrium DVX software – Compute Nodes are equipped with local flash storage to run the primary workloads – more information can be found here:: https://www.datrium.com/resources/dvx-compute-node-specifications/

• Data Node – Datrium supplied storage appliance that maintains the persistent (final) copy of all data accessed on the Compute Nodes as well as backup copies (snapshots) of local data or data that has been replicated in from another DVX system – more information can be found here: https://www.datrium.com/resources/dvx-data-node-specifications/

• LVM - In Linux, Logical Volume Manager (LVM) is a device mapper target that provides logical volume management for the Linux kernel. Most modern Linux distributions are LVM-aware to the point of being able to have their root file systems on a logical volume

• Oracle ASM - In Oracle, ASM (Oracle Automatic Storage Management) is a volume manager and a file system for Oracle database files that supports single-instance Oracle Database and Oracle Real Application Clusters (Oracle RAC) configurations. Oracle ASM is Oracle’s recommended storage management solution that provides an alternative to conventional volume managers, file systems, and raw devices.

• Oracle Enterprise Linux (OEL) - Oracle Linux is a Linux distribution packaged and freely distributed by Oracle, available partially under the GNU General Public License since late 2006. It is compiled from Red Hat Enterprise Linux source code, replacing Red Hat branding with Oracle’s.



Datrium DVXThe Datrium DVX system is a combination of Compute Nodes and Data Nodes networked together to provide a scalable environment for independent processing (compute) and data (storage) management.

The figure below shows the separation of the compute and storage nodes enabled by the DVX architecture. This split provisioning deployment (Figure 1) is then connected back together through appropriate networking.

Compute NodeThe Datrium Compute Node is a physical server running VMware vSphere or KVM hypervisor and installed with the Datrium hyperdriver software. The Compute Node runs the application virtual machines directly from local flash on the host. Compute Node configurations can vary as it is possible to use Datrium Compute Nodes or customer supplied Compute Nodes.

Data NodeThe Datrium Data Node comes preconfigured with fully redundant, hot swappable components. This includes mirrored NVRAM for fast writes, and redundant 10Gb or 25Gb network ports with load balancing and path failover for high speed data traffic. The Data Node provides the resilient backing storage for the Compute Node flash-based data copy as well as the built-in backup storage for self-protecting datasets.



Oracle Single-Instance Database Benchmark SetupFor this benchmark we chose to use SLOB (Silly Little Oracle Benchmark) created by Kevin Closson, a tool focused on generating Oracle I/O workloads to stress the storage sub-system infrastructure.

Additionally, instead of pursuing complex and expensive virtual machine and database configurations for peak performance we have opted for a more simplistic and real-world approach that doesn’t produce a solution that would be difficult to support in production deployments.

VM and OS Configuration

• VMware ESXi, 6.7.0, 8941472

• Linux CentOS/7 and Oracle Enterprise Linux

• 12 vCPU

• 32GB RAM

• 1 x 100GB vDISK

• 6 x 250GB vDISK PVSCSI w/ LVM disk aggregation and XFS file-system for the Oracle database

• 1 Terabyte database

• Oracle 12.2.0.1 Enterprise Edition – Single Instance

Hardware Configuration

• PowerEdge R930 – E7-8890 v4 @ 2.20GHz / 2016

• 8 x 1.9TB SATA Samsung GC57 (MZ7LM240HMHQ0D3/2016)

• 1 x Datrium Datanode F12X2 2x25G-23TB

The hardware and software configurations above represent in its totality one compute node, one datanode, and one virtual machine. In a Datrium architecture datanodes are responsible for storing data permanently using erasure coding (similar to triple mirroring) , while compute nodes are responsible for keeping data local to the host on flash devices for providing data locality and fast access to the data.

[+] The SLOB configurations used during tests can be found in the Appendix.

2-Hour Oracle Burn-In Sustained Workload

Datrium handles high-performance and sustained workloads while providing high read I/O efficiency from host attached flash devices, and erasure coding data protection for persistent data.

The picture below demonstrates the virtual machine level metrics, as seen by the VMware hypervisor and perceived by users. At the peak, SLOB generated 68,336 IOPS with an average latency of 0.99 milliseconds, and the number of IO operations per second is fairly constant over the 2-hour sustained benchmark.



[+] There is a small latency fluctuation towards the end of the run with the maximum average latency of 3 milliseconds, when Oracle prefetches data. Some vendors disable the native and hidden Oracle prefetch feature to obtain lower latency results, but we decided to keep it similar to a real-world production scenario, where prefetching is often enabled.

6-Hour Oracle Burn-In with 10-minute Snapshots

Datrium provides seamless data protection during high-performance and sustained workloads, without disrupting applications.

The image below shows two 6-hour Oracle burn-in tests: the first being without snapshots, and the second with Datrium native snapshots taken every 10 minutes, and retained for 24 hours.

The second run, on the right, demonstrates the same workload with real-time native snapshots every 10 minutes (retained for 24 hours), and the Oracle databases experience no meaningful disruption to the running workload.



Know more about Datrium Native Snapshots

Datrium maintains two distinct “namespaces.” The “datastore” contains the current, live version of all VMs and files. This is what the hypervisor sees. The contents of these files always contain the most recently written data.

The “snapstore” on the other hand contains previous point-in-time snapshots of the live datastore as it existed earlier. Every time a protection group causes a snapshot to be taken, entries are made in the snapstore with the contents of files/VMs that are subscribed by the protection group, at that instant in time.

Datrium DVX uses a “redirect on write” (ROW) technique to store incoming data. New data is always written to new locations (vs. copy-on write techniques that can introduce delays as changes are copied). Because only changes are stored in a snapshot, and because DVX only stores compressed and deduplicated data, snapshots consume relatively little capacity.

Encryption with Near-Zero Performance ImpactDatrium’s encryption is an industry-first, implementing software-based end-to-end encryption with full data reduction (deduplication, compression, and erasure coding), across hosts, host SSDs, datanodes, and also for data in-flight (on the wire) and at-rest.

Datrium’s encryption uses FIPS AES-XTS-256 military grade crypto algorithm and leverage Intel Intelligent Storage Acceleration Library (Intel ISA-L) with Intel AES new encryption instructions set (Intel AES-NI), that provide <5% performance hit at worst case scenario, but in most cases is negligible. The Datrium encryption satisfies regulatory requirements for government agencies, banking, financial, healthcare and other G2000 enterprise customers.

• AES – Advanced Encryption Standard specified in FIPS 197.

• FIPS – Federal Information Processing Standard.

• XTS-AES – Mode of Operation defined in IEEE Std. 1619-2007 and approved by SP800-38E with one additional requirement on the lengths of the data units.

There are two types of encryption to be known:

• Approved mode is the FIPS mode that is using the encryption algorithm which is approved by FIPS, but the code path is not getting through the FIPS validation process.

• Validated mode is the FIPS mode that is using the encryption algorithm that is validated through the FIPS validation process and certified by NIST.

The FIPS-Validated mode provides excellent performance, but it does impose some additional CPU load, while FIPS-Approved provides minimal performance degradation. Both methods have no impact on data reduction.

The detailed NIST report for Datrium FIPS Object Module FIPS 140-2 can be found at https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp3079.pdf

The image below demonstrates three 1-hour Oracle burn-in tests: the first without encryption, the second with encryption set to Approved mode, and the third with encryption set to Validated mode. At the peak, SLOB generated 67,020 IOPS with an average latency of 1 millisecond.

The image below represents 5-minute roll-ups for all three 1-hour runs, and there is negligible I/O and Throughput variance – the numbers only vary a little because SLOB workload isn’t static. The VM-Level latency varies according to the workload, but the average latency is steady with a maximum peak of 1.3 milliseconds.

We also notice that deduplication and compression ratios remained basically the same during the entire set of tests, with and without end-to-end encryption.



[+] We measured I/O requests per Second issued by Oracle, as driven by SLOB. The 2nd run with the FIPS Approved-Mode presents a near-zero performance impact, and the 3rd run, with FIPS Validated-Mode presents a small variance of 1.8K I/Os. As mentioned, in FIPS Validated-Mode Datrium expect <5% performance hit at worst case scenario, and in this scenario the impact is only 3%.



Oracle Databases on Datrium Tuning PracticesStorage configuration is critical for a successful database deployment, especially in virtual environments where you may consolidate different workloads on a single host. Datrium storage subsystem provide sufficient I/O throughput as well as storage capacity to accommodate the cumulative needs of Oracle databases running in virtual machines.

IMPORTANT: VMware vSphere Best Practices is used extensively throughout this section. It is recommended to have a good understanding of VMware’s Storage concepts in order to fully comprehend Datrium’s best practices. The latest Oracle Databases on VMware Best Practices Guide can be found here: https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/solutions/vmware-oracle-databases-on-vmware-best-practices-guide.pdf

• Logical Volume Manager - Linux LVM or Oracle ASM (Oracle Automatic Storage Management) provide good performance with their default configuration. The difference between 512KB and 4MB block sizes is marginal, but larger block sizes should be used for read intensive workloads with Large Read I/O.

Our tests demonstrated that XFS with CentOS/7 provide the highest number of Read IO/sec in a single sample, and ASM + OEL (Oracle Enterprise Linux) provided the highest number of Write IO/sec (15,594). However, the highest average IO/sec for both Read and Write IOs across all benchmarks was provided by Oracle ASM + OEL, making ASM with Oracle Enterprise Linux the best option in terms of performance and manageability.

Example LVM:

fdisk /dev/sd{x} vgcreate vg-01 /dev/sdc1 /dev/sdd1 /dev/sde1 lvcreate -n oracle --extents 100%FREE -i3 -I512Kb vg-01 mkdir /oracle mount /dev/vg-01/oracle /oracle mkfs.xfs /dev/vg-01/oracle

• File System - When using Linux LVM the XFS file system provides best performance when paired with high speed disks, such as SSDs, that can take real advantage from parallelized access and multi-thread designs. EXT4 and EXT3 similarly provide good performance. Note: Datrium Guest File Restore is only supported on EXT3, EXT4, NTFS and FAT.

• Virtual Controller - Multiple paravirtual SCSI (PVSCSI) controllers for the database virtual machines or virtual machines with high I/O load. The use of multiple paravirtual SCSI controllers allows the execution of several parallel I/O operations inside the guest operating system.

• Virtual Disk - Multiple Hard disks evenly distributed across paravirtual SCSI (PVSCSI) controllers provide the enhanced parallel I/O operations inside the guest operating system.

Our tests demonstrate that peak performance is achieved with 3 Oracle data disks per controller, but 2 Oracle data disks per controller offer very similar performance results while keeping the solution simpler.



• Host Flash - Host flash play a big part on both read and write performance, and finding the right balance between quantity, capacity and performance is important. Our tests demonstrate that for Oracle databases, four or more cheaper and slower SSDs will perform at lower latency and higher IOPS than just two fast and expensive SSDs due to Datrium I/O parallelization and multithreading.

• Oracle Redo/Log - From a performance standpoint Datrium does not require separate Redo/Log I/O traffic from data file I/O traffic through separate virtual SCSI controllers. Some Oracle administrators may want to segment for database hygiene perspective.

• Encryption - Datrium’s encryption uses FIPS AES-XTS-256 military grade crypto algorithm and leverage Intel Intelligent Storage Acceleration Library (Intel ISA-L) with Intel AES encryption instructions set (Intel AES-NI), that provide <5% performance hit at worst case scenario, but in most cases it is negligible. In our tests, the performance impact from encryption was negligible. We recommend encryption to be Enabled for Oracle applications that carry sensitive data.

• Packet Size - The simplest and recommended setup of the DVX Data Network can use the default MTU of 1500 between Compute Nodes and Data Nodes with no significant performance considerations. Using Jumbo frames configured with an MTU of 9000 is not a requirement with DVX.

• Backup - Datrium protection groups technology provide native crash-consistent protection for Oracle database with instant recovery from local and remote snapshots. Where app-consistent data protection is required, Datrium compliments Oracle RMAN and Data Guard. RMAN is widely used for Oracle backup and restore functionality.

ConclusionDatrium DVX supports all the features needed to allow organizations to fully virtualize Oracle database solutions including RAC implementations. IT and Oracle administrators can take full advantage of the simplicity of management and performance gains of the DVX solution and deploy production Oracle databases – single instance or clustered – and leverage the advances brought by newer CPUs, local flash storage access, data protection management to meet the needs of today’s modern data centers.

This benchmark report demonstrates Datrium capabilities, including snapshotting and encryption, running under strenuous real-world conditions. The same results attained during the tests can be obtained by customers using standard and widely available server hardware.



AppendixSLOB

SLOB provides a multitude of possible configurations. We used what seems to be the most common configuration used by vendors – 70:30 Read/Write ratio. The SLOB execution config. 24 users were used to run the stress test and I used the following command: sh ./runit.sh 24

slob.conf PDATE_PCT=30 RUN_TIME=7200 SCAN_TABLE_SZ=1M WORK_UNIT=64 REDO_STRESS=LITE THREADS_PER_SCHEMA=1

• UPDATE_PCT - Percentage of SLOB update operations

• RUN_TIME - Wall- clock duration of a SLOB test in seconds

• SCAN_TABLE_SZ - size of the short scan table(s)

• WORK_UNIT - scope of blocks being manipulated by each operation

• REDO_STRESS - amounts of redo logging

• THREADS_PER_SCHEMA – sessions against each schema

Oracle

Here are the list of Oracle initialization parameters modified from default during tests.

filesystemio_options=setall db_files=2000 processes=8000 shared_pool_size=4G db_cache_size=1536M parallel_max_servers=0 pga_aggregate_target=9G db_block_checksum=false



Linux Kernel

Here are the Linux kernel parameters used during tests, with all other settings the same as the default installation.

fs.aio-max-nr = 1048576 fs.file-max = 6815744 kernel.shmall = 1073741824 kernel.shmmax = 4398046511104 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 kernel.panic_on_oops = 1 net.ipv4.ip_local_port_range = 9000 65500 net.ipv4.conf.all.rp_filter = 2 net.ipv4.conf.default.rp_filter = 2 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048576 vm.swappiness = 0

WHITE PAPER Oracle Single-Instance Database Benchmark and ... · ORACLE INGLE-INSTANCE DATABASE...

Documents

Transcript of WHITE PAPER Oracle Single-Instance Database Benchmark and ... · ORACLE INGLE-INSTANCE DATABASE...