OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST...

65
Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux and UNIX Systems E77753-01 October 2017

Transcript of OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST...

Page 1: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Oracle® Certification EnvironmentOAST Installation and User's Guide for OracleDatabase

12c Release 2 (12.2) for Linux and UNIX SystemsE77753-01October 2017

Page 2: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Oracle Certification Environment OAST Installation and User's Guide for Oracle Database, 12c Release 2(12.2) for Linux and UNIX Systems

E77753-01

Copyright © 2009, 2017, Oracle and/or its affiliates. All rights reserved.

Primary Author: Bharathi Jayathirtha

Contributors: Janelle Simmons, Jason Wu, Jing Liu, Lewis Kaplan, and Yongfeng Zhou

This software and related documentation are provided under a license agreement containing restrictions onuse and disclosure and are protected by intellectual property laws. Except as expressly permitted in yourlicense agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify,license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means.Reverse engineering, disassembly, or decompilation of this software, unless required by law forinteroperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. Ifyou find any errors, please report them to us in writing.

If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it onbehalf of the U.S. Government, then the following notice is applicable:

U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software,any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are"commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of theprograms, including any operating system, integrated software, any programs installed on the hardware,and/or documentation, shall be subject to license terms and license restrictions applicable to the programs.No other rights are granted to the U.S. Government.

This software or hardware is developed for general use in a variety of information management applications.It is not developed or intended for use in any inherently dangerous applications, including applications thatmay create a risk of personal injury. If you use this software or hardware in dangerous applications, then youshall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure itssafe use. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of thissoftware or hardware in dangerous applications.

Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks oftheir respective owners.

Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks areused under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron,the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced MicroDevices. UNIX is a registered trademark of The Open Group.

This software or hardware and documentation may provide access to or information about content, products,and services from third parties. Oracle Corporation and its affiliates are not responsible for and expresslydisclaim all warranties of any kind with respect to third-party content, products, and services unless otherwiseset forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not beresponsible for any loss, costs, or damages incurred due to your access to or use of third-party content,products, or services, except as set forth in an applicable agreement between you and Oracle.

Page 3: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Contents

Preface

Audience vii

Documentation Accessibility vii

Related Documents vii

Conventions viii

1 Introduction to OAST

1.1 Overview of OAST 1-1

1.2 What's New in OAST Kit Version 12.2 1-1

1.3 OAST Certification Requirements 1-3

1.4 Checklist for OAST 1-3

2 OAST Preinstallation Requirements

2.1 Software Requirements for Installing OAST 2-1

2.2 About OAST Distribution 2-3

2.3 Disk Space Requirements for Installing OAST 2-3

2.4 Database Requirements for Installing OAST Software in Server/Server Mode 2-4

3 About Installing OAST

3.1 Installing OAST Software 3-1

4 Postinstallation Tasks

4.1 About Validating OAST Installation 4-1

4.2 About Compiling the Loader and the Driver Binaries 4-2

4.3 About Validating the Loader and Driver Binaries 4-3

5 About Creating a Database

5.1 Creating a Database 5-1

iii

Page 4: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

5.1.1 About Validating Database Creation 5-2

5.2 Creating OAST Schema Creation and Loading Data 5-2

5.2.1 About Validating OAST Schema Creation and Data Load 5-3

5.3 OAST Offline Data Load Tool 5-4

6 Running OAST Workloads

6.1 About OAST Workloads 6-1

6.2 Running OAST Workload for a non-CDB 6-3

6.3 Stress and Destructive Tests 6-3

6.4 Running OAST Workload with CDB Enabled 6-4

6.4.1 Running Single Workload 6-4

6.4.2 Running Multiple OAST Workloads Concurrently 6-5

6.5 Running OAST Oracle RAC Elevator Test 6-7

7 Tuning OAST Workloads to Stress System Resource

7.1 Running the Stress Tests for OAST Certification 7-1

7.1.1 Four Stress Workloads 7-3

7.2 Additional Requirements for Oracle Database 12c 7-7

8 About Configuring OAST Database and Workloads for OracleDatabase 12c Certification

8.1 Configuring OAST Database and Workloads for Oracle Database 12cCertification 8-1

9 Troubleshooting

9.1 Utility Scripts 9-1

9.2 About General OAST Troubleshooting 9-1

9.2.1 Adding Data Files for out of space Condition 9-3

9.2.2 Adding Data Files to Support Multiple Devices 9-3

9.3 OAST Troubleshooting for Oracle Database 12c Release 1 9-5

9.3.1 createuser Process Fails 9-5

9.3.2 How to Compile and Build for Separate PDB Homes 9-6

9.3.3 Why Does OAST Installation Not Work in EMACS 9-6

9.3.4 How to Differentiate the Resources for Different PDBs 9-7

9.3.5 How to Add and Modify the Database Parameters 9-7

9.4 OAST Troubleshooting for Oracle Database 12c Release 2 9-7

9.4.1 OAST Installation Fails for a Workload 9-8

9.4.2 OAST Kit Compilation Fails for a Workload 9-9

iv

Page 5: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

9.4.3 OAST Database Build Fails for a Workload 9-9

9.4.4 OAST Data Load Fails for a Workload 9-11

9.4.5 Running Workloads in a Cluster with Insufficient Memory 9-12

9.4.6 Custom Workloads Versus All Workloads 9-12

9.4.7 Supported Shells for Login ID 9-12

9.4.8 Incorrect ASM Values Displayed in the Installer 9-12

9.4.9 User ID Different for OAST Home and Oracle Home 9-12

9.4.10 User Group is Different on Remote Client Node 9-13

9.4.11 Client Software Install Requirements 9-13

9.4.12 Oracle RAC Software Requirements 9-13

9.4.13 Creating non-CDB Databases for CRSTRESS and PQSTRESS Tests 9-13

9.4.14 How to Run RAC Elevator Test 9-13

A About CDB and PDB Utilities

B Workloads Configuration and FAQs

B.1 OAST Workloads Configuration B-1

B.2 Frequently Asked Questions B-2

Index

v

Page 6: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

List of Tables

2-1 Prerequisite for storage types 2-2

2-2 Disk Space Requirements for OAST Installation on a Server 2-3

2-3 An example of Disk Space Requirements for OAST Installation on a Server 2-3

vi

Page 7: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Preface

Use this guide to certify the system for Oracle Database 12c Release 2 (12.2) andOracle Real Application Clusters 12c Release 2 (12.2). This guide describes how toinstall the Oracle product software, the Oracle Certification Environment (OCE) OASTsoftware, and how to certify the system.

AudienceThis guide is intended for software engineers who are certifying Oracle Database 12cRelease 2 (12.2) on UNIX and Linux systems.

This guide assumes that users of Oracle Certification Environment (OCE) kit have noprior experience of certifying Oracle products on UNIX and Linux systems. However,users must have Oracle Database and Oracle RAC experience before startingcertification work.

Documentation AccessibilityFor information about Oracle's commitment to accessibility, visit the OracleAccessibility Program website at http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc.

Access to Oracle Support

Oracle customers that have purchased support have access to electronic supportthrough My Oracle Support. For information, visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trsif you are hearing impaired.

Related DocumentsThis guide contains information about installing Oracle Automated Stress/SystemTesting (OAST) software, which is part of the OCE kit for Oracle Database 12cRelease 2 and Oracle Real Application Clusters for Oracle Database 12c Release 2. Italso describes how to run the certification tests.

Installation and administrative tasks for Oracle products are provided in thefollowing guides:

The following documentation and all the relevant additional documentation is availableat the following location:

http://docs.oracle.com/en/database/

• Oracle Database Client Installation Guide

vii

Page 8: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

• Oracle Database Installation Guide

• Oracle Database Release Notes

• Oracle Grid Infrastructure Installation Guide

• Oracle Real Application Clusters Installation Guide

Ordering Documentation

To order documentation, call the appropriate number:

• In the United States, call Documentation Sales at 1.800.252.0303.

• In the United Kingdom, call Oracle Direct Response at +44.990.332200.

• In other European countries, contact your local Oracle Support office.

• In the Asia-Pacific region, contact your Oracle sales representative.

Shipping Enquiries

For shipping inquiries, product exchanges, or returns, call the appropriate number:

• In the United States, call Client Relations at 1.650.506.1500.

• In the United Kingdom, call Customer Relations at +44.990.622300.

• In other European countries, contact your local Oracle Support office.

• In the Asia-Pacific region, contact your Oracle sales representative.

ConventionsThe following text conventions are used in this document:

Convention Meaning

boldface Boldface type indicates graphical user interface elements associatedwith an action, or terms defined in text or the glossary.

italic Italic type indicates book titles, emphasis, or placeholder variables forwhich you supply particular values.

monospace Monospace type indicates commands within a paragraph, URLs, codein examples, text that appears on the screen, or text that you enter.

Preface

viii

Page 9: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

1Introduction to OAST

This chapter provides an overview of Oracle Automated Stress/System Testing(OAST) for certification testing and contains information about the following topics:

• Overview of OAST (page 1-1)

• What's New in OAST Kit Version 12.2 (page 1-1)

• OAST Certification Requirements (page 1-3)

• Checklist for OAST (page 1-3)

1.1 Overview of OASTOAST is an Oracle partner standard stress test suite intended to test Oracle Databaseon a particular system by simulating a real world environment.

OAST includes a set of test drivers for stress testing platform specific CPU, I/O,Interconnect, and Memory subsystem when you run Oracle database server software.

OAST workloads can be tuned to maximize the load on all nodes within a cluster or asingle node, when running a single instance database.

OAST takes a user input of the system and feeds them into its built-in sizing engine.The sizing engine produces an input parameter (number of warehouses) to the stressOLTP kit generator, which creates a stress OLTP kit. This OLTP kit is then used to:

• Prepare the database

• Create the schema

• Populate the tables and views

• Create indexes

The run script (nrunoastoltpXXX.sh) prepares database backup and restore, andperforms stress test runs.

Note:

In 12c Release 2, OAST run script for CDB-enabled configurations isoast_main.sh.

1.2 What's New in OAST Kit Version 12.2This topic mentions the new features in the OAST kit version 12.2.

• OAST kit version 12.2 includes a new installer with the following features:

1-1

Page 10: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

– Menu-driven customizable workloads, such as CPU, IOP, DSS, CRSTRESS,and PQSTRESS supporting multitenant container database (CDB) and non-CDB configurations.

– Auto discovery of GRID, Oracle RAC, and Client information.

– Menu driven configuration for Oracle ASM.

– Maximum PDB limit enhanced to 4k for Cloud and Exadata customers. Themaximum PDB limit for on-premise customers is limited to 252.

– Automation of execution of parallel installation tasks, such as, kit compilation,database build, and data load.

– Ability to have unique OAST homes built for a specified subset of PDBs.

– Dependency on ORACLE_HOME environment variable removed.

– Listener configuration process streamlined.

– Auto extend implemented on all data files. So, data files are no longerrestricted to install time configuration size.

– Improved secure passwords.

– Database build and delete scripts have been updated.

– Enabled display of on-screen install configuration information.

– Total elapsed time displayed for all installation components.

– Enabled visual feedback of installation progress.

– Enabled on-screen help.

• The following new workloads have been included:

– Multitenant Operations (MTOPS) workload automatically performs the typicalDBA day-to-day tasks against Oracle Multitenant Database Architecturefeatures. This workload is designed to run in parallel with existing stress testworkloads.

– Parallel Query Interconnect Stress (PQSTRESS) workload drives cache fusiontraffic across the private interconnect through global cache traffic. This test isbased on the TPC-H benchmark, simulating complex warehouse queries.

• Application Container support is now added that consists the following:

– Application root

– Application container

– Application name

– Application versions

• PDB undo tablespace (Local UNDO) is enabled by default.

• Advanced Index Compression added to provide improved index compressionratios while still providing an efficient OLTP access to indexes.

• Support to Multi-Process Multi-Threaded (MPMT).

• RAC elevator can now be run using any one work load home. Oracle recommendsto use the CPU workload.

• Removed RAC elevator node naming requirement limitation.

Chapter 1What's New in OAST Kit Version 12.2

1-2

Page 11: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

1.3 OAST Certification RequirementsTo achieve OCE standards, OAST must satisfy the standards mentioned in thissection.

Hardware and Software Configurations:

The vendor hardware system must be capable of running a balanced databaseworkload with sufficient memory, CPU, network, and storage resources. A separateclient system is required to run OAST stress workloads in client/server mode for bothSingle Instance and Oracle RAC configurations. For Oracle RAC testing, you must runthe OAST stress tests and Oracle RAC destructive tests on the same hardwareconfiguration. Refer to the Destructive and Stress Test Plan for Vendor CompatibilityWith Oracle RAC 12cR2 document for specific details. You can download thisdocument from the following location:

http://www.oracle.com/technetwork/database/index.html

The Oracle database configuration required to run OAST stress workloads depends onthe vendor technology stacks that needs to be certified.

Oracle and the vendor will work together to define the minimum requirement for OracleRAC 12c Release 2 configuration. The configuration must be recorded when initiatingthe certification project using Certification Management System (CMS).

Workload Requirements:

1. Run OAST workloads successfully for 48 hours to stress the system resource tothe required level on a Single instance or an Oracle RAC configuration as follows:

a. The CPU utilization must be greater than or equal to 90 percent.

b. Network traffic (interconnect and storage usage throughput) must reachapproximately 70 percent of their capacities. It is recommended to use OracleOrion to calibrate the I/O subsystem throughput.

Multiple OAST stress workloads, such as, OLTP, DSS, and Oracle RACInterconnect stress workload may be required to run concurrently to achieve thestress level. Refer to Tuning OAST Workloads to Stress System Resource(page 7-1) and About Configuring OAST Database and Workloads for OracleDatabase 12c Certification (page 8-1) for recommendations.

2. For an Oracle RAC configuration, you must run the Oracle RAC elevator test for12 hours.

1.4 Checklist for OASTAfter successfully running the OAST tests, you must complete the OAST section withinthe certification checklist. You can download the checklist from the following website:

http://www.oracle.com/technetwork/database/index-089070.html

The certification checklist helps you to record the successful completion of all thetests. If you have any questions related to the checklist, contact the OCE group at [email protected].

Chapter 1OAST Certification Requirements

1-3

Page 12: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

2OAST Preinstallation Requirements

This chapter describes the preinstallation requirements for installing OAST. It includesinformation about the following topics:

• Software Requirements for Installing OAST (page 2-1)

• About OAST Distribution (page 2-3)

• Disk Space Requirements for Installing OAST (page 2-3)

• Database Requirements for Installing OAST Software in Server/Server Mode(page 2-4)

2.1 Software Requirements for Installing OASTBefore installing OAST, you must install the following Oracle products depending onthe type of testing you want to perform:

• Oracle Database — Software only option

• Oracle Database Client (Recommended)

Note:

Oracle Database Client software is required to install OAST in the client/servermode. Refer to “Installing OAST Software (page 3-1)” for more details.

You must select the Custom installation type to install Oracle Database Client12c Release 2. In the Client Selection screen during installation, chooseSelect All to install all Oracle Database clients.

• Oracle Grid Infrastructure

Note:

Oracle Grid Infrastructure is required to install OAST in an Oracle RACenvironment. You must install and configure Oracle Grid Infrastructure.

Before you install OAST in client/server mode, the client/server must meet thefollowing requirements:

• The installation must use the same user name that was used when installingOracle Database on a client or a server.

• Passwordless SSH connectivity must be enabled between all the nodes and theclient, and from the client to all the nodes.

• Oracle client node name is required during OAST installation in Client/Servermode.

2-1

Page 13: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Note:

The client information is required for installing OAST in client/server mode.

When you run oastinstall to install OAST in the client/server mode, the driver isdistributed to the client during the installation process. The Oracle Database versionand the Oracle Database Client version must be identical. If you have any questionsrelated to the installation, then contact the Oracle Certification Environment Group at https://oce.oracle.com/cms/welcome.do and create a CMS issue for the certificationmanagement system identification number (CMSID).

You can run OAST for all storage types supported by Oracle Database, such as, NFS,dNFS, Oracle Cluster File System, Oracle Automatic Storage Management, andOracle Automatic Storage Management Cluster File System (Oracle ACFS). Thefollowing table lists the prerequisites for the storage types. OAST enables OracleManaged Files.

OAST is restricted to data file creation within a single mount, share, or OracleAutomatic Storage Management disk group. Use the db_create_file_dest parameter tocreate the destination location. If additional mount points or file locations are needed,then refer to Adding Data Files to Support Multiple Devices (page 9-3) .

Table 2-1 Prerequisite for storage types

Storage Type Prerequisite

Oracle Automatic Storage ManagementCluster File System

Create the shared volumes and validateaccess for each node in the cluster.

Oracle Automatic Storage Management Create the required disk groups before youinstall OAST. You can use a single disk groupto store both the data files and the redo logs.

Cluster File System Create the shared volumes and validateaccess for each node in the cluster.

NFS Create the NFS mounts and validate accessfor each node in the cluster.

dNFS Refer to Oracle Database Administrator’sGuide for details about enabling dNFS.

All the storage types support the following file types:

• .dbf

• .arc

• .arch

• .log

• .ctl

• .spfile

• .ora

• .tmp

Chapter 2Software Requirements for Installing OAST

2-2

Page 14: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

2.2 About OAST DistributionThe OAST installation contains the following directories and files:

Directory and Files Contains

install The installation script and other supportdirectories. OAST is installed in the OASThome directory.

orion The Oracle Orion scripts that are used to verifythe I/O throughput.

readme.oast The readme file.

2.3 Disk Space Requirements for Installing OASTThe following are the disk space requirements for OAST on a 100 GB database, whichis equal to 1500 warehouses on an Oracle Database 11g Release 2 (11.2) and 500warehouses on an Oracle Database 12c Release 2 (12.2).

Table 2-2 Disk Space Requirements for OAST Installation on a Server

Item Disk Space Requirement

OAST software 100 MB

OAST home 10 GB to 1 TB depending on workloadselection and database size for offline dataload.

An example for disk space requirements for OAST installation on a server is listed in Table 2-3 (page 2-3). The server details are as follows:

• 1500 Warehouses, which is equal to 100 GB seed

• 4 Nodes

• 2 Log Groups

• 1 Member per Log Group

Table 2-3 An example of Disk Space Requirements for OAST Installation on aServer

Item Disk Space Requirement

Seed data 100 GB

Logs 2 GB

Free space 30 GB

Total space without archiving (A sum of seeddata, logs, and free space)

132 GB

Archive Directory 200 GB

Total space with archiving 332 GB

Chapter 2About OAST Distribution

2-3

Page 15: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Note:

Table 2-2 (page 2-3) and Table 2-3 (page 2-3) do not include the disk spacerequirements for the Oracle product software. For more information about diskspace requirements, refer to the documents listed in "Related Documents(page vii)."

The workload details required for installing OAST in server/server mode are as follows:

Workload Parameter Length

I/O-CPU W2 24 hours

Interconnect W7 3 hours

Oracle RAC Elevator Not Applicable 144 Steps

2.4 Database Requirements for Installing OAST Software inServer/Server Mode

This section describes the requirements for installing OAST in an OCE environment.OCE certification requires OAST to be installed in the server/server mode. A singledatabase is sufficient with each workload executed in a serialized fashion.

When you install OAST in server/server mode, the OAST driver is executed on thesame servers that contains the Oracle database. OAST automatically executes thedriver on each node in the cluster using the driver shell script on node 1.

OAST installation in server/server mode requires the following database:

OAST Version Database Name Number of warehouses

12.1 oastdb 500

12.2 Based on the selection youmake during installation.

Varies based on the data sizeyou choose.

Chapter 2Database Requirements for Installing OAST Software in Server/Server Mode

2-4

Page 16: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

3About Installing OAST

To install OAST, you need a local file system with sufficient disk space. Typically, youneed 1 GB of OAST home for a recommended configuration with online data load.

For offline data load, you must reserve disk space to generate data. For an OracleRAC configuration, you must have a shared file system disk space, such as, NFS orCFS.

Before you install OAST, ensure that your system meets the preinstallationrequirements as described in OAST Preinstallation Requirements (page 2-1).

Before installing OAST, you must complete all the preinstallation tasks. It is importantthat the system meets all the requirements as described in Oracle Grid InfrastructureInstallation Guide, Oracle Database Installation Guide, and Oracle Real ApplicationClusters Installation Guide for a successful installation. Refer to your platform-specificdocumentation for more details.

The vendor must validate the hardware and the supporting software.

Note:

Running OAST on a system that does not meet the hardware requirementsmay negatively impact the ability of OAST to generate stress.

3.1 Installing OAST SoftwareYou must install OAST software for certifying and testing Oracle Database on aparticular system by simulating a real world environment.

Before you run the installer script:

• Set the target OAST home location on local disk on all the nodes.

For example, /u01/oast_root.

• Ensure that the terminal window size is greater than 42 lines and 80 columns.

The recommended terminal window size is 50 (lines) x 120 (columns).

• Set the user equivalence for passwordless access to all the server nodes, andoracle client node in the client/server mode.

Note:

Shared OAST home is not supported.

Contact OCE team if you need access to any OCE kits.

3-1

Page 17: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

To install OAST, perform the following procedure:

1. Download the OAST package, which is in the zip format.

2. Use the following commands to extract the archive to a temporary kit staging arealocation. For example, /u01/OAST_KIT:

$ mkdir/u01/OAST_KIT$ cd /u01/OAST_KIT

3. Download the zip file to the current location.

4. Use the unzip utility to extract the contents of the zip file:

$ unzip OAST_version number.zip

After extracting the files, the following files and directories are listed:

$ ls

install OAST_version number.zip orion readme.oast readme.windows

sample.windows version.info

5. Run the following commands to install:

$ cd install

$ ./oastinstall

As the OAST RAC installation progresses, the installer screen displays informationabout GRID HOME, ORACLE HOME, Cluster nodes, and so on. For example, the followingfigure shows the information displayed for the following configuration:

• GRID HOME: /scratch/oce/grid [12.2.0.1.0]

• ORACLE HOME: /scratch/oce/db122 [12.2.0.1.0]

• cluster nodes: slc11nbj slc11nbm slc11nbk slc11nbl

• client/server mode

• OAST home root: /scratch/oce/oast_root

OAST Configuration Tool

GRID HOME:/scratch/oce/grid [12.2.0.1.0]

Cluster Node list: slc11nbj slc11nbm slc11nbk slc11nbl

Version - type - Oracle Home(s)

1. 12.1.0.2.0 - [ RAC ] - /scratch/oce/db121

2. 12.2.0.1.0 - [ RAC ] - /scratch/oce/db122

Choose : 2

Note: [RAC] - Oracle Real Application Cluster Home.

[SI] - Oracle Single Instance Home.Generic help

Make choice

Configuration information

progress

6. Enter 2 to install OAST 12.2.0.1.0.

7. Follow the prompts and make appropriate choices for your configuration. Anexample of OAST 12.2.0.1.0 installation is as shown in the following figure:

Chapter 3Installing OAST Software

3-2

Page 18: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

OAST Configuration Tool

GRID HOME:/scratch/oce/grid [12.2.0.1.0]

Cluster Node list: slc11nbj slc11nbm slc11nbk slc11nblORACLE HOME:/scratch/oce/db122 [12.2.0.1.0]

ORACLE BASE:/scratch/oce/orabase

1.OAST RAC instance mode.

2.OST single instance mode.

Choose mode :1

Note: OAST Kit Installation mode.

8. Enter 1 to install OAST in RAC mode.

9. Follow the prompts and provide appropriate information. For example:

• Install Home root: /scratch/oce/oast_root

Ensure that the install home root directory is empty, writable, and hassufficient space on all the nodes.

• Client/Server mode [y/n]?

– Enter y to install OAST in the client/server mode. The installer prompts forthe client node name. Follow the prompts to select the client home fromthe client server.

Note:

OAST expects actual client software installation. OAST will not use or acceptOracle grid or database home for client installation.

– Enter n to skip client information.

• Storage configured with HCC [y/n]?

– Enter y if you are sure that the storage supports Hybrid ColumnarCompression (HCC).

– Enter n if you are unsure. This is a recommended and a safe option.

Note:

HCC feature is supported only on storage of Exadata/ZFS/Axiom.

After the information provided by you is validated, the installer prompts you tochoose a custom configuration or all the workloads with default settings. Anexample of custom configuration is as shown in the following figure:

Chapter 3Installing OAST Software

3-3

Page 19: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

OAST Configuration Tool

GRID HOME :/scratch/oce/grid [12.2.0.1.0]

Cluster Node list :slc11nbj slc11nbm slc11nbk slc11nblORACLE HOME :/scratch/oce/db122 [12.2.0.1.0]

ORACLE BASE :/scratch/oce/orabase

TEST Mode :RAC

Database Admin managed :true

Install Home root : /scratch/oce/oast_root

Home root path shared :false

Client/Server mode :true

Client node : slc03eta

Oracle Client Home : /scratch/oce/client/12201 [12.2.0.1.0]

HCC enable storage :false

ASM DG/size :DATA:4324MB, DATABASE:818964MB

Remaining DG/size ;DATA:4324MB, DATABASE:818964MB

Selected workloads : [ ]

1.cpu load.

2.iop load.

3.dss load.

4.crstress load.

5.pqstress load.

d.Done.

Re. Reset workloads.

Choose :

The installer goes through each workload type configuration. You can modify eachworkload for size, storage, archive mode, redo logs, and so on.

10. After configuring each workload type, enter d to continue.

You will be prompted again to confirm or reconsider your choices. Choosing thesame workload repeatedly will overwrite the existing configuration. Only the finalselection is retained.

11. Enter No to retain the default configuration and proceed with the installation.

Note:

If you enter Yes, then the installer resets all the configuration. If you provideany other input, then the installer goes back to the previous menu withoutaltering any values.

12. If required, customize an individual workload and modify the allowed parametersthat are relevant to OAST.

You can view, modify, or continue to the next stage of the installation. Thefollowing figure shows an example of the installer screen where you can modifythe OAST parameters:

Chapter 3Installing OAST Software

3-4

Page 20: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

OAST Configuration Tool

Parameter

1. CPU DB NAME

2. CPU CDB ENABLE

3. CPU PDB COUNT

4. CPU MEMORY PERCENT

5. CPU DB BLOCK SIZE

6. CPU ENABLE ARCHIVE MODE

7. CPU INMEMORY CLAUSE DEFAULT

8. CPU MEMORY SIZE (MB)

9. CPU MEMORY TARGET (MB)

10. CPU PROCESSES

11. CPU REDO LOG SIZE (MB)

r. Resset New Values

c. Continue

Choose:

Note : Choose a work load’s parameter value(s) you wish to change.

Current Value

cpudb

true

1

30

4k

false

inmemory

1024

6554

5000

512

New Value

Each option allows you to change the default value of the OAST parameter. Aftermodifying the parameters, the configuration is recreated with the new values.

13. Provide your choices for the following queries:

• Build database [y/n]?

Enter y to create database after the installation

Enter n to create the database manually.

• LOAD data [y/n]?

Enter y to load the seed data during installation.

Enter n to load the seed data manually after the installation is complete.

The following figure shows an example of OAST configuration for a singleworkload:

Chapter 3Installing OAST Software

3-5

Page 21: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

OAST Configuration Tool

GRID HOME :/scratch/oce/grid [12.2.0.1.0]

Cluster Node list :slc11nbj slc11nbm slc11nbk slc11nblORACLE HOME :/scratch/oce/db122 [12.2.0.1.0]

ORACLE BASE :/scratch/oce/orabase

TEST Mode :RAC

Database Admin managed :true

Install Home root : /scratch/oce/oast_root

Home root path shared :false

Client/Server mode :true

Client node : slc03eta

Oracle Client Home : /scratch/oce/client/12201 [12.2.0.1.0]

HCC enable storage :false

ASM DG/size :DATA:4324MB, DATABASE:818964MB

Remaining DG/size ;DATA:4324MB, DATABASE:818964MB

Selected workload :cpu:1GB

CPU=>Install:In-progress, kitcomplie:Pending,

database build;Manual, DataLoad:Manual

You can choose any custom set or all workload with default settings as required.For each workload, kit installation and kit compile are completed by default. If youenabled database build, then all the databases for each workload will be built inparallel. If you enabled data load, then the data load will continue after eachdatabase is built.

The installer displays the time and date every 5 seconds to ensure that theinstallation tasks are in progress.

After the tasks are completed, the installer returns the prompt.

During the installation, you can monitor the logs in the OAST ROOT/workload/log/*.log files for each stage.

14. When the installer prompts, execute the manual steps in the OAST HOME directory. .

15. If you enabled database build and data load, then the server is ready to run thetests after the installation is complete.

16. To build a database manually, run the following command in each OAST home:

$ oast_main.sh -dbca y

After the database is built and there are no errors, you can proceed to load thedata depending on the workload type.

17. For CPU, DSS, and IOP workloads, run the following command in OAST home:

$ oast_main.sh -d y

18. For CRSTRESS workload, run the following command in OAST home:

$ oast_main.sh -d c

19. For PQSTRESS workload, run the following command in OAST home:

$ oast_main.sh -d p

After you complete all the manual steps, the OAST kit is ready to run the stressworkloads.

Chapter 3Installing OAST Software

3-6

Page 22: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

4Postinstallation Tasks

After installing OAST, you must validate the installation. The following sectionsdescribe the procedure to validate the OAST installation:

• About Validating OAST Installation (page 4-1)

• About Compiling the Loader and the Driver Binaries (page 4-2)

• About Validating the Loader and Driver Binaries (page 4-3)

4.1 About Validating OAST InstallationUse the tee command to validate the logs generated as part of the redirection. Reviewthe output for any command errors.

For an Oracle RAC installation, all the nodes in the cluster must containthe $OAST_HOME/oast_cluster directory. This directory must contain multiple files anddirectories.

You must also review the following files and subdirectories in the OAST_HOME directoryon the primary node. However, this is not a complete list of the generated files andsubdirectories.

• oast_main.sh

Contains the main script if multitenant container database (CDB) is enabled. Referto About CDB and PDB Utilities (page A-1) for more information.

• $CDBNAME_pdb* directory

Contains the PDB OAST home if CDB is enabled. Each directory is a valid OASThome. Refer to About CDB and PDB Utilities (page A-1) for more information.

• nrunoastoltpnum_ware.sh

Contains the main drive script, the value for num_ware depends on the size of theseed data entered.

• oastoltpnum_ware directory

Contains the scripts to generate tablespaces and tables, and start the loaderprogram.

• oastoltpnum_ware/log

Contains the output of the -d y option used with the nrunoastoltpnum_ware.sh scriptto execute the schema creation and load.

• dbca_template

Contains the files required to run DBCA in silent mode. OAST uses these files tobuild the final DBCA template, which is a part of the installation process.

• oast_cluster (Oracle RAC installation only)

4-1

Page 23: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Contains the directory with multiple subdirectories, supporting scripts, and sourcecode. There are several primary files under this directory structure as follows:

– oast_cluster/oastoltpnum_ware

Contains the rac_elevator.ksh driver script and supporting files. Refer to Running OAST Oracle RAC Elevator Test (page 6-7) for more details.

– oastoltpnum_ware/oastenv.sh

Contains various export variables.

– oastoltpnum_ware/tnsnames.ora

Contains the tnsnames.ora file that is used by the drivers. OAST uses theTNS_ADMIN environment variable to control the location of the tnsnames.ora file.

– oastoltpnum_ware/stressrun/scripts

Contains the scripts that support the execution of the driver. During theexecution, the output is temporarily stored in this directory, including thetps.out script. After the execution, the logs are copied to the oastoltpnum_ware/stressrun/results/testname directory.

– oastoltpnum_ware/stressrun/source/server

Contains the Oracle Call Interface (OCI) code for the loader and driver. Thecode in this directory is compiled with the -kitcompile command to generatethe driver and the loader binaries.

– oastoltpnum_ware/stressrun/bin

Contains the following loader and driver binaries that are generated after asuccessful -kitcompile command:

oastoltpload.exerunoastoltpb.exe

– oastoltpnum_ware/stressrun/results

Contains the final results of the execution and completion of a stress run increating a testname subdirectory. For example, execution and completion ofthe following command:

nrunoastoltp1500.sh -n iorun -u 100 -w 1 -t 600

where, 1500 is num_ware and oast_home is /home/oracle/oast/home will create theresults subdirectory as follows:

/home/oracle/oast/home/oast_cluster/oastoltp1500/stressrun/results/iorun

4.2 About Compiling the Loader and the Driver BinariesTo compile the loader and the driver binaries, namely, the oastoltpload.exe andrunoastoltpb.exe , you must pass the -kitcompile option to thenrunoastoltpnum_ware.sh script. This script will attempt to locate the compiler andgenerate files for the current platform. However, if a location cannot be determined,then the script will prompt for the path as necessary. Use the following commands:

cd $OAST_HOME./nrunoastoltp1500.sh -kitcompile 2>&1 | tee kcompile.txt

Chapter 4About Compiling the Loader and the Driver Binaries

4-2

Page 24: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

If CDB is enabled, then use the following commands to compile all the binaries for allthe PDB OAST homes:

cd $OAST_HOME./oast_main.sh -kitcompile y 2>&1 | tee compile_all.out

The oastoltpload.exe binary is responsible for loading the initial seed metadata for thevarious tables under the oastoltp user name.

The runoastoltpb.exe binary is a stress-harnessed OCI-multithread client. This binarysupports all workload types, except crstress, which are workload 6 and workload 7.Workload 6 and workload 7 do not use the runoastoltpb.exe for stress generation, butuses a set of supporting shell scripts and SQL commands to drive interconnect trafficin an Oracle RAC environment.

4.3 About Validating the Loader and Driver BinariesThe -kitcompile option generates the following two binaries in the oastoltpnum_ware/stressrun/bin directory:

• oastoltpload.exe

• runoastoltpb.exe

When you install OAST for an Oracle RAC configuration, the install pushes thebinaries to each node. You must verify the binaries that are located in theoastoltpnum_ware/stressrun/bin directory.

Note:

Binaries generated on all platforms will have the .exe extension. This is theexpected behavior.

Chapter 4About Validating the Loader and Driver Binaries

4-3

Page 25: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

5About Creating a Database

This chapter contains information about creating a database and the OAST schema. Itincludes the following topics:

• Creating a Database (page 5-1)

• Creating OAST Schema Creation and Loading Data (page 5-2)

5.1 Creating a DatabaseDatabase creation and data loading can be initiated during installation. If databasecreation and data load are part of the installation, then the tasks run in parallel for eachwork load type. At the end of the installation, the databases are created, the data isloaded, and ready to run the tests. If you choose not to build the database or not toload the data during installation, then follow the procedure mentioned in this section tocreate a database.

Although database creation and the oastoltp schema creation and data loading can beexecuted sequentially by combining the -dbca y -d y|c options that control thesesteps, Oracle recommends issuing each command separately and validating eachstep. This approach saves time and allows for easier isolation of the issue becauseOAST does not validate creation of the database before executing the schemacreation. The schema creation and data loading is a complex step and producesmultiple pages of output.

Executing the -dbca y option of the nrunoastoltpnum_ware.sh script deletes an existingdatabase by the same name and then creates the database. OAST uses OracleDatabase Configuration Assistant (DBCA) in silent mode to control the deletion andcreation steps. Perform the following steps to create a database:

For a non-CDB

1. Navigate to the OAST home directory:

cd $OAST_HOME

2. Run the following script:

./nrunoastoltpnum_ware.sh -dbca y 2>&1 | tee dbcreate.txt

For a CDB

1. Navigate to the OAST home directory:

cd $OAST_HOME

2. Run the following script:

./oast_main.sh -dbca y 2>&1 | tee dbca_all.out

If you are using Oracle Automatic Storage Management (Oracle ASM), then DBCAprompts for the ASMSNMP password. This is the same password that you entered as part

5-1

Page 26: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

of the Oracle Automatic Storage Management creation step during an Oracle GridInfrastructure installation.

If you do not remember the ASMSNMP password, then you must reset the password. Usethe following commands to reset the ASMSNMP password:

srvctl config asm -a displays the grid infrastructure oracle homeexport ORACLE_HOME= grid_oracle_homeexport ORACLE_SID= ASM_instance_name for current nodeasmcmdASMCMD> password ASMSNMP

5.1.1 About Validating Database CreationValidate the database creation by reviewing the generated output. If an error is found,review the Oracle Database Configuration Assistant log files in the followingdirectories:

• $ORACLE_BASE/cfgtoollogs/dbca

• $ORACLE_BASE/cfgtoollogs/dbca/db_name

where oracle_base and db_name are values entered during the installation phase. IfOracle Database Configuration Assistant fails to create the database, then you mustreview the alert and trace logs in the following directory:

$ORACLE_BASE/diag/rdbms/db_name/instance_name/trace

Common Issues

The common issues during validating database creation include:

Not enough memory defined as shared memory for the SGA.System configuration settings and undefined or invalid database file locations.

When a failure occurs, understanding and resolving the issue is critical. Reexecutionof the creation phase works very rarely without additional modifications to the system.If the failure is related or traced back to invalid input entered during the installationphase, then you must start the installation at the oastinstall phase. Otherwise,reexecuting the database creation will be necessary.

5.2 Creating OAST Schema Creation and Loading DataOAST schema creation and data load are initiated by using the -d y option with thenrunoastoltpnum_ware.sh script. This phase will also execute Oracle's calibrate I/OPLSQL package. This package determines I/O throughput by IOPS and MBPS for theexisting file names used within the OAST database.

The freeextent.rpt and extent.rpt reports, which provide usage values are alsogenerated at the end of this phase. Both these reports are generated inthe $OAST_HOME/oastoltpnum_ware directory. Perform the following steps to load the data:

1. Navigate to the OAST home directory:

cd $OAST_HOME

2. Run the following script:

./nrunoastoltpnum_ware.sh -d y 2>&1 | tee schema.txt

Chapter 5Creating OAST Schema Creation and Loading Data

5-2

Page 27: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Note:

If CDB is enabled, then you must use the following command to load data forall PDB OAST homes:

cd $OAST_HOME./oast_main.sh -d y 2>&1 | tee load_all.out

You will be prompted with the following options:

Would you like to use offline loading?

1 Yes

2 No

If you choose yes, then OAST will use offline load tools to generate the .csvdata files and then to load the data. You must provide the data directory tostore the .csv files. Otherwise, OAST will use online loading, which will notgenerate the .csv files.

Schema creation and data loading can take several hours to complete depending onthe number of warehouses and the processor speed of the nodes. The schema iscreated under the database user name OASTOLTP. This phase uses the environmentvariables in oastenv.sh located in the $OAST_HOME/oastoltpnum_ware directory.

The crstress workload (-w 6 an -w 7) requires to use the -d c options to generate therequired schema and tables. The crstress schema creation and load does not use theoastloader for data load. Data load is managed using .sql files.

Note:

If CDB is enabled, then oast_main.sh does not support to create crstressworkload. You must navigate to a certain PDB OAST home to use the -d coption instead.

5.2.1 About Validating OAST Schema Creation and Data LoadAs with other tasks, validation starts with reviewing the redirected output. OASTterminates the step if an error occurs. OAST generates schema and loader logs inthe $OAST_HOME/oastoltpnum_ware/log directory. You must review the trace.log file forerrors or issues. The trace.log file provides the start and stop entries, with a timestamp for each step in the process. All the tasks in the file must have a done entry. IfOAST fails during a step, then a stop file may be generated in the $OAST_HOME/oastoltpnum_ware directory. This stop file must be removed before you run the SchemaCreation - Data Load phase. Since OAST does not clear the $OAST_HOME/ostoltpnum_ware/log directory when you rerun the schema creation phase, it isrecommended that you archive or move the directory to a different location before yourerun the schema creation phase.

Before you continue with the next phase, you must manually validate the tables withinthe schema by logging in as the oastoltp user.

Chapter 5Creating OAST Schema Creation and Loading Data

5-3

Page 28: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

cd oast_home/oastoltpnum_ware. ./oastenv.shsqlplus oastoltpEnter password: passwordSQL>set pages 100SQL>select table_name from user_tables;SQL>select count(*) from $table_name;

Where table_name can refer to any of the following tables:

• WARE

• DIST

• ITEM

• NORD

• CUST

• ORDR

• HIST

• STOK

• ORDL

These tables must exist under the oastoltp user. The row count of each table dependson the number of warehouses.

The following table lists the results of a SQL query for an OAST database of 1500warehouses after the data load phase. The database size is approximately 100 GB.The WARE table must have 1500 rows and the DIST table row count must be ten timesthe WARE table row count, which is 15000 in this case.

TABLE_NAME NUM_ROWS

WARE 1500

DIST 15000

ITEM 100000

NORD 13500000

CUST 45000000

ORDR 45000000

HIST 45000000

STOK 150000000

ORDL 450000000

5.3 OAST Offline Data Load ToolOAST includes a separated offline data load tool that can be used to generate OASTdata to .csv files and to separately load the oastoltp schema. This tool is a newfeature with Oracle Database release 12.2, but it can be used for both OracleDatabase release 11.2 and Oracle Database release 12.2.

OAST offline data load generates data to .csv file first, then loads the data to OASTdatabase from .csv file in bulk, which greatly improves the data load speed. Previous

Chapter 5OAST Offline Data Load Tool

5-4

Page 29: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

version of OAST uses online data load, which generates and loads data record byrecord.

OAST offline data load tool is located in the $OAST_HOME/oastoltpXXX/loader directory.You can use oastdata.sh –g y to generate data and oastdata.sh –l y to load data.

The detailed usage is as follows:

[aime@adc2171369 loader]$ ./oastdata.shUsage:oastdata.sh [-h] [-g y/n] [-l y/n] [-w WARE] [-d DIR] [-o LOG] [-r ROW] [-R ROW]

Where:-h help-g [y/n] generic data-l [y/n] load data-w [WARE] warehouse number-d [DIR] data directory-o {OUT] log directory-r [ROW] fixed duplicate columns in a row-r [ROW] random duplicate columns in a row

For example,

oastdata.sh -g y

Generates the data.

oastdata.sh -l y

Loads the data.

oastdata.sh -g y -l y

Generates and loads the data.

Chapter 5OAST Offline Data Load Tool

5-5

Page 30: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

6Running OAST Workloads

This chapter contains information and procedures about running OAST workload,running the stress tests and destructive tests, and running Oracle RAC Elevator test. Itincludes the following topics:

• About OAST Workloads (page 6-1)

• Running OAST Workload for a non-CDB (page 6-3)

• Stress and Destructive Tests (page 6-3)

• Running OAST Workload with CDB Enabled (page 6-4)

• Running OAST Oracle RAC Elevator Test (page 6-7)

6.1 About OAST WorkloadsThe details of the OAST workloads that are required to run destructive testing are asfollows:

• OLTP - workload 1

The OLTP workload is used to drive IOPS against the storage subsystem thatcontains the database and the redo files. Using workload 1, OAST will execute thetransactions in terms of small random access, which in turn drives IOPS. OASTmaintains a very consistent rate of IOPS by using a working set that is larger thanthe current buffer cache size. OAST achieves this by randomly selecting primarykey values before execution of the transaction. This randomization produces ahigh probability that the block needed for the current transaction is not in the buffercache, provided the working set is larger than the buffer cache.

• I/O/CPU - workload 2

The I/O and CPU workload is used to drive both IOPS and CPU. The smallrandom I/O and CPU stress is obtained by utilizing a working set that allows foreach thread to have distinct warehouse and district IDs. These IDs are permanentand do not change when a new transaction is generated for a particular client.

• Interconnect - workload 3

Note:

This workload is different from workload 6 and workload 7.

This workload does not provide high interconnect traffic, but generates a workloadthat causes all clients to access the same warehouse, district, and customer. Thiswill force cache buffer exchanges, but will not generate any significant datatransfer due to compression and other cache fusion performance enhancements.This workload causes a significant amount of cache fusion locking.

6-1

Page 31: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

• CPU - workload 4

The CPU workload is used to drive high CPU consumption by using an in-memorydatabase. A CPU-only workload is achieved by using a read only weightedtransactional deck and using a working set that is smaller than the buffer cache.The resulting workload will show initial read activity as the buffer cache is primedwith data. After the buffer cache is primed, any requests for blocks will comedirectly from the buffer cache and not from the disk. The primary keys areconsistent and not regenerated on each pass.

• Decision Support System - workload 5

The Decision Support System (DSS) workload is used to drive MBPS against thestorage subsystem containing the database. Using workload 5, OAST executesfull table selects that aggregate the individual rows of the tables. With ParallelQuery enabled, the DSS workload uses the PGA as the receiving area for the dataread. Direct Path reads are executed and the buffer cache is bypassed withmultiple parallel query slaves providing additional throughput by allowingconcurrent reads from the disk.

• Consistent Read Transfer Single block Workload - workload 6

The CRSTRESS single block workload is used to drive cache fusion traffic acrossthe interconnect against the storage subsystem containing the database. Usingworkload 6, each node executes a full table scan from an in-memory table residingon another node. The full table scan causes the node to request a single blockdata transfer from another node. This action generates significant interconnecttraffic.

• Consistent Read Transfer Multiblock Workload - workload 7

The CRSTRESS multiblock workload is used to drive cache fusion traffic acrossthe interconnect against the storage subsystem containing the database. Usingworkload 7, each node executes a full table scan from an in-memory table residingon another node. The full table scan causes the node to request a multi-nodeblock data transfer from another node. This action generates significantinterconnect traffic.

• Parallel Query Interconnect Stress - workload 8

The Parallel Query Interconnect Stress (PQSTRESS) workload is used to drivecache fusion traffic across the private interconnect through global cache traffic.This test is based on the TPC-H benchmark, simulating complex warehousequeries.

• Multitenant Operations - workload 9

The Multitenant Operations (MTOPS) workload is used to automatically performthe typical DBA day-to-day tasks against Oracle’s Multitenant DatabaseArchitecture features. This workload is designed to run in parallel with the existingCDB stress test workloads. The main features of this workload include:

– Cloning of PDBs

– Creation and management of common and local users and privileges

– Tablespace resizing and new feature schema, tables, and index additions

– Database Backup and Recovery

– Unplugging and dropping of PDBs

– Application Container support

Chapter 6About OAST Workloads

6-2

Page 32: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

6.2 Running OAST Workload for a non-CDBTo start an OAST workload for a non-CDB, perform the following steps:

1. Navigate to the OAST home directory.

For a server/server mode, use the following command:

cd $OAST_HOME

For a client/server mode, use the following command:

cd $OAST_HOME/oast_clients

2. Run the following command:

./nrunoastoltp1500.sh -n test1 -u 10 -t 300 2>&1 | tee test1.outwhere:1500 is the number of warehousestest1 is the test name10 is the number of users300 is the time in seconds-w 1 is the I/O workload

After you run the command, the driver/workload results are copied to the followingdirectory:

For a server/server mode:

$OAST_HOME/oast_cluster/oastoltpnum_ware/stressrun/results/testname

For a client/server mode:

$OAST_HOME/oast_clients/oastoltpnum_ware/stressrun/results/testname

where test_name is the value passed to the -n option. During the execution, OASTtemporarily generates the results in the $OAST_HOME/oast_cluster/oastoltp num_ware/stressrun/scripts directory.

Use the tail -f command to view tps.out file during execution. OAST generatesAutomatic Workload Repository (AWR) report. If you execute the workload for theoastoltp user name, then an invalid user name or invalid object error occurs, whichindicates a failure in the Schema Creation and Data Load Phase.

6.3 Stress and Destructive TestsAfter you configure the cluster, databases, and workload to the required specificationsand before you perform the destructive tests, you must run the long running stresstests. This test requires that all the workloads are run at the pre-configured network,storage, and cpu balanced saturation levels for 48 hours.

After the criteria is met, the destructive tests must be run successfully. The mainsubset of these destructive tests that must be run is described in the white paperDestructive and Stress Test Plan for Vendor Compatibility With Oracle RAC 12cR1,you can download this whitepaper from the following website:

www.oracle.com/technetwork/index.htm

Chapter 6Running OAST Workload for a non-CDB

6-3

Page 33: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Because the vendor certification involves hardware technologies that may not havebeen envisioned and are not specialized and do not fit into generic or generalizedcategory, the destructive tests (and sometimes also in the specification of the stressand function tests) must be designed specifically for the technology. For example, anew storage technology will most likely need increased and possibly more complexdestructive storage tests. An FCOE implementation, especially one that may bevirtualized sharing private, public, and storage network data over the same hardwarewould need specialized and increased testing across the network interfaces, namelycable, switches, and CNA adapters. A stretch cluster would need destructive distancetesting.

A new requirement that has been added to the destructive tests is that snapshots forAWRs are now taken in the following scenarios:

• After a system is running stable, which is before failure inducement.

• After a system has returned to full stability, which is after failure recovery.

By adding these two snapshots, there are now three AWR views available. The fullyconfigured working system, the system during failure, and the system after failure.

Blackouts, brownouts, steady state can now be analysed both through AWR reportsand the tps.out report. However, the tps.out report is of no use for the CRS Stressand does not granularize the results per node. OSWatcher results also provide a goodlook from the system side.

6.4 Running OAST Workload with CDB EnabledStarting from Oracle Database 12c Release 1 (12.1.0.1), OAST supports pluggabledatabase (PDB) and multitenant container database (CDB) features. If you enableCDB option during OAST installation, then you can run OAST workload with CDB.There are two methods to run the OAST workload with CDB:

• Running Single Workload (page 6-4)

• Running Multiple OAST Workloads Concurrently (page 6-5)

6.4.1 Running Single WorkloadIn this method, OAST performs workloads within the same PDB. However, differentworkloads must be executed separately. This is similar to earlier versions Oracledatabase where CDB feature was not supported. For example, to start an I/Oworkload, you must perform the following steps:

1. Navigate to the OAST home directory.

cd $OAST_HOME

2. Run the following command:

./nrunoastoltpXXXX.sh -n test1 -u 10 -t 300 2>&1 | tee test1.outwhere:XXXX is the number of warehousestest1 is the test name10 is the number of users300 is the time in seconds-w 1 is the I/O workload

Chapter 6Running OAST Workload with CDB Enabled

6-4

Page 34: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

6.4.2 Running Multiple OAST Workloads ConcurrentlyIf you want to run multiple workloads, then you must create multiple PDBs for eachworkload. During installation, specify the PDB number and the size for each PDB. Forexample, if the target OAST database name is cdb, then during OAST installation, youmust answer the following questions:

1. Would you like to create OAST db as a CDB?

You chose: Yes

2. How many PDBs do you want to create in this container db? [1-4098]:

3

3. The PDBs are created as follows:

PDB Name: cdb_pdb1 PDB_OAST_HOME: /home/oracle/oast/home/cdb_pdb1

PDB Name: cdb_pdb2 PDB_OAST_HOME: /home/oracle/oast/home/cdb_pdb2

PDB Name: cdb_pdb3 PDB_OAST_HOME: /home/oracle/oast/home/cdb_pdb3

where cdb can be one of the following:

• cpu

• iop

• dss

• crstress

• pqstress

4. Please enter data size for pdb cdb_pdb1 [10 Gigabytes]:

1

Size for pdb cdb_pdb1 is 1 GB.

Please enter data size for pdb cdb_pdb2 [10 Gigabytes]:

2

Size for pdb cdb_pdb2 is 2 GB

Please enter data size for pdb cdb_pdb3 [10 Gigabytes]:

3

Size for pdb cdb_pdb3 is 3 GB

Note:

During OAST installation, the following pdb oast homes are installed in themain oast home directory:

• $OAST_HOME/cdb_pdb1

• $OAST_HOME/cdb_pdb2

• $OAST_HOME/cdb_pdb3

Chapter 6Running OAST Workload with CDB Enabled

6-5

Page 35: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

5. If the database is not built, then create the database by running the followingcommands:

cd $OAST_HOME./oast_main.sh -dbca y

Note:

Alternatively, you can also run the command, nrunoastoltpxxxx.sh -dbca yfrom each pdb oast home.

6. If the data is not loaded, then build the oast schema and load the data for eachpdb oast home. Run the following commands:

cd $OAST_HOME./oast_main.sh -d y

Note:

Alternatively, you can also run the command, nrunoastoltpxxxx.sh -d y fromeach pdb oast home.

7. To start multiple workloads concurrently, for example 4 workloads together, youmust run the following commands from different terminals simultaneously:

OLTP terminal:

cd $OAST_HOME/cdb_pdb1./nrunoastoltp5.sh -no test1 -w 1 -u 10 -t 900 2>&1 | tee test1.out

CPU terminal:

cd $OAST_HOME/cdb_pdb2./nrunoastoltp10.sh -kitcompile./nrunoastoltp10.sh -no test4 -w 4 -u 10 -t 900 2>&1 | tee test4.out

DSS terminal:

cd $OAST_HOME/cdb_pdb3./nrunoastoltp15.sh -kitcompile./nrunoastoltp15.sh -no test5 -w 5 -u 10 -t 900 2>&1 | tee test5.out

MTOPS terminal:

cd $OAST_HOME/oastoltp num_warehouses/mtops./ mtopsmain.sh 2>&1 | tee test6.out

Chapter 6Running OAST Workload with CDB Enabled

6-6

Page 36: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Note:

• Do not use the -n option. This will restart the database, use -no instead.

• The main script, oast_main.sh can only execute -dbca. This script cannotbe used to run stress tests. The -kitcompile, -d y, and no are executedfrom the main oast home directory.

• The CRSSTRESS and MTOPS workloads must be executed separately.

6.5 Running OAST Oracle RAC Elevator TestTo run OAST Oracle RAC Elevator test, perform the following steps:

1. If necessary, reload the OAST data using the following commands:

$ cd $OAST_HOME$ nrunoastoltpXXX.sh -d y

2. Run the following commands to check the database by starting and stopping theOracle RAC database:

$ cd $OAST_HOME/oast_cluster$ ./multinodedb.sh node1 node2 nodeX

3. Ensure that the listener is correctly running:

Note:

The main driver of the Oracle RAC elevator script will use the bequeathprotocol to connect. However the Oracle RAC elevator script on the main nodewill attempt to validate user connect counts by using the tnsnames entry oast1.

cd $OAST_HOME/oast_cluster/oastoltpXXX. ./oastenv.shsqlplus oastoltp/oastoltp@oast1

4. Run the elevator test as follows:

$ cd $OAST_HOME/oast_cluster/oastoltpXXX/nxhome$ ./rac_elevator.ksh -n rac-elevator-test -oasthome $OAST_HOME -w XXX -steps 144 -elevator_kind r -shutdown_mode bystep > elevator.out 2>&1

5. Check the generated output files elevator.out for errors.

Chapter 6Running OAST Oracle RAC Elevator Test

6-7

Page 37: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

7Tuning OAST Workloads to Stress SystemResource

This chapter contains information and procedures about tuning OAST workload,running the stress tests and destructive tests, and the requirements to run the stresstests for Oracle Database 12c. It includes the following topics:

• Running the Stress Tests for OAST Certification (page 7-1)

• Additional Requirements for Oracle Database 12c (page 7-7)

7.1 Running the Stress Tests for OAST CertificationTo run the stress tests for OAST certification, perform the following steps:

1. At least one separate client machine is required and the client machine must beoperating system resource capable in terms of cpu, memory, and network, to drivethe necessary client requests to the database cluster. Since OAST is an openworkload, an additional client machine may be required to drive CPU utilization onthe cluster. If the cluster database is unable to achieve a significant CPU value,which is above 90 percent, then you must monitor the CPU utilization on the clientmachine for high system CPU utilization. If the client machine is 100 percentutilized, then adding an additional client machine may be necessary.

Note:

The client machine may be undersized to drive the necessary workload stresson the cluster. You must specifically monitor the CPU on the client todetermine if the client is CPU bound. You can copy the OAST client homefrom client1 to client2 if an additional client is needed, Oracle database mustbe installed on the second client using the same oracle home location.

2. At least three Oracle ASM diskgroups are required: Each Oracle ASM diskgroupmust be mounted on physically separate spindles on the shared storage. AnOracle ASM diskgroup and its underlying disks and volumes must not share thephysical disks with any other Oracle ASM diskgroup. The minimum threediskgroups are: DATADG1, DATADG2, and REDODG1. The first diskgroup, DATADG1 must beused to mount three databases to drive CPU, IOPS/OLTP, and Interconnect/Network databases. The redo logs for each of these databases must reside on thethird diskgroup REDODG1. The second diskgroup, DATADG2 must be used to mount theDSS datafiles and its redo logs. Place the redo logs for the DSS on DATADG2. Theseparation of OLTP (IOPS) and DSS (MBPS) on physical spindles will helpachieve a controlled and consistent IOPS and MBPS flow by reducing physicaldisk arm movement for these distinctively different I/O workloads.

3. If the operating system supports ASMLIB, then it must be correctly configured to usethe multipath name and exclude single port device names.

7-1

Page 38: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

4. Storage validation: Orion is an Oracle provided standalone binary and can be usedto validate and determine the IOPS and MBPS values of the storage. The bestpractices for using Orion are as follows:

a. Run IOPS against the LUNs that will be used for DATADG1 and REDODG1sequentially on each node in the cluster. All IOPS results for each node mustbe within a close range. If the values are different, then you must determinethe specific reason and reexecute.

b. Run MBPS against the LUNs that will be used for DATADG2 sequentially on eachnode in the cluster. All MBPS results for each node must be within a closerange. If the values are different, then determine the specific reason andreexecute.

c. After you run IOPS and MBPS and you have consistent values for each on allthe nodes, you must run IOPS against the LUNs that will be used for DATADG1and REDODG1 in parallel on each node in the cluster. All IOPS results for eachnode must be within a close range. If the values are different, then you mustdetermine the specific reason and reexecute.

d. Run MBPS against the LUNs that will be used for DATADG2 in parallel on eachnode in the cluster. All MBPS results for each node must be within a closerange. If the values are different, then you must determine the specific reasonand reexecute.

e. Individual values for IOPS must approximately be equal to the sum of thecorresponding IOPS that you run in parallel.

Individual values for MBPS must approximately be equal to the sum of thecorresponding MBPS that you run in parallel.

For example if the MBPS value for a sequential run as mention is step 4.b(page 7-2) is 800 for each node, then the MBPS value for a parallel run asmention in step 4.d (page 7-2) should sum upto to 800. Each node mustexecute at 100MBPS.

5. Redo logs: Use a minimum of 6 redo log groups. The default for OAST is 2, so youmust enter a value of 6 during oastinstall. IOPS workload can generate asignificant amount of redo logs, if the storage array is very efficient in IOPS andlatency. Monitor the alert.log file for checkpoint not complete messages, whichindicate that the logs are switching too fast to complete the checkpoint. You mayneed to increase the size of the redo logs.

6. If you use multiplexing, ensure that each thread resides on a different REDOdiskgroup. Multiplexing writes the same redo into multiple location. Using the sameredo diskgroup for each thread will only cause contention.

7. Install in the client/server mode.

8. The client operating system and the servers operating system must be the same.

9. Oracle Database version on the client and the servers must be the same.

10. The installation user name must be the same for both the client and the servers.

11. The client must have the ability to use password-less ssh to all nodes in thecluster.

12. Each node must have the ability to use password-less ssh to the client. Thismeans that each node must have the ability to communicate with all the clientmachines without a password, using ssh.

Chapter 7Running the Stress Tests for OAST Certification

7-2

Page 39: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Note:

You must use only ssh to communicate remotely. Other remote communicationmethods, such as, rsh and dsh are not supported.

13. Execute each workload independently to verify the resource and stress utilization.Validate if each workload is running as expected and independent of anotherworkload. This procedure is critical in validating each workload in the standalonemode before executing the workload in parallel.

14. When you run the test concurrently, disable the OS Watcher on all stress runsexcept one. Use the -osw n option for nrunoastoltpXXX.sh.

15. The client's clock must have the same time zone, date, and time as the servers.

7.1.1 Four Stress WorkloadsFour databases are typically necessary to drive the resource utilization and increaseload. Resource utilization in this document is defined as CPU, Memory, Networkinterconnect traffic, and I/O to the shared storage subsystem. For I/O, both largerandom I/O in terms of MBPS and small random I/O in terms of IOPS are targeted.

Each workload is a separate oastinstall scenario with interconnect requiring additionalattention during the database and seed load steps. Refer to New Workload forInterconnect Traffic section in the readme.oast file for specific details.

Verify each workload independently. Execute each workload in the standalone modeto verify if the targeted operating system resource is fully utilized. Once each resourceis verified, execute all the four workloads concurrently. When running the testconcurrently, disable the OS Watcher (as the default mode is enabled) on all thestress runs except one. Use the -osw n option with nrunoastoltpXXX.sh script to disable.

The four workloads required are as follows:

Note:

Modify the source code in the correct location and recompile by using the -kitcompile command.

The SCALE value is defined in the oastoltpvars.sh file located on the client atthe : OAST_HOME/oast_clients/oastoltpXXX/stressrun/scripts directory.Updating the SCALE value does not require a recompile of the code.

OLTP:CPU:Database 1

• Workload 4 ( -w 4)

• Datafiles: DATADG1

• Redo logs: REDODG1

• Database Name: oastcpu

• User count: Depends

Chapter 7Running the Stress Tests for OAST Certification

7-3

Page 40: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

This workload drives CPU utilization on all nodes in the cluster by using a read-onlytransaction set that is fully cached in the SGA. The buffer cache must be large enoughto hold the working set. The buffer cache working set (the number of buffers used bythe workload) is controlled by an export value called, SCALE.

To execute the CPU workload you must modify the SCALE export value inoastoltpvars.sh script on the client, by setting it to 5 * number of nodes. Note that theoastoltpvars.sh is modified, as each workload will have this file. This will initially beequal to the number of warehouses. Reduce the value to 5 * number of nodes. Youmust have at least SCALE value for warehouses, otherwise the driver will generate anerror.

For example, if the total warehouse value is 15, with 8 nodes, then setting the SCALE toa value above 15 causes an error. The SCALE value is used to limit the number ofbuffers consumed by OAST by reducing the warehouse ID primary key range. A largerwarehouse range and a higher SCALE value will result in more rows being read from thedisk.

Multiple Client machines

Multiple client machines could be required to support the necessary workload (userthreads -u option) to drive CPU on the cluster. The number of client machinesdepends on the individual server performance and specifications, the number ofnodes, and the client's machine specification. Monitor the client machine for CPUutilization at incremental increases in the users threads. Use the -u option withnrunoastolptXXX.sh script. If client CPU is over 90 percent, then additional user threadsor workers (increasing the –u value) on the same client will have a negative impact andcause a reduction in resource utilization on the cluster. At this point an additional clientmachine will be necessary to drive additional user threads and achieve a highpercentage of CPU on the server.

For example:

• Vendor cluster: 8 nodes

• CPU: 2 clients running each with 110 users. Total users would be 220 from boththe clients. User count would be 220/8 users per node

OLTP: IOPS

• Workload 1 ( -w 1)

• Datafiles: DATADG1

• Redo logs: REDODG1

• Database Name: oastiop

• User count: 25 * number of nodes

Do not modify the SCALE value initially. The OAST driver will divide and partition thewarehouses based on the number of nodes in the cluster. Since this is an IOPSworkload, you must have sufficient warehouses to cause a miss for the current primarykey. Adjusting the SCALE value reduces the read and write ratio. If all the blocks fit intobuffer cache, then the workload will be more write intensive than read. As the LGWR andDBWR will be active, no additional reads are required after the cache is primed.

For example:

• Vendor cluster: 8 nodes

Chapter 7Running the Stress Tests for OAST Certification

7-4

Page 41: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

• Total users 200 per node. User count would be 200/8, which is equal to 25 usersper node

CRS:Interconnect

• Workload 7 (-w 7). Oracle recommends to use workload 8 (-w 8).

• Datafiles: DATADG1

• Redo logs: REDODG1

• Database Name: oastcs

• User count: 16

Do not modify the SCALE value. The load option for this workload to build the requiredtables and seed data is -d c. OAST automatically generates a workload that is capableof handling the number of nodes running in the cluster. This workload causesmultiblock transfers using full table scan, from and to all the nodes in the cluster. The -u option is not recommended if you are using more than 16 users. If you do not seehigh interconnect traffic during the run, then the buffer cache is probably too small.The buffer cache size must be large enough to handle both sending blocks andreceiving blocks. If the buffer cache is not large enough, then the workload will beforced to disk and no interconnect traffic will be generated. The –blocks parameter canbe used to increase or decrease the number of blocks. The value provided is the totalsum of blocks that will be used for sending and receiving data divided by 2. Fiftypercent of this value is allocated for sending the blocks and the remaining fifty percentis allocated for receiving the blocks.

For example:

• Vendor cluster: 8 nodes

• Total users 16, users per node is 16, multiblock interconnect traffic.

DSS:MBPS

• Workload 5 (-w 5)

• Datafiles: DATADG2

• Redo logs: REDODG2

• Database Name: oasdss

• User count: 3 * number of nodes

Set the following init.ora parameters:

1. Assign the DOP for the tables that queried for DSS.

alter table ordl parallel 4;alter table stok parallel 4;alter table cust parallel 4;alter table hist parallel 4;alter table ordr parallel 4;

2. Set the following common parameters:

alter system set parallel_adaptive_multi_user=FALSE scope=both;alter system set parallel_min_servers=8 scope=both;alter system set parallel_max_servers=64 scope=both;alter system set parallel_degree_policy=FALSE scope=both;

Chapter 7Running the Stress Tests for OAST Certification

7-5

Page 42: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

3. The following settings depend on the name of the dss database. This exampleassumes that the db_name is set to oastdss with an admin database. If you areusing a policy-managed database, then the sid will have an underscore and thenumber of nodes will be equal to 8.

Add or subtract depending on the number of nodes in the cluster. Modify the sidvalue to the corresponding instance names on your system.

Values for the instance_groups and parallel_instance_group parameters do notneed to be modified, the following values will work on all systems.

Note:

For an Oracle RAC non-cross node PQ, add or subtract based on the numberof nodes. Isolate PQ to individual nodes.

The following example is for an 8 node cluster:

alter system set instance_groups='g1','all' scope=spfile sid='oastdss1';alter system set instance_groups='g2','all' scope=spfile sid='oastdss2';alter system set instance_groups='g3','all' scope=spfile sid='oastdss3';alter system set instance_groups='g4 ,'all' scope=spfile sid='oastdss4';alter system set instance_groups='g5','all' scope=spfile sid='oastdss5';alter system set instance_groups='g6','all' scope=spfile sid='oastdss6';alter system set instance_groups='g7','all' scope=spfile sid='oastdss7';alter system set instance_groups='g8','all' scope=spfile sid='oastdss8';

alter system set parallel_instance_group='g1' scope=spfile sid='oastdss1';alter system set parallel_instance_group='g2' scope=spfile sid='oastdss2';alter system set parallel_instance_group='g3' scope=spfile sid='oastdss3';alter system set parallel_instance_group='g4' scope=spfile sid='oastdss4';alter system set parallel_instance_group='g5' scope=spfile sid='oastdss5';alter system set parallel_instance_group='g6' scope=spfile sid='oastdss6';alter system set parallel_instance_group='g7' scope=spfile sid='oastdss7';alter system set parallel_instance_group='g8' scope=spfile sid='oastdss8';

Do not modify the SCALE value. The OAST driver executes full table scan queries.Due to the fundamental characteristic differences when executing IOPS or MBPSagainst a physical disk, you must isolate each workload on separate physicalspindles (having separate Oracle ASM diskgroups) on the shared storage. This isto prevent thrashing of the physical disk arm. There have been instances wherethe use of the same physical disk to handle both larger random sequential reads(MBPS) and small random I/O (OLTP) causes a large queuing of read and write inthe servers due to thrashing of the physical disk arm.

This queuing can have a negative impact on IOPS and cause a major bottleneckfor the other workloads that are executed concurrently.

For example:

• Vendor cluster: 8 nodes

• Total user count 24

• Per node user count at 3

• PQ on: parallel minimum and maximum servers at values indicated in Step 1(page 7-5).

Chapter 7Running the Stress Tests for OAST Certification

7-6

Page 43: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Note:

The specific SCALE value and number of users (-u) depends on multiple factorson the server and bottlenecks that are introduced.

The physical I/O ratios are not accurate as a large buffer cache causes morelogical reads. Very big buffer caches have a high ratio of physical write tophysical read in the AWR reports. Due to this, the physical writes must bepresent on the disk. However, the physical reads can be either logical, whichis from the memory or physical, which is from the disk.

7.2 Additional Requirements for Oracle Database 12cThis section provides the modified and additional requirements to stress test an OracleRAC Database environment for 12c Release 2. The following information denoteschanges in reference to Running the Stress Tests for OAST Certification (page 7-1).

Note:

Refer to Running the Stress Tests for OAST Certification (page 7-1) and FourStress Workloads (page 7-3) for generic requirements to perform the stresstests.

• A minimum of 4 disk groups are required. In Oracle Database 12c, an additionaldisk group is required for Oracle ACFS. The remaining 3 diskgroups are asspecified in Running the Stress Tests for OAST Certification (page 7-1). TheOracle homes must be installed in Oracle ACFS that is contained in the ACFSDGdisk group. All the disk groups must be formed from separate spindles and notfrom luns which span the same disks.

• Oracle Cluster Registry (OCR) and voting disk are also required to be in OracleASM with normal redundancy. OCR must have at least two failure groups andvoting disk must have at least three failure groups. Oracle recommends to haveOCR and voting disk in a fifth disk group, however, this is not mandatory.

• The OAST install process is executed four times to create three multitenantcontainer databases (CDBs) and one non-CDB. The first three CDBs contain onePDB each. The OLTP:CPU CDB is created with 4k block size and the DSS:MBPS CDB iscreated with 16k block size.

• The CRS:Interconnect stress test uses a non-CDB. The final configuration will beone CDB containing three PDBs and one non-CDB. The PDB from the OLTP:CPUCDB and the PDB from DSS:MBPS CDB will be unplugged from their respectiveCDBs and plugged into the OLTP:IOPS CDB. Refer to About Configuring OASTDatabase and Workloads for Oracle Database 12c Certification (page 8-1) fordetails regarding the CDB and PDB setup requirement.

• Set up the cluster with mgmtdb.

• Flex mode (Oracle Flex ASM) is the required ASM mode. The ASM network willshare the private network.

Chapter 7Additional Requirements for Oracle Database 12c

7-7

Page 44: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

• The required cpu usage on the Oracle RAC nodes must be equal to or greaterthan 90 percent. Both the desired private network usage and the desired storagenetwork usage must exceed 70 percent.

Chapter 7Additional Requirements for Oracle Database 12c

7-8

Page 45: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

8About Configuring OAST Database andWorkloads for Oracle Database 12cCertification

The configuration and tuning procedures described in Tuning OAST Workloads toStress System Resource (page 7-1) must be modified to reflect the use of CDBs.Instead of tuning at the database level, the tuning must be at the PDB level. You mustconnect to the PDB before making any changes, for instance the changes using SQL/Plus listed in the configuration for DSS:MBPS. Apply the steps outlined in Running theStress Tests for OAST Certification (page 7-1) to the PDBs in the 3 CDBs. No changeare required to the CRS:Interconnect configuration as this will be done with theremaining non-CDB.

After tuning the four configurations and achieving a consistent 90 percent plus cpuusage with all four databases running, the three PDBs will be combined into a singleCDB configuration. Save the results of this configuration to use it as a base.

Unplug the pdb 1 oastcpu_pdb from the OLTP:CPU CDB database, which is oastcpu.Unplug the pdb oastdss_pdb from the DSS:MBPS CDB database, which is oastdss.Plug these pdbs into the OLTP:IOPS CDB database, which is oastiop. Apply all theconfigurations to each PDB as done previously for the separate CDBs. Add the 4k and16k caches, set the SGA size to the size of the 3 previous CDBs combined. Setparallel query setting that were set in the DSS:MBPS CDB, the oastdss database. Thepresent main methodology for tuning the PDBs together is to use the Oracle ResourceManager. Use the resource manager to tune the combined PDBs toward the basenumbers previously obtained in the separate CDBs.

8.1 Configuring OAST Database and Workloads for OracleDatabase 12c Certification

Following is an example to move from a CDB containing a single PDB configuration toa CDB containing three PDBs:

1. Unplug the PDBs from their CDBs:

$ sqlplus / as sysdba (oracle sid is oastdss1)SQL> alter pluggable database oastdss_pdb close immediate instances=all;SQL> alter pluggable database oastdss_pdb unplug into '/home/oracle/homedss/oastdss_pdb.xml';SQL> drop pluggable database oastdss_pdb keep datafiles;SQL> quit$ sqlplus / as sysdba (oracle sid is oastcpu1)SQL> alter pluggable database oastcpu_pdb close immediate instances=all;SQL> alter pluggable database oastcpu_pdb unplug into '/home/oracle/homedss/oastcpu_pdb.xml';SQL> drop pluggable database oastcpu_pdb keep datafiles;SQL> quit

8-1

Page 46: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

$ srvctl stop database –d oastdss$ srvctl stop database –d oastcpu

2. Modify the SPFILE for the OLTP:IOPS CDB (oastiop) to accept or handle the newPDBs that will be plugged in. Run the following commands:

SQL> alter system set db_4k_cache_size=6560M sid='*' scope=spfile6560M was the db_cache_size that was found in the AWR report to be the maximum allocated to any node during the base run for oastcpu.SQL> alter system set db_16k_cache_size=6880M sid='*' scope=spfile6880M was the db_cache_size that was found in the AWR report to be the maximum allocated to any node during the base run for oastdssSQL> alter system set db_cache_size=6496M sid='*' scope=spfile6496M was the db_cache_size that was found in the AWR report to be the maximum allocated to any node during the base run for oastiop.SQL> alter system set memory_target=43488M sid='*' scope=spfile;SQL> alter system set memory_max_target=43488M sid='*' scope=spfileSQL> alter system set sga_max_size=43488M sid='*' scope=spfile;Set memory_target=memory_max_target=sga_max_size = max(memory_target(oastdss)) + max(memory_target(oastcpu) + max(memory_target(oastdss)) where max is the maximum allocated to any node during the base run as found in AWR report.SQL> alter system set sga_target=26240M sid='*' scope=spfile;Set sga_target = max(sga_target(oastdss)) + max(sga_target(oastcpu) + max(sga_target(oastdss)) where max is the maximum allocated to any node during the base run as found in AWR report. SQL> alter system set parallel_adaptive_multi_user=false sid='*' scope=spfile;SQL> alter system set parallel_max_servers=64 sid='*' scope=spfile;SQL> alter system set parallel_min_servers=8 sid='*' scope=spfile;SQL> alter system set instance_groups='g1','all' scope=spfile sid='oastiop1';SQL> alter system set instance_groups='g2','all' scope=spfile sid='oastiop2';SQL> alter system set instance_groups='g3','all' scope=spfile sid='oastiop2';SQL> alter system set instance_groups='g4','all' scope=spfile sid='oastiop4';SQL> alter system set parallel_instance_group='g1' scope=spfile sid='oastiop1';SQL> alter system set parallel_instance_group='g2' scope=spfile sid='oastiop2';SQL> alter system set parallel_instance_group='g3' scope=spfile sid='oastiop3';SQL> alter system set parallel_instance_group='g4' scope=spfile sid='oastiop4';

Applying the oastdss CDB parallel server settings to oastiop.

3. Restart the oastiop database to apply the new spfile:

$ srvctl stop database -d oastiop$ srvctl start database -d oastiop

4. Plug PDBs oastdss_pdb and oastcpu_pdb into the oastiop CDB:

$ sqlplus / as sysdba /* ORACLE_SID=oastiop1 */SQL> create pluggable database oastdss_pdb using '/home/oracle/homedss/oastdss_pdb.xml' nocopy tempfile reuse;SQL> create pluggable database oastcpu_pdb using '/home/oracle/homecpu/oastcpu_pdb.xml' nocopy tempfile reuse;

5. Run the CDB and non-CDB together using default resource management for thePDBs in the CDB:

$ sqlplus / as sysdba /* ORACLE_SID=oastiop1 */SQL> alter pluggable database all open instances = all; /* always make sure that all pdbs are up on all instances */SQL> quit$ ./nrunoastoltp1500.sh –no new_pdb1 -u 100 -t 3600 -w 1 2>&1 | tee new_pdb1.out$ ./nrunoastoltp150.sh –no new_pdb1 -u 100 -t 3600 -w 4 2>&1 | tee new_pdb1.out$ ./nrunoastoltp750.sh –no new_pdb1 -u 12 -t 3600 -w 5 2>&1 | tee new_pdb1.out$ ./nrunoastoltp150.sh –no new_pdb1 -u 16 -t 3600 -w 7 2>&1 | tee new_pdb1.out

Chapter 8Configuring OAST Database and Workloads for Oracle Database 12c Certification

8-2

Page 47: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

6. Validate the results, compare the results to the base, and modify the resourcemanager as required:

exec DBMS_RESOURCE_MANAGER.CREATE_PENDING_AREA();BEGIN DBMS_RESOURCE_MANAGER.CREATE_CDB_PLAN( plan => 'cpu_oltp_dss_plan', comment => 'CDB Resource plan for OAST stress minus crs');END;BEGIN DBMS_RESOURCE_MANAGER.CREATE_CDB_PLAN_DIRECTIVE( plan => 'cpu_oltp_dss_plan', pluggable_database => 'oastcpu_pdb', shares => 2, utilization_limit => 60, parallel_server_limit => 0);END;/BEGIN DBMS_RESOURCE_MANAGER.CREATE_CDB_PLAN_DIRECTIVE( plan => 'cpu_oltp_dss_plan', pluggable_database => 'oastiop_pdb', shares => 3, utilization_limit => 25, parallel_server_limit => 0); END;/BEGIN DBMS_RESOURCE_MANAGER.CREATE_CDB_PLAN_DIRECTIVE( plan => 'cpu_oltp_dss_plan', pluggable_database => 'oastdss_pdb', shares => 1, utilization_limit => 15, parallel_server_limit => 100); END;/EXEC DBMS_RESOURCE_MANAGER.VALIDATE_PENDING_AREA();EXEC DBMS_RESOURCE_MANAGER.SUBMIT_PENDING_AREA();alter system set resource_manager_plan='cpu_oltp_dss_plan' scope=spfile sid='*';

Based on the parameters set earlier, 50 percent, which corresponds to 3 shares ofthe resource are allocated to the OLTP:IOPS PDB, 33.3 percent, whichcorresponds to 2 shares of the resource are allocated to the OLTP:CPU PDB and16.7 percent, which corresponds to 1 share of the resource is allocated to theDSS:MBPS PDB. OLTP:CPU receives 60 percent of the cpu, OLTP:IOPS receives25percent of the cpu, and DSS:MBPS receives 15 percent of the cpu. FinallyDSS:MBPS receives 100 percent of the parallel servers.

7. Repeat steps 5 (page 8-2) and 6 (page 8-3). Change value for theresource_manager_plan parameter for each iteration to get base results.

Currently there is no method to partition the SGA usage, in particular, the buffer cacheallocation for a PDB. There are 3 workloads in the PDB, the oltp, cpu, and dss. Thecpu workload should be cached as much as possible, oltp workload must be cachedless than the cpu workload and the dss workload must be cached a minimum. The cpuusage (important to give preference to the cpu workload) and parallel servers usage(important for DSS workload preference) can be managed by the Resource Manager.Otherwise the Resource Manager allocates on the gross level - general resourcepercentage. By using 3 different default cache sizes, the workloads are allocated theirown buffer cache area, which is not shared.

Chapter 8Configuring OAST Database and Workloads for Oracle Database 12c Certification

8-3

Page 48: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

9Troubleshooting

This section provides information about troubleshooting with OAST. It includes thefollowing topics:

• Utility Scripts (page 9-1)

• About General OAST Troubleshooting (page 9-1)

• OAST Troubleshooting for Oracle Database 12c Release 1 (page 9-5)

• OAST Troubleshooting for Oracle Database 12c Release 2 (page 9-7)

9.1 Utility ScriptsThe scripts discussed in this section are called through nrunoastoltpnum_ware.sh.These scripts must not be used directly:

• oastoltpXXX/oastenv.sh: contains environment variables.

• oastoltpXXX/addts.sh: for dropping a tablespace and creating a tablespace.

• oastoltpXXX/addfile.sh: for adding a data file to the tablespace.

• oast_cluster/multinodedb.sh: for startup and shutdown of all instances.

• oast_cluster/multinodedb_shut.sh: for shutdown of all instances.

• oast_cluster/oastoltpnum_ware/oastenv.sh: contains environment variables for oneinstance.

• oast_cluster/listener.ora and oast_cluster/tnsnames.ora: for TNS settings for oneinstance.

• $OAST_HOME/utils/freespace: determines the free space of Oracle files.

• $OAST_HOME/utils/sysstat.sh: to get a snapshot of wait events.

9.2 About General OAST TroubleshootingIssues that you encounter during installation, database creation, data loading, andexecution of OAST workload stress are typically the result of not validating each step,as suggested in this document.

For every task, you must redirect the standard.out and standard.error to a unique fileusing the tee command.

Note:

You must check the file for errors and other issues to prevent additional down-stream problems.

9-1

Page 49: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Many issues that occur are traced back to incorrect configuration of system resources.OAST uses Automatic Memory Management. The MEMORY_TARGET parameter uses 50percent of the defined system memory for this value. Review your operating systemspecific installation guide to determine the correct procedure.

OAST uses a template file, hostname_*.dbt for Oracle Database ConfigurationAssistant silent installation. The template file is constructed using the installationvalues entered during OAST installation. Various parts of the template are shippedwith OAST and the final template formed depends on system resources, the specificoperating system, and the type of file system being used.

Use nrunoastoltpnumware.sh to create the OAST database. OAST copies thecurrent $OAST_HOME/dbca_template/hostname_*.dbt to the ORACLE_HOME/assistants/dbca/templates/ directory. Do not modify the version at assistants/dbca/templates directory,as OAST overwrites the version.

The different phases where you may encounter problems are as follows:

• OAST Installation

OAST Installation fails if you use an existing OAST home directory or if youprovide an invalid value for the $OAST_HOME variable during the installationprocedure.

• Oracle Database Configuration Assistant Creation

OAST leverages DBCA in silent mode to create the database required by OAST.DBCA generates directories and logs in the ORACLE_BASE/cfgtoollogs/dbcadirectory. The DBCA logs provides additional information for the DBCA error andcause.

Database creation using DBCA may fail due to insufficient resources, invalidoperating system settings, and invalid directories. DBCA may also fail due toOracle ASM being offline or due to improper cleanup of a previous database withthe same name.

You must validate if the database is correctly generated by DBCA before movingto the next step.

• Schema Creation and Data Load

OAST uses SQL scripts and the binary executable, oastoltpload.exe to create theOAST schema and to load data.

If the -d y phase suspends on the first step of the process, which is Create Useroastoltp, then it indicates that the database is not up and running and the -dbca ystep failed. Execution of the script to create the user name will be immediate andshould not suspend. Run the following command to find the createuser process:

ps -ef |grep createuser

Run the following command to end the process:

kill -9 $process_number

OAST generates the trace.log file in the OAST_HOME/oastoltpnum_ware/log directory.If an error occurs during schema and data load, then the specific step that failedwill be provided in the trace.log file.

Verify and confirm the following:

– The -dbca y step has completed without errors.

Chapter 9About General OAST Troubleshooting

9-2

Page 50: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

– The binary, oastoltpload.exe was generated as part of the -kitcompilecommand. The binary oastoltpload.exe is generated in the OAST_HOME/oastoltpnum_ware/stressrun/bin directory.

– Manual deletion of the stop file. OAST will place a zero length file called stopin the OAST_HOME/oastoltpnum_ware directory if an error occurs. You mustremove this file before you rerun the -d y option.

9.2.1 Adding Data Files for out of space ConditionOAST uses autoextend for the data files and does not use big files. The maximum sizeof the data file is operating system dependent. If you reach the maximum size, thenadding additional data files will remove the free space issue. Perform the followingsteps to add data files:

1. Navigate to the OAST home directory:

cd $OAST_HOME

2. Source the environment variables:

../oastoltpnum_ware/oastenv.sh

3. Log into SQL *Plus as follows:

SQL> CONNECT OASTOLTPEnter password: password

4. Add an Oracle Managed File to an existing tablespace using the device or mountindicated by the db_create_file_dest:

alter tablespace $tablespace_name add datafile;

9.2.2 Adding Data Files to Support Multiple DevicesIn some scenarios adding additional data files or mounts to the tablespaces used byOAST may be desirable to achieve a distribution of I/O. This scenario allows foradding additional mount points to the tablespaces before table creation and data load.To achieve the desired effect of I/O distribution across multiple data files, thetablespace must be locally managed and must use a uniform extend. Auto-allocatedextend creation, which is the default for tablespaces, does not distribute extends butcreates all extends in a single data file before moving to the next. If you add data filesafter the data seed load, then the existing data will not be redistributed. You must addthe data files before the table is created and the seed data is loaded. Adding newmounts or data files to the tablespace before seed load will allow Oracle to use all datafiles for creating table extends and will produce a more balanced distribution of I/O.

This activity requires using oastdb.sh script directly to start specific sections of theschema creation and data load process. OAST provides the ability to supply start andstop points through the file oastdb.sh.

Perform the following steps to add additional data files:

1. Modify addts.sh to use uniform extends to create tablespace.

Chapter 9About General OAST Troubleshooting

9-3

Page 51: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Note:

It is recommended to use uniform extends from the start, otherwise Oracledatabase will not stripe across the mount points for the same tablespace.

2. Run oastdb.sh to create the oastoltp user and the tablespaces used by oast:

a. Navigate to the $OAST_HOME directory:

cd $OAST_HOME

b. Source the environment variables:

../oastoltpnum_ware/oastenv.sh

c. Run oastdb.sh as follows:

./oastdb.sh -oastoltp num_ware 5 6

where, num_ware is the number of warehouse.

3. Set the mount point/directory location using the db_create_file_dest to identify thelocation:

a. Navigate to the oast_home directory:

cd oast_home

b. Source the environment variables:

../oastoltpnum_ware/oastenv.sh

c. Log into SQL *Plus as follows:

SQL> CONNECT OASTOLTPEnter password: password

d. Run the alter system set command for the db_create_file_dest parameter:

alter system set db_create_file_dest=new mount location

4. Add the Oracle Managed Files data file to the existing tablespaces:

Note:

OAST uses partitioned and nonpartitioned tables. The number of tablespacescreated is a factor of the warehouse size. You can query the tablespacesdefined using user_tab_partitions and user_tables or other system tables.

a. Navigate to the oast_home directory:

cd oast_home

b. Source the environment variables:

../oastoltpnum_ware/oastenv.sh

c. Log into SQL *Plus as follows:

SQL> CONNECT OASTOLTPEnter password: password

Chapter 9About General OAST Troubleshooting

9-4

Page 52: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

d. Add the Oracle Managed Files to an existing tablespace using the device/mount indicated by the db_create_file_dest parameter:

alter tablespace $tablespace_name add datafile

5. Repeat step 2 (page 9-4) and step 3 (page 9-4) for each new mount point.

6. Complete the remaining steps in Schema Creation and Data Load Phase:

a. Navigate to the oast_home directory:

cd oast_home

b. Source the environment variables:

../oastoltpnum_ware/oastenv.sh

c. Run oastdb.sh

./oastdb.sh -oastoltp num_ware 7 42

where num_ware is the number of warehouse.

Steps 7 through 42 in the oastdb.sh script will complete the Schema Creationand Data Load Phase. Data loading during this process will be load balancedacross the devices configured.

9.3 OAST Troubleshooting for Oracle Database 12cRelease 1

This section provides information about OAST troubleshooting for Oracle Database12c Release 1 (12.1.0.1). It includes the following topics:

• createuser Process Fails (page 9-5).

• How to Compile and Build for Separate PDB Homes (page 9-6)

• Why Does OAST Installation Not Work in EMACS (page 9-6)

• How to Differentiate the Resources for Different PDBs (page 9-7)

• How to Add and Modify the Database Parameters (page 9-7)

9.3.1 createuser Process FailsOAST may stop to respond when the creat user process fails. To verify if the createuser process is running, run the following command:

ps -ef | grep createuser

If the create user process exists, then run the following command to stop the process:

kill -9 $process_number

Note:

Creating a user is the initial step of the oastdb.sh script.

Chapter 9OAST Troubleshooting for Oracle Database 12c Release 1

9-5

Page 53: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

After you stop the process, check the error in the log file located in the $OAST_HOME/oastoltpxxxx directory. If you have enabled CDB, then the log file may contain thefollowing error, which indicates that the PDB you are connecting to is not running:

ERROR:ORA-01033: ORACLE initialization or shutdown in progressProcess ID: 0Session ID: 0Serial Number: 0

The workaround for this issue is to start the corresponding PDB for the current OASTworkload. For more information on starting or closing PDBs, refer to About CDB andPDB Utilities (page A-1).

The log file may also contain the following error:

ERROR:ORA-12514: TNS:listener does not currently know of service requested in connect descriptor.

This means that the information in the tnsnames.ora file is incorrect. The oast1 entry forthe PDB does not exist. Check the following files for the oast1 entry:

$OAST_HOME/oastoltpxxxx/oastenv.sh$TNS_ADMIN/tnsnames.ora

Verify if the PDB is the current PDB used. The PDBs should be the same afterinstallation.

9.3.2 How to Compile and Build for Separate PDB HomesWhen CDB is enabled, you can run the compiler and load the data using theoast_main.sh script from the screen output. The oast_main.sh file is used to invokeDBCA and to run commands like compile and load only once. However, you can alsorun the compiler from a separate pdb oast home. Use the following commands:

cd $OAST_HOME/cdb_pdb1./nrunoastoltp5.sh -kitcompile./nrunoastoltp5.sh -d y

If you have run the oast_main.sh -d y command for all the PDBs, then you will noticethat the data is loaded in the same manner. If you choose loading the data offline, thenthe DATA_DIR is used for .csv files. To use a different method for loading the data fordifferent PDBs or to perform offline loading of data where you can store all the .csvfiles in different directories, perform the following steps:

1. Load the data using the -d y option from each pdb oast home.

2. Run the oast_main.sh -d y command to load data silently in all the PDBs.

9.3.3 Why Does OAST Installation Not Work in EMACSOAST installation does not work while running under EMACS. This issue is alsonoticed when you configure an Oracle RAC installation and depends on how EMACShandles the control characters.

Chapter 9OAST Troubleshooting for Oracle Database 12c Release 1

9-6

Page 54: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

9.3.4 How to Differentiate the Resources for Different PDBsThe different stress tests in OAST are meant to stress different resources. With theuse of PDBs in the same CDB the resources are shared as the database to eachworkload can not be tuned. The Resource Manager can turn the cpu and parallelserver usage between the PDBs, but not SGA and I/O usage. The initializationparameter OPTIMIZER_INDEX_COST_ADJ can be used to differentiate the preference forusing the index access against full table scan for each PDB. We must validate thatwith the use of the Resource Manager for cpu, the initialization parameterOPTIMIZER_INDEX_COST_ADJ for index preference and data sizing that we can obtainenough differentiation in the configuration of the CPU, OLTP, and DSS tests in asingle CDB.

9.3.5 How to Add and Modify the Database ParametersOAST uses a dbca template file to create a database and some parameters likememory_target are set during installation. If you want to add some new parameters ormodify the value of existing parameters, then you must modify the template before yourun dbca y.

Perform the following steps:

1. Navigate to the $OAST_HOME/dbca_template directory.

2. Open the hostname_*.dbt file.

3. Find the lines for which you want to modify the value. For example, to modify thevalue for db_files parameter, you should search for the following line:

<initParam name="db_files" value="500"/>

4. Change the value from 500 to 1000.

5. Save the hostname_*.dbt file.

6. Run -dbca y to create the database with the new parameter values.

9.4 OAST Troubleshooting for Oracle Database 12cRelease 2

If the installer fails at any stage, then you must review the log files for each workloadhome in the following directory:

OAST install/workload_home/log/*[.log,.out]

where, workload can relate to CPU, IOP, DSS, CRSTRESS, and PQSTRESS.

The different log files are:

• Kit workload installation: workload_install.log

• Kit compile: workload_kitcompile.log

• Database creation: workload_dbca.log

• Load data creation: workload_dataload.log

• Elapsed time: nohup_workload_build.log

Chapter 9OAST Troubleshooting for Oracle Database 12c Release 2

9-7

Page 55: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

The possible causes for a failure could be:

• Lack of disk space in the OAST installation area, shared file system, or ASM.

• No permission to write to ORACLE_HOME/assistants/templates/dbca/* directory.Ensure that the write permission is available to create and link files.

• Out of memory when all the instances are started. Adjust the memory for eachworkload.

9.4.1 OAST Installation Fails for a Workload

If OAST installation fails for a workload, then you must verify the corresponding installlogs in the OAST_install_location/workload_install.log directory.

where:

OAST_install_location is /scratch/oce

workload can be one of the following:

• cpu_home

• iop_home

• dss_home

• crstress_home

• pqstress_home

For a CPU workload, the log file is located in the /scratch/oce/cpu_home/cpu_install.log directory.

For a server/server installation, the installation time for a single CDB/PDB is less than5 minutes. Installation for each additional PDB with OAST home takes 2 to 3 minutesto complete on most clusters. However, if the installation for a single CDB/PDB takesmore than 10 minutes, then verify the log files. If the installation does not progress formore than 20 minutes or if the system is suspended, then verify:

• The local directory disk space for OAST install location. Oracle recommends thefollowing values:

– A minimum of 150 GB space on each node for a non-shared local file system.

– 76 GB space to support data load, which is the sum of default values for allworkloads.

• If the remote node access is slow or suspended for ssh or scp.

• The remote read, write, and execute permissions for the OAST user ID. Ensurethat all the nodes have the same user ID and group values.

• That the /tmp directory is not full.

• That the OAST user has the read and write permissions for ORACLE_HOME/assistants/dbca/templates. Oracle recommends to use the same user ID andgroup as the Oracle software owner.

All the verifications for a server/server installation applies to a client/server installation.In addition, you must ensure that:

• The user ID and group matches with the server nodes.

Chapter 9OAST Troubleshooting for Oracle Database 12c Release 2

9-8

Page 56: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

• Oracle client is fully installed on the remote client node.

• OAST user ID has execute permissions for an Oracle client installation.

9.4.2 OAST Kit Compilation Fails for a WorkloadIf OAST kit compilation fails for a workload, then you must verify theworkload_kitcompile.log for any errors:

• If you notice any linking errors, then contact the OCE team.

• Retry to install CDB/PDB by using the following commands:

$ cd OAST_INSTALL_LOCATION/work load_home;$ oast_main.sh –kitcompile y

Verify if the linking errors are resolved and executables are produced.

• Ensure that the gmake and make utilities are installed on the server nodes.

OAST Kit Compilation on Client Node

By default, the executables are copied to the client node If the executables are notcopied, then ensure that the gmake and make utilities are installed on the client node andrun the following command:

nrunoastoltp -kitcompile y

9.4.3 OAST Database Build Fails for a WorkloadIf the OAST Database build fails for a workload, then you must look for any errors inthe workload_dbca.log file and also other log locations that are mentioned in the logfiles.

The possible reasons for the OAST database build failure could be:

• ORA-04301, ORA-00821: Specified value of sga_target XXXXM is too small, needsto be at least YYYYM

where XXXX refers to a smaller memory value in megabytes and YYYY refers to alarger memory value in megabytes.

This error indicates that the memory allocated to the workload instance isinsufficient. You must increase the memory from the default value to a highervalue either by editing the dbt file or by updating the parameter during installation.

Check and update the following parameters during installation:

– m to modify the memory

– number to select the workload

Calculate the percentage memory to SGA and allocate the desired memory. Forexample, for a single CDB and 2 PDB configuration, you must allocate 9 GBmemory. Modify the percentage memory to SGA value. For example, if the mainmemory is 32 GB and the chosen memory for an instance is 9 GB, then the newpercentage memory is calculated as follows:

New percentage value = (Required memory * 100)/total memory = (9*100)/32 = app 28%

Chapter 9OAST Troubleshooting for Oracle Database 12c Release 2

9-9

Page 57: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Complete the installation after allocating the appropriate values to all theinstances.

If the values were not modified and the database failed with the above error, thenyou must edit the workload specific dbt file located at:

OAST_INSTALL_LOCATION/work load_home/dbca_template/hostname_*.dbt

Update the parameter percentageMemToSGA?? percentageMemToSGA and rerundatabase building using following commands:

$ cd OAST_INSTALL_LOCATION/work load_home;$ oast_main.sh –dbca y

Note:

If the number of PDB count is higher, then allocate Base Instance Memory +(PDB count * .5) GB. Compute as appropriate and update the parametervalue.

• Low memory:

Database build fails if you attempt to build all the workloads together on a serverthat has less memory. Usually, each CDB requires around 8 GB to 10 GB memorydepending on the configuration created for the workload. Allow the installer tocomplete all the tasks. After installation, shut down the databases for which thetasks are complete and build the database manually. Use the followingcommands:

$ cd OAST_INSTALL_LOCATION/work load_home; $ oast_main.sh –dbca y

• Low disk space on shared file system, no space left messages:

Check the file system disk space status by running the operating systemcommands. For example:

$ df –m OAST_INSTALL_LOCATION

• Low disk space on ASM disk group used for shared location, no space leftmessages:

Check if the ASM disk group used has sufficient space to create a database. Usethe following commands:

$ gridhome/bin/asmcmdASMCMD> lsdg -s

Increase the disk space on the shared file system or ASM and rerun the followingcommands to manually build the database:

$ cd OAST_INSTALL_LOCATION/work load_home; $ oast_main.sh –dbca y

Note:

The initial database creation for ASM uses more space than the shared filesystem.

Chapter 9OAST Troubleshooting for Oracle Database 12c Release 2

9-10

Page 58: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

9.4.4 OAST Data Load Fails for a WorkloadIf OAST data load fails for a workload, then you must verify the workload_dbca.log filefor errors. Follow additional clues to identify the reasons for data load failure.

For a failed OAST data load, run the following commands to reload data:

$ cd OAST_INSTALL_LOCATION/workload_home;$ find . -name dbcre done -exec rm -f {}\;$ oast_main -dy/c/p

use the flag:

• y for CPU, IOP, and DSS workloads.

• c for CRSTRESS workload.

• p for PQSTRESS workload.

Note:

All the PDBs in this case will have the same type of data loaded.

OAST Data Load Fails for a PDB

If OAST data load fails for an individual PDB, then use the following commands to loador reload data:

$ cd OAST_INSTALL_LOCATION/workload_home/workload_pdbcount;$ nrunoastoldbnum_ware -dy/c/p

use the flag:

• y for CPU, IOP, and DSS workloads.

• c for CRSTRESS workload.

• p for PQSTRESS workload.

Note:

You can customize data load for a single PDB to run specific workloads.

OAST Data Load Fails for a Non-CDB

If OAST data load fails for a non-CDB, which includes CRSTRESS, PQSTRESS, andRAC Elevator tests, then use the following commands to load or reload data:

$ cd OAST_INSTALL_LOCATION/workload_home;$ nrunoastoldbnum_ware -dy/c/p

use the flag:

• y for RAC Elevator workload.

• c for CRSTRESS workload.

Chapter 9OAST Troubleshooting for Oracle Database 12c Release 2

9-11

Page 59: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

• p for PQSTRESS workload.

9.4.5 Running Workloads in a Cluster with Insufficient MemoryOAST fails to accommodate all workloads when the cluster nodes have less than 64GB of main memory.

To resolve this issue, do the following:

1. Allow the OAST installer to build as many workloads as possible.

2. Shutdown successful CDBs.

3. Build the remaining databases.

4. Manually load the data.

Once you complete building all the databases, you can unplug the PDBs from the CDBand consolidate them into a single CDB. You can allocate all or the desired memory tothe target CDB.

9.4.6 Custom Workloads Versus All WorkloadsSelecting custom workload allows a combination of workloads.

It also allows to change data sizes, the number of PDBs, and number of PDBs withOAST homes, CDB, and non-CDB configurations.

Selecting all workloads skips customization and creates 1CDB/1PDB fixed sizeworkloads. The customization in this workload is however very limited.

9.4.7 Supported Shells for Login IDOAST supports only ksh and bash shells for login default shells.

Any other shell may work, but, it is not guaranteed. There are known issues for theclient/server mode. Oracle recommends to configure bash as the login default shell. Ifnecessary, run the following command on all the server nodes and client nodes:

$ chsh path to bash/ksh

For example:

$ chsh /bin/bash

9.4.8 Incorrect ASM Values Displayed in the InstallerCurrent version of OAST does not accurately compute ASM disk group spacerequirements. Ensure that there is enough space for ASM disk group. Oraclerecommends 400 GB for ASM disk group. This requirement may increase if morenumber of PDBs are configured.

9.4.9 User ID Different for OAST Home and Oracle HomeHaving different user IDs for OAST home and Oracle home may result in databasebuild failure. To resolve this issue, ensure that you create write permission to thetemplates directory as follows:

Chapter 9OAST Troubleshooting for Oracle Database 12c Release 2

9-12

Page 60: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

$ chmod ug+rwx ORACLE_HOME/assistants/dbca/templates

9.4.10 User Group is Different on Remote Client NodeOAST kit installation on client node fails if the user group is different on the remoteclient node.

9.4.11 Client Software Install RequirementsTo run stress tests successfully, use custom install during Oracle Client installationand select all Oracle clients.

9.4.12 Oracle RAC Software RequirementsOAST needs software only Oracle installation. Do not create a database. If a databaseis created, then either shutdown the database or delete the database instance. Oracleinstance that is not created by OAST consumes the server resources.

9.4.13 Creating non-CDB Databases for CRSTRESS and PQSTRESSTests

To create non-CDB databases for CRSTRESS (workload 7) and PQSTRESS(workload 8), you must set the PDB count to zero or disable CDB in the configuration.

To set the PDB count to zero or to disable CDB:

1. Configure the CRSTRESS and PQSTRESS workloads with single a PDB.

2. Select and configure all the workloads.

The installer then prompts the following options:

v to view, m to modify, and c to continue.

3. Select m to modify the workload parameters.

4. Select any workload of your choice. For example, CRSTRESS, PQSTRESS.

5. In the next installer screen, you can set PDB count to zero or disable CDB.

Selecting either of the options will have the same effect.The configurationgenerated now will not have PDB and the database will be built on a non-CDBversion.

6. Proceed with the remaining steps to complete the installation.

9.4.14 How to Run RAC Elevator TestRAC elevator test runs in a non-CDB server/server environment. RAC elevator testmust not be started from client in a client/server mode. Select any one workload anddisable CDB as mentioned in Creating non-CDB Databases for CRSTRESS andPQSTRESS Tests (page 9-13)

Chapter 9OAST Troubleshooting for Oracle Database 12c Release 2

9-13

Page 61: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

AAbout CDB and PDB Utilities

Multitenant container database (CDB) is a new feature in Oracle Database 12cRelease. To support this feature, OAST provides a set of utilities. You can use theseutilities to perform the following:

• Start or stop one or more plugabble databases (PDBs).

• Rename a PDB.

• Clone a PDB.

OAST provides the following CDB and PDB utilities:

oastoltpxxxx/cdb/pdb_showstatus.sh

cd $OAST_HOME/oastoltpxxxxcdb/pdb_showstatus.sh all

displays the following results:

CON_ID INST_ID NAME OPEN_MODE

2 1 PDB$SEED READ ONLY

3 1 CDB_PDB1 MOUNTED

4 1 CDB_PDB2 MOUNTED

5 1 CDB_PDB3 MOUNTED

The command, cdb/pdb_showtatus.sh cdb_pdb1 displays the following result:

CON_ID INST_ID NAME OPEN_MODE

3 1 CDB_PDB1 MOUNTED

oastoltpxxxx/cdb/pdb_get_allnames.sh

This utility is used to display one or all the existing PDBs and their correspondingstatus. For example, the command,

cd $OAST_HOME/oastoltpxxxxcdb/pdb_get_allnames.sh

displays the following results:

CDB_PDB1CDB_PDB2CDB_PDB3

oastoltpxxxx/cdb/pdb_start_allnodes.sh

This utility starts a single PDB on a single instance when Single Instance is enabledand on all nodes when Oracle RAC is enabled. For example,

A-1

Page 62: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

cd $OAST_HOME/oastoltpxxxxcdb/pdb_start_allnodes.sh cdb_pdb1

displays the following results:

Starting PDB cdb_pdb1 on all nodesStart PDB cdb_pdb1 open on node bej301301

oastoltpxxxx/cdb/pdb_close_allnodes.sh

This utility is used to shut down a single PDB on a single instance when SingleInstance is enabled and on all nodes when Oracle RAC is enabled. For example,

cd $OAST_HOME/oastoltpxxxxcdb/pdb_close_allnodes.sh cdb_pdb1

displays the following results:

Close PDB cdb_pdb1 open on node bej301301Pluggable database altered.

oastoltpxxxx/cdb/pdb_rename.sh

This utility renames an existing PDB. For example,

cd $OAST_HOME/oastoltpxxxxcdb/pdb_rename.sh cdb_pdb1 newpdbRename PDB cdb_pdb1 to newpdbStopping CDB cdbStarting CDB cdbElapsed:00:00:00.06Pluggable database altered.Elapsed:00:00:02.60Session altered.Elapsed:00:00:00.10Pluggable database altered.Elapsed:00:00:05.88

CON_ID INST_ID NAME OPEN_MODE

3 1 NEWPDB1 READ WRITE

The PDB cdb_pdb1 has been renamed to newpdb now.

oastoltpxxxx/cdb/pdb_clone.sh

This utility clones a new PDB from an existing PDB. For example,

cd $OAST_HOME/oastoltpxxxxcdb/pdb_clone.sh cloned_pdb_cdb_pdb1

Appendix A

A-2

Page 63: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

BWorkloads Configuration and FAQs

This appendix describes the workloads configuration and frequently asked questions.

B.1 OAST Workloads ConfigurationAll Workloads Configuration

All Workloads Configuration mode configures all the workloads with the fixed data sizeas mentioned in the following table. Oracle recommends to use this mode:

Workload Type Data Size

CPU 1 GB

IOP 5 GB

DSS 50 GB

CRSTRESS 10 GB`

PQSTRESS 10 GB

The default configuration contains 1CDB/1PDB. Use the modify parameter option toenable or disable CDB. You can also use this option to modify:

• OAST PDB count

• Memory percentage Mem to SGA

• DB block size

• Archive mode

• In-memory

Custom Workloads Configuration

Custom Workloads Configuration mode configures the selected workloads with thedefault data size as mentioned in the following table:

Workload Type Data Size

CPU 1 GB

IOP 5 GB

DSS 50 GB

CRSTRESS 10 GB`

PQSTRESS 10 GB

You can modify the data size for each workload, if the workload is not scaling up to theoptimal level.

B-1

Page 64: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

B.2 Frequently Asked QuestionsWhat are the server nodes configuration to build all workloads simultaneously?

The following are the hardware requirements to build all workloads simultaneously:

• Memory - 64 GB or more

• Shared disk space for database ASM or shared file system - 350 GB

• Shared disk space for data load - 300 GB

Should the workloads be run in non-CDB mode or legacy mode?

You must run the CRSTRESS and PQSTRESS workloads in the non-CDB mode. Usethe modify parameter option to disable CDB.

Note:

The PDB count will be zero when CDB is disabled.

About Running RAC Elevator tests

Choose any non-CDB legacy mode workload to run RAC elevator tests. TheCRSTRESS and PQSTRESS workload homes works in this mode.

Database Fails to build

If the memory is less than 64 GB, then the database may fail to build. If any workloadsfail to build, then:

1. Manually rebuild the database and load data using the oast_main.sh script.

Ensure that you shutdown the workload databases that were successfully built,before you rebuild the database.

2. Load the data after rebuilding the database.

3. When all the databases are built and data is loaded to the PDBs, unplug the PDB.

4. Plug the PDB to the IOP workload CDB.

Note:

Unplugging and plugging databases applies only for PDBs, to keep all thePDBs in a single CDB. Non-CDB workloads will remain as is.

Appendix BFrequently Asked Questions

B-2

Page 65: OAST Installation and User's Guide for Oracle Database · Oracle® Certification Environment OAST Installation and User's Guide for Oracle Database 12c Release 2 (12.2) for Linux

Index

CCustom workload, 9-12

IInstalling OAST

OAST Installation, 3-1

Installing OAST (continued)

TTroubleshooting, 9-8

Insufficient Memory, 9-12OAST Data Load Fails, 9-11

Index-1