Oracle 11g Real Application Clusters for Administrators€¦ · Oracle 11g RAC for Administrators...

Oracle 11g RealApplication Clusters

for Administrators

Student Workbook

Page ii Rev 06-30-10 © 2010 ITCourseware, LLC

Oracle 11g RAC for Administrators


Published by ITCourseware, LLC., 7245 South Havana Street, Suite 100, Centennial, CO 80112

Contributing Authors: John Mullins and Rob Roselius

Technical Editor: Danielle Hopkins

Editors: Danielle North and Jan Waleri

Special thanks to: Many instructors whose ideas and careful review have contributed to the quality of thisworkbook, including Julie Johnson and John McAlister, and the many students who have offered comments,suggestions, criticisms, and insights.

Copyright © 2010 by ITCourseware, LLC. All rights reserved. No part of this book may be reproducedor utilized in any form or by any means, electronic or mechanical, including photo-copying, recording, or byan information storage retrieval system, without permission in writing from the publisher. Inquiries should beaddressed to ITCourseware, LLC., 7245 South Havana Street, Suite 100, Centennial, Colorado, 80112.(303) 302-5280.

All brand names, product names, trademarks, and registered trademarks are the property of their respectiveowners.

© 2010 ITCourseware, LLC Rev 06-30-10 Page iii


Contents

Chapter 1 - Course Introduction ............................................................................................................... 9

Course Objectives ............................................................................................................................ 10Course Overview .............................................................................................................................. 12Using the Workbook ......................................................................................................................... 13Suggested References ....................................................................................................................... 14

Chapter 2 - Cluster Architecture .............................................................................................................. 17

Cluster Architecture........................................................................................................................... 18Shared Storage ................................................................................................................................. 20Nodes and Interconnects .................................................................................................................. 22Virtual IP Addresses .......................................................................................................................... 24Oracle Software ................................................................................................................................ 26Oracle Cluster Registry (OCR) ......................................................................................................... 28The RAC Voting Disk ....................................................................................................................... 30Labs ................................................................................................................................................. 32

Chapter 3 - Oracle Clusterware .............................................................................................................. 35

What is Oracle Clusterware?............................................................................................................. 36Oracle Clusterware Files ................................................................................................................... 38Oracle Clusterware Processes........................................................................................................... 40The Grid Home ................................................................................................................................. 42Oracle Clusterware Preinstallation Steps ........................................................................................... 44Node Preparation ............................................................................................................................. 46Configuring SSH User Equivalency ................................................................................................... 48cluvfy — The Cluster Verification Utility ............................................................................................ 50Installing the Grid Infrastructure ......................................................................................................... 52Oracle Clusterware Postinstallation Steps .......................................................................................... 54Labs ................................................................................................................................................. 56Lab Datasheet ................................................................................................................................... 58

Page iv Rev 06-30-10 © 2010 ITCourseware, LLC


Chapter 4 - Installing DB Software and Creating a RAC Database ......................................................... 61

Installation Overview ......................................................................................................................... 62Configuring the OS Environment ....................................................................................................... 64Verifying System Readiness with the CVU ......................................................................................... 66Installing the Database Software ........................................................................................................ 68Troubleshooting Installation Setup ..................................................................................................... 70Recommended Postinstallation Tasks ................................................................................................ 72Running the VIPCA .......................................................................................................................... 74Creating a Cluster Database Using DBCA ........................................................................................ 76Database Pre-creation Tasks............................................................................................................. 78Initialization Parameters ..................................................................................................................... 80Labs ................................................................................................................................................. 82

Chapter 5 - RAC Database Architecture ................................................................................................. 85

Oracle Single-Instance Architecture ................................................................................................... 86Overview of RAC ............................................................................................................................. 88RAC Architecture ............................................................................................................................. 90RAC Instances and Parameter Files .................................................................................................. 92RAC Database Components ............................................................................................................. 94RAC Instance Background Processes ............................................................................................... 96Global Resource Directory ................................................................................................................ 98Overview of Cache Fusion .............................................................................................................. 100Cache Fusion Components — GES and GCS ................................................................................ 102Cache Fusion Components — Resource Master and GRD ............................................................ 104Application Connection to RAC ...................................................................................................... 106Labs ............................................................................................................................................... 108

Chapter 6 - Managing Oracle Clusterware ............................................................................................ 111

About Oracle Clusterware .............................................................................................................. 112Backing Up and Recovering Voting Disks ....................................................................................... 114Adding and Removing Voting Disks ................................................................................................. 116The OCR ........................................................................................................................................ 118Changing the OCR Configuration .................................................................................................... 120Adding and Removing an OCR Location......................................................................................... 122Backing Up and Recovering the OCR ............................................................................................. 124Restoring from Automatic OCR Backups ........................................................................................ 126Moving or Replacing the OCR ........................................................................................................ 128Repairing the OCR Configuration .................................................................................................... 130Troubleshooting the OCR................................................................................................................ 132Labs ............................................................................................................................................... 134

© 2010 ITCourseware, LLC Rev 06-30-10 Page v


Chapter 7 - RAC Instance Management ............................................................................................... 137

Overview of RAC Instance Management ........................................................................................ 138Starting and Stopping a RAC Database........................................................................................... 140Starting and Stopping a RAC Instance ............................................................................................ 142RAC Database Identical Parameters ............................................................................................... 144RAC Database Unique Parameters ................................................................................................. 146Changing Parameter Values ............................................................................................................. 148Administering Undo Tablespaces in RAC ........................................................................................ 150Administering Redo Logs in RAC ................................................................................................... 152Labs ............................................................................................................................................... 154

Chapter 8 - RAC Utilities ...................................................................................................................... 157

The ocrcheck Utility ........................................................................................................................ 158The ocrdump Utility......................................................................................................................... 160The crs_stat Utility .......................................................................................................................... 162The crsctl Utility .............................................................................................................................. 164The Server Control (SRVCTL) Utility ............................................................................................. 166SRVCTL ADD ............................................................................................................................... 168SRVCTL CONFIG ........................................................................................................................ 170SRVCTL ENABLE and DISABLE ................................................................................................ 172SRVCTL GETENV ........................................................................................................................ 174SRVCTL MODIFY ........................................................................................................................ 176SRVCTL RELOCATE .................................................................................................................... 178SRVCTL STATUS.......................................................................................................................... 180SRVCTL REMOVE ....................................................................................................................... 182SRVCTL START ............................................................................................................................ 184SRVCTL STOP .............................................................................................................................. 186Labs ............................................................................................................................................... 188

Chapter 9 - Services ............................................................................................................................. 191

Overview of Services ...................................................................................................................... 192Types of Services ............................................................................................................................ 194Creating Services with DBCA ......................................................................................................... 196Creating Services with srvctl ............................................................................................................ 198Preferred and Available Instances for Services ................................................................................. 200Using Services ................................................................................................................................ 202Managing Services .......................................................................................................................... 204Service Views ................................................................................................................................. 206Tracing with Services....................................................................................................................... 208Labs ............................................................................................................................................... 210

Page vi Rev 06-30-10 © 2010 ITCourseware, LLC


Chapter 10 - Failover ............................................................................................................................ 213

Transparent Application Failover (TAF)........................................................................................... 214Client-Side vs. Server-Side TAF ..................................................................................................... 216Configuring TAF on the Client ......................................................................................................... 218Configuring TAF on the Server ........................................................................................................ 220Using OEM to Configure TAF ........................................................................................................ 222Using svrctl to Configure TAF ......................................................................................................... 224Using DBCA to Configure TAF ...................................................................................................... 226The DBMS_SERVICE Package ..................................................................................................... 228Connecting to the Database with TAF ............................................................................................. 230Monitoring TAF Connections .......................................................................................................... 232Labs ............................................................................................................................................... 234

Chapter 11 - RAC Backup and Recovery ............................................................................................. 237

Overview of RAC Backup and Recovery ....................................................................................... 238Log Archiving in RAC ..................................................................................................................... 240Undo Tablespaces in RAC .............................................................................................................. 242Using Flashback Features in RAC ................................................................................................... 244Deploying a Flash Recovery Area in RAC ....................................................................................... 246Performing RMAN Backups of a RAC Database ........................................................................... 248Performing Non-RMAN Backups .................................................................................................. 250Preparing to Restore and Recover with RMAN .............................................................................. 252Recover the RAC Database with RMAN ....................................................................................... 254Recovering Without RMAN ............................................................................................................ 256Labs ............................................................................................................................................... 258

Chapter 12 - Cluster Management ........................................................................................................ 261

Overview of Cluster Management Tasks ......................................................................................... 262Extending the Clusterware Home Directory ..................................................................................... 264Extending the Database Software Home Directory .......................................................................... 266Creating the New Instance .............................................................................................................. 268Verifying the New Instance .............................................................................................................. 270Removing a Node ........................................................................................................................... 272Labs ............................................................................................................................................... 274

Chapter 13 - Automatic Storage Management ....................................................................................... 277

ASM Overview .............................................................................................................................. 278ASMLib ......................................................................................................................................... 280Installing ASM ................................................................................................................................ 282

© 2010 ITCourseware, LLC Rev 06-30-10 Page vii


Creating an ASM Instance .............................................................................................................. 284ASM Initialization Parameters ......................................................................................................... 286Accessing an ASM Instance ............................................................................................................ 288ASM Startup and Shutdown ........................................................................................................... 290Creating a Disk Group .................................................................................................................... 292Changing a Disk Group ................................................................................................................... 294Creating and Managing a Database ................................................................................................. 296Using Oracle-Managed Files ........................................................................................................... 298Creating and Managing a Tablespace .............................................................................................. 300Creating and Managing Redo Logs ................................................................................................. 302Creating and Copying Control Files ................................................................................................. 304Backing up and Restoring Control Files ........................................................................................... 306Creating and Managing Archive Logs .............................................................................................. 308ASMCMD ..................................................................................................................................... 310Labs ............................................................................................................................................... 312

Chapter 14 - ASM in a RAC Environment ............................................................................................ 315

Overview of ASM and RAC ........................................................................................................... 316Migrating a RAC Database to ASM................................................................................................ 318Overview of Recovery Manager (RMAN) and ASM ...................................................................... 320Creating ASM Instances for a RAC Database ................................................................................. 322Migrating SPFILEs ......................................................................................................................... 324Migrating Tablespaces ..................................................................................................................... 326ASM and RAC Specifics ................................................................................................................ 328Labs ............................................................................................................................................... 330

Chapter 15 - RAC Troubleshooting ...................................................................................................... 333

The Oracle Clusterware Alert Log ................................................................................................... 334Clusterware Component Log Files .................................................................................................. 336Using crsctl To Diagnose Cluster Issues........................................................................................... 338Using diagcollection.pl ..................................................................................................................... 340Checking Interconnect Settings........................................................................................................ 342cluvfy — Verifying Clusterware Component Integrity ....................................................................... 344cluvfy —Verifying Cluster Registry Integrity ..................................................................................... 346cluvfy — Verifying Cluster Integrity ................................................................................................. 348ocrcheck — Verifying the Oracle Cluster Repository ....................................................................... 350RAC Database Alerts ..................................................................................................................... 352The racdiag.sql Script ..................................................................................................................... 354The oradebug Utility ........................................................................................................................ 356Labs ............................................................................................................................................... 358

Page viii Rev 06-30-10 © 2010 ITCourseware, LLC


Chapter 16 - RAC Tuning ..................................................................................................................... 361

RAC Tuning Methodology .............................................................................................................. 362Using Performance Views in RAC ................................................................................................... 364Monitoring Cache Fusion ................................................................................................................ 366Global Cache Latencies .................................................................................................................. 368Monitoring Cache Transfers ............................................................................................................ 370OEM — Cluster Database Performance ......................................................................................... 372OEM — RAC-Related Reports ..................................................................................................... 374Using AWR in the RAC Environment .............................................................................................. 376Generating AWR Reports ................................................................................................................ 378Analyzing AWR Reports ................................................................................................................. 380Using ADDM in the RAC Environment ........................................................................................... 382Analyzing ADDM Reports .............................................................................................................. 384RAC Tuning Tips ............................................................................................................................. 386Labs ............................................................................................................................................... 388

Appendix - Preparing a RAC Node ...................................................................................................... 391

Checking the Hardware Requirements ............................................................................................. 392Identifying Network Requirements .................................................................................................. 394Configuring Operating System Users and Groups ............................................................................ 398Generating RSA and DSA Keys ...................................................................................................... 400Adding the Keys to an Authorized Key File ..................................................................................... 402Configuring SSH User Equivalency ................................................................................................. 404Configuring the Operating System Environment ............................................................................... 406Configuring the Network ................................................................................................................. 408Verifying the Network Configuration ................................................................................................ 410Preparing the Operating System and Software ................................................................................. 412Configuring Installation Directories and Shared Storage ................................................................... 414Choosing Directories ....................................................................................................................... 416Labs ............................................................................................................................................... 418

Solutions ............................................................................................................................................... 421

Index..................................................................................................................................................... 427

Course Introduction

© 2010 ITCourseware, LLC Rev 06-30-10

Chapter 1

Chapter 1 - Course Introduction


Rev 06-30-10 © 2010 ITCourseware, LLC

Explain the architecture of a RAC system.

Install and configure Oracle Grid Infrastructure and Oracle RAC DatabaseSoftware in a cluster environment.

Add, remove, backup, and recover voting disks and the OCR.

Correctly set RAC initialization parameters.

Start and stop individual RAC instances and an entire RAC database.

Manage redo log files and undo tablespaces for a RAC database.

Configure and Manage Services in a RAC Environment.

Configure Transparent Application Failover for client load balancing, connect-time, and application failover.

Configure a RAC database for archiving redo logs and flashback databasemode.

Backup and recover a RAC database and its components.

Prepare, configure, remove, and add nodes to an existing cluster.

Set up and manage an ASM instance.

Migrate a RAC database to ASM.

Locate and interpret the contents of critical RAC alert and RAC component logfiles.

Monitor and diagnose performance issues with built-in utilities, views, reports,and the OEM.

Course Objectives

Course Introduction


Chapter 1



Audience: Database administrators new to the RAC environment.

Prerequisites: Oracle 11g Database Administration and at least 6 monthsof administration experience recommended.

Classroom Environment:

A workstation per student.

Course Overview

Course Introduction


Chapter 1

Using the Workbook

Chapter 2 Servlet Basics

© 2002 ITCourseware, LLC Rev 2.0.0 Page 17

Add an init() method to your Today servlet that initializes a bornOn date, then print the bornOn date

along with the current date:

Today.java

...

public class Today extends GenericServlet {

private Date bornOn;

public void service(ServletRequest request,

ServletResponse response) throws ServletException, IOException

{

...

// Write the document

out.println("This servlet was born on " + bornOn.toString());

out.println("It is now " + today.toString());

}

public void init() {

bornOn = new Date();

}

}

Hands On:

The init() method is

called when the servlet is

loaded into the container.

This workbook design is based on a page-pair, consisting of a Topic page and a Support page. When youlay the workbook open flat, the Topic page is on the left and the Support page is on the right. The Topicpage contains the points to be discussed in class. The Support page has code examples, diagrams, screenshots and additional information. Hands On sections provide opportunities for practical application of keyconcepts. Try It and Investigate sections help direct individual discovery.

In addition, there is an index for quick lookup. Printed lab solutions are in the back of the book as well asonline if you need a little help.

Java Servlets

Page 16 Rev 2.0.0 © 2002 ITCourseware, LLC

� The servlet container controls the life cycle of the servlet.

� When the first request is received, the container loads the servlet class

and calls the init() method.

� For every request, the container uses a separate thread to call

the service() method.

� When the servlet is unloaded, the container calls the destroy()

method.

� As with Java’s finalize() method, don’t count on this being

called.

� Override one of the init() methods for one-time initializations, instead of

using a constructor.

� The simplest form takes no parameters.

public void init() {...}

� If you need to know container-specific configuration information, use

the other version.

public void init(ServletConfig config) {...

� Whenever you use the ServletConfig approach, always call the

superclass method, which performs additional initializations.

super.init(config);

The Servlet Life Cycle

The Topic page providesthe main topics for

classroom discussion.

The Support page hasadditional information,

examples, and suggestions.

Code examples are in afixed font and shaded. Theonline file name is listedabove the shaded area.

Screen shots showexamples of what youshould see in class.

Topics are organized intofirst ( ), second ( ), and

third ( ) level points.

Pages are numberedsequentially throughout

the book, making lookupeasy.

Callout boxes point outimportant parts of the

example code.



Dyke, Julian and Shaw, Steve. 2006. Pro Oracle Database 10g RAC on Linux: Installation,Administration, and Performance. Apress, Berkeley, CA. ISBN 1590595246.

Gopalakrishnan, K. 2006. Oracle Database 10g Real Application Clusters Handbook. McGrawHill Osborne Media, Emeryville, CA. ISBN 007146509X.

Vallath, Murali. 2006. Oracle 10g RAC Grid, Services & Clustering. Elsevier Digital Press,Burlington, MA. ISBN 1555583210.

The following books are available online at http://www.oracle.com/pls/db112/homepage :Oracle Database Administrator's Guide 11g Release 2 (11.2)Oracle Database 2 Day + Real Application Clusters Guide 11g Release 2 (11.2)Oracle Clusterware Administration and Deployment Guide 11g Release 2 (11.2)Oracle Real Application Clusters Administration and Deployment Guide 11g Release2 (11.2)Oracle Database Storage Administrator's Guide 11g Release 2 (11.2)Grid Infrastructure Installation Guide for LinuxReal Application Clusters Installation Guide for Linux and Unix

Suggested References

Course Introduction


Chapter 1

Chapter 2 Cluster Architecture


Chapter 2 - Cluster Architecture

Objectives

Explain the general architecture of acluster.

Identify the locations of the various filetypes within a cluster.

List the different shared storage filesystem options.



A cluster consists of multiple interconnected hosts (nodes), that appear to usersand applications as if they are one host.

The nodes maintain close communication with one another, typically overhigh-speed LAN connections.

Typically, the nodes use shared storage, such as a Storage Area Network(SAN), Network Attached Storage (NAS), or other devices.

Software, often called clusterware, running with operating system privileges oneach node, coordinates activities in the cluster.

Tasks of the clusterware include:

Identifying which nodes are currently members, and which mightneed to be removed (evicted) from cluster activities.

Monitoring application software (database servers, applicationservers, other services), and stopping them, starting them, ormoving their work to another node (failing over) as necessary.

The clustering software may use shared storage to record its own stateand other information.

Clustering can provide one or more of several benefits:

High availability — Failure of one or more nodes, and the bringing up oftheir replacements, is hidden from the user and the overall cluster remainsavailable.

Scalability — More nodes can be brought online to meet growing userdemand.

Load balancing — New requests are distributed equally (or at leastintelligently) among available nodes.

High performance — Portions of large tasks can be parcelled out toseparate nodes to be executed in parallel.

Cluster Architecture



In a cluster:Each node is in communication with all other nodes.All nodes agree on which nodes are valid and working.

Clusterware on each node provides heartbeat information, visible to all other nodes, at frequent intervals.When other members don't see a node's heartbeat, it can mean:

The node has crashedThe node's communication link has failed.The node is hung.

In this case, the remaining nodes may vote to evict the node from the cluster. Remaining nodes will then takeover the failed node's provided services as well as the shared resources it controlled.

If multiple nodes lose contact with one another, but remain running, each may think it is the only remainingnode. Applications on the separated nodes could write to shared resources simultaneously, each assumingit's safe to do so. Either the clusterware or the application software must therefore be designed to quicklydetect and resolve this split-brain situation.



Shared Storage

The nodes in a cluster need shared access to data in various files.

In a RAC Database, at a minimum, shared storage is needed for:

The clusterware Oracle Cluster Registry (OCR)The clusterware voting diskThe database files

The Oracle RDBMS software can be installed on shared storage or locallyon each node.

Following are shared storage file system options and the types of files that arevalid on each:

Shared cluster filesystem - Oracle Cluster File System (OCFS V1, V2), orsupported NFS

Oracle RDBMS software (required for OCFS V2 only)The Oracle Cluster Registry (OCR)The voting diskDatabase filesSPFILE

Automatic Storage Management (ASM)

Database files



Raw Devices

Previous versions of Oracle Database have supported raw devices, unmounted disk partititions accesseddirectly by Oracle software as character or block devices, for database and clusterware files. As of Oracle11g, raw devices are no longer recommended or supported. Oracle highly recommends using ASM forRAC database files, though shared cluster filesystems (OCFS2, NFS, etc.) are still supported.

Starting with 11gR2, Oracle's ASM supports storage of non-database files using an ASM extension calledACFS (ASM Cluster File System). For example, the Oracle RDBMS software home (ORACLE_HOME)could be installed into ASM's ACFS. However, some files inherently cannot be stored under ASM, such asthe Oracle Clusterware software home.



Nodes and Interconnects

To use the cluster, a client application connects to one of the nodes using apublic IP address for the node, on a public network.

The clusterware instances on the nodes, and applications running on the cluster,communicate with each other using a private interconnect, sometimes called thecluster interconnect.

The private interconnect is a very high-speed link that allows processes ondifferent nodes to interact as one system.

Many cluster architectures use gigabit (or faster) ethernet for the privateinterconnect.

Other implementations include, among others:

HP HyperfabricHP Memory ChannelSun Scalable Coherent Interface (SCI)Veritas LLT

If a node's private interconnect fails, it can no longer function as part of thecluster.

Clusterware on it and the other nodes will fence it out (prevent outputfrom the node onto shared storage), evict the node from the cluster, andpossibly initiate a reboot of the node.

If the entire private network fails (say, the gigabit switch goes down), theclusterware on the nodes must detect and immediately resolve the potential split-brain situation, where several nodes each think they're the only surviving memberof the cluster.

Fully redundant private interconnects are a wise choice in any productioncluster, and can be accomplished through:

Multiple network interface cards for the private network on each node.

Plugging each node in to multiple network switches, with switches ondifferent circuits and power supplies.



Interconnect Switch 2

NIC Interconnect Switch 1

NIC NIC

NIC

Public Network

VIP1 VIP2

Node 1 Node 2

Two Node Cluster with Redundant Interconnects

In an Oracle RAC installation, the private interconnect is used both by the clusterware itself (for clustersynchronization, membership services, etc.), and by the RAC database instances (for exchanging datablocks, managing locks, etc.). High bandwidth and low latency are important for the best performance ofthe RAC database. On many systems, multiple network interface cards (NIC) can be bonded, creating asingle logical interface of higher bandwidth. Some of these solutions provide load balancing across theNICs, and may include automatic failover if one of the NICs fails.

Public and Private IP Addresses

In general, IP addresses in the ranges 10., 172., and 192.168. are reserved for "private" (non-Internet-connected) networks. Most addresses outside these ranges are "public", globally unique addresses that canbe routed across the Internet. Routers do not directly route traffic between private networks and theInternet.

In Oracle RAC, "public" just means the network that clients will use to connect to RAC instances. "Private"is the network used by the clusterware and by the RAC instances for communicating with other RACnodes.

The Cluster Verification Utility for Linux assumes the public IP address for a node is an actual public Internetaddress, and reports an error if you have chosen to use a private (10., 172., or 192.168.) IP. However, ifall clients are within the same private network, a private IP works just fine for the RAC "public" interface.



Virtual IP Addresses

To use a service on a cluster, a client application connects to one of the nodes.

The client may first connect to a dispatcher program, which can determine(on the basis of load balancing needs, or attributes of the client, forexample) the node to which the client should connect.

Oracle's TNSLISTENER serves this function in an Oracle RACcluster.

Once "connected," the client sends TCP/IP packets to the node's IP address.

The lower-level ARP protocol associates this IP address with the physicalMAC address of the node's network interface card.

If the node is down (crashed, hung, or disconnected from the publicnetwork), a client normally waits a specified amount of time (possiblyseveral minutes) before giving up and generating an error.

Virtual IP addresses allow for quicker failover when a node goes down.

Each node has a second, "virtual" IP address (VIP) and hostnameconfigured for it.

Clients connect to the virtual IP address, rather than the direct IP address.

When a node goes down, the clusterware detects the failure almostimmediately — much more quickly than the TCP/IP timeout.

A different node is assigned the failed node's VIP, and broadcastsits hardware MAC address and the VIP to the network.

The client will receive a TCP/IP RESET message, indicatingimmediately that the old node is gone, and can connect to the newnode.

Applications can be written in such a way as to gracefully handle the TCP/IPRESET, and continue working.



VIP1 VIP2

Node 1 Node 2

VIP1

Node 1 has experienced a failure, causing its VIP address to transfer over to Node 2.

Users attempting to connect to Node 1 will be redirected to Node 2.



Oracle Software

There are three software components involved in configuring a RACenvironment.

Oracle ClusterwareOracle ASMOracle RAC Database

The Clusterware and ASM software together comprise the Oracle GridInfrastructure, which is installed in a single location (the Grid Home) on the localhard drive.

Prior to Oracle 11g, Oracle ASM and Oracle Database were the samesoftware and could be installed in the same ORACLE_HOME, or inseparate ORACLE_HOME and ASM_HOME locations.

The Oracle Database software for the RDBMS can either be installed on eachnode or on the shared storage device.

As of 11gR2, Oracle Database can be installed under ASM, using ASM'sACFS.

There are pros and cons for installing the Oracle Database Software on eachnode or on a shared storage device.

Installing on each node increases maintenance because files must becopied to each node.

Installing on each node allows for increased flexibility when applyingpatches or upgrades, as one node at a time can be patched while the othernodes continue to function.

However, if installed on a shared storage device, all nodes wouldneed to be stopped at once to apply the patch.



Node 1

Installed Software

Oracle Clusterware

Oracle Database — ASM

Oracle Database — RDBMS

Node 2

Installed Software

Oracle Clusterware



Shared Storage

Database Files

Node 1

Installed Software

Oracle Clusterware

Node 2

Installed Software

Oracle Clusterware

Shared Storage

Installed Software



Oracle Software File Locations — Option 1

Oracle Software File Locations — Option 2



Oracle Cluster Registry (OCR)

The OCR is a file that maintains cluster and database configuration information,such as:

Cluster NameDatabase Name(s)Instance NamesNode NamesNumber of NodesListenersVoting Disk(s)OCR LocationsOCR Backup LocationsServicesIP Addresses (VIP, Public, Interconnect)Oracle RDBMS VersionDatabase LanguageORACLE_HOME DirectoryCRS_HOME (Clusterware) DirectoryASM Instance InformationSPFILE(s) Locations

The OCR must be located on shared storage.

The OCR file is created during the installation of the Oracle Clusterwaresoftware.

The OCR can be read and updated using Enterprise Manager (EM), theSRVCTL utility, the Database Configuration Assistant (DBCA), and a numberof other utilities and tools, including SQL*Plus.

In Oracle 11g R2 and later, the OCR can be stored in an ASM diskgroup.

If the OCR is located on a shared cluster filesystem, it can be mirrored(maximum five copies).



OCR Backup LocationsService(s)IP Addresses (VIP, Public, Interconnect)Oracle RDBMS VersionDatabase LanguageORACLE_HOME DirectoryCRS_HOME (Clusterware) DirectoryASM Instance InformationSPFILE(s) Locations

Cluster Information Registeredin OCR

EnterpriseManager

DatabaseConfiguration

Assistant

SRVCTLUtility

Cluster NameDatabase Name(s)Instance NamesNode Names# of NodesListenersVoting Disk(s)OCR Locations

OCR

OCRMirror



The RAC Voting Disk

The voting disk is a file on shared storage that is used by the ClusterSynchronization Services (CSS) daemon OCSSD to manage node membershipin the cluster.

Nodes are notified any time a node joins or leaves the cluster.

The voting disk file is created during the installation of the Oracle Clusterwaresoftware.

In Oracle 11g R2 and later, the voting disk file stored on an ASM diskgroup.

If the voting disk is located on a shared cluster filesystem, it can bemirrored.

You can create up to 15 voting disk mirrors, with a minimum ofthree and a maximum of five recommended.



Node 1

Instance 1

CSS

Shared Storage

Node 2

Instance 2

CSS

Voting

Disk

Voting

Disk

Voting

Disk

Mirror

Mirror

I am here!I am here!

Prior to 11gR2, the OCR and voting disks were required to be located on shared storage outside of ASM,either as raw shared volumes or as files on a shared cluster filesystem. Only two OCR and three voting diskmirrors were possible in 10gR2.



Name the cluster component that is responsible for maintaining cluster and database configurationinformation.

The Oracle database files can be created onto what type(s) of file systems?

What role do virtual IP addresses play in the event of a failed node?

Which cluster component maintains a cluster membership list and provides timely notification ofmembership changes?

What is the main purpose of the private (cluster) interconnect?

From the following list of files, list the file under its correct designation on the shared storage of aRAC system. The Control Files item has been done for you.

OCRRedo Log Groups and MembersSPFILEArchived Redo LogsData FilesPassword FileFlash Recovery AreaVoting DiskControl FilesTemp FilesUndo Tablespace

Instance-Specific Files Shared Files

Control Files

Labs

Chapter 7 RAC Instance Management

© 2010 ITCourseware, LLC Rev 06-30-10 Page 137

Chapter 7 - RAC Instance Management

Objectives

Correctly set RAC initialization parameters.

Start and stop an individual RAC instance.

Start and stop an entire RAC database.

Add, remove, and monitor redo log files fora RAC database.

Manage undo tablespaces in a RACdatabase.


Page 138 Rev 06-30-10 © 2010 ITCourseware, LLC

Overview of RAC Instance Management

You can manage, start up, and shut down instances with OEM, SQL*Plus, orSRVCTL.

Both OEM and SRVCTL provide options to start up and shut down allof the instances in an Oracle RAC database with a single step.

Using SQL*Plus requires mulitple steps to start up and shut down all ofthe instances in an Oracle RAC database.

The OEM is a browser-based tool that enables you to start up, shut down, andmonitor databases, instances, etc., and is available with two different controls.

Database Control — Manages a single RAC instance/database.Grid Control — Manages multiple RAC instances/databases.

The OEM, however, allows you to manage other items beyond SRVCTL.

Schemas, Tablespaces, Storage, Segments, PerformanceMonitoring, and more.

SQL*Plus allows you to operate only the current instance.

With the appropriate privileges you can shut down and start up instances,but only one instance at a time.

SRVCTL is a command-line utility that can perform many of the same instance/database functions as OEM.

SRVCTL stores configuration data in the OCR.

This configuration data can also be used by the OEM.



Tool Format Advantages

OracleEnterpriseManager

Browser-Based

Easy to use.

Can be used to manage the complete instance and database.With Grid Control can be used to manage multiple instances.

Integrated with the OCR.

SRVCTL Command Line

Can be used to manage multiple instances such as services orlisteners.

Integrated with the OCR.

SQL*Plus

Browser-Based

Command Line

GUI-Based

Easy to use.

Oracle Instance Management Tools Summary



Starting and Stopping a RAC Database

In a RAC environment, starting and stopping a RAC database means startingand stopping all of the RAC instances for that database.

The OEM and SRVCTL have the ability to shut down and start up a RACdatabase with a single step.

SQL*Plus must shut down each RAC instance one at time in multiple steps.

In the case where you want to shut down every RAC instance in thecluster, OEM and SRVCTL offer a big advantage over SQL*Plus,especially in large clusters.

Starting a RAC database with SRVCTL:

srvctl start database -d <databasename>[-o <startupoptions>] [-c <connectstring> | -q]

Stopping a RAC database with SRVCTL:

srvctl stop database -d <databasename> [-o <stopoptions>][-c <connectstring> | -q]



Valid startup options:

NOMOUNTMOUNTOPEN (default)

# srvctl start database -d xyz -o OPEN

Valid stop options:

NORMALIMMEDIATE (default)TRANSACTIONALABORT

# srvctl stop database -d xyz -o IMMEDIATE



Starting and Stopping a RAC Instance

In a RAC environment, multiple instances can have the same database open atthe same time.

The procedures for starting and stopping a RAC instance are the same asstarting and stopping a single instance.

A RAC instance can be started or stopped with either OEM, SQL*Plus, or withSRVCTL.

When using ASM, the ASM instances must be started before the databaseinstances.

Starting a RAC instance options with SRVCTL:

srvctl start instance -d <databasename> -i<instancenamelist> [-o <startupoptions>][-c <connectstring> | -q]

The startup options for an instance are the same as the start options for aRAC database.

Stopping a RAC instance options with SRVCTL:

srvctl stop instance -d <databasename> -i <instancenamelist>[-o <stopoptions>] [-c <connectstring> | -q]

The stop options for an instance are the same as the stop options for aRAC database.

As with single instance databases, details about each instance startup andshutdown can be found in that instance's alert log.



Stop RAC instances xyz1 and xyz2 that are part of RAC database xyz with the IMMEDIATE option:

srvctl stop instance -d xyz -i xyz1,xyz2 -o immediate

Stop RAC instances xyz1 and xyz2 that are part of RAC database xyz with the OPEN option:

srvctl start instance -d xyz -i xyz1,xyz2 -o open

Start all RAC instances that are registered with the RAC database xyz:

srvctl start database -d xyz -o open



RAC Database Identical Parameters

Using a single SPFILE for all your RAC instances is preferred.

Set the SPFILE parameter to the location, on shared storage, of yourSPFILE.

Set CLUSTER_DATABASE=true.

Setting CLUSTER_DATABASE_INSTANCES to the correct number allowsOracle to tune memory use appropriately.

If there is more than one cluster interconnect, list them inCLUSTER_INTERCONNECTS.

You do not need to set this parameter if:

Your interconnects are bonded at the OS level.

You have only one interconnect interface.



The following parameters must be set identically for all RAC database instances:

ACTIVE_INSTANCE_COUNTARCHIVE_LAG_TARGETCLUSTER_DATABASECONTROL_FILESDB_BLOCK_SIZEDB_FILESDB_DOMAINDB_RECOVERY_FILE_DESTDB_RECOVERY_FILE_DEST_SIZEDB_UNIQUE_NAMEMAX_COMMIT_PROPAGATION_DELAYTRACE_ENABLEDUNDO_MANAGEMENT



RAC Database Unique Parameters

Each instance must have its own unique setting for the following parameters:

THREAD

Used in archived redo log file names.

Use the same number as INSTANCE_NUMBER.

ROLLBACK_SEGMENTS

Not needed, and in fact ignored, if you setUNDO_MANAGEMENT to AUTO as recommended.

INSTANCE_NAME

Normally, the INSTANCE_NUMBER concatenated toDB_NAME.

INSTANCE_NUMBER

UNDO_TABLESPACE

Each instance uses its own undo tablespace, which is visible (onshared storage) and used by the other instances.



Changing Parameter Values

How you change your initialization parameter values depends on whether youare using a PFILE or an SPFILE for your parameters.

The PFILE is a text file you can edit with your operating system editor.

The SPFILE is a binary file and its contents can be changed with theALTER SYSTEM command.

You cannot edit your SPFILE with an operating system editor.

The ALTER SYSTEM command can be used to change the current value for aparameter or to change the value of a parameter in an SPFILE or both.

It cannot be used to change the value in a PFILE.

The SPFILE has several advantages over the PFILE and is recommended.

Simplifies administrationMaintains parameter setting consistencyGuarantees parameter setting persistence across database / instanceshutdown and startup events

All instances in the cluster database use the same SPFILE at startup.

The SPFILE should reside on the shared file system or ASM in a RACenvironment.

The PFILE in $ORACLE_HOME/dbs contains the location of theshared SPFILE.

Be careful not to overwrite or modify this default PFILE!

Parameter values can also be changed from Enterprise Manager.



When changing the values of your SPFILE parameters there is a very important option to consider:

ALTER SYSTEM SET <parametername> = <value> [SIDoption] [SCOPEoption] ;

The SIDoption updates the specified parameter in the SPFILE and associates the parameter and its valueto a particular instance.

ALTER SYSTEM SET db_cache_size=500M SID='xyz2' ;

Resulting SPFILE Entryxyz2.db_cache_size=500M

If you do not specify the SIDoption, the default is *, which means the parameter and its value apply toall of the instances in the RAC environment.

ALTER SYSTEM SET db_cache_size=500M ;

Resulting SPFILE Entry*.db_cache_size=500M



Administering Undo Tablespaces in RAC

Each RAC instance has its own undo tablespace.

All undo tablespaces reside on shared storage.

All instances can always read all undo blocks throughout the cluster environmentfor consistent read purposes.

Automatic Undo is the recommended procedure.

It eases the management of undo segments.

Oracle automatically manages undo segments (how many and their spaceusage) among the various active sessions.

To enable automatic undo management, set theUNDO_MANAGEMENT initialization parameter to AUTO.

Oracle automatically manages undo segments within a specific undo tablespacethat is assigned to an instance.

Only the instance assigned to the undo tablespace can modify thecontents of that tablespace.

Undo tablespaces in your Oracle RAC database are assigned byspecifying a different value for the UNDO_TABLESPACE parameterfor each instance.

Each instance in your RAC environment should have its own undotablespace.

All instances of an Oracle RAC database must operate in the same undo mode.



To verify your undo management mode and the undo tablespaces being used, check theUNDO_MANAGEMENT and UNDO_TABLESPACE parameters. Remember, all instances of anOracle RAC database must operate in the same undo mode.

SELECT inst_id, instance_name, name, value FROM gv$instance NATURAL JOIN gv$spparameter WHERE name = 'undo_management' OR name = 'undo_tablespace'ORDER BY instance_name, name;

INSTANCE_NAME NAME VALUE---------------- --------------- ------------xyz1 undo_management AUTOxyz1 undo_tablespace UNDOTBS2xyz1 undo_tablespace UNDOTBS1xyz2 undo_management AUTOxyz2 undo_tablespace UNDOTBS1xyz2 undo_tablespace UNDOTBS2

Both instances are operating in Automatic Undo Management mode.

If you need to change the assignment of an undo tablespace to an instance, use the ALTER SYSTEMcommand. Remember to use the SIDoption to assign the undo tablespace to the appropriate instance.

ALTER SYSTEM SET undo_tablespace = 'UNDOTBS3' SCOPE = BOTH SID='xyz2';

Note:The SCOPEoption must occur before the SIDoption in the ALTER SYSTEM statement.

After the last ALTER SYSTEM command, your parameter file contents will be:

*.undo_management='AUTO'xyz1.undo_tablespace='UNDOTBS1'xyz2.undo_tablespace='UNDOTBS3'



Administering Redo Logs in RAC

In a single-instance Oracle database environment, redo logs are stored in two ormore redo log file groups.

Each of these groups contains a redo log file, and possibly one or moremirrored copies of that file.

In a RAC database, each instance requires its own set of redo log groups.

All redo log groups reside on shared storage.

Each set is known as a thread of redo logs.

Each redo log thread must contain at least two redo log groups.

Each instance writes and archives the redo log groups in its own thread.

This behavior is the same as with single-instance Oracle databases.

In recovery mode, the instance performing the recovery is able to readand process all of the redo threads for the database, regardless of whichinstance generated the redo threads.

If your database is in ARCHIVELOG mode, then each instance mustsave filled log files into its own archive log thread.



To view your current redo log groups and the instances/threads they belong to:

SELECT DISTINCT thread#, instance, group# FROM gv$log JOIN gv$thread USING (thread#)ORDER BY thread#;

THREAD# INSTANCE GROUP#---------- ---------- ---------- 1 xyz1 1 1 xyz1 2 2 xyz2 3 2 xyz2 4

To add a redo log group to a specific thread/instance:

ALTER DATABASEADD LOGFILE THREAD 1 GROUP 5 ('/u02/oradata/xyz/redo05.log')SIZE 50M;

Note:If you do not specify the thread, the redo log group gets added to the current instance.



Labs

For each node in your RAC cluster, use SQL*Plus to add a new redo log group to each instance.

Use SQL*Plus commands to increase the db_cache_size by 10% for each instance in your cluster.Make sure the change is persistent across instance restarts.

Chapter 12 Cluster Management


Chapter 12 - Cluster Management

Objectives

Add a node to an existing cluster.

Remove a node from an existing cluster.



Once your cluster is in place, there are some typical cluster management tasksthat may need to be performed, such as:

Adding a new node to the cluster.

The new node should have the same version of the operatingsystem as the existing nodes in your cluster.

Removing a node from the cluster.

You may need to add a new node to your cluster due to the the need foradditional resources.

Additional resources (memory, cpu) are not always available on anexisting server.

The need to remove a node from your cluster may arise due to:

Node failure.

The node is underutilized and needed elsewhere.

Overview of Cluster Management Tasks



General steps for adding a new node to your cluster:

Check for Dependencies and Prerequisites.- Operating system versions, packages, user accounts, groups.Configure Network Components.- Add network information to the /etc/hosts file:

Public node names and ip addresses.Private node names and ip addresses.VIP node names and ip addresses.

Install Oracle Clusterware.- Run the Cluster Verification Utility (CVU) before installing.Configure Oracle Clusterware.- Run the addNode.sh script provided by Oracle.

The script is located in the <Clusterware Home>/oui/bin directory.Install Oracle Software.Add New Instance(s).- Execute the DBCA utility.Do Housekeeping Chores.- Set up environment variables.- Configure Oracle Net.

General steps to remove a node from your cluster:

Remove the instance.- Run the DBCA or SRVCTL utility.Remove the node from the cluster.Reconfigure the OS and remaining hardware.- If necessary, remove all Oracle installation files from the removed node.



Extending the Clusterware Home Directory

Once your new node has been configured for Oracle Clusterware, use the OUIto add an Oracle Grid Infrastructure home to the new node.

To extend your Grid Home directory you must have a successfully installedOracle Clusterware version on another node.



To extend the Oracle Clusterware installation to include your new node:

1. Verify the new node has been fully prepped to join the cluster.cluvfy stage -pre nodeadd -n node3 -verbose

2. Go to Grid_home/oui/bin and run the addNode.sh script.

cd /u01/app/11.2.0/grid/oui/bin./addNode.sh -silent "CLUSTER_NEW_NODES=(node3)" \ "CLUSTER_NEW_VIRTUAL_HOSTNAMES=(node3-vip)"

The OUI starts and displays the Welcome window. Choose Next.

3. Select the node or nodes that you want to add.- Verify the entries that OUI displays on the Summary Page.

4. Run the rootaddNode.sh script from the Grid_home/install/ directory on node1 when prompted to doso.

5. Run the orainstRoot.sh script on the new node.

6. Run the Grid_home/root.sh script on the new node. This starts the Clusterware.

Verify the status of the Clusterware extension on your new node by using the CVU.

cluvfy stage -post nodeadd -n newnode -verbose



Extending the Database Software HomeDirectoryAfter extending the Oracle Clusterware and ASM homes to your new node, youmust extend the Oracle Database home (RDBMS) to your new node.

The steps to add an Oracle RDBMS home to the new node are very similar tothe steps you just completed for extending ASM to the new node.



To extend the Oracle RDBMS installation to include the new node:

Note:Make sure that you have successfully installed the Oracle RDBMS software on at least one node in yourcluster environment.

1. Go to the $ORACLE_HOME/oui/bin directory on node1 and run the addNode.sh script.

./addNode.sh -silent "CLUSTER_NEW_NODES=(node3)"

2. Run the root.sh script on the new node from the $ORACLE_HOME directory on that node.



Creating the New Instance

Once your new node is in place and you have installed the Oracle software andconfigured the listener, you are ready to create a new instance for your cluster.

To create a new instance on your new node use the DBCA utility.

The DBCA utility will update the OCR with the new instance's information.



Upon starting up the DBCA to create a new instance, it will prompt you for whether you are using a RACdatabase or a single instance database.

Choose RAC database.From here, you will have eight steps to complete.1. Choose Instance Management.2. Choose Add an Instance.3. Select the cluster database to which the instance will be added.4. Review list of existing RAC instance names.5. Enter the name of your new instance and which node to create it on.6. For each database service defined, specify its preferred and available instance. You can also specify

a service to not be used for a particular instance.7. Enter the storage information for your new instance.

- Redo log storage specifics.- Undo Tablespace.

8. Review the summary of your input and then the instance will be created.

Note:If you had been using ASM in the existing cluster, the DBCA will prompt you for whether you wish to createan ASM instance on your new node also.



Verifying the New Instance

The DBCA should automatically add your new instance to the RAC cluster, butit is always a good idea to check for yourself.

To verify that the RAC cluster recognizes your new instance:

Use the SRVCTL utility.

Use the ocrdump utility.



The SRVCTL utility will report the status of all recognized instances in your cluster.

$ srvctl status database -d xyz

Instance xyz1 is running on node cl01node54Instance xyz2 is running on node cl01node51

The ocrdump utility will create a text file containing the recognized configuration of your cluster, including itsinstances.

$ ocrdump latest_ocr_dumpfile

$ more latest_ocr_dumpfile

...[DATABASE.DATABASES.xyz.INSTANCE.xyz1.NODE]ORATEXT : cl01node54SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION :PROCR_ALL_ACCESS, OTHER_PERMISSION : PROCR_ALL_ACCESS, USER_NAME : oracle, GROUP_NAME: dba}...[DATABASE.DATABASES.xyz.INSTANCE.xyz2.NODE]ORATEXT : cl01node51SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION :PROCR_ALL_ACCESS, OTHER_PERMISSION : PROCR_ALL_ACCESS, USER_NAME : oracle, GROUP_NAME: dba}...



Removing a Node

A node may need to be removed from your cluster for various reasons.

There is a problem with the node.

The node is not needed due to under-utilization.

To remove a node, several components need to be taken care of:

The instance on the node

The OCR

The Software



Steps to Remove a Node:

1. Remove the database instance.Use the DBCA to remove the instance.- Run the DBCA from one of your remaining nodes.

2. Remove the ASM instance if present.Use the SRVCTL utility to remove the instance.$ srvctl remove asm1 -n nodename

3. Delete the Listener Process.This can be done with the NETCA.

4. Delete the Oracle Software.Run the OUI and delete the ORACLE_HOME.

5. Disable the Clusterware for the node being removed.Run the rootdelete.sh script.Run as the root user on the node being removed.

6. Remove the node from the OCR.Run the rootdeletenode.sh script as user root.

7. Delete the Clusterware Software.Run the OUI on the node being removed.

8. Verify the node's deletion from the cluster.Run the cluvfy utility.$ cluvfy comp crs -n all



Labs

Extend Oracle Clusterware to your newly prepared node.

Extend Oracle RAC Database to your new node.

Establish and verify an instance on the new node.

Oracle 11g Real Application Clusters for Administrators€¦ · Oracle 11g RAC for Administrators...

Documents

Transcript of Oracle 11g Real Application Clusters for Administrators€¦ · Oracle 11g RAC for Administrators...