RACGuides°²×°Ö¸ÄÏ_Rac11gR1OnHPUX

Table of ContentsRac11gR1OnHPUX............................................................................................................................................1

1. *Introduction.......................................................................................................................................1 1.1. *What you need to know....................................................................................................1

1.1.1. Software required for install (Itanium or PA-RISC)..........................................1 1.1.2. Processor Model.................................................................................................1

1.2. *Installation steps..........................................................................................................................................1 1.3. *Schematic....................................................................................................................................................2

1.3.1. Hardware/software configuration BEFORE Oracle software install............................................2 1.3.2. Hardware/software configuration AFTER Oracle software install..............................................2

1.4. *Installation Method.....................................................................................................................................3 2. *Prepare the cluster nodes for Oracle RAC.....................................................................................................3

2.1. *User Accounts and Software Directories.......................................................................................4 2.1.1. *Creating the OSDBA (DBA) Group..............................................................................4 2.1.2. *Creating an OSOPER Group (Optional)........................................................................4 2.1.3. *Creating the Oracle Software Owner User....................................................................4 2.1.4. *Creating an Oracle Software Owner User.....................................................................5 2.1.5. *Modifying an Oracle Software Owner User..................................................................5 2.1.6. *Creating Software Directories.......................................................................................5

2.2. *SSH Setup...................................................................................................................................................6 2.2.1. *Checking Existing SSH Configuration on the System...............................................................6 2.2.2. Configuring SSH...........................................................................................................................6

2.3. *Hardware Requirements and Preparation...................................................................................................7 2.3.1. *Hardware Requirements..............................................................................................................7

2.4. *Kernel Parameters.......................................................................................................................................8 2.5. *Required HPUX Patches.............................................................................................................................9

2.5.1. *HPUX 11.23 Patches..................................................................................................................9 2.5.2. *HPUX 11.31 Patches..................................................................................................................9

2.6. *Network Requirements and Preparation...................................................................................................10 2.6.1. *Network Ping Tests...................................................................................................................10 2.6.2. *Network Adapter configuration................................................................................................11

2.7. *Stage the Oracle Software.........................................................................................................................11 2.8. *Check Preparations...................................................................................................................................11 3. *Prepare the Shared Storage for Oracle RAC This section describes how to prepare the shared storage for Oracle RAC..................................................................................................................................................16

3.1. Configuring Raw Disk for Storage To configure raw disks for database file storage, follow the following steps:...............................................................................................................................16

4. Oracle Clusterware Installation and Configuration.......................................................................................18 4.1. Oracle Clusterware Install..............................................................................................................18

5. Oracle Clusterware patching..........................................................................................................................25 5.1. Oracle Clusterware patching complete..........................................................................................32

6. Oracle ASM Home Software Install..............................................................................................................32 6.1. CVU check.....................................................................................................................................32 6.2. Oracle ASM Home Software Install..............................................................................................32

6.2.1. Oracle ASM software home has been installed.............................................................40 7. Oracle ASM Software Home Patching..........................................................................................................40

7.1. Start the OUI for Oracle ASM Software Home Patching..............................................................40 8. Oracle RAC Database Listener Creation.......................................................................................................47

8.1. Create Node specific network listeners..........................................................................................47 8.2. Completed Node specific network listeners creation.....................................................................52

9. Oracle ASM Instance and diskgroup Creation..............................................................................................52 9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups..........................................52 9.2. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups............61

10. Oracle RAC Database Home Software Install.............................................................................................61 10.1. CVU check - Pre Database Install...............................................................................................61 10.2. Oracle RAC Database Home Software Install.............................................................................61

i

Table of ContentsRac11gR1OnHPUX 10.2.1. Oracle RAC Database software home has been installed............................................69

11. Oracle RAC Software Home Patching........................................................................................................69 11.1. Using the installer to patch the Oracle RAC Software Home.....................................................69

12. Oracle RAC Database Creation...................................................................................................................76 12.1. use dbca to create the RAC database...........................................................................................76

ii

Rac11gR1OnHPUX

1. *Introduction

1.1. *What you need to know

For those of you starting an Oracle RAC install for the first time, all this will be new. For those of you withOracle 10g RAC on HPUX experience, there are just a few changes.

Disk requirements for Oracle Clusterware devicesThe storage requirement for the OCR & vote devices used by Oracle Clusterware has been increasedto 280 MB (256 MB, plus an additional buffer to allow for fdisk variation in space allocation).

Better support for IP addressThe previous release had some restrictions that could be easily worked around, with regard to IPaddresses from the non-routable range. These restrictions have been removed in the OracleClusterware 11g version and the workarounds are no longer required.

1.1.1. Software required for install (Itanium or PA-RISC)

hpia64_11gR1_database_1of2.zip and hpia64_11gR1_database_2of2.zip• hpux.parisc_11gR1_database_1of2.zip and hpux.parisc_11gR1_database_2of2.zip•

1.1.2. Processor Model

This paper covers both HPUX(PA-RISC) and HPUX(IA64) installs. Please note that The OS Version ofOracle you install must match the OS Chip Version. The following HPUX versions are supported:

HP-UX 11.31 Itanium• HP-UX 11.31 PA-RISC• HP-UX 11.23 Itanium• HP-UX 11.23 PA-RISC•

1.2. *Installation steps

To summarise the install steps are:

Preparation

Pre-reqs. to make sure the cluster is setup OK.♦ Stage all the software on one node, typically Node1♦

•

Establish Oracle Clusterware

Install the Oracle Clusterware (using the push mechanism to install on the other nodes in thecluster)

♦

Patch the Clusterware to the latest patchset♦

•

Establish ASM

Install an Oracle Software Home for ASM♦

•

Rac11gR1OnHPUX 1

Patch the ASM Home to the latest patchset♦ Create the Listeners♦ Create the ASM Instances and initial Disk Group♦

Establish RAC Database

Install an Oracle Software Home for RAC Database♦ Patch the RAC Database Home to the latest patchset♦ Create the RAC Database Instances♦

•

1.3. *Schematic

The following is a schematic of the software & hardware layout of a 2 node RAC cluster. As explained in thisdocument the actual number of LUN’s required will vary depending on your mirroring requirements.

1.3.1. Hardware/software configuration BEFORE Oracle software install

1.3.2. Hardware/software configuration AFTER Oracle software install

Notice that there are a number of LUNs present to both machines from the shared storage.

4 of 500MB LUNs•

1.2. *Installation steps 2

2 are used for the OCR devices (/dev/rdsk/c4t0d1,/dev/rdsk/c6t0d1)♦ 2 are used for the vote devices (/dev/c4t0d2,/dev/c6t0d2)♦

12 of 15890MB LUNs

10 are used for the +DATA ASM diskgroup (/dev/rdsk/c4t0d[3-7],/dev/rdsk/c6t0d[3-7])♦ 2 are used for the +FLASH diskgroup. (/dev/rdsk/c4t0d10,/dev/rdsk/c6t0d10)♦

•

Also notice there is a dedicated switch on the cluster interconnect.

1.4. *Installation Method

This document details one method of installing a 2-node Oracle 11gR1 RAC cluster on HPUX:

Character Device: The Oracle Clusterware, ASM & Oracle Database Homes binaries are establishedon the local disk of each of the RAC nodes. The files required by Oracle Clusterware must be onCharacter Devices. The database data-files are on ASM. The local Homes for Oracle Clusterware,ASM and RAC take up a total of 6.5GB on each node's local disk filesystem.

•

It should be noted that there are other installation configurations. This document does not cover thesemethods:

OCFS: The Oracle Cluster, ASM & Oracle Database Homes are established on the local disk of eachof the RAC nodes. The files required by Oracle Clusterware are on OCFS. The database data-files arealso on OCFS.

•

NFS: The Oracle Cluster, ASM & Oracle Database Homes are established on the local disk of each ofthe RAC nodes. The files required by Oracle Clusterware are on Block devices. The databasedata-files are also on NFS File Server.

•

ServiceGuard: Various combinations of HP-ServiceGuard and Oracle Clusterware/ASM aresupported for the Oracle 11gR1 Database. Installation details of these configurations are not coveredin this document.

•

There are other possible combinations although Oracle recommends one of the above methods.

You cannot place the Clusterware devices on NFS File Server. NFS File Server is not covered here•

2. *Prepare the cluster nodes for Oracle RAC

This installation routine presumes that you have a 2-node HPUX cluster. There are a number of items thatrequire checking before the install commences. Getting this bit right will enhance your install experience.

User Accounts and Software Directories1. SSH Setup2. Hardware Requirements and Preparation3. Kernel Parameters4. Required HPUX Patches5. Network Requirements and Preparation6. Stage the Oracle Software7. Check Preparations8.

It is essential that these items are checked and are configured correctly before the install commences.

1.3.2. Hardware/software configuration AFTER Oracle software install 3

2.1. *User Accounts and Software Directories

2.1.1. *Creating the OSDBA (DBA) Group

You must create an OSDBA group in the following circumstances:

An OSDBA group does not exist, for example, if this is the first installation of Oracle Databasesoftware on the system

•

An OSDBA group exists, but you want to give a different group of operating system users databaseadministrative privileges in a new Oracle installation

•

To determine whether the OSDBA group exists, enter the following command:

# grep OSDBA_group_name /etc/group

If the OSDBA group does not exist or if you require a new OSDBA group, then create it as follows. In thefollowing command, use the group name dba unless a group with that name already exists. The defaultOSDBA group name is dba.

# /usr/sbin/groupadd -g 201 dba

Check too whether the oinstall group exists. If it does not, then add this group too.

# /usr/sbin/groupadd -g 200 oinstall

2.1.2. *Creating an OSOPER Group (Optional)

Create an OSOPER group only if you want to identify a group of operating system users with a limited set ofdatabase administrative privileges (SYSOPER operator privileges). For most installations, it is sufficient tocreate only the OSDBA group. If you want to use an OSOPER group, then you must create it in the followingcircumstances:

If an OSOPER group does not exist, for example, if this is the first installation of Oracle Databasesoftware on the system

•

If an OSOPER group exists, but you want to give a different group of operating system users databaseoperator privileges in a new Oracle installation

•

If you require a new OSOPER group, then create it as follows. In the following command, use the group nameoper unless a group with that name already exists.

# /usr/sbin/groupadd -g 202 oper

2.1.3. *Creating the Oracle Software Owner User

You must create an Oracle software owner user in the following circumstances:

If an Oracle software owner user does not exist, for example, if this is the first installation of Oraclesoftware on the system

•

If an Oracle software owner user exists, but you want to use a different operating system user, withdifferent group membership, to give database administrative privileges to those groups in a newOracle Database installation

•

To determine whether an Oracle software owner user named oracle exists, enter the following command:

2.1. *User Accounts and Software Directories 4

# id oracle

If the oracle user exists, then the output from this command is similar to the following:

uid=440(oracle) gid=200(oinstall) groups=201(dba),202(oper)

If the user exists, then determine whether you want to use the existing user or create another oracle user.

If you want to use the existing user, then ensure that the user's primary group is the Oracle Inventory groupand that it is a member of the appropriate OSDBA and OSOPER groups.

2.1.4. *Creating an Oracle Software Owner User

In the following procedure, use the user name oracle unless a user with that name already exists. If the Oraclesoftware owner user does not exist or if you require a new Oracle software owner user, then create it asfollows:

1.To create the oracle user, enter a command similar to the following:

# /usr/sbin/useradd -g oinstall -G dba[,oper] oracle

In this command:

The -g option specifies the primary group, which must be the Oracle Inventory group, for exampleoinstall

•

The -G option specifies the secondary groups, which must include the OSDBA group and if required,the OSOPER group

•

2.Set the password of the oracle user:

# passwd oracle

2.1.5. *Modifying an Oracle Software Owner User

If the oracle user exists, but its primary group is not oinstall or it is not a member of the appropriate OSDBAor OSOPER groups, then enter a command similar to the following to modify it. Specify the primary groupusing the -g option and any required secondary group using the -G option:

# /usr/sbin/usermod -g oinstall -G dba[,oper] oracle

2.1.6. *Creating Software Directories

On each of the nodes, issue the following commands as root, to create the required directories with the correctownership and permissions:

# mkdir /var/opt/oracle # chown oracle:oinstall /var/opt/oracle # chmod 755 /var/opt/oracle

# mkdir -p /opt/oracle/product/CRS

# mkdir -p /opt/oracle/product/RDBMS # chown -R oracle:dba /opt/oracle

2.1.3. *Creating the Oracle Software Owner User 5

# chmod -R 755 /opt/oracle

Make sure that the user environment variables are set correctly:

# export ORACLE_BASE=/opt/oracle/product # export ORACLE_HOME=$ORACLE_BASE/RDBMS # export ORA_CRS_HOME=$ORACLE_BASE/CRS

2.2. *SSH Setup

To configure SSH, you must first create RSA or DSA keys on each cluster node, and then copy all the keysgenerated on all cluster node members into an authorized keys file that is identical on each node. Note that theSSH files must be readable only by root and by the software installation user (oracle, crs, asm), as SSHignores a private key file if it is accessible by others. When this is done, then start the SSH agent to load keysinto memory. In the examples that follow, the RSA key is used. You must configure SSH separately for eachOracle software installation owner that you intend to use for installation.

2.2.1. *Checking Existing SSH Configuration on the System

To determine if SSH is running, enter the following command:

# pgrep sshd

If SSH is running, then the response to this command is one or more process ID numbers. In the homedirectory of the software owner that you want to use for the installation (crs, oracle), use the command ls-al to ensure that the .ssh directory is owned and writable only by the user.

You need either an RSA or a DSA key for the SSH protocol. RSA is used with the SSH 1.5 protocol, whileDSA is the default for the SSH 2.0 protocol. With OpenSSH, you can use either RSA or DSA. Theinstructions that follow are for SSH1. If you have an SSH2 installation, and you cannot use SSH1, then referto your SSH distribution documentation to configure SSH1 compatibility or configure SSH2 with DSA.

2.2.2. Configuring SSH

The following steps are to be run by the software owner ("oracle" user in our case)

On the First Node, then on the Second Node:

# mkdir ~/.ssh # chmod 755 ~/.ssh # /usr/bin/ssh-keygen -t rsa

Note that in this case, we will leave the passphrase empty. The identification information has been saved in/home/oracle/.ssh/id_rsa. The public key has been saved in/home/oracle/.ssh/id_rsa.pub.

The contents of the id_rsa.pub file of both nodes need to be put into a file called/home/oracle/.ssh/authorized_keys on both nodes. Basically, the files are identical, so createthe file on node 1 and copy it to the same location on node 2.

# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys # ssh oracle@node2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys # chmod 644 ~/.ssh/authorized_keys

2.1.6. *Creating Software Directories 6

# scp ~/.ssh/authorized_keys node2:~/.ssh/authorized_keys

Now, check connectivity in each direction, from each of the cluster servers, first from node1 and then repeatfrom node2.

# ssh node1 date # ssh node2 date

At the end of this process, the public hostname for each member node should be registered in theknown_hosts file for all other cluster member nodes.

2.3. *Hardware Requirements and Preparation

2.3.1. *Hardware Requirements

Cluster servers must all be configured and prepared sufficient to the following specifications:

at least 1GB of physical RAM. Use either of the following commands to verify the amount ofphysical memory installed:

•

# /usr/contib/bin/machinfo | grep -i Memory

# /usr/sbin/dmesg | grep "Physical:"

swap space equivalent to a multiple of the physical memory, according to the following situations. IfRAM is less than 2GB, swap space should be double the size of RAM. If RAM is between 2GB and8GB, then the recommended swap space should be equal to the RAM. Otherwise, if RAM is over8GB, then allocate a swap space .75 times the size of the RAM. Use the following command toconfirm the size of swap:

•

# /usr/sbin/swapinfo -a

ensure that there is at least 400MB free space in the /tmp directory. To verify the free space availablein /tmp, issue the following command:

•

# bdf /tmp

the Oracle Clusterware home must have at least 650MB of available disk space.• the Oracle database home should have at least 4GB of available space• confirm that the OS-version is correct, and 64-bit, issue the follow commands:•

# uname -a # /bin/getconf KERNEL_BITS

confirm that Async I/O is enabled (default for HPUX 11.31 and 11.23):•

# ll /dev/async

the output should look something like: # crw-rw-rw- 1 bin bin 101 0x000000 May 16 07:23/dev/async

•

in order to successfully relink the Oracle executables (if required) after installation, ensure that thefollowing symbolic links have been created:

•

2.2.2. Configuring SSH 7

# cd /usr/lib # ln -s /usr/lib/libX11.3 libX11.sl # ln -s /usr/lib/libXIE.2 libXIE.sl # ln -s /usr/lib/libXext.3 libXext.sl # ln -s /usr/lib/libXhp11.3 libXhp11.sl # ln -s /usr/lib/libXi.3 libXi.sl # ln -s /usr/lib/libXm.4 libXm.sl # ln -s /usr/lib/libXp.2 libXp.sl # ln -s /usr/lib/libXt.3 libXt.sl # ln -s /usr/lib/libXtst.2 libXtst.sl

make sure that each of the cluster nodes is set to the same date and time. Oracle strongly recommendsusing the Network Time Protocol feature of most operating systems for this purpose, with all nodesusing the same reference Network Time Protocol server.

•

2.4. *Kernel Parameters

Verify that the kernel parameters shown listed below are set correctly (higher values are allowed). Also, checkthe most recently published documentation for the Oracle Cluseterware and Database software for anychanges.

nproc => 4096• ksi_alloc_max => (nproc*8)• executable_stack => 0• max_thread_proc => 1024• maxdsiz => 1073741824 (1 GB)• maxdsiz_64bit => 2147483648 (2 GB)• maxssiz => 134217728 (128 MB)• maxssiz_64bit => 1073741824 (1 GB)• maxuprc => ((nproc*9)/10)• msgmap (11.23 only) => (msgtgl+2)• msgmni => nproc• msgseg => (nproc*4) (at least 32767)• msgtgl => nproc• ncsize => ninode+1024• nfile => (15*nproc+2048) (for db's with a high number of data files, assume (number of Oracleprocesses)*(number of Oracle data files)+2048)

•

nflocks => nproc• ninode => (8*nproc+2048)• nkthread => (((nproc*7)/4)+16)• semmni => nproc• semmns => (semmni*2)• semmnu => (nproc-4)• semvmx => 32767• shmmax => size of physical memory or 1073741824, whichever is greater, must be equal to or greaterthan the size of the SGA

•

shmmni => 512• shmseg => 120• vps_ceiling => 64 (up to 16384 for a large SGA)•

Modify the kernel parameter settings by using either the kcweb application

# /usr/sbin/kcweb -F

2.3.1. *Hardware Requirements 8

or by using the kmtune command line utility (kctune on Itanium):

# kmtune parameter>=value

List the parameters changed:

# kmtune -D

2.5. *Required HPUX Patches

To determine which operating system patches are installed, enter the following command:

# /usr/sbin/swlist -l patch

To determine if a specific operating system patch has been installed, enter the following command:

# /usr/sbin/swlist -l patch <patch_number>

To determine which operating system bundles are installed, enter the following command:

# /usr/sbin/swlist -l bundle

2.5.1. *HPUX 11.23 Patches

HPUX 11.23 with Sept 2004 or newer base, and Mar 2007 Patch bundle for HP-UX11iV2-B.11.23.0703

•

PHKL_33025 file system tunables cumulative patch• PHKL_34941 Improves Oracle Clusterware restart and diagnosis• PHCO_32426 reboot(1M) cumulative patch• PHCO_36744 LVM patch [replaces PHCO_35524]• PHCO_37069 libsec cumulative patch• PHCO_37228 libc cumulative patch) [replaces PHCO_36673]• PHCO_38120 kernel configuration commands patch• PHKL_34213 vPars CPU migr, cumulative shutdown patch• PHKL_34989 getrusage(2) performance• PHKL_36319 mlockall(2), shmget(2) cumulative patch) [replaces PHKL_35478]• PHKL_36853 pstat patch• PHKL_37803 mpctl(2) options, manpage, socket count) [replaces PHKL_35767]• PHKL_37121 sleep kwakeup performance cumulative patch [replaces PHKL_35029]• PHKL_34840 slow system calls due to cache line sharing• PHSS_37947 linker + fdp cumulative patch) [replaces PHSS_35979]• PHNE_37395 cumulative ARPA Transport patch•

2.5.2. *HPUX 11.31 Patches

PHKL_37296 vfs module patch• PHKL_37452 vm cumulative patch [replaces PHKL_35900, PHKL_35936]• PHKL_37453 esdisk cumulative patch [replaces PHKL_36249]• PHKL_37454 esctl cumulative patch [replaces PHKL_36248]• PHCO_37476 libc cumulative patch• PHCO_37807 Cumulative Patch for bcheckrc• PHSS_37948 linker + fdp cumulative patch• PHSS_37954 Integrity Unwind Library•

2.4. *Kernel Parameters 9

PHNE_35894 Networking commands cumulative patch•

2.6. *Network Requirements and Preparation

You will need a minimum of 2 network adapters per machine, 1 adapter is the public interface. The otheradapter, the private interface, is used for Oracle RAC node membership and inter-instance traffic. The privateinterface adapters should be connected to each other via a network switch which is used exclusively for theinterconnect traffic. In 2 node environments do not be tempted to use a 'Crossover Cable'. Best practice is touse more than 2 adapters, Before installation of the Oracle software you take 2 of the adapters on the privateinterface and team them together to create what looks like a single, highly available adapter for theinterconnect traffic. You can then take the remaining 3rd adapter and use it for the public interface; somecustomers even team their public adapters as well. In that case you would need to start with 4 adapters, 2public & 2 private. Ensure you team the adapters before starting installation.

Please be aware that even with 2-node cluster you must use a dedicated network switch on the interconnect,Crossover cables are not supported

You will need a total of 3 IP addresses per node:

The public IP address, which should be recorded in hosts file on each node and, if available, DNS.This IP Address should be bound to the public adapter before starting the install. It should be a static,not DHCP, address.

•

The private IP address, which should be from a different subnet than the public IP address. Thisaddress does not require registering in DNS but you should place an entry in the hosts file on eachnode. This IP Address should be bound to the private adapter before starting the install. It should be astatic, not DHCP, address. Oracle recommends that you use a subnet reserved for private networks,such as 10.0.0.0 or 192.168.0.0

•

A VIP address, which should be from the same subnet as the public IP address and should berecorded in DNS and the hosts file on each node. This IP Address should NOT be bound to the publicadapter before starting the install. Oracle Clusterware is responsible for binding this address. It shouldbe a static, not DHCP, address. Refer to Metalink Note 296874.1 "Configuring the HPUX OperatingSystem for the Oracle 10g VIP".

•

If you do not have a DNS server then make sure both the public and the VIP addresses are entered into all therelevant hosts files that are normally found in the /etc/hosts file.

2.6.1. *Network Ping Tests

There are a series of 'ping' tests that should be completed, and then the network adapter binding order shouldbe checked. You should ensure that the public IP addresses resolve correctly and that the private addresses areof the form 'nodename-priv' and resolve on both nodes via the hosts file.

Public Ping testPinging node1 from node1 should return node1's public IP addressPinging node2 from node1 should return node2's public IP addressPinging node1 from node2 should return node1's public IP addressPinging node2 from node2 should return node2's public IP address

•

Private Ping testPinging node1 private from node1 should return node1's private IP addressPinging node2 private from node1 should return node2's private IP addressPinging node1 private from node2 should return node1's private IP addressPinging node2 private from node2 should return node2's private IP address

•

2.5.2. *HPUX 11.31 Patches 10

VIP Ping testPinging the VIP address at this point should fail. VIPs will be activated at the end of the OracleClusterware install.

•

If any of the above tests fail you should fix name/address resolution by updating the DNS or local hosts fileson each node before continuing with the installation.

2.6.2. *Network Adapter configuration

If your network adapters allow configuration you should make sure that they are configured for Full Duplexand at the 'fastest' speed consistent among nodes. They should not be left to 'auto-negotiate'.

2.7. *Stage the Oracle Software

It is recommended that you stage the required software onto a local drive on Node 1 of your cluster.

The 11gR1 now integrates the Oracle Clusterware, Database and Client install into one DVD with onerunInstaller program.

2.8. *Check Preparations

Now you can run the CVU to check the state of the cluster prior to the install of the Oracle ClusterwareSoftware. CVU can be run from the installation media.

From the clsuterware directory (just below the stage directory), issue the following command to run the CVUutitility:

# runcluvfy.sh stage -pre crsinst -n node1,node2 -r 11gR1 -verbose

This will produce a listing of pre-installation tests and success or failure states for each test, much like thefollowing:

Show CVU output Hide CVU outputruncluvfy.sh stage -pre crsinst -n atlrp441,atlrp442 -r 11gR1 -verbose

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "atlrp441" Destination Node Reachable? ------------------------------------ ------------------------ atlrp441 yes atlrp442 yes Result: Node reachability check passed from node "atlrp441".

Checking user equivalence...

Check: User equivalence for user "oracle" Node Name Comment ------------------------------------ ------------------------ atlrp442 passed

2.6.1. *Network Ping Tests 11

atlrp441 passed Result: User equivalence check passed for user "oracle".

Checking administrative privileges...

Check: Existence of user "oracle" Node Name User Exists Comment ------------ ------------------------ ------------------------ atlrp442 yes passed atlrp441 yes passed Result: User existence check passed for "oracle".

Check: Existence of group "oinstall" Node Name Status Group ID ------------ ------------------------ ------------------------ atlrp442 exists 402 atlrp441 exists 402 Result: Group existence check passed for "oinstall".

Check: Membership of user "oracle" in group "oinstall" [as Primary] Node Name User Exists Group Exists User in Group Primary Comment ---------------- ------------ ------------ ------------ ------------ ------------ atlrp442 yes yes yes yes passed atlrp441 yes yes yes yes passed Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed.

Administrative privileges check passed.

Checking node connectivity...

Interface information for node "atlrp442" Interface Name IP Address Subnet Subnet Gateway Default Gateway Hardware Address ---------------- ------------ ------------ ------------ ------------ ------------ eth0 10.143.181.34 10.143.181.0 10.143.181.34 UNKNOWN 00:0E:0C:08:06:4B eth2 10.143.207.17 10.143.207.0 10.143.207.17 UNKNOWN 00:04:23:A6:CD:BE

Interface information for node "atlrp441" Interface Name IP Address Subnet Subnet Gateway Default Gateway Hardware Address ---------------- ------------ ------------ ------------ ------------ ------------ eth0 10.143.181.33 10.143.181.0 10.143.181.33 UNKNOWN 00:0E:0C:08:08:F4 eth2 10.143.207.16 10.143.207.0 10.143.207.16 UNKNOWN 00:04:23:A6:CD:6A

Check: Node connectivity of subnet "10.143.181.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- atlrp442:eth0 atlrp441:eth0 yes Result: Node connectivity check passed for subnet "10.143.181.0" with node(s) atlrp442,atlrp441.

Check: Node connectivity of subnet "10.143.207.0" Source Destination Connected? ------------------------------ ------------------------------ ---------------- atlrp442:eth2 atlrp441:eth2 yes Result: Node connectivity check passed for subnet "10.143.207.0" with node(s) atlrp442,atlrp441.

2.8. *Check Preparations 12

Interfaces found on subnet "10.143.181.0" that are likely candidates for a private interconnect:atlrp442 eth0:10.143.181.34atlrp441 eth0:10.143.181.33

Interfaces found on subnet "10.143.207.0" that are likely candidates for a private interconnect:atlrp442 eth2:10.143.207.17atlrp441 eth2:10.143.207.16

WARNING: Could not find a suitable set of interfaces for VIPs.Result: Node connectivity check passed.

Checking system requirements for 'crs'...

Check: Total memory Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- atlrp442 7.98GB (8371808KB) 1GB (1048576KB) passed atlrp441 7.98GB (8371808KB) 1GB (1048576KB) passed Result: Total memory check passed.

Check: Free disk space in "/tmp" dir Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- atlrp442 5.02GB (5262320KB) 400MB (409600KB) passed atlrp441 5.02GB (5262320KB) 400MB (409600KB) passed Result: Free disk space check passed.

Check: Swap space Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- atlrp442 15.98GB (16760416KB) 1.5GB (1572864KB) passed atlrp441 15.98GB (16760416KB) 1.5GB (1572864KB) passed Result: Swap space check passed.

Check: System architecture Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- atlrp442 PA-RISC PA-RISC passed atlrp441 PA-RISC PA-RISC passed Result: System architecture check passed.

Check: Kernel version Node Name Available Required Comment ------------ ------------------------ ------------------------ ---------- atlrp442 HP-UX B.11.31 HP-UX B.11.31 passed atlrp441 HP-UX B.11.31 HP-UX B.11.31 passed Result: Kernel version check passed.

Check: Package existence for "OS-Core:B.11.31" Node Name Status Comment ------------------------------ ------------------------------ ---------------- atlrp441 OS-Core:B.11.31 passed atlrp442 OS-Core:B.11.31 passed


Result: Package existence check passed for "OS-Core:B.11.31".

Check: Operating system patch for "PHKL_36248" => Superceded by PHKL_37454 Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- atlrp441 missing PHKL_36248 failed atlrp442 missing PHKL_36248 failedResult: Operating system patch check failed for "PHKL_36248".

Check: Operating system patch for "PHKL_36249" => Superceded by PHKL_37453 Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- atlrp441 missing PHKL_36249 failed atlrp442 missing PHKL_36249 failedResult: Operating system patch check failed for "PHKL_36249".

Check: Operating system patch for "PHKL_35900" => Superceded by PHSS_37452 Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- atlrp441 missing PHKL_35900 failed atlrp442 missing PHKL_35900 failedResult: Operating system patch check failed for "PHKL_35900".

Check: Operating system patch for "PHKL_35936" => Superceded by PHSS_37452 Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- atlrp441 missing PHKL_35936 failed atlrp442 missing PHKL_35936 failedResult: Operating system patch check failed for "PHKL_35936".

Check: Operating system patch for "PHSS_37948" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- atlrp441 PHSS_37948 PHSS_37948 passed atlrp442 PHSS_37948 PHSS_37948 passedResult: Operating system patch check passed for "PHSS_37948".

Check: Package existence for "ACXX:C.06.10" Node Name Status Comment ------------------------------ ------------------------------ ---------------- atlrp441 ACXX:C.06.10 passed atlrp442 ACXX:C.06.10 passedResult: Package existence check passed for "ACXX:C.06.10".

Check: Package existence for "C-ANSI-C:C.06.10" Node Name Status Comment ------------------------------ ------------------------------ ---------------- atlrp441 C-ANSI-C:C.06.10 passed atlrp442 C-ANSI-C:C.06.10 passedResult: Package existence check passed for "C-ANSI-C:C.06.10".

Check: Operating system patch for "PHKL_37296" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- atlrp441 PHKL_37296 PHKL_37296 passed


atlrp442 PHKL_37296 PHKL_37296 passedResult: Operating system patch check passed for "PHKL_37296".

Check: Operating system patch for "PHCO_37476" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- atlrp441 PHKL_34941 PHCO_37476 passed atlrp442 PHKL_34941 PHCO_37476 passedResult: Operating system patch check passed for "PHCO_37476".

Check: Operating system patch for "PHCO_37807" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- atlrp441 missing PHCO_37807 passed atlrp442 missing PHCO_37807 passedResult: Operating system patch check passed for "PHCO_37807".

Check: Operating system patch for "PHSS_37948" Node Name Applied Required Comment ------------ ------------------------ ------------------------ ---------- atlrp441 PHSS_37948 PHSS_37948 passed atlrp442 PHSS_37948 PHSS_37948 passedResult: Operating system patch check passed for "PHCO_37807".

Check: Package existence for "Perl5-64:D.5.8.8" Node Name Status Comment ------------------------------ ------------------------------ ---------------- atlrp441 Perl5-64:D.5.8.8.B passed atlrp442 Perl5-64:D.5.8.8.B passedResult: Package existence check passed for "Perl5-64:D.5.8.8".

Check: Group existence for "dba" Node Name Status Comment ------------ ------------------------ ------------------------ atlrp442 exists passed atlrp441 exists passed Result: Group existence check passed for "dba".

Check: Group existence for "oinstall" Node Name Status Comment ------------ ------------------------ ------------------------ atlrp442 exists passed atlrp441 exists passed Result: Group existence check passed for "oinstall".

Check: User existence for "nobody" Node Name Status Comment ------------ ------------------------ ------------------------ atlrp442 exists passed atlrp441 exists passed Result: User existence check passed for "nobody".

System requirement failed for 'crs'

Pre-check for cluster services setup was unsuccessful on all the nodes.


Show all Hide all

Note that in this case the CVU failed, but only because some of the patches it was checking for had beensuperceded with more recent versions.

3. *Prepare the Shared Storage for Oracle RAC This sectiondescribes how to prepare the shared storage for Oracle RAC

For all installations, you must choose the storage option that you want to use for Oracle Clusterware files,Automatic Storage Manager (ASM) and Oracle Real Application Clusters databases (Oracle RAC).

Oracle Clusterware files include:

Voting disks, used to monitor cluster node status, and• Oracle Cluster Registry (OCR) which contains configuration information about the cluster.•

There are two ways of storing Oracle Clusterware files:

Raw Devices: Oracle Clusterware files can be placed on RAW devices based on shared diskpartitions.

•

A supported shared file system - Supported file systems include the following:

A supported cluster file system♦ Network File System (NFS): A file-level protocol that enables access and sharing of files♦

•

In the example given in this document, Raw Devices are used.

3.1. Configuring Raw Disk for Storage To configure raw disks fordatabase file storage, follow the following steps:

Make sure that the disks are available, enter the following command on every node:•

# ioscan -fnNkCdisk

If the ioscan command does not display device name information for a device that you want to use,enter the following command to install the special device files for any new devices:

•

# insf -e

The disk names for the same disk can be different on each node. A disk can be identified as the sameone via the WWID. The WWID of a disk can be checked via the following command:

•

# scsimgr lun_map -D /dev/rdsk/disk25 | grep WWID

The System Management Homepage also shows the WWID for each disk:

3. *Prepare the Shared Storage for Oracle RAC This section describes how to prepare the shared storage for Oracle RAC16

For each disk that you want to use, enter the following command on any node to verify that it is notalready part of an LVM volume group:

•

# pvdisplay /dev/rdsk/ora_data1

If this command displays volume group information, the disk is already part of a volume group. Thedisks that you choose must not be part of an LVM volume group.

•

We recommend to create a special Oracle device directory, and to use mknod to create device paths inthis special Oracle folder. This has the advantage, that you get same names for OCR and Voting filesacross all nodes in the cluster. In addition, it ensures that the permissions of these Oracle device filesremain untouched of 'insf −e'.

•

# mkdir /dev/oracle # ll /dev/rdsk/disk25 crw-r----- 1 bin sys 23 0x000019 Jan 16 12:16 /dev/rdisk/disk25 # mknod /dev/oracle/asmdisk1 c 23 0x000019

Later during set-up, for the ASM instance, set the ASM_DISKSTRING parameter to /dev/oracle/*Now when 'insf -e' is run, it only touches 'standard' DSF, rather than this special ones.

•

Modify the owner, group, and permissions on the character raw device files (ASM and Voting Disks)on all nodes:

•

# chown oracle:dba /dev/oracle/* # chmod 660 /dev/oracle/*

Modify the owner, group, and permissions on the character raw device files (OCR files) on all nodes:•

# chown root:dba /dev/oracle/OCR # chmod 640 /dev/oracle/OCR

3.1. Configuring Raw Disk for Storage To configure raw disks fordatabase file storage, follow the following steps:17

4. Oracle Clusterware Installation and Configuration

The next step is to install the Oracle Clusterware layer. Oracle Clusterware is an essential component of theOracle RAC database infrastructure. Oracle Clusterware does not require any other clustering software.

4.1. Oracle Clusterware Install

Start the installer by running runInstaller from the staged installation media.

$ ./runInstaller &

Notes

Notice that the Oracle 11g Installer now combines the Oracle Database, Client andClusterware components

♦

•

Action

Select the Oracle Clusterware radio button♦ Click Next♦

•

Notes•

4. Oracle Clusterware Installation and Configuration 18

The OUI will name the Oracle Clusterware Home'OraCrs11g_home'. If you change this youshould make sure that the name you use is unique

♦

Actions

Specify a location for the Oracle Clusterware Home♦ Click Next♦

•

Notes

The installer will validate the state of the cluster before continuing. If there are issues youshould rectify them before continuing

♦

•

Actions

Click Next♦

•

Notes

Each Cluster requires a name, this should be unique within your organisation, The default is asubstring of the node name followed by _cluster

♦

This is where you specify details of all the nodes in the cluster. The installer will defaultnames for the node it is running on. You must add other nodes manually

♦

Oracle defaults the names to 'nodename', 'nodename-priv', 'nodename-vip'♦

•

Actions•

4.1. Oracle Clusterware Install 19

Confirm the Cluster Name selected is acceptable♦ Confirm the Details for the current node are OK. The defaults are:

Public Node Name : must resolve via hosts and or DNS to the public IP address andmust be live

◊

Private Node Name : must resolve via hosts to the interconnect IP address and mustbe live

◊

Virtual Host Name : must resolve via hosts and or DNS to a new IP address and mustnot be live

◊

If these are not correct select the node entry and click Edit... to modify OR◊

♦

Click Add... to add more nodes to the cluster♦

Notes

If you have more nodes repeast the Add... cycle♦

•

Actions

Click Next♦

•

Notes

Here you can see we have successfully configured the network adapter usage. Ideally you willhave only 1 adapter set as public and 1 adapter set as private. Other adapters, if available, set

♦

•


to ‘do not use’. If you have multiple public or multiple private adapters it is better to teamthem at the OS adapter driver level before commencing the install.Type : Private

Actions

Click Next♦

•

Notes

Here we specify the shared storage devices that will be used by Oracle Clusterware. Ideallyyou will have 2 devices for the OCR, Oracle will mirror to these devices to protect you from asingle OCR device failure. Also you will have an additional 3 vote devices to protect yourcluster from the failure of a single vote device.

♦

•

Actions

Select the Normal radio button♦ Enter the device to be used for the First OCR♦ Enter the device to be used for the Second OCR♦ Click Next♦

•

Notes

Next we specify the devices to be used for the Oracle Clusterware vote disks♦

•

Actions•


Select the Normal radio button♦ Enter the device to be used for the First vote disk♦ Enter the device to be used for the Second vote disk♦ Enter the device to be used for the Third vote disk♦ Click Next♦

Notes

the installer lists a summary of the planned actions♦

•

Actions

Click Install♦

•

Notes

the OUI first copies software to the local node and then copies the software to the remotenodes

♦

•

Actions

none required♦

•


Notes

The installer requires commands to be run as root on each of the nodes♦

•

Actions

On the first node open a root shell window & run the root.sh script♦

•

Show root.sh output Hide root.sh output# /opt/oracle/product/CRS/root.shWARNING: directory '/opt/oracle/product' is not owned by rootWARNING: directory '/opt/oracle' is not owned by rootWARNING: directory '/opt' is not owned by rootChecking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directorySetting up Network socket directoriesOracle Cluster Registry configuration upgraded successfullyThe directory '/opt/oracle/product' is not owned by root. Changing owner to rootThe directory '/opt/oracle' is not owned by root. Changing owner to rootThe directory '/opt' is not owned by root. Changing owner to rootSuccessfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node : node 1: atlrp441 atlrp441-priv atlrp441 node 2: atlrp442 atlrp442-priv atlrp442 Creating OCR keys for user 'root', privgrp 'sys'.. Operation successful. Now formatting voting device: /dev/oracle/vote1 Now formatting voting device: /dev/oracle/vote2 Now formatting voting device: /votedisk/vote3 Format of 3 voting devices complete. Startup will be queued to init within 30 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. Cluster Synchronization Services is active on these nodes. atlrp441 Cluster Synchronization Services is inactive on these nodes. atlrp442 Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.

Show all Hide all

Once this has completed successfully on the first node, open a root shell window on the nextnode & run the root.sh script

♦ •

Show root.sh output Hide root.sh output# /opt/oracle/product/CRS/root.shWARNING: directory '/opt/oracle/product' is not owned by rootWARNING: directory '/opt' is not owned by rootChecking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directorySetting up Network socket directories


Oracle Cluster Registry configuration upgraded successfullyThe directory '/opt/oracle/product' is not owned by root. Changing owner to rootThe directory '/opt' is not owned by root. Changing owner to rootclscfg: EXISTING configuration version 4 detected.clscfg: version 4 is 11 Release 1.Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node : node 1: atlrp441 atlrp441-priv atlrp441 node 2: atlrp442 atlrp442-priv atlrp442 clscfg: Arguments check out successfully. NO KEYS WERE WRITTEN. Supply -force parameter to override. -force is destructive and will destroy any previous cluster configuration. Oracle Cluster Registry for cluster has already been initialized Startup will be queued to init within 30 seconds. Adding daemons to inittab Expecting the CRS daemons to be up within 600 seconds. Cluster Synchronization Services is active on these nodes. atlrp441 atlrp442 Cluster Synchronization Services is active on all the nodes. Oracle CRS stack installed and running under init(1M) Running vipca(silent) for configuring nodeapps Creating VIP application resource on (2) nodes... Creating GSD application resource on (2) nodes... Creating ONS application resource on (2) nodes... Starting VIP application resource on (2) nodes... Starting GSD application resource on (2) nodes... Starting ONS application resource on (2) nodes... Done.

Show all Hide all

Notes

a series of configuration assistants are run at the end of the install♦

•

Actions

none required♦

•

Notes

The installer has completed installation and configuration of Oracle Clusterware♦

•


Actions

Click Exit♦

•

Verify your CRS installation by executing the olsnodes command from the/opt/oracle/product/CRS/bin directory:

•

# olsnodes -n # atlrp441 1 # atlrp442 2

Now you should see the following processes running:

oprocd -- Process monitor for the cluster.♦ evmd -- Event manager daemon that starts the racgevt process to manage callouts.♦ ocssd -- Manages cluster node membership and runs as oracle user; failure of this processresults in cluster restart.

♦

crsd -- Performs high availability recovery and management operations such as maintainingthe OCR. Also manages application resources and runs as root user and restarts automaticallyupon failure.

♦

•

You can check whether the Oracle processes evmd, occsd, and crsd are running by issuing thefollowing command.

•

# ps -ef | grep d.bin

At this point, you have completed phase one, the installation of Cluster Ready Services•

Please note that Oracle added the following three lines to the automatic startup file /etc/inittab

h1:3:respawn:/sbin/init.d/init.evmd run >/dev/null 2>&1 </dev/null♦ h2:3:respawn:/sbin/init.d/init.cssd fatal >/dev/null 2>&1 </dev/null♦ h3:3:respawn:/sbin/init.d/init.crsd run >/dev/null 2>&1 </dev/null♦

•

Oracle Support recommends NEVER modifying these entries in the inittab or modifying the initscripts unless you use this method to stop a reboot loop or are given explicit instructions from Oraclesupport.

•

To ensure that the Oracle Clusterware install on all the nodes is valid, the following should bechecked on all the nodes:

•

# /opt/oracle/product/CRS/bin/crsctl check crs

5. Oracle Clusterware patching

At this point we have installed Oracle Clusterware 11.1.0.6. In this section we will patch the OracleClusterware to the latest release of Oracle 11gR1 - 11.1.0.7. The patchset can be downloaded from metalink.

Note: The Oracle Clusterware software must be at the same or newer level as any other ORACLE_HOMEdirectory. Therefore, you should always upgrade Oracle Clusterware before you upgrade any otherORACLE_HOME directory. This patch does not allow you to upgrade other ORACLE_HOME directories,before you upgrade Oracle Clusterware home directory. We will apply this patchset in a rolling upgradefashion.

5. Oracle Clusterware patching 25

Before we start we can query the clusterware versions

# /opt/oracle/product/CRS/bin/crsctl query crs softwareversionOracle Clusterware version on node [atlrp441] is [11.1.0.6.0]# /opt/oracle/product/CRS/bin/crsctl query crs activeversionOracle Clusterware active version on the cluster is [11.1.0.6.0]Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directorywhere you unpacked the patch set software:

$ cd patchset_directory/Disk1$ ./runInstaller

Notes

The installer appears♦

•

Actions

Click Next♦

•


Notes

You should ensure that the Clusterware home is selected in the first drop down list box♦ The installer should default the directory to the correct location♦

•

Actions

Ensure the correct home is selected♦ Click Next♦

•


Notes

the installer detects that this is a clustered home and automatically selects all the nodes in thecluster

♦

•

Actions

Click Next♦

•

Notes

Some parameters are validated by the installer♦

•

Actions

Click Next♦

•


Notes

This is a summary of the actions the installer will complete♦

•

Actions

Click Install♦

•

Notes•


the installer stages the patch on all the nodes in the cluster♦ Actions

No action required♦

•

Notes

At the end the installer lists the mandatory steps that must be completed to apply this patch♦

•

Actions

Log in as the root user and enter the following command to shut down the OracleClusterware:

♦

•

# CRS_home/bin/crsctl stop crs

Run the root111.sh script. It will automatically start the Oracle Clusterware on the patchednode:

♦ •

# CRS_home/install/root111.sh

Show root111.sh output (all nodes except last) Hide root111.sh output (all nodes except last)# /opt/oracle/product/CRS/install/root111.shCreating pre-patch directory for saving pre-patch clusterware filesCompleted patching clusterware files to /opt/crsRelinking some shared libraries.Relinking of patched files is complete.Preparing to recopy patched init and RC scripts.Recopying init and RC scripts.Startup will be queued to init within 30 seconds.Starting up the CRS daemons.


Waiting for the patched CRS daemons to start.This may take a while on some systems..11107 patch successfully applied.clscfg: EXISTING configuration version 4 detected.clscfg: version 4 is 11 Release 1.Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node : node 1: atlrp441 atlrp441-priv atlrp441 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully Creating '/opt/oracle/product/CRS/install/paramfile.crs' with data used for CRS configuration Setting CRS configuration values in /opt/oracle/product/CRS/install/paramfile.crs Setting cluster unique identifier Restarting Oracle clusterware Stopping Oracle clusterware Stopping resources. This could take several minutes. Successfully stopped Oracle Clusterware resources Stopping Cluster Synchronization Services. Shutting down the Cluster Synchronization Services daemon. Shutdown request successfully issued. Waiting for Cluster Synchronization Services daemon to stop Cluster Synchronization Services daemon has stopped Starting Oracle clusterware Attempting to start Oracle Clusterware stack Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Waiting for Cluster Synchronization Services daemon to start Cluster Synchronization Services daemon has started Event Manager daemon has started Cluster Ready Services daemon has started

Show root111.sh output on last node Hide root111.sh output on last node# /opt/oracle/product/CRS/install/root111.shCreating pre-patch directory for saving pre-patch clusterware filesCompleted patching clusterware files to /opt/crsRelinking some shared libraries.Relinking of patched files is complete.Preparing to recopy patched init and RC scripts.Recopying init and RC scripts.Startup will be queued to init within 30 seconds.Starting up the CRS daemons.Waiting for the patched CRS daemons to start.This may take a while on some systems..11107 patch successfully applied.clscfg: EXISTING configuration version 4 detected.clscfg: version 4 is 11 Release 1.Successfully accumulated necessary OCR keys.Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.node : node 2: atlrp442 atlrp442-priv atlrp442 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully Creating '/opt/oracle/product/CRS/install/paramfile.crs' with data used for CRS configuration Setting CRS configuration values in /opt/oracle/product/CRS/install/paramfile.crs

Continue with these two steps on all nodes, one by one, this will achieve a rolling upgrade of the OracleClusterware. When done, verify the Oracle Clusterware is running on all nodes before exiting the installer.

$ CRS_home/bin/crsctl check crsCluster Synchronization Services appears healthyCluster Ready Services appears healthyEvent Manager appears healthy

# /opt/oracle/product/CRS/bin/crsctl query crs softwareversionOracle Clusterware version on node [atlrp441] is [11.1.0.7.0]# /opt/oracle/product/CRS/bin/crsctl query crs activeversionOracle Clusterware active version on the cluster is [11.1.0.7.0]


Notes

Once the Oracle Clusterware is running on all nodes, exit the installer♦

•

Actions

Click Yes♦

•

5.1. Oracle Clusterware patching complete

6. Oracle ASM Home Software Install

6.1. CVU check

6.2. Oracle ASM Home Software Install

Start the installer by running "runInstaller" from the staged installation media.

export ORACLE_HOME=/opt/oracle/product/asm./runInstaller &

In this section we establish the Oracle software for the ASM home.

5.1. Oracle Clusterware patching complete 32

Notes

Here we will create a ASM Software home on all the nodes in the cluster♦

•

Actions

Select the Oracle Database 11g radio button♦ Click Next♦

•

6.2. Oracle ASM Home Software Install 33

Notes

Here we can choose the type of database software to be installed♦

•

Actions

Select the Enterprise Edition radio button♦ Click Next♦

•

Notes

Here we specify the location of various components. An 11g install makes more use of theORACLE_BASE. Most logs will be stored in sub directories under the oracle base. Theoracle base will be common to all installs. Modify as required. Also this is where you specifythe location of the ASM software home. I usually modify the Home name and Home Path toinclude the word ASM. This makes it easier to identify later on. If you change the path youshould ensure that you do not use the exact same path as the Oracle Clusterware home

♦

•

Actions

Confirm entries are OK♦ Click Next♦

•


Notes

The installer will detect the presence of Oracle Clusterware and uses this to populate thedialog box. To build a cluster which includes all nodes you must ensure that there arecheck-boxes next to the node names

♦

•

Actions

Click Select All♦ Click Next♦

•


Notes

The installer will then complete some Product-Specific Prerequisite checks. These should allpass OK – as you have already run the CVU check

♦

•

Actions

Click Next♦

•


Notes

We are going to install a Software only home and then subsequently configure the software♦

•

Actions

Select the Install Software Only radio button♦ Click Next♦

•

Notes

We must assign the Unix group name of the priveleged users♦

•

Action

Choose oinstall (default) from the drop down lists♦ Click Next♦

•


Notes

A summary of the install will be displayed♦ In the first step, the installer will copy the software to all of the identified cluster nodes♦

•

Actions

Click Install♦

•


Notes

At various steps, the installer will pause, with instructions to run scrips (such as root.sh)♦

•

Action

open a shell window on each node and follow instructions provided by installer♦

•

# /opt/oracle/product/asm/root.sh Running Oracle 11g root.sh script...

The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /opt/oracle/product/asm

Enter the full pathname of the local bin directory: [/usr/local/bin]:

/usr/local/bin is read only. Continue without copy (y/n) or retry (r)? [y]: y

Warning: /usr/local/bin is read only. No files will be copied.

Creating /etc/oratab file... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed. Finished product-specific root actions.

Notes

Run the root scripts - they should only take a few seconds to run on each node, one at a time♦

•

Action

run the scripts indicated in the previous screen on both nodes♦ then return to the installer and Click OK♦

•


Notes

After the software install completes you will see this End of Installation dialog♦

•

Actions

Click Exit♦

•

6.2.1. Oracle ASM software home has been installed

7. Oracle ASM Software Home Patching

Once ASM software has been installed, the next step in the process is to apply the 11.1.0.7 patchset to theASM ORACLE_HOME.

7.1. Start the OUI for Oracle ASM Software Home Patching

Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directorywhere you unpacked the patch set software:


6.2.1. Oracle ASM software home has been installed 40

Notes

the Universal Installer screen appears♦

•

Action

On the Welcome screen, click Next.♦

•

Notes•

7.1. Start the OUI for Oracle ASM Software Home Patching 41

Specify the name and the location of the asm home♦ Action

Check that the name and location are correct♦

•

Notes:

here you can specify your metalik credentials for this install.♦ If you leave both fields blank you can opt out of notifications (see next screen).♦

•

Action

Enter your email address♦ Enter your Metalink password♦ Click Next♦

•


Notes:*

If you leave both fields blank, you will opt out of notifications and get the above screen.♦

•

Action

Click Yes♦

•

Notes•



♦

Actions

Click Next♦

•

Notes


•

Action

Click Next♦

•


Notes


•

Actions

Click Install♦

•

Notes•


The installer copies the patch to all the nodes in the cluster♦ Actions


•

Notes

The installer pauses, root.sh need to be run as root on both nodes of the cluster♦

•

Action

open a shell window on each node and run root.sh (one after the other)♦ then return to the installer and Click OK♦

•

#/opt/oracle/product/asm/root.sh Running Oracle 11g root.sh script...

The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /opt/oracle/product/asm




Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed.


Finished product-specific root actions.

Notes

The output from the root.sh script should be similar to this♦ it should only take a few seconds to run on each node♦

•

Action

run the scripts on all nodes♦ then return to the installer and Click OK♦

•

Notes

After the software install completes you will see this End of Installation dialog, exit theinstaller

♦

•

Actions

Click Yes♦

•

8. Oracle RAC Database Listener Creation

8.1. Create Node specific network listeners

The Oracle network listeners traditionally run from the ASM home. Here we are going to create the listenersusing netca from the ASM home.

8. Oracle RAC Database Listener Creation 47

Notes

Ensure the ORACLE_HOME environement variable isse tto the recently installed ASM homelocation

♦

•

Action

change to the ASM home bin directory♦ run ./netca♦

•

Notes

Netca detects that the Oracle Clusterware layer is running and offers Cluster or Single Nodeconfiguration

♦

•

Actions

Select the Cluster configuration radio button♦ Click Next♦

•

Notes

Netca uses Oracle Clusterware to determine all the nodes in the cluster♦

•

Actions

Click Select all nodes♦ Click Next♦

•

8.1. Create Node specific network listeners 48

Notes

You get various options – we need to configure listeners♦

•

Actions

Select the Listener configuration radio button♦ Click Next♦

•

Notes

We need to add a listener♦

•

Actions

Select the Add radio button♦ Click Next♦

•


Notes

Here you get the opportunity to name the listener – Do not change this. The listeners willeventually be called LISTENER_nodename1 & LISTENER_nodename2. This is importantfor RAC

♦

•

Actions

Click Next♦

•

Notes

Oracle Net supports various network protocols, although TCP is the most common.♦

•

Actions

Ensure the Selected Protocols list includes TCP♦ Click Next♦

•


Notes

It is possible to choose a non-default Port – I do not recommend this♦

•

Actions

Ensure the Use the Standard port number of 1521 radio button is set♦ Click Next♦

•

Notes

After configuring the node listeners you get the opportunity to configure more networkcomponents

♦

•

Actions

Select the No radio button♦ Click Next♦

•

Notes•


You get the opportunity to configre other networking components♦ Action

Click the Finish button to exit the tool♦

•

Notes

You can see the listener resources inside Oracle Clusterware♦

•

Action

run the ./crs_stat -t comand from the Oracle Clusterware home bin directory♦

•

8.2. Completed Node specific network listeners creation

You have now completed the RAC cluster nodes Network listener configuration. A new managed resourcewill have been added to Oracle Clusterware OCR for each listener.

9. Oracle ASM Instance and diskgroup Creation

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups

Notes

We use the dbca from the ASM install to create the ASM instances♦

•

Actions

ensure the ORACLE_HOME environment variable is set to the ASM home directory♦ run ./dbca from the ASM home bin directory♦

•

8.2. Completed Node specific network listeners creation 52

Notes

dbca detects the Oracle Clusterware layer is running and offers to create either cluster orsingle instance database

♦

•

Actions

Select the Oracle Real Application Clusters database radio button♦ Click Next♦

•

9.1. Create ASM Instance and add the +DATA and +FLASH diskgroups 53

Notes

We use dbca to create RAC databases and ASM instances♦

•

Actions

Select the Configure Automatic Storage Management radio button♦ Click Next♦

•

Notes

You need to make sure you create ASM instances on all the cluster nodes♦

•

Actions


•


Notes

Here we specify the password for the ASM Oracle SYS user♦

•

Actions

Enter the SYS password♦ Enter the same password for the Confirm SYS password♦ Click Next♦

•


Notes

dbca will create and start ASM instances♦

•

Actions

Click OK♦

•


Notes

This should take a few seconds♦

•

Actions

no action♦

•

Notes

ASM requires disks to be group together into diskgroups. This section will be used to create 2disk groups +DATA and +FLASH

♦

•

Actions

Click Create New♦

•


Notes

At the moment no disks are visible♦

•

Actions

Click Change Discovery Path♦

•


Notes

Now we will assign disks to specific disk groups and create the DATA diskgroup♦ Use a discovery string to enable dbca to find the available disks♦

•

Actions

In the Disk Group Name enter DATA♦ Select the External Redundancy radio button♦ Select the Show All radio button♦ Select the 6 disks to be used for the DATA diskgroup♦ Click OK♦

•

Notes

A progress message – will be displayed - this should not take very long♦

•

Actions

none♦

•

Notes

Here we can see the DATA diskgroup has been created and is mounted on 2/2 instances. Wenow need to create the FLASH diskgroup

♦

•

Actions

Click Create New♦

•

Notes

Repeat the above steps, this time allocating the disks for the FLASH disk group♦

•


Notes

Here we can see the DATA and FLASH diskgroups have been created and are mounted on2/2 instances. This completed the ASM configuration.

♦

•

Actions

Click Finish♦

•


Notes

A confirmation dialog box♦

•

Actions

Click No♦ run ./crs_stat -t from the Oracle Clusterware bin directory♦

•

9.2. Completed creation of ASM Instance and add of the +DATA and+FLASH diskgroups

Congratulations you have installed ASM, Network listeners and created 2 ASM diskgroups.

10. Oracle RAC Database Home Software Install

10.1. CVU check - Pre Database Install

Before installing the RAC Database software you should use the CVU utility to confirm the status of thecluster.

10.2. Oracle RAC Database Home Software Install

Start the installer by running "runInstaller" from the staged installation media.

export ORACLE_HOME=/opt/oracle/product/RAC11./runInstaller &

9.2. Completed creation of ASM Instance and add of the +DATA and +FLASH diskgroups 61

In this section we establish the Oracle software for the RAC Database home.

Notes

We use the same installer for Database homes as Clusterware homes♦

•

Actions

Select the Oracle Database 11g radio button♦ Click Next♦

•

10.2. Oracle RAC Database Home Software Install 62

Notes

Here we can choose the type of database software to be installed♦

•

Actions

Select the Enterprise Edition radio button♦ Click Next♦

•


Notes

Here we specify the location of various components. If you change the path you shouldensure that you do not use the exact same path as the Oracle

♦

•

Clusterware or ASM homes

Actions

Confirm entries are OK♦ Click Next♦

•

Notes

The installer will detect the presence of Oracle Clusterware and uses this to populate a dialogbox. To build a cluster which includes all nodes you must ensure that there are check-boxesnext to the node names

♦

•

Actions


•


Notes

The installer will then complete some Product-Specific Prerequisite checks. These should allpass OK - as you have already run the CVU check

♦

•

Actions

Click Next♦

•


Notes

We are going to install a Software only home and then subsequently configure the software♦

•

Actions

Select the Install Software Only radio button♦ Click Next♦

•

Notes

Assign the Unix groups for the privileged db users (default is oinstall)♦

•

Action

♦ Click Next♦

•


Notes

Here we can see a summary of the install.♦

•

Actions

Click Install♦

•

Notes•


The installer will copy the software to all nodes in the cluster♦ The installer will pause and display instructions for running scripts, such as root.sh♦ Be sure to follow directions carefully, running scripts one at a time, on each node♦

Action

open a shell and follow instructions closely♦ then return to the installer and Click OK♦

•

Notes

After the software install completes you will see this End of Installation dialog♦

•

Actions

Click Exit♦

•


Notes

Are you sure ?♦

•

Actions

Click Yes♦

•

10.2.1. Oracle RAC Database software home has been installed

11. Oracle RAC Software Home Patching

Once RAC software has been installed, the next step in the process is to apply the 11.1.0.7 patchset to theRAC ORACLE_HOME.

11.1. Using the installer to patch the Oracle RAC Software Home

Enter the following commands to start Oracle Universal Installer, where patchset_directory is the directorywhere you unpacked the patch set software:


10.2.1. Oracle RAC Database software home has been installed 69

Notes

the Universal Installer screen appears♦

•

Actions

On the Welcome screen, click Next.♦

•

Notes•

11.1. Using the installer to patch the Oracle RAC Software Home 70

Specify the name and the location of the Database home♦ Actions

Check that the name and location are correct, Click Next♦

•

Notes

here you can specify your metalik credentials for this install.♦ If you leave both fields blank you can opt out of notifications (see next screen).♦

•

Actions

Enter your email address♦ Enter your Metalink password♦ Click Next♦

•


Notes

If you leave both fields blank, you will opt out of notifications and get the above screen.♦

•

Actions

Click Yes♦

•

Notes•



♦

Actions

Click Next♦

•

Notes


•

Actions

Click Next♦

•


Notes


•

Actions

Click Install♦

•

Notes•


The installer copies the patch to all the nodes in the cluster♦ Actions


•

Notes

The installer pauses, root.sh need to be run as root on both nodes of the cluster♦

•

Actions

open a shell window on each node and run root.sh (one after the other), output is shownbelow

♦

then return to the installer and Click OK♦

•

# /opt/oracle/product/RAC11/root.shRunning Oracle 11g root.sh script...

The following environment variables are set as: ORACLE_OWNER= oracle11 ORACLE_HOME= /opt/oracle/product/RAC11




Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root.sh script.


Now product-specific root actions will be performed.Finished product-specific root actions.

Notes

Exit the installer♦

•

Actions

Click Yes♦

•

12. Oracle RAC Database Creation

12.1. use dbca to create the RAC database

Notes

You must launch dbca from the RAC database home♦

•

Action

Ensure the ORACLE_HOME environment variable is set to the new RAC Home♦ change to the RAC Home bin directory♦ run ./dbca♦

•

12. Oracle RAC Database Creation 76

Notes

dbca detects the Oracle Clusterware layer is running and offers to create either cluster orsingle instance database

♦

•

Actions

Select the Oracle Real Application Clusters database radio button♦ Click Next♦

•

12.1. use dbca to create the RAC database 77

* Notes

We use dbca to create RAC databases♦ •

Actions

Select the Create a Database radio button♦ Click Next♦

•

Notes

You need to make sure you create RAC database instances on all the cluster nodes♦

•

Actions


•


Notes

Here you get to select the type of database♦

•

Actions

Select the General Purpose... radio button♦ Click Next♦

•


Notes

Now you specify the preface for the SIDs for the RAC database instances – recommendationis to keep it simple

♦

•

Actions

Enter APX in the Global Database name, The SID should autofill♦ Click Next♦

•


Notes

Here you get to configure Oracle Enterprise Manager. If you have no Grid Control server thenthe best method is to use dbcontrol as detailed here.

♦

•

Actions

Check the Configure Enterprise manager checkbox♦ Select the Configure Database Control for local management radio button♦ Click Next♦

•


Notes

Here we specify the password for the Oracle Database users. In this example we are settingthe same password for all users. You should set a password scheme that meets yourrequirements.

♦

•

Actions

Select the Use the Same Administrative Password… radio button♦ Enter the password♦ Enter the same password for the Confirm password♦ Click Next♦

•


Notes

Here you specify where you would like your database datafiles stored. We are going to usethe ASM diskgroups we created earlier.

♦

•

Actions

Select the Automatic Storage Management (ASM) radio button♦ Click Next♦

•


Notes

dbca displays the diskgroups we previously created. We are going to use the DATAdiskgroup

♦

•

Actions

Check the box on the DATA diskgroup line♦ Click Next♦

•


Notes

Here we can adjust where various files are placed♦

•

Actions

Select the Use Oracle-managed Files radio button♦ Ensure +DATA is in the Database Area field♦ Click Next♦

•


Notes

Here we specify recovery configuration information. We are going to use a flash recoveryarea.

♦

•

Actions

Check the Specify Flash Recovery Area checkbox♦ In the Flash recovery area enter +FLASH♦ Set the flash recovery size – remember the size of the +FASH diskgroup♦ Check the Enable Archiving checkbox♦ Click Next♦

•


Notes

Here you can choose to add the Oracle Sample schemas♦

•

Actions

Check the Sample schemas checkbox♦ Click Next♦

•


Notes

Here you can modify the default initialisation parameters.♦

•

Actions

Click Next♦

•


Notes

11g introduces enhanced security settings♦

•

Actions

Select the Keep the enhanced… radio button♦ Click Next♦

•

Notes

11g will set up some automatic maintenance tasks♦

•

Actions

Check the Enable automatic maintenance tasks checkbox♦ Click Next♦

•


Notes

Here you can review the placement of various database files♦

•

Actions

Click Next♦

•


Notes

You can decide to generate database creation scripts♦

•

Actions

Check the Create Database checkbox♦ Click Next♦

•

Notes

A summary screen will be displayed♦

•

Actions

Click Next♦

•

Notes

The database is created♦

•

Actions

none♦

•


Notes

After the database is created the summary screen is displayed – note the URL for theDatabase Control

♦

•

Actions

Click Exit♦

•

Notes

The Database instances will be started on all nodes♦

•

Actions

none♦

•

Notes

You can see that cluster has started the database instances on each n node♦

•

Action

check the status of the db instances♦ run ./crs_stat -t♦

•


RACGuides°²×°Ö¸ÄÏ_Rac11gR1OnHPUX

Documents

Transcript of RACGuides°²×°Ö¸ÄÏ_Rac11gR1OnHPUX