Oracle 11g R1 Installation Procedure (NON-RAC With NON-ASM) on RHEL5
Installation SUN 10gR2 RAC ASM
Transcript of Installation SUN 10gR2 RAC ASM
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Oracle Deutschland GmbH
Oracle RAC with ASM
Authors: Jürgen DömelOracle Deutschland GmbH
Michael Ahrens
SUN Microsystems GmbH
Created 05/05/2006 14:08:00
Last change: 29/11/2006 14:11:35 A11/P11
Version: 2.0
Page 1 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Table of ContentsTable of Contents......................................................................................................................................................2Document Info...........................................................................................................................................................4
Content..............................................................................................................................................................4Cluster / host information..................................................................................................................................4RAC...................................................................................................................................................................4
Hardware Installation................................................................................................................................................5Architecture.......................................................................................................................................................5Drawing.............................................................................................................................................................5
Solaris Installation.....................................................................................................................................................6Solaris 10 u1......................................................................................................................................................6Setup slvm.........................................................................................................................................................6EIS-CD..............................................................................................................................................................7Setup IPMP groups on all nodes for all public network interfaces...................................................................8Modify NVRAM parameter:.............................................................................................................................8Setup Trunking for the Clusterinterconnect......................................................................................................9Last Systemanalyse...........................................................................................................................................9
Prepare Oracle Installation......................................................................................................................................10Set kernel parameters......................................................................................................................................10Create Users and Groups.................................................................................................................................10Share staging area............................................................................................................................................10Mount staging area..........................................................................................................................................10Modify /etc/hosts.............................................................................................................................................10Check SSH daemon.........................................................................................................................................11Setup SSH for oracle.......................................................................................................................................11Setup userenv for oracle..................................................................................................................................11Perform cluster check......................................................................................................................................11Workaround Bug.............................................................................................................................................12Prepare storage................................................................................................................................................12Verify device order..........................................................................................................................................12Trobleshooting device size..............................................................................................................................13Label devices...................................................................................................................................................13
Install Clusterware 10.2.0.1.....................................................................................................................................14Run installer....................................................................................................................................................14Trobleshooting Assistants...............................................................................................................................23Check Cluster Status.......................................................................................................................................25Install crsstat.sh...............................................................................................................................................26
Install ASM Home 10.2.0.1....................................................................................................................................27Run installer....................................................................................................................................................27
Create ASM Instance..............................................................................................................................................35Run DBCA......................................................................................................................................................35
Install Database Home 10.2.0.1..............................................................................................................................41Run installer ...................................................................................................................................................41
Install Management Agent......................................................................................................................................47Run installer....................................................................................................................................................47
Create Database ZDA..............................................................................................................................................55Run DBCA......................................................................................................................................................55Create login script for user oracle...................................................................................................................70
Install Patchset 10.2.0.2 on CRS Home..................................................................................................................71Stop all oracle processes.................................................................................................................................71Install binaries.................................................................................................................................................71Restart Custerware..........................................................................................................................................75
Install Patchset 10.2.0.2 on ASM Home.................................................................................................................76Stop Nodeapps.................................................................................................................................................76Install binaries.................................................................................................................................................76
Install Patchset 10.2.0.2 on DB Home....................................................................................................................81Verify service state..........................................................................................................................................81Install binaries.................................................................................................................................................81
Install Interim Patch 5117016 on DB Home...........................................................................................................87Install with opatch...........................................................................................................................................87Start services required to perform database upgrade......................................................................................89
Page 2 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Upgrade database ZDA...................................................................................................................................89Restart cluster..................................................................................................................................................90How to Setup IPMP as Cluster Interconnect Doc ID: Note:368464.1..........................................................90
Add new node to cluster..........................................................................................................................................91Create links to devices.....................................................................................................................................91Change permissions.........................................................................................................................................91Modify /etc/hosts.............................................................................................................................................92Check SSH daemon.........................................................................................................................................92Enable user equivalence..................................................................................................................................92Perform cluster check......................................................................................................................................93Workaround OUI Bug.....................................................................................................................................93Install clusterware on new node......................................................................................................................93Install ASM home on new node......................................................................................................................97Install DB home on new node.......................................................................................................................101Create listener in DB Home..........................................................................................................................105Create ASM instance on new node...............................................................................................................110Create database instance on new node..........................................................................................................113End of Installation.........................................................................................................................................119
Remove Node from Cluster...................................................................................................................................120Remove instance...........................................................................................................................................120Remove Listener...........................................................................................................................................124Clean-up the ASM.........................................................................................................................................128Removing node from the database................................................................................................................128Removing node from the clusterware:..........................................................................................................128
Cluster Performance Tuning.................................................................................................................................130Apply Brownout Patches...............................................................................................................................130Set the CSS misscount to match the service level.........................................................................................130Run CSS in Real Time..................................................................................................................................130Set the check_interval for VIP addresses......................................................................................................130Remove the dependency between the VIP and ASM...................................................................................131Reduce the ping timeout................................................................................................................................131
Troubleshooting VIP Failover...............................................................................................................................132Adjusting Listener Configuration..................................................................................................................132Upgrade Oracle Home of Swingbench .........................................................................................................132Adjust SQLNet settings.................................................................................................................................132Set the ping timeout back to default..............................................................................................................132
Addendum.............................................................................................................................................................133Metalink Note 284752.1................................................................................................................................133Metalink Note 294430.1................................................................................................................................134Metalink Note 294430.1................................................................................................................................140
Page 3 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Document Info
Content
This is an installation summary of the installation and patch process of the ZDA environment. It is based on the official procuct documentation provided by oracle cannot replace the those documents.
Cluster / host information
Hostnames: orac-rac-1, ora-rac-2, ora-rac,4
OS: Solaris 10
RAC
Oracle Cluster Name: crs
Oracle Versions 10.2.0.1
Page 4 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Hardware Installation
Architecture
The cluster is based on two SF440 with 4 CPU's, 16GB RAM and 4 internal Disks. The internaldisks are mirrored with SLVM. Two disks are used for the OS and the other two are used for Oracle. Oracle will not be installed on shared disks because of the rolling upgrade capability. The shared disks are luns from a SAN and they are mirrored with Oracle ASM. The clusterinterconnect is realized with two trunked gigabitethernet connections.
Drawing
Page 5 of 141
2 x Interconnect
Storage A
Oracle RAC Cluster Node: ora-rac-1
Oracle RAC ClusterNode: ora-rac-2
Storage B
IPMP Groupipmp0
IPMP Groupipmp0
MpxIO MpxIO
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Solaris Installation
Solaris 10 u1
Solaris 10 u1 SUNWCXall is installed.
• For SVM reserve slice 7 with 32MB (EIS recommendation) for metaDB/replicas.
• Ensure that the SAN Foundation Suite packages are installed and patched. (SAN is necessary for fibre server/storage)
change the home of root
#mkdir /root
#vi /etc/passwd
root:x:0:0:Super-User:/root:/sbin/sh
logout,login
Setup slvm
#cat /etc/lvm/md.tab
mddb1 -c 3 /dev/dsk/c3t0d0s7
d10 -m d11
d11 1 1 /dev/dsk/c3t0d0s0
d12 1 1 /dev/dsk/c3t1d0s0
d20 -m d21
d21 1 1 /dev/dsk/c3t0d0s1
d22 1 1 /dev/dsk/c3t1d0s1
d30 -m d31
d31 1 1 /dev/dsk/c3t0d0s6
d32 1 1 /dev/dsk/c3t1d0s6
Page 6 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
EIS-CD
Setup EIS environment
cd /cdrom/eis-dvd/sun/install
./setup-standard.sh
Install actual Solaris 10 patches
cd /cdrom/eis-dvd/patch/10
unpack-patches
cd /tmp/10
./install-patches
Analyse basic systemstatus
explorer
cd /opt/SUNWexplo/output/explorer*
/cdrom/eis-dvd/sun/tools/explominer/explominer
see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and install all neccessary patches.
To enable MpxIO on Solaris 10 start
stmsboot -e before OracleCluster will be installed !!
For safety reasons clean the devices with:
devfsadm -C on all nodes.
Page 7 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Setup IPMP groups on all nodes for all public network interfaces.
At least you need one test IP for each network adapter and one production IP for each IPMP group. e.g: IPMP group active-active with interfaces ce0 and ce4 with one physical IP.
ora-rac-1
Entry of /etc/hostname.ce0 :
ora-rac-1-1 netmask + broadcast + group ipmp0 up \
addif ora-rac-1 netmask + broadcast + deprecated -failover up
Entry of /etc/hostname.ce4 :
ora-rac-1-2 netmask + broadcast + group ipmp0 deprecated -failover up
ora-rac-2
Entry of /etc/hostname.ce0 :
ora-rac-2-1 netmask + broadcast + group ipmp0 up \addif ora-rac-3 netmask + broadcast + deprecated -failover up
Entry of /etc/hostname.ce4 :
ora-rac-2-2 netmask + broadcast + group ipmp0 deprecated -failover up
If the defaultrouter is NOT 100% available please read infodocs
70060 and 70062.
Notes:
• Do not use test IP for normal applications
• Test IP for all adapters in the same IPMP group must belong to a single IP subnet
• You need an addtional IP for each logical host.
• In the /etc/default/mpathd file, the value of
TRACK_INTERFACES_ONLY_WITH_GROUPS must be yes. (default)
Modify NVRAM parameter:
Set boot-device & diag-device to both sides of the mirror.
Suggested naming convention:
• Primary: rootdisk
• Secondary: rootmirror
Page 8 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Setup Trunking for the Clusterinterconnect
Install Sun Trunking 1.3 for the ce interfaces.
pkgadd -d ./SUNWtrku
pkgadd -d ./SUNWtrkm
#vi /etc/opt/SUNWconn/bin/nettr.sh
Add at the end:
nttr -setup 1 device=ce members=1,3 policy=1
#reboot
Check the configuration:
#/etc/opt/SUNWconn/bin/nettr -conf
Name Head Policy DEV Type Original-Mac-Addr Speed Duplex Link
ce1 ce1 1 pci 8:0:20:8f:be:24 1000 Full Up
ce3 ce1 pci 8:0:20:8f:be:24 1000 Full Up
Last Systemanalyse
explorer
cd /opt/SUNWexplo/output/explorer*
/cdrom/eis-dvd/sun/tools/explominer/explominer
see the outputfile ExploMiner_EIS_Patchlist_nobody of the explominer and install all neccessary patches.
Page 9 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Prepare Oracle Installation
Set kernel parameters
* Oracle settings *
set noexec_user_stack=1
set rlim_fd_cur=8192
* end of Oracle settings *
Create Users and Groups
>> ora-rac-1, ora-rac-2 as root
/usr/sbin/groupadd -g 61000 oinstall/usr/sbin/groupadd -g 61001 dba/usr/sbin/useradd -u 61000 -g oinstall -G dba oracle -s /usr/bin/bash -d /export/home/oracle
mkdir -p /export/home/oraclechown -R oracle:oinstall /export/home/oracle
passwd oracle> oracle> oracle
Share staging area
All Oracle software is located on Host ora-poc2:/oracle/install. The directory is shared as /install/oracle and can be mounted on every node if needed.
Share stage directory:
>> ora-poc2 as root
chown -R oracle:oinstall /oracleshare -rw /oracle/install
Mount staging area
In order to install oracle software, the staging area must be mounted on the instaation node of the clus-ter.
>> ora-rac-1 as root
mkdir /installmount ora-poc2:/oracle/install /install
Page 10 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Modify /etc/hosts
127.0.0.1 localhost192.168.10.212 ora-rac-1 loghost ora-rac-1.192.168.10.234 ora-rac-1-1192.168.10.233 ora-rac-1-210.0.0.1 ora-rac-1-priv10.0.0.2 ora-rac-1-priv-110.0.0.3 ora-rac-1-priv-2192.168.10.225 ora-rac-1-vip
192.168.10.214 ora-rac-2 loghost ora-rac-2.192.168.10.232 ora-rac-2-1192.168.10.231 ora-rac-2-210.0.0.4 ora-rac-2-priv10.0.0.5 ora-rac-2-priv-110.0.0.6 ora-rac-2-priv-2192.168.10.229 ora-rac-2-vip
192.168.10.223 ora-rac-3-vip
192.168.10.239 ora-rac-4-vip
Check SSH daemon
ps -ef | grep sshd
Setup SSH for oracle
Note: This steps are different for any node in the cluster. Do complete the steps on the indicated nodes.
>> ora-rac-1, ora-rac-2 :
mkdir ~/.ssh
chmod 700 ~/.ssh
/usr/bin/ssh-keygen -t rsa/usr/bin/ssh-keygen -t dsa
>> ora-rac-1:
cat id_dsa.pub > authorized_keyscat id_rsa.pub >> authorized_keys
scp authorized_keys ora-rac-2:/export/home/oracle/.ssh/authorized_keys
>> ora-rac-2:
cat id_dsa.pub >> authorized_keyscat id_rsa.pub >> authorized_keys
>> ora-rac-1:
scp ora-rac-2:/export/home/oracle/.ssh/authorized_keys authorized_keys
>> ora-rac-1, ora-rac-2:
ssh ora-rac-1 datessh ora-rac-2 date
exec /usr/bin/ssh-agent $SHELL/usr/bin/ssh-add
Page 11 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Setup userenv for oracle
>> ora-rac-1, ora-rac-2
Add line to ~/.profile
# Set Environment for oracle
. ~/set_oracle_env.sh
Perform cluster check
>> ora-rac-1
/install/10gR2/clusterware/runcluvfy.sh stage -pre crsinst -n ora-rac-1,ora-rac-2 -verbose
Workaround Bug
The OUI may fail if it does not find ssh and scp binaries in /usr/local/bin.
>> ora-rac-1, ora-rac-2 as root
mkdir -p /usr/local/binln -s /usr/bin/ssh /usr/local/bin/sshln -s /usr/bin/scp /usr/local/bin/scp
Prepare storage
Create symbolic links for all disk devices needed for oracle
>> ora-rac-1, ora-rac-2 as root
mkdir -p /dev/oracle
ln -s /dev/rdsk/c7t600015D00022B3000000000000005012d0s1 /dev/oracle/ocr1
ln -s /dev/rdsk/c7t600015D00022B3000000000000005018d0s1 /dev/oracle/ocr2
ln -s /dev/rdsk/c7t600015D00022B300000000000000501Ed0s1 /dev/oracle/vote1
ln -s /dev/rdsk/c7t600015D00022B3000000000000005024d0s1 /dev/oracle/vote2
ln -s /dev/rdsk/c7t600015D00022B300000000000000502Ad0s1 /dev/oracle/vote3
ln -s /dev/rdsk/c7t600015D00022B3000000000000005042d0s1 /dev/oracle/asm01
ln -s /dev/rdsk/c7t600015D00022B3000000000000005048d0s1 /dev/oracle/asm02
ln -s /dev/rdsk/c7t600015D00022B300000000000000504Ed0s1 /dev/oracle/asm03
ln -s /dev/rdsk/c7t600015D00022B3000000000000005054d0s1 /dev/oracle/asm04
ln -s /dev/rdsk/c7t600015D00022B300000000000000505Ad0s1 /dev/oracle/asm05
ln -s /dev/rdsk/c7t600015D00022B3000000000000005060d0s1 /dev/oracle/asm06
ln -s /dev/rdsk/c7t600015D00022B3000000000000005066d0s1 /dev/oracle/asm07
ln -s /dev/rdsk/c7t600015D00022B300000000000000506Cd0s1 /dev/oracle/asm08
ln -s /dev/rdsk/c7t600015D00022B3000000000000005072d0s1 /dev/oracle/asm09
ln -s /dev/rdsk/c7t600015D00022B3000000000000005078d0s1 /dev/oracle/asm10
Change permissions for devices
chown oracle:dba /dev/oracle/*
Verify device order
Test if devices appear in same order on every node of the cluster.
Page 12 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
>> ora-rac-1
Create some textfiles for testing...
echo "This is orc1" > /tmp/ocr1echo "This is orc2" > /tmp/ocr2echo "This is vote1" > /tmp/vote1echo "This is vote2" > /tmp/vote2echo "This is vote3" > /tmp/vote3
Copy text to device
dd if=/tmp/ocr1 of=/dev/oracle/ocr1 dd if=/tmp/ocr2 of=/dev/oracle/ocr2 dd if=/tmp/vote1 of=/dev/oracle/vote1dd if=/tmp/vote2 of=/dev/oracle/vote2dd if=/tmp/vote3 of=/dev/oracle/vote3
>> ora-rac-2
Copy device contents to file on ervery node
dd if=/dev/oracle/ocr1 of=/tmp/ocr1 dd if=/dev/oracle/ocr2 of=/tmp/ocr2 dd if=/dev/oracle/vote1 of=/tmp/vote1dd if=/dev/oracle/vote2 of=/tmp/vote2dd if=/dev/oracle/vote3 of=/tmp/vote3
Check if content is the same as on the other node
cat /tmp/ocr1 cat /tmp/ocr2 cat /tmp/vote1cat /tmp/vote2cat /tmp/vote3
Trobleshooting device size
Count blocks
dd if=/dev/oracle/ocr1 of=/tmp/ocr1 bs=1k count=10
dd if=/dev/oracle/ocr2 of=/tmp/ocr2 bs=1k count=10
dd if=/dev/oracle/vote1 of=/tmp/vote1 bs=1k count=10
dd if=/dev/oracle/vote2 of=/tmp/vote2 bs=1k count=10
dd if=/dev/oracle/vote3 of=/tmp/vote3 bs=1k count=10
Label devices
Prepare devices for usage.
Template Device:
/dev/rdsk/c7t600015D00022B3000000000000005030d0s1
Label devices using template device:
prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s - /dev/rdsk/c7t600015D00022B3000000000000005012d0s2
prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s - /dev/rdsk/c7t600015D00022B3000000000000005018d0s2
prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s - /dev/rdsk/c7t600015D00022B300000000000000501Ed0s2
prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s - /dev/rdsk/c7t600015D00022B3000000000000005024d0s2
prtvtoc /dev/rdsk/c7t600015D00022B3000000000000005030d0s2 | fmthard -s - /dev/rdsk/c7t600015D00022B300000000000000502Ad0s2
Page 13 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Install Clusterware 10.2.0.1
Run installer
>> ora-rac-1
/install/10gR2/clusterware/Disk1/runInstaller
Page 14 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 15 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 16 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 17 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 18 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 19 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 20 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 21 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 22 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Trobleshooting Assistants
bash-3.00# ./root.sh
WARNING: directory '/oracle/product' is not owned by root
WARNING: directory '/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/oracle/product' is not owned by root
WARNING: directory '/oracle' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: ora-rac-1 ora-rac-1-priv ora-rac-1
node 2: ora-rac-2 ora-rac-2-priv ora-rac-2
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
root.shExpecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
ora-rac-1
ora-rac-2
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
The given interface(s), "ce0, ce2" is not public. Public interfaces should be used to configure virtual IPs.
Page 23 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
/oracle/product/crs/bin/racgons add_config ora-rac-1:6200 ora-rac-2:6200
/oracle/product/crs/bin/oifcfg setif -global ce0/192.168.10.0:public ce1/10.0.0.0:cluster_interconnect ce2/192.168.10.0:public ce3/10.0.0.0:cluster_interconnect
/oracle/product/crs/bin/cluvfy stage -post crsinst -n ora-rac-1,ora-rac-2
Page 24 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Check Cluster Status
>> ora-rac-1
ORACLE_HOME=/oracle/product/crs/bin
export ORACLE_HOME
PATH=$ORACLE_HOME/bin:$PATH
export PATH
$ crs_stat -t
Name Type Target State Host ----------------------------------------------------------- ora....c-1.gsd application OFFLINE OFFLINE ora....c-1.ons application ONLINE ONLINE ora-rac-1ora....c-1.vip application ONLINE ONLINE ora-rac-1 ora....c-2.gsd application OFFLINE OFFLINE ora....c-2.ons application ONLINE ONLINE ora-rac-2ora....c-2.vip application ONLINE ONLINE ora-rac-2
Page 25 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Install crsstat.sh
Because of the unreadable output of „crs_stat -t“ setup a wrapper for crs_stat.
mkdir /oracle/tools
vi /oracle/tools/crsstat.sh
#!/usr/bin/ksh
#
# Sample 10g CRS resource status query script
#
# Description:
# - Returns formatted version of crs_stat -t, in tabular
# format, with the complete rsc names and filtering keywords
# - The argument, $RSC_KEY, is optional and if passed to the script, will
# limit the output to HA resources whose names match $RSC_KEY.
# Requirements:
# - $CRS_HOME should be set in your environment
RSC_KEY=$1
QSTAT=-u
AWK=/usr/xpg4/bin/awk
# Table header:echo ""
$AWK \
'BEGIN {printf "%-45s %-10s %-18s\n", "HA Resource", "Target", "State";
printf "%-45s %-10s %-18s\n", "-----------", "------", "-----";}'
# Table body:
$CRS_HOME/bin/crs_stat $QSTAT | $AWK \
'BEGIN { FS="="; state = 0; }
$1~/NAME/ && $2~/'$RSC_KEY'/ {appname = $2; state=1};
state == 0 {next;}
$1~/TARGET/ && state == 1 {apptarget = $2; state=2;}
$1~/STATE/ && state == 2 {appstate = $2; state=3;}
state == 3 {printf "%-45s %-10s %-18s\n", appname, apptarget, appstate; state=0;}'
Output of crsstat.sh
HA Resource Target State
----------- ------ -----
ora.ora-rac-1.gsd ONLINE ONLINE on ora-rac-1
ora.ora-rac-1.ons ONLINE ONLINE on ora-rac-1
ora.ora-rac-1.vip ONLINE ONLINE on ora-rac-1
ora.ora-rac-2.gsd ONLINE ONLINE on ora-rac-2
ora.ora-rac-2.ons ONLINE ONLINE on ora-rac-2
ora.ora-rac-3.vip ONLINE ONLINE on ora-rac-2
Page 26 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Install ASM Home 10.2.0.1
Run installer
>> ora-rac-1
/install/10gR2/database/Disk1/runInstaller
Page 27 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Ora_asmhome
Page 28 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 29 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 30 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 31 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 32 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 33 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 34 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Create ASM Instance
Run DBCA
/oracle/product/asm/bin/dbca
Page 35 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 36 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 37 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 38 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 39 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 40 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Install Database Home 10.2.0.1
Run installer
>> ora-rac-1
/install/10gR2/database/Disk1/runInstaller
Page 41 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 42 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 43 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 44 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 45 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 46 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Install Management Agent
Run installer
ora-rac-1
/install/10gR2/database/Disk1/runInstaller
Page 47 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 48 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 49 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 50 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 51 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Password: oracle1
Page 52 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 53 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 54 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Create Database ZDA
Run DBCA
>> ora-rac-1
/oracle/product/db/bin/dbca
Page 55 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 56 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 57 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Password: oracle
Page 58 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 59 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 60 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 61 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 62 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 63 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 64 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 65 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 66 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 67 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 68 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 69 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Create login script for user oracle
The script is located at ~/ set_oracle_env.sh
---------------This script starts here ------------
#!/usr/bin/bashclear
# Menu section. If user does not choose any oracle home, no actions takes place.
echo "Choose your Oracle Home and hit RETURN"echo "Enter 1 to use Oracle Home DB"echo "Enter 2 to use Oracle Home ASM"echo "Enter 3 to use Oracle Home Agent"echo "Enter 4 to use Oracle Home CRS"echo "Hit any other key to skip Oracle Home selection"
read answer
if [[ $answer = 1 ]] then ORACLE_HOME=/oracle/product/db ; ORACLE_SID=ZDA1elif [[ $answer = 2 ]] then ORACLE_HOME=/oracle/product/asm ; ORACLE_SID="+ASM1" ;elif [[ $answer = 3 ]] then ORACLE_HOME=/oracle/product/agent/agent10g ;elif [[ $answer = 4 ]] then ORACLE_HOME=/oracle/product/crs ;else returnfi
# If the user choose an oracle home, the following actions are performed:
ORACLE_BASE=/oracleexport ORACLE_BASE
export PATH=$ORACLE_HOME/bin:$PATHexport ORACLE_SIDexport ORACLE_HOME
EDITOR=viexport EDITOR
if [ -t 0 ]; then stty intr ^Cfi
# Alias section. Some usefull shortcuts ...
alias crsi='/oracle/product/crs/bin/crs_stat -t'alias xxx=". ~/set_oracle_env.sh"alias oh='cd $ORACLE_HOME'
echo "Tip: Enter xxx to change your current setting"
---------------The script ends here ------------
Page 70 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Install Patchset 10.2.0.2 on CRS Home
Stop all oracle processes
As oracle:
srvctl stop database -d ZDA
srvctl stop asm -n ora-rac-1
srvctl stop asm -n ora-rac-2
srvctl stop nodeapps -n ora-rac-1
srvctl stop nodeapps -n ora-rac-2
As root:
/etc/init.d/init.crs stop
Install binaries
./runInstaller
Page 71 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 72 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 73 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 74 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Restart Custerware
>> ora-rac-1 root:
/etc/init.d/init.crs start
Page 75 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Install Patchset 10.2.0.2 on ASM Home
Stop Nodeapps
>> ora-rac-1:
srvctl stop nodeapps -n ora-rac-1srvctl stop nodeapps -n ora-rac-2
Install binaries
./runInstaller
Page 76 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 77 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 78 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 79 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 80 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Install Patchset 10.2.0.2 on DB Home
Verify service state
All Services must be down.
crs_stat -t
Name Type Target State Host------------------------------------------------------
ora....A1.inst application OFFLINE OFFLINEora....A2.inst application OFFLINE OFFLINEora.ZDA.db application OFFLINE OFFLINEora....SM1.asm application OFFLINE OFFLINEora....-1.lsnr application OFFLINE OFFLINEora....c-1.gsd application OFFLINE OFFLINEora....c-1.ons application OFFLINE OFFLINEora....c-1.vip application OFFLINE OFFLINEora....SM2.asm application OFFLINE OFFLINEora....-2.lsnr application OFFLINE OFFLINEora....c-2.gsd application OFFLINE OFFLINEora....c-2.ons application OFFLINE OFFLINEora....c-2.vip application OFFLINE OFFLINE
Install binaries
./runInstaller
Page 81 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 82 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 83 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 84 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 85 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 86 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Install Interim Patch 5117016 on DB Home
Install with opatch
>> ora-rac-3 as user oracle
#cd /install/10gR2/5117016
#opatch apply -local
Page 87 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
>> ora-rac-4 as user oracle
#cd /install/10gR2/5117016
#opatch apply -local
Page 88 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Change /etc/init.d/init.crsd:
>> ora-rac-3 and ora-rac-4 as user root
from: exec $ORA_CRS_HOME/bin/crsd $BOOTFLAG $*
to : exec newtask $ORA_CRS_HOME/bin/crsd $BOOTFLAG $*
Setup system parameters on both Nodes:
>> ora-rac-3 and ora-rac-4 as user root
#projadd -p 61002 -c "Oracle RAC" -U oracle \-K 'project.max-sem-ids=(priv,1024,deny)' \-K 'process.max-sem-nsems=(priv,2048,deny)' \-K 'project.max-shm-ids=(priv,256,deny)' \-K 'project.max-shm-memory=(priv,18446744073709551615,deny)' \MAH
cat >> /etc/user_attroracle::::project=RAC<CTRL>-d
Check all by reboot
>> ora-rac-3 as user root
#scshutdown -g0 -y
>> ora-rac-3 as user root
ok boot
>> ora-rac-4 as user root
ok boot
Start services required to perform database upgrade
Restart the listener:
srvctl start listener -n ora-rac-1srvctl start listener -n ora-rac-2
Start asm:
srvctl start asm -n ora-rac-1srvctl start asm -n ora-rac-2
Upgrade database ZDA
sqlplus / as sysdbaSTARTUP NOMOUNTALTER SYSTEM SET CLUSTER_DATABASE=FALSE SCOPE=spfile;shutdown
STARTUP UPGRADESPOOL patch_10202.log@?/rdbms/admin/catupgrd.sqlSPOOL OFF
Check log for errors:
grep ORA- patch_10202.log
ORA-00600: internal error code, arguments: [qkacon:NFswrwo], [1], [], [], [],
This is probably the bug 5089217, fixed in version 11.1 and should be fixed in version 10.2.0.3.
Workaround bug 5089217:
Page 89 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Set "_optimizer_connect_by_cost_based" = false
shutdown
startup@?/rdbms/admin/utlrp.sql
select distinct status from dba_objects;=> valid
ALTER SYSTEM SET CLUSTER_DATABASE=TRUE SCOPE=spfile;ALTER SYSTEM SET "_optimizer_connect_by_cost_based" = false SCOPE=spfile;
shutdown;startup;
Restart cluster
srvctl stop database -d ZDAsrvctl stop asm -n ora-rac-1srvctl stop asm -n ora-rac-2srvctl stop nodeapps -n ora-rac-1srvctl stop nodeapps -n ora-rac-2
srvctl start nodeapps -n ora-rac-1srvctl start nodeapps -n ora-rac-2srvctl start asm -n ora-rac-1srvctl start asm -n ora-rac-2srvctl start database -d ZDA
crs_stat -t
How to Setup IPMP as Cluster Interconnect Doc ID: Note:368464.1
In order to setup IPMP as cluster interconnect note 283107.1 should be used.
Page 90 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Add new node to cluster
Create links to devices
>> ora-rac-4 as root
mkdir -p /dev/oracle
ln -s /dev/rdsk/c9t600015D00022B3000000000000005012d0s1 /dev/oracle/ocr1
ln -s /dev/rdsk/c9t600015D00022B3000000000000005018d0s1 /dev/oracle/ocr2
ln -s /dev/rdsk/c9t600015D00022B300000000000000501Ed0s1 /dev/oracle/vote1
ln -s /dev/rdsk/c9t600015D00022B3000000000000005024d0s1 /dev/oracle/vote2
ln -s /dev/rdsk/c9t600015D00022B300000000000000502Ad0s1 /dev/oracle/vote3
ln -s /dev/rdsk/c9t600015D00022B3000000000000005042d0s1 /dev/oracle/asm01
ln -s /dev/rdsk/c9t600015D00022B3000000000000005048d0s1 /dev/oracle/asm02
ln -s /dev/rdsk/c9t600015D00022B300000000000000504Ed0s1 /dev/oracle/asm03
ln -s /dev/rdsk/c9t600015D00022B3000000000000005054d0s1 /dev/oracle/asm04
ln -s /dev/rdsk/c9t600015D00022B300000000000000505Ad0s1 /dev/oracle/asm05
ln -s /dev/rdsk/c9t600015D00022B3000000000000005060d0s1 /dev/oracle/asm06
ln -s /dev/rdsk/c9t600015D00022B3000000000000005066d0s1 /dev/oracle/asm07
ln -s /dev/rdsk/c9t600015D00022B300000000000000506Cd0s1 /dev/oracle/asm08
ln -s /dev/rdsk/c9t600015D00022B3000000000000005072d0s1 /dev/oracle/asm09
ln -s /dev/rdsk/c9t600015D00022B3000000000000005078d0s1 /dev/oracle/asm10
Change permissions
>> ora-rac-4 as root
chown oracle:dba /dev/oracle/*
Page 91 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Modify /etc/hosts
Add IP-Adresses.
>> ora-rac-1, ora-rac-2, ora-rac-4 as root
127.0.0.1 localhost192.168.10.212 ora-rac-1 loghost ora-rac-1.192.168.10.234 ora-rac-1-1192.168.10.233 ora-rac-1-210.0.0.1 ora-rac-1-priv10.0.0.2 ora-rac-1-priv-110.0.0.3 ora-rac-1-priv-2192.168.10.225 ora-rac-1-vip
192.168.10.214 ora-rac-2 loghost ora-rac-2.192.168.10.232 ora-rac-2-1192.168.10.231 ora-rac-2-210.0.0.4 ora-rac-2-priv10.0.0.5 ora-rac-2-priv-110.0.0.6 ora-rac-2-priv-2192.168.10.229 ora-rac-2-vip
192.168.10.223 ora-rac-3-vip
192.168.10.240 ora-rac-4 loghost ora-rac-4.192.168.10.241 ora-rac-4-1192.168.10.242 ora-rac-4-210.0.0.7 ora-rac-4-priv10.0.0.8 ora-rac-4-priv-110.0.0.9 ora-rac-4-priv-2192.168.10.239 ora-rac-4-vip
Check SSH daemon
>> ora-rac-4
ps -ef | grep sshd
Enable user equivalence
>> ora-rac-4
mkdir /export/home/oracle/.ssh
chmod 700 ~/.ssh
/usr/bin/ssh-keygen -t rsa/usr/bin/ssh-keygen -t dsa
scp authorized_keys ora-rac-1:/export/home/oracle/.ssh/authorized_keys
cat id_dsa.pub >> authorized_keyscat id_rsa.pub >> authorized_keys
scp authorized_keys ora-rac-1:/export/home/oracle/.ssh/authorized_keys
scp authorized_keys ora-rac-2:/export/home/oracle/.ssh/authorized_keys
Page 92 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Perform cluster check
/install/10gR2/clusterware/Disk1/cluvfy/runcluvfy.sh stage -pre crsinst -n ora-rac-1,ora-rac-2,ora-rac-4 -verbose
Workaround OUI Bug
>> ora-rac-4
The OUI may fail if it does not find ssh and scp binaries in /usr/local/bin.
>> ora-rac-4 as root
mkdir -p /usr/local/bin
ln -s /usr/bin/ssh /usr/local/bin/sshln -s /usr/bin/scp /usr/local/bin/scp
Install clusterware on new node
>> ora-rac-1
/oracle/product/crs/oui/bin/addNode.sh
Page 93 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
<Enter node name>
Page 94 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 95 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 96 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
/oracle/product/crs/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....DA1.srv application ONLINE ONLINE ora-rac-1
ora....atch.cs application ONLINE ONLINE ora-rac-1
ora....A1.inst application ONLINE ONLINE ora-rac-1
ora....A2.inst application ONLINE ONLINE ora-rac-2
ora.ZDA.db application ONLINE ONLINE ora-rac-2
ora....SM1.asm application ONLINE ONLINE ora-rac-1
ora....-1.lsnr application ONLINE ONLINE ora-rac-1
ora....c-1.gsd application ONLINE ONLINE ora-rac-1
ora....c-1.ons application ONLINE ONLINE ora-rac-1
ora....c-1.vip application ONLINE ONLINE ora-rac-1
ora....SM2.asm application ONLINE ONLINE ora-rac-2
ora....-2.lsnr application ONLINE ONLINE ora-rac-2
ora....c-2.gsd application ONLINE ONLINE ora-rac-2
ora....c-2.ons application ONLINE ONLINE ora-rac-2
ora....c-2.vip application ONLINE ONLINE ora-rac-2
ora....c-4.gsd application ONLINE ONLINE ora-rac-4
ora....c-4.ons application ONLINE ONLINE ora-rac-4
ora....c-4.vip application ONLINE ONLINE ora-rac-4
Install ASM home on new node
/oracle/product/asm/oui/bin/addNode.sh
Page 97 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 98 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 99 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 100 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Install DB home on new node
/oracle/product/db/oui/bin/addNode.sh
Page 101 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 102 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 103 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 104 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Create listener in DB Home
/oracle/product/db/bin/netca
Page 105 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Fehler! Hier nur Node ora-rac-4 auswählen!
Page 106 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
LISTENER
LISTENER
Page 107 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
LISTENER
Page 108 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 109 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Create ASM instance on new node
>> ora-rac-1
/oracle/product/asm/bin/dbca
Page 110 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 111 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
password: oracle
Page 112 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Create database instance on new node
>> ora-rac-1
/oracle/product/db/bin/dbca
Page 113 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 114 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
password sys / oracle
Page 115 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 116 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 117 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Verify that group# is not already used (i.e. by standby redolos. In this case, alter group# to an unused value.
Page 118 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
End of Installation
Due to time constraints, the installtion was not completed. To be continued.
Page 119 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Remove Node from ClusterSteps to perform a remove node from cluster following Note 317516.1
Remove instance
>> ora-rac-1
/oracle/product/db/bin/dbca
Page 120 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 121 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 122 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 123 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Remove Listener
>> ora-rac-1
/oracle/product/db/bin/dbca/netca
Page 124 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 125 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 126 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 127 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Clean-up the ASM
>> ora-rac-1
srvctl stop asm -n ora-rac-4srvctl remove asm -n ora-rac-4
>> ora-rac-4
rm -rf $ORACLE_HOME/admin/+ASMrm -f $ORACLE_HOME/dbs/*ASM*
Edit the /etc/oratab file and remove entries beginning with +ASM.
Removing node from the database
>> ora-rac-4
/oracle/product/db/oui/bin/runInstaller -updateNodeList ORACLE_HOME= /oracle/product/crs "CLUSTER_NODES={ora-rac-4}" -local
>> ora-rac-4
/oracle/product/crs/oui/bin/runInstaller #=> Choose de-install products and select the dbhome
Screenshots here!
>> ora-rac-1
/oracle/product/crs/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/oracle/product/crs "CLUSTER_NODES={node1,node2}"
Removing node from the clusterware:
>> ora-rac-1 as root
Page 128 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Get Remote Port number:
cat /oracle/product/crs/opmn/conf/ons.config
/oracle/product/crs/bin/racgons remove_config ora-rac-4:6200
>> ora-rac-4 as root
/oracle/product/crs/install/rootdelete.sh
>> ora-rac-1 as root
/oracle/product/crs/bin/olsnodes -n
/oracle/product/crs/install/rootdeletenode.sh ora-rac-4,3
>> ora-rac-4
/oracle/product/crs/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/oracle/product/crs/ "CLUSTER_NODES={ora-rac-4}" CRS=TRUE -local
/oracle/product/crs/oui/bin/runInstaller #=> Choose Deinstall software and re-move the CRSHOME
>> ora-rac-1
/oracle/product/crs/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/oracle/product/crs "CLUSTER_NODES={ora-rac-1,ora-rac-2}" CRS=TRUE
/oracle/product/crs/bin/olsnodes -n
Page 129 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Cluster Performance TuningOverview and best practices for setting up a "zero brownout" configuration to achieve maximum high availability. More info in note 369101.1.
Note: Patch 5024639 is not yet available and skipped therefore.
Apply Brownout Patches
This patches are installed by using OPatch on DB and ASM Home:
5186679 (This became patch 4960705)
5186674 (This became patch 5016142)
5127434
5127472
5127482
5176453
Set the CSS misscount to match the service level
1. Shut down the clusterware stack on all nodes but one
2. On the remaining node, run as root:
$ORA_CRS_HOME/bin/crsctl set css misscount 5
$ORA_CRS_HOME/bin/crsctl set css disktimeout 200
3. Reboot the nodes for this to take affect.
4. Confirm the new values for the CSS parameters
$ORA_CRS_HOME/bin/crsctl get css misscount
$ORA_CRS_HOME/bin/crsctl get css disktimeout
Run CSS in Real Time
1. On each node as root use, crsctl set css priority 4
2. Restart each node in series, using crsctl stop crs followed by crsctl start crs
Set the check_interval for VIP addresses
>> ora-ra-1 as root
1. Generate the resource profile for the VIP resource
cd $ORA_CRS_HOME/crs/profile
crs_stat -p ora.ora-rac-1.vip > ora. ora-rac-1.vip.capcrs_stat -p ora.ora-rac-2.vip > ora. ora-rac-2.vip.cap
(current value of check_interval = 60 (default))
2. Edit ora.<hostname>.vip.cap and set the parameter CHECK_INTERVAL to 5
Page 130 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
vi ora.ora-rac-1.vip.capvi ora.ora-rac-2.vip.cap
3. Update the resource profile as the root user:
crs_register -u ora.ora-rac-1.vipcrs_register -u ora.ora-rac-2.vip
Note: The check interval should be long enough to sustain the refresh time for the VIP.
Remove the dependency between the VIP and ASM
1. Generate the resource profile for the ASM instance resource
cd $ORA_CRS_HOME/crs/profile
crs_stat -p ora.<hostname>.<ASM instance #>.asm > ora.<hostname>.<ASM instance #>.asm.cap
2. Edit ora.<hostname>.<ASM instance #>.asm.cap and remove 'ora.<hostname>.vip as the value of the REQUIRED_RESOURCES
3. Change the parameter REQUIRED_RESOURCES
vi ora.<hostname>.<ASM instance #>.asm.cap
REQUIRED_RESOURCES=
4. Update the resource profile
crs_register -u ora.<hostname>.asm
Note: There is a hard dependency from the ASM to the VIP. This means that when the virtual IP is relocated, ASM is shut down. It is fully supported to remove this CRS dependency, and is the default from 10.2.0.3 and later releases.
Reduce the ping timeout
1. Open the $CRS_HOME/bin/racgvip script.
(current value of PING_TIMEOUT = 3 (default))
2. Edit to set PING_TIMEOUT to:
PING_TIMEOUT="-w 1 -c 1"
This change shortens the delay between begin of CRS recovery and start of the VIP.
Note: If any patches are applied that change the racgvip script, this variable would need to be changed again.
alter system set disk_asynch_io=true scope=spfile ;alter system set fast_start_parallel_rollback=high scope=spfile ;alter system set "_imr_disk_voting_interval"=0 scope=spfile;alter system set "_first_spare_parameter"=0 scope=spfile ;alter system set "_imr_splitbrain_res_wait"=20 scope=spfile ;alter system set "_db_mttr_advice"=off scope=spfile ;
Page 131 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Troubleshooting VIP Failover
Adjusting Listener Configuration
Make the local IP-address the first listening entry in the listener.ora. More infos in note 3589757 and 339107.1
>> All nodes
Example on node ora-rac-1. Before:
LISTENER_ORA-RAC-1 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = ora-rac-1-vip) (PORT = 1521) (IP = FIRST)) (ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.10.212) (PORT = 1521) (IP = FIRST)) ) )
[...]
After:
LISTENER_ORA-RAC-1 = (DESCRIPTION_LIST = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.10.212) (PORT = 1521) (IP = FIRST)) (ADDRESS = (PROTOCOL = TCP) (HOST = ora-rac-1-vip) (PORT = 1521) (IP = FIRST)) ) )
[...]
Upgrade Oracle Home of Swingbench
Upgrade Oracle Home of Swingbench to Rel. 10.2.0.
Adjust SQLNet settings
Edit sqlnet.ora of client. Add / change the following line
SQLNET.OUTBOUND_CONNECT_TIMEOUT=10
Set the ping timeout back to default
1. Open the $CRS_HOME/bin/racgvip script.
(current value of PING_TIMEOUT = 3 (default))
2. Edit to set PING_TIMEOUT to:
PING_TIMEOUT="3"
Page 132 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Addendum
Metalink Note 284752.1
Page 133 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Metalink Note 294430.1
Page 134 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 135 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 136 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 137 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 138 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 139 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Metalink Note 294430.1
Page 140 of 141
Document: Installation_Documentation_RAC_ASM.odtAuthors: Juergen Doemel, Oracle Deutschland GmbH Michael Ahrens SUN Microsystems GmbH
Page 141 of 141