CONTENTSappsdba.info/docs/RAC/install/Install_RAC_using_OracleVM_on_OEL6… · Configure SCAN DNS...

Author – A.Kishore

http://www.appsdba.info

CONTENTS

Introduction ............................................................................................................................................................ 2

Oracle RAC Software .......................................................................................................................................... 3

Oracle VirtualBox Software ................................................................................................................................ 4

Oracle Enterprise Linux 6.7 software ................................................................................................................. 4

Oracle VirtualBox Installation ............................................................................................................................. 8

Virtual Machine Setup ........................................................................................................................................ 8

Guest Operating System Installation ................................................................................................................ 23

Prepare Steps........................................................................................................................................................ 46

Edit the “hosts” file of rac1 . ......................................................................................................................... 46

Install Guest Additions .................................................................................................................................. 47

Configure SCAN DNS for RAC 11G RAC using dnsmasq in OEL 6 .................................................................. 51

Disable Firewall ............................................................................................................................................. 56

Disable secure linux ...................................................................................................................................... 56

Install oracle-rdbms-server-11gR2-preinstall using yum .............................................................................. 57

Deconfigure Network Time Protocol ............................................................................................................ 57

Install and Configure Automatic Storage Management and Disks. .............................................................. 61

Verify ASMLib Packages. ............................................................................................................................... 62

Configuring and Loading the ASMLib Packages. ........................................................................................... 62

Backup VIRTUAL MACHINE. .......................................................................................................................... 66

TO add shared disks click on storage ............................................................................................................ 68

Clone the Virtual Machine ............................................................................................................................ 75

Configure the Shared Disks ........................................................................................................................... 93

Create ASM disks. ......................................................................................................................................... 96

Take Snapshot node1 and nodes2 ................................................................................................................ 99

Grid Installation .................................................................................................................................................. 102

NFS sharing of the software ....................................................................................................................... 103

Install the missing package on node1 and node2 ....................................................................................... 105

Setup SSH .................................................................................................................................................... 105

Verify using Cluvfy utility ............................................................................................................................ 106

KICK OFF THE INSTALLATION ...................................................................................................................... 107

Post-Install Actions ..................................................................................................................................... 133

http://www.appsdba.info/



Install Database .................................................................................................................................................. 136

References .......................................................................................................................................................... 149

Problems & Solutions ........................................................................................................................................ 149

INTRODUCTION

Here is my setup, I have 16 GB Desktop with Linux installed and with 250 SSD on which the

installation was done. I have created 2 VMs using Oracle Virtual Machine

If you want to get through all steps of the Oracle RAC installation and your laptop or desktop

computer has 8 GB or more of RAM, then this is entirely feasible using Oracle VirtualBox as

demonstrated in this article. You can get a running RAC system which can host a small test database.

The created system is not, and should not be considered, a production-ready system. It's simply to

allow you to get used to installing and using RAC and test various administration procedures. The

article also explains how to save the images and restore RAC from the images in a matter of minutes.

Even if you break your test system, it will be easy to restore.

We have used dnsmasq for scan configuration, we can now have 3 scan listeners on our 2 node

RAC. I had to refer multiple docs to make it work and appreciate all the people who have contributed.

This article uses the 64-bit versions of Oracle Linux, version 6.7, and Oracle 11g Release 2, version

11.2.0.3. Using VirtualBox you can run multiple Virtual Machines (VMs) on a single server, allowing

you to run both RAC nodes on a single machine. In addition, it allows you to set up shared virtual

disks. The finished system includes two guest operating systems, two sets of Oracle Grid

Infrastructure (Clusterware + ASM) and two Database instances all on a single server.

Linux Desktop

16 GB RAM / 250 SSD

RAC1 - Linux 6.7

3 GB RAM

RAC2 - Linux 6.7

3 GB RAM




This article was originally inspired by the article "Oracle Database 11g Release 2 RAC On Linux

Using VirtualBox" written by Sergei Romanenko and published in his blog. Then it was almost entirely

revised and reworked, now this article seems to have very little resemblance with the original work.

Note. When this article was written, Oracle Database 11g Release 2 (11.2.0.3) for Linux 64-bit (both

clusterware and database) was available through the Oracle support to licensed customers only.

As of now, 11.2.0.3 can be downloaded in Oracle support site, in "Patches & Updates", then select

"Latest Patchsets", then select "Oracle Database", then select "Linux x86-64", then select

"11.2.0.3.0". The number of this patch set is 10404530, it is possible to jump to the download page

using this number. This patch set is a full installation of the Oracle Database software. This means

that you do not need to install Oracle Database 11g Release 2 (11.2.0.1) before installing Oracle

Database 11g Release 2 (11.2.0.3). For installing RAC database you will need only 3 files:

ORACLE RAC SOFTWARE

I have used 11.2.0.3 which comes as a patch 10404530, if you are not having support userid you can

download 11.2.0.1 from otn.oracle.com

11.2.0.3 software

Oracle Database (includes Oracle Database and Oracle RAC), part 1:

p10404530_112030_Linux-x86-64_1of7.zip 1.3G

Oracle Database (includes Oracle Database and Oracle RAC), part 2:

p10404530_112030_Linux-x86-64_2of7.zip 1.1G

Oracle Grid Infrastructure (includes Oracle ASM, Oracle Clusterware):

p10404530_112030_Linux-x86-64_3of7.zip 933M




11.2.0.1 Software from OTN

http://otn.oracle.com

Oracle 11g client on Windows for testing TAF (transaction application failover)

ORACLE VIRTUALBOX SOFTWARE

On Linux host

VirtualBox-5.0-5.0.2_102096_el5-1.x86_64.rpm

ORACLE ENTERPRISE LINUX 6.7 SOFTWARE

http://edelivery.oracle.com/linux

Click Sign In/Register


http://otn.oracle.com/



1.1.2. Enter Username & Password




Accept Terms & Restrictions, Click Continue

Select Oracle Linux from Product Pack & select x86 (64 bit) from Platform, Click Go




Select Oracle Linux Release 6 Update 7 Media Pack for x86 (64 bit), Click Continue.

V77197-01.iso

Note:, you need to burn them using the image option available with your cd-writer software

Refer: http://www.petri.co.il/how_to_write_iso_files_to_cd.htm


http://www.petri.co.il/how_to_write_iso_files_to_cd.htm



ORACLE VIRTUALBOX INSTALLATION

rpm -Uvh VirtualBox-5.0-5.0.2_102096_el5-1.x86_64.rpm

Create below folders on the HOST machine

cd /d05/oracle/RAC

mkdir rac1 rac2 asm

VIRTUAL MACHINE SETUP

Go to System Tools – Oracle VM VirtualBox




Now we must define the two virtual RAC nodes. We can save time by defining one VM, then cloning it when it is installed.

Start VirtualBox and click the "New" button on the toolbar. Click the "Next" button on the first page of the Virtual Machine Wizard.




Enter the name "rac1", OS "Linux" and Version "Oracle (64 bit)", then click the "Next" button.




Set Base Memory to 3 GB or more, then click the “Next” button




Accept the default option to create a new virtual hard disk by clicking the “Create” button




Select the default VMDK type and click the “Next” button on the Create Virtual Hard Drive screen

Select Dynamically Allocated




Provide the location where you want to save the VM files. In my case I have provided path as below. Also

increase the size of your vm from default to say 30GB and Click “Create”




The "rac1" VM will appear on the left hand pane. Scroll down the "Details" tab on the right




Click on General->Advanced and change shared clipboard to Bidirectional and Change Drag ‘n’ Drop to

Bidirectional and click Ok




Click on Network




Add two Adaptors.

Make sure “Adapter 1” is enabled, attached to “Bridged Adapter”. This interface will be used

for public network, for example, for connection to the RAC database from other applications.




Then click on the “Adapter 2” tab. Make sure “Adapter 2” is enabled and attach to “Internal Network”. Name

this network “privnet”:




Click on storage to load OEL6.7 ISO




GUEST OPERATING SYSTEM INSTALLATION

Please note that during installation Virtual Box will be keeping the mouse pointer inside VM area. To exit, press Right Control key on the keyboard.

Click on Start as shown below




After starting we get below screen and select Install or Upgrade an Existing System

Select skip




Scroll Down and click NEXT




Select basic Storage and click Next

Select Discard any Data




Change the hostname and Scroll down and click on “Configure Network




Select System eth0 and click on edit

Select Connect Automatically and select IPV4 Setting




Make sure that “Connect automatically” is checked. Select “IPv4 Settings” tab; change Method to “DHCP”;

Press “Add” and fill Address: 192.168.1.6; Netmask: 255.255.255.0; Gateway: 192.168.1.1 Press “Apply” then

done:

In the Network Connections screen select “System eth1” interface, this will be used for private network, then

press “Edit”. Then check the box “Connect automatically”. Select “IPv4 Settings” tab; change Method to

“Manual”. Press “Add” and fill Address: 192.168.0.100; Netmask: 255.255.255.0; Gateway: 0.0.0.0. When

done, press “Apply”:




192.168.0.100

255.255.255.0




Finally select “System eth2” interface, this will be used for Internet, then press “Edit”. Check the box “Connect

automatically”. Select “IPv4 Settings” tab make sure the Method is set to “Automatic (DHCP)”.Press “Apply”

button:

Click Next




Give password for root

Select Replace Existing Linux




Select Write Changes to disk

Select Database Server option and select Customize now and Click Next




Select Tiger VNC




Click all the options under Desktops and Click Next




Click Yes

Reboot the Guest Machine




Click Forward




PREPARE STEPS

First of all check whether we can connect to the internet or not

ping www.google.com

PING www.google.com (74.125.130.103) 56(84) bytes of data.

64 bytes from sb-in-f103.1e100.net (74.125.130.103): icmp_seq=1 ttl=46 time=63.4 ms



EDIT THE “HOSTS” FILE OF RAC1 .

In this chapter we are going to prepare RAC1 and later clone to RAC2

Follow the below steps on rac1

vi /etc/hosts

127.0.0.1 localhost.localdomain localhost

# Public Network - (eth0)

192.168.1.6 rac1.appsdba.info rac1

192.168.1.7 rac2.appsdba.info rac2

# Private Interconnect - (eth1)

192.168.0.100 rac1-priv.appsdba.info rac1-priv

192.168.0.101 rac2-priv.appsdba.info rac2-priv

# Public Virtual IP (VIP) addresses for - (eth0)

192.168.1.200 rac1-vip.appsdba.info rac1-vip

192.168.1.201 rac2-vip.appsdba.info rac2-vip




INSTALL GUEST ADDITIONS

Click Ok




Click on RUN




If It fails. We need to update packages

Solution:

To overcome this error update gcc using

yum update gcc

yum update - It will take 20 minutes

yum install kernel-uek-devel

Note: Reboot the Server once above steps are completed




Once again try to install VM tools

Purpose we will be able to view the shared folders in Linux Guest




CONFIGURE SCAN DNS FOR RAC 11G RAC USING DNSMASQ IN OEL 6

Install and configure dnsmasq

1. To install dnsmasq run as root following command

yum install dnsmasq

Loaded plugins: refresh-packagekit, security

Setting up Install Process

Resolving Dependencies

--> Running transaction check

---> Package dnsmasq.x86_64 0:2.48-13.el6 will be updated

---> Package dnsmasq.x86_64 0:2.48-14.el6 will be an update

--> Finished Dependency Resolution

Dependencies Resolved

================================================================================

Package Arch Version Repository Size

================================================================================

Updating:

dnsmasq x86_64 2.48-14.el6 public_ol6_latest 149 k

Transaction Summary

================================================================================

Upgrade 1 Package(s)

Total download size: 149 k

Is this ok [y/N]: y

Downloading Packages:

dnsmasq-2.48-14.el6.x86_64.rpm | 149 kB 00:00

warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOK EY

Retrieving key from http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

Importing GPG key 0xEC551F03:

Userid: "Oracle OSS group (Open Source Software group) <[email protected]>"

From : http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6




Is this ok [y/N]: y

Running rpm_check_debug

Running Transaction Test

Transaction Test Succeeded

Running Transaction

Updating : dnsmasq-2.48-14.el6.x86_64 1/2

Cleanup : dnsmasq-2.48-13.el6.x86_64 2/2

Verifying : dnsmasq-2.48-14.el6.x86_64 1/2

Verifying : dnsmasq-2.48-13.el6.x86_64 2/2

Updated:

dnsmasq.x86_64 0:2.48-14.el6

Complete!

2. Configure dnsmasq

create new file “/etc/racdns” with settings for SCAN

[root@rac1 ~]# vi /etc/racdns

#scan

192.168.1.20 rac-scan rac-scan.appsdba.info



modifydnsmasq default configuration file “/etc/dnsmasq.conf”. One parameter addn-hosts should be changed to point to file “/etc/racdns”.

cat /etc/dnsmasq.conf | grep addn-hosts

addn-hosts=/etc/racdns

3. Start dnsmasq

service dnsmasq start

chkconfig dnsmasq on




4. Next step is to resolve problem with file “/etc/resolv.conf”

I’m using third network card as NAT with DHCP so each time you restart network card or reboot host the file is overwritten with automatically generated settings. Nameserver points for 192.168.1.1 which is required to resolve internet entries but not enough to resolve SCAN settings via dnsmasq.

cat /etc/resolv.conf

# Generated by NetworkManager

search appsdba.info

nameserver 192.168.1.1

As default dnsmasq is running on ipadress 127.0.0.1 so it requires following settings in “/etc/resolv.conf”.

cat /etc/resolv.conf

# Generated by NetworkManager


search appsdba.info


You must protect the file from being automatically overwritten by host reboot, network card restart etc.

[root@rac1 ~]# chattr +i /etc/resolv.conf

To write we use below command

chattr -i /etc/resolv.conf




Verification

It’s just running nslookup to verify all is working fine

nslookup rac-scan

Server: 127.0.0.1

Address: 127.0.0.1#53

Name: rac-scan.appsdba.info

Address: 192.168.1.21


Address: 192.168.1.22


Address: 192.168.1.20

[root@rac1 ~]# nslookup rac-scan

Server: 127.0.0.1

Address: 127.0.0.1#53


Address: 192.168.1.22


Address: 192.168.1.20


Address: 192.168.1.21




[root@rac1 ~]# nslookup rac1

Server: 127.0.0.1

Address: 127.0.0.1#53

Name: rac1.appsdba.info

Address: 192.168.1.6

[root@rac1 ~]# nslookup rac1-priv

Server: 127.0.0.1

Address: 127.0.0.1#53

Name: rac1-priv.appsdba.info

Address: 192.168.0.100

[root@rac1 ~]# nslookup rac1-vip

Server: 127.0.0.1

Address: 127.0.0.1#53

Name: rac1-vip.appsdba.info

Address: 192.168.1.200




DISABLE FIREWALL

Now we need to disable the firewall: Linux Main menu | System | Administration | Firewall. Click on “Disable”

icon, then on “Apply”.

Click on "Disable" icon, then on "Apply".

DISABLE SECURE LINUX

Disable secure linux by editing the “/etc/selinux/config” file, making sure the SELINUX flag

is set as follows.

SELINUX=permissive




INSTALL ORACLE-RDBMS-SERVER-11GR2-PREINSTALL USING YUM

yum install oracle-rdbms-server-11gR2-preinstall

DECONFIGURE NETWORK TIME PROTOCOL

Either configure NTP, or make sure it is not configured so the Oracle Cluster Time Synchronization Service

(ctssd) can synchronize the times of the RAC nodes. If you want to deconfigure NTP do the following.

Below steps will disable NTP - we have followed below steps only

# service ntpd stop

Shutting down ntpd: [ OK ]

# chkconfig ntpd off

# mv /etc/ntp.conf /etc/ntp.conf.orig

# rm /var/run/ntpd.pid

Below steps will enable NTP

If you want to use NTP, you must add the “-x” option into the following line in the “/etc/sysconfig/ntpd” file.

OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"

Then restart NTP.

# service ntpd restart




Edit the “sysctl.conf” of rac1 with the following values.

vi /etc/sysctl.conf

kernel.core_uses_pid = 1

kernel.shmmax=4294967295

kernel.shmmni=4096

kernel.sem=256 32000 100 142

fs.file-max = 6815744

kernel.msgmnb = 65535

kernel.msgmni = 2878

net.core.rmem_default = 262144

net.core.rmem_max= 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 1048576

net.ipv4.tcp_wmem = 262144 262144 262144

net.ipv4.tcp_rmem = 262144 262144 262144

fs.aio-max-nr = 1048576

net.ipv4.ip_local_port_range = 9000 65500




Edit the “limits.conf” file of rac1 with the following values.

vi /etc/security/limits.conf

* hard nofile 327680

* soft nofile 262144

* hard nproc 327680

* soft nproc 262144

* hard memlock 3145728

* soft memlock 3145728

* hard stack 16384

* soft stack 10240

Create "oracle/grid" User and Directories in both nodes.

groupadd -g 1000 oinstall

groupadd -g 1020 asmadmin

groupadd -g 1021 asmdba

groupadd -g 1022 asmoper

groupadd -g 1031 dba

useradd -u 1100 -g oinstall -G asmoper,asmdba,asmadmin,dba grid

userdel oracle

useradd -u 1101 -g oinstall -G dba,asmdba oracle

cd /home

chown -R oracle:dba oracle




Create directories and grant permissions

cd /home

chown -R oracle:oinstall oracle/

mkdir -p /u01/app/grid

chown -R grid:oinstall /u01/app/

chmod -R 775 /u01/app/grid

mkdir -p /u01/app/oracle

chown oracle:oinstall /u01/app/oracle

chmod -R 775 /u01/app/oracle

Change the password of grid and oracle users

passwd grid

passwd oracle

passwd grid

Changing password for user grid.

New password:

BAD PASSWORD: it is too short

BAD PASSWORD: is too simple

Retype new password:

passwd: all authentication tokens updated successfully.

[root@rac1 home]# passwd oracle

Changing password for user oracle.

New password:

[root@rac1 home]# passwd oracle

Changing password for user oracle.

New password:

BAD PASSWORD: it is based on a dictionary word

BAD PASSWORD: is too simple




Retype new password:

passwd: all authentication tokens updated successfully.

INSTALL AND CONFIGURE AUTOMATIC STORAGE MANAGEMENT AND DISKS.

Install Oracleasmlib. I have downloaded file from

http://www.oracle.com/technetwork/server-storage/linux/asmlib/ol6-1709075.html

yum install oracleasm-support

oracleasmlib-2.0.4-1.el6.i686.rpm

rpm -Uvh oracleasmlib-2.0.4-1.el6.i686.rpm

Preparing... ########################################### [100%]

1:oracleasmlib ########################################### [100%]


http://www.oracle.com/technetwork/server-storage/linux/asmlib/ol6-1709075.html



VERIFY ASMLIB PACKAGES.

This installation needs to be performed on all nodes as the root user account:

rpm -qa |grep asm

oracleasmlib-2.0.4-1.el6.i686

oracleasm-support-2.1.8-1.el6.x86_64

CONFIGURING AND LOADING THE ASMLIB PACKAGES.

Now that we downloaded and installed the ASMLib Packages for Linux, we need to configure and load the ASM kernel module. This task needs to be run on all nodes as root:

$ su –

/etc/init.d/oracleasm configure

/etc/init.d/oracleasm configure

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver is

loaded on boot and what permissions it will have. The current values

will be shown in brackets ('[]'). Hitting <ENTER> without typing an

answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: grid

Default group to own the driver interface []: asmadmin

Start Oracle ASM library driver on boot (y/n) [n]: y




Scan for Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: done

Initializing the Oracle ASMLib driver: [ OK ]

Scanning the system for Oracle ASMLib disks: [ OK ]

Login as the grid user and add the following lines at the end of the .bash_profile file.

su - grid

vi .bash_profile

# Oracle Settings

TMP=/tmp; export TMP

TMPDIR=$TMP; export TMPDIR

ORACLE_HOSTNAME=rac1.appsdba.info; export ORACLE_HOSTNAME

ORACLE_BASE=/u01/app/grid; export ORACLE_BASE

GRID_HOME=/u01/app/11.2.0/grid; export GRID_HOME

ORACLE_SID=+ASM1; export ORACLE_SID

ORACLE_HOME=$GRID_HOME; export ORACLE_HOME

PATH=$PATH:$ORACLE_HOME/bin; export PATH

ORACLE_TERM=xterm; export ORACLE_TERM

LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

if [ $USER = "oracle" ]; then

if [ $SHELL = "/bin/ksh" ]; then

ulimit -p 16384




ulimit -n 65536

else

ulimit -u 16384 -n 65536

fi

fi

Login as the oracle user and add the following lines at the end of the .bash_profile file.

su – oracle

vi .bash_profile

# User specific environment and startup programs

# database name - RACDB

# instance name 1 - RACDB1

# instance name 2 - RACDB2

PATH=$PATH:$HOME/bin

export PATH

PATH=$PATH:$HOME/bin

export PATH

# Oracle Settings

TMP=/tmp; export TMP

TMPDIR=$TMP; export TMPDIR

DB_HOME=/u01/app/oracle/product/11.2.0/dbhome_1

ORACLE_HOSTNAME=lb.appsdba.info; export ORACLE_HOSTNAME

export ORACLE_UNQNAME=RACDB

ORACLE_SID=RACDB1; export ORACLE_SID

ORACLE_HOME=$DB_HOME; export ORACLE_HOME

PATH=$ORACLE_HOME/bin:$PATH; export PATH




LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH

CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH

ORACLE_TERM=xterm; export ORACLE_TERM

if [ $USER = "oracle" ]; then

if [ $SHELL = "/bin/ksh" ]; then

ulimit -p 16384

ulimit -n 65536

else

ulimit -u 16384 -n 65536

fi

fi




BACKUP VIRTUAL MACHINE.

Now we will shut down the server as we need to add asm disks. Before we add asm disks.

We will backup our vm

Click File export Appliance




Select our VM and click next Save to some location

Click next and click export

Ok We done with the backup now add disk’s for ASM storage.




TO ADD SHARED DISKS CLICK ON STORAGE




Pending




CLONE THE VIRTUAL MACHINE

Later versions of VirtualBox allow you to clone VMs, but these also attempt to clone the shared disks, which is not what we want. Instead we must manually clone the VM.

Shut down the "rac1" virtual machine using the following command.

# shutdown -h now

If you are using VirtualBox version 4.0.0 or later, so the following.

VBoxManage clonehd rac1.vdi /d05/oracle/RAC/rac2/rac2.vdi

Create rac2




Change the disks to shared that were created earlier for ASM




Do the same for all the ASMDISK’s




Now once this is done click on RAC2 Vm and Storage (Attach the disks)




Modify the Network Settings




Now start the Rac2 server make sure RAC1 IS DOWN

The main purpose of starting the server is that we have to configure Network

If we see here it is still rac1. Hostname has to be changed




Change the hostname and reboot rac2

cat /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=rac2.appsdba.info

Start "Network Connections" tool (Main menu | System | Preferences | Network Connections). Edit eth0 and

eth1 interfaces and set in IPv4 addresses 192.168.1.7 and 192.168.0.101 correspondingly.




REBOOT NODE2




Check the MAC address of each of the available network connections. Don't worry that they are listed as

"eth2" and "eth3". These are dynamically created connections because the MAC address of the "eth0" and

"eth1" connections is incorrect.

ifconfig -a | grep eth

eth2 Link encap:Ethernet HWaddr 08:00:27:57:F1:AB

eth3 Link encap:Ethernet HWaddr 08:00:27:64:EC:15

Edit the "/etc/sysconfig/network-scripts/ifcfg-eth0", amending only the IPADDR and HWADDR settings

as follows and deleting the UUID entry. Note, the HWADDR value comes from the "eth2" interface

displayed above.




eth2 Link encap:Ethernet HWaddr 08:00:27:57:F1:AB

eth3 Link encap:Ethernet HWaddr 08:00:27:64:EC:15

Edit the "/etc/sysconfig/network-scripts/ifcfg-eth1", amending only the IPADDR and HWADDR settings as

follows and deleting the UUID entry. Note, the HWADDR value comes from the "eth3" interface displayed

above.

If the adapter names do not reset properly, check the HWADDR in the "/etc/udev/rules.d/70-persistent-

net.rules" file. If it is incorrect, amend it to match the settings described above.

init 0

Verify the adapter settings




Verify the ipaddress




CONFIGURE THE SHARED DISKS

Start the "rac1" virtual machine by clicking the "Start" button on the toolbar. When the server has started, log in as the root user so you can configure the shared disks. The current disks can be seen by issuing the following commands.

# cd /dev

# lssd*

sda sda1 sda2 sdbsdcsdd

fdisk –l

Use the "fdisk" command to partition the disks sdb to sdf. The following output shows the expected fdisk output for the sdb disk.

# fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF

disklabel

Building a new DOS disklabel. Changes will remain in memory only,

until you decide to write them. After that, of course, the previous

content won't be recoverable.




The number of cylinders for this disk is set to 1305.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

(e.g., DOS FDISK, OS/2 FDISK)

Warning: invalid flag 0x0000 of partition table 4 will be corrected by

w(rite)

Command (m for help): n

Command action

e extended

p primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-1305, default 1):

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305):

Using default value 1305

Command (m for help): p

Disk /dev/sdb: 10.7 GB, 10737418240 bytes

255 heads, 63 sectors/track, 1305 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System




/dev/sdb1 1 1305 10482381 83 Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

#

In each case, the sequence of answers is "n", "p", "1", "Return", "Return", "p" and "w".

Once all the disks are partitioned, the results can be seen by repeating the previous "ls" command.

# cd /dev

# lssd*

sda sda1 sda2 sdb sdb1 sdc sdc1 sdd sdd1

#




CREATE ASM DISKS.

Create the ASM disks on any one node as the root user.

/usr/sbin/oracleasm createdisk DISK1 /dev/sdb1

/usr/sbin/oracleasm createdisk DISK2 /dev/sdc1

/usr/sbin/oracleasm createdisk DISK3 /dev/sdd1

Verify that the ASM disks are visible from every node. (Right now it will not work)

# /etc/init.d/oracleasm scandisks

Scanning system for ASM disks: [ OK ]

We can see the disk are now visible to ASM using the "listdisks" command.

# /usr/sbin/oracleasm listdisks

DISK1

DISK2

DISK3




Edit the "/home/oracle/.bash_profile" file on the "rac2" node to correct the ORACLE_SID and ORACLE_HOSTNAME values.

ORACLE_SID=RACDB2; export ORACLE_SID

ORACLE_HOSTNAME=rac2.appsdba.info; export ORACLE_HOSTNAME

Also, amend the ORACLE_SID setting in the "/home/oracle/db_env" and "/home/oracle/grid_env" files.

ORACLE_SID=+ASM2; export ORACLE_SID

Start the "rac1" virtual machine and restart the "rac2" virtual machine. When both nodes have started, check they can both ping all the public and private IP addresses using the following commands.

ping -c 3 rac1

ping -c 3 rac1-priv

ping -c 3 rac2

ping -c 3 rac2-priv

At this point the virtual IP addresses defined in the "/etc/hosts" file will not work, so don't bother testing them.

The virtual machine setup is now complete.

rac-scan

[root@rac2 ~]# nslookup rac2-vip

Server: 127.0.0.1

Address: 127.0.0.1#53

Name: rac2-vip.appsdba.info

Address: 192.168.1.201

[root@rac2 ~]# nslookup rac2

Server: 127.0.0.1

Address: 127.0.0.1#53

Name: rac2.appsdba.info

Address: 192.168.1.7




[root@rac2 ~]# nslookup rac-scan

Server: 127.0.0.1

Address: 127.0.0.1#53


Address: 192.168.1.21


Address: 192.168.1.22


Address: 192.168.1.20




TAKE SNAPSHOT NODE1 AND NODES2

Shutdown node1 and node2

After shutdown go to Oracle VM

Click Snapshots on the right hand side




LOCATION OF SNAPSHOTS




GRID INSTALLATION

Clean the old installation (on both nodes, if required)

cd /etc/init.d

rm –rf ohasdinit.ohasd

rm -rf /etc/oracle

rm -rf /etc/oraInst.loc

rm -rf /etc/oratab

cd /u01/app

rm -rf oracle

rm -rf /u01/app/*.*

rm -rf /u01/app/11.2.0/grid/*.*

cd /u01/app/grid

rm -rforaInventory

mkdir -p /u01/app/grid

chown -R grid:oinstall /u01/app/

chmod -R 775 /u01/app/grid

mkdir -p /u01/app/oracle

chownoracle:oinstall /u01/app/oracle

chmod -R 775 /u01/app/oracle

rm -rf /u01/app/grid/*

rm -rf /usr/tmp/.oracle/

rm -rf /u01/app/oraInventory/*




-- On one of the node

-- If there are 3 diskgroups then use the below commands

/usr/sbin/oracleasmdeletedisk DISK1



/usr/sbin/oracleasmcreatedisk DISK1 /dev/sdb1

/usr/sbin/oracleasmcreatedisk DISK2 /dev/sdc1

/usr/sbin/oracleasmcreatedisk DISK3 /dev/sdd1

/etc/init.d/oracleasmlistdisks

Always Verify

Public ip’s

Private ip’s

Date between the both the nodes are same

NFS SHARING OF THE SOFTWARE

The software is available on host machine – 192.168.1.31

On the server (host machine)-

vi /etc/exports

/software/RAC_SOFTWARE/ 192.168.1.10(ro)

Service nfs restart

On the client – node1

mkdir –p /software




mount 192.168.1.31:/d04/oracle/RAC_SOFTWARE /software

Software

p10404530_112030_Linux-x86-64_3of7.zip

unzipp10404530_112030_Linux-x86-64_3of7.zip




INSTALL THE MISSING PACKAGE ON NODE1 AND NODE2

cd /software/11gR11203/grid/rpm

rpm -Uvh cvuqdisk-1.0.9-1.rpm

Preparing... ########################################### [100%]

Using default group oinstall to install package

1:cvuqdisk ########################################### [100%]

SETUP SSH

Connect as grid user

cd /software/11gR11203/grid/sshsetup

./sshUserSetup.sh -user grid -hosts "rac1 rac2" -noPromptPassphrase -advanced -exverify

Verifying SSH connectivity has been setup from rac1 to rac1

------------------------------------------------------------------------

IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL.

Tue Sep 1 05:12:35 EDT 2015

------------------------------------------------------------------------

------------------------------------------------------------------------


------------------------------------------------------------------------


Tue Sep 1 05:12:35 EDT 2015

------------------------------------------------------------------------

-Verification from rac1 complete-

------------------------------------------------------------------------


------------------------------------------------------------------------


Tue Sep 1 05:12:35 EDT 2015




------------------------------------------------------------------------

------------------------------------------------------------------------


------------------------------------------------------------------------


Tue Sep 1 05:12:36 EDT 2015

------------------------------------------------------------------------

-Verification from rac2 complete-

SSH verification complete.

VERIFY USING CLUVFY UTILITY

Prior to 11gR2 we would probably use the "runcluvfy.sh" utility in the clusterware root directory to check the prerequisites have been met. If you are intending to configure SSH connectivity using the installer this check should be omitted as it will always fail. If you want to setup SSH connectivity manually, then once it is done you can run the "runcluvfy.sh" with the following command.

[grid@racnode1 ~]$ cd /software/11gR11203/grid

[grid@racnode1 grid]$ sh runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose

If you get any failures be sure to correct them before proceeding.




KICK OFF THE INSTALLATION




Every ASM disk is divided into allocation units (AU). An AU is the fundamental unit of allocation within a disk group. A file extent consists of one or more AU. An ASM file consists of one or more file extents. When you create a disk group, you can set the ASM AU size to be between 1 MB and 64 MB in powers of two, such as, 1, 2, 4, 8, 16, 32, or 64. Larger AU sizes typically provide performance advantages for data warehouse applications that use large sequential reads.




Only for 11.2.0.3 grid installation (this is already done)




The results of prerequisite checks are shown on the next screen. You should see two warnings and one failure. The failure was caused

by inability to lookup SCAN in DNS and that should be expected. Check "Ignore All" box and press "Next".




/u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@rac1 ~]# /u01/app/11.2.0/grid/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0/grid




Enter the full pathname of the local bin directory: [/usr/local/bin]:

Copying dbhome to /usr/local/bin ...

Copying oraenv to /usr/local/bin ...

Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

User ignored Prerequisites during installation

OLR initialization - successful

root wallet

root wallet cert

root cert export

peer wallet

profile reader wallet

pa wallet

peer wallet keys

pa wallet keys

peer cert request

pa cert request

peer cert

pa cert

peer root cert TP

profile reader root cert TP

pa root cert TP

peer pa cert TP

pa peer cert TP

profile reader pa cert TP

profile reader peer cert TP

peer user cert

pa user cert




Adding Clusterware entries to upstart

CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'

CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'

CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'

CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'

CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded

CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.cssd' on 'rac1'

CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'

CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded

CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded

ASM created and started successfully.

Disk Group DATA created successfully.

clscfg: -install mode specified

Successfully accumulated necessary OCR keys.

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

CRS-4256: Updating the profile

Successful addition of voting disk 5f5866ae4c0e4fafbffab1ab690a130b.

Successfully replaced voting disk group with +DATA.

CRS-4256: Updating the profile

CRS-4266: Voting file(s) successfully replaced

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 5f5866ae4c0e4fafbffab1ab690a130b (/dev/oracleasm/disks/DISK1) [DATA]

Located 1 voting disk(s).

CRS-2672: Attempting to start 'ora.asm' on 'rac1'

CRS-2676: Start of 'ora.asm' on 'rac1' succeeded

CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1'

CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeeded

Configure Oracle Grid Infrastructure for a Cluster ... succeeded




On node2

/u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@rac2 ~]# /u01/app/11.2.0/grid/root.sh



ORACLE_OWNER= grid

ORACLE_HOME= /u01/app/11.2.0/grid


Copying dbhome to /usr/local/bin ...

Copying oraenv to /usr/local/bin ...

Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...





Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

User ignored Prerequisites during installation

OLR initialization - successful

Adding Clusterware entries to upstart

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating




An active cluster was found during exclusive startup, restarting to join the cluster

Configure Oracle Grid Infrastructure for a Cluster ... succeeded




Hurray :) we have 3 scans

On node1

[root@rac1 ~]# ps -ef |grep tns

root 15 2 0 Sep01 ? 00:00:00 [netns]

grid 15436 1 0 00:25 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit


grid 16190 1 0 00:31 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit

root 18572 3820 0 00:33 pts/0 00:00:00 greptns

On node2

ps -ef |grep tns

root 15 2 0 Sep01 ? 00:00:00 [netns]


grid 14410 1 0 00:31 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit




root 16571 11656 0 00:33 pts/0 00:00:00 greptns

[root@orcl1 ~]# su - grid

cluvfy comp scan -verbose

[grid@rac1 ~]$ cluvfy comp scan -verbose

Verifying scan

Checking Single Client Access Name (SCAN)...

SCAN Name Node Running? ListenerName Port Running?

---------------- ------------ ------------ ------------ ------------ ------------

rac-scan.appsdba.info rac2 true LISTENER_SCAN1 1521 true



Checking TCP connectivity to SCAN Listeners...

Node ListenerName TCP connectivity?

------------ ------------------------ ------------------------

rac1 LISTENER_SCAN1 yes



TCP connectivity to SCAN Listeners exists on all cluster nodes

Checking name resolution setup for "rac-scan.appsdba.info"...

SCAN Name IP Address Status Comment

------------ ------------------------ ------------------------ ----------

rac-scan.appsdba.info 192.168.56.112 passed



Verification of SCAN VIP and Listener setup passed

Verification of scan was successful.




crsctl stat res -t

[grid@rac1 ~]$ crsctl stat res -t

--------------------------------------------------------------------------------

NAME TARGET STATE SERVER STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ora.LISTENER.lsnr

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ora.asm

ONLINE ONLINE rac1 Started

ONLINE ONLINE rac2 Started

ora.gsd

OFFLINE OFFLINE rac1

OFFLINE OFFLINE rac2

ora.net1.network

ONLINE ONLINE rac1

ONLINE ONLINE rac2

ora.ons

ONLINE ONLINE rac1

ONLINE ONLINE rac2

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE rac2








ora.cvu


ora.oc4j


ora.rac1.vip


ora.rac2.vip


ora.scan1.vip


ora.scan2.vip


ora.scan3.vip


POST-INSTALL ACTIONS

By default, the Global Services Daemon (GSD) is not started on the cluster. To start GSD, change

directory to the <CRS_HOME> and issue the following commands:

srvctl enable nodeapps –n rac1

srvctl enable nodeapps –n rac2

srvctl start nodeapps

Successful Oracle Clusterware operation can also be verified using the following command:

[grid@racnode1 oraagent_grid]$ crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online




CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

olsnodes -n

racnode1 1

racnode2 2

srvctl status asm -a

ASM is running on rac2,rac1

ASM is enabled.

ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 3

Total space (kbytes) : 262120

Used space (kbytes) : 2676

Available space (kbytes) : 259444

ID : 720927388

Device/File Name : +DATA

Device/File integrity check succeeded

Device/File not configured




Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user




crsctl query css votedisk


-- ----- ----------------- --------- ---------



crsctl check cluster -all

**************************************************************


-- ----- ----------------- --------- ---------



[grid@rac1 ~]$ crsctl check cluster -all

**************************************************************

rac1:




**************************************************************

rac2:




**************************************************************




INSTALL DATABASE

Confirm that the oracle user in this session is a member of the required groups

Connect as oracle user setup SSH

cd /software/11gR11203/grid/sshsetup

./sshUserSetup.sh -user grid -hosts "rac1 rac2" -noPromptPassphrase -advanced -exverify

root@rac1 Desktop]#su - oracle

[oracle@rac1 ~]$ cd /tmp/database/

[oracle@rac1 database]$ ls

doc readme.html rpm sshsetup welcome.html

install responserunInstaller stage

[oracle@rac1 database]$ ./runInstaller




On node1 and node2

/u01/app/oracle/product/11.2.0/dbhome_1/root.sh



ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/oracle/product/11.2.0/dbhome_1


The contents of "dbhome" have not changed. No need to overwrite.

The contents of "oraenv" have not changed. No need to overwrite.

The contents of "coraenv" have not changed. No need to overwrite.








Finished product-specific root actions.




REFERENCES

http://balaoracledba.com/2013/12/23/step-by-step-install-of-oracle-rac-11gr2-on-oel-6-5-on-oracle-vm-virtual-box/

http://dbaora.com/configure-scan-dns-for-rac-11g-rac-12c-using-dnsmasq-in-oel5-oel6-2/

http://www.lab128.com/rac_installation_using_vb/article_text.html#install_grid_infrastructure

https://oracle-base.com/articles/11g/oracle-db-11gr2-rac-installation-on-oracle-linux-6-using-virtualbox#clone_the_vm

http://forums.oracle.com/forums/thread.jspa?messageID=4345930

http://www.oracle.com/technology/pub/articles/wartak-rac-vm_3.html

http://wiki.oracle.com/page/11gR2+RAC+on+a+Mac+-+Part+6 –nice one

PROBLEMS & SOLUTIONS

11gR2 Grid: root.sh fails to start the clusterware on the second node (Doc ID 981357.1)

How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation [ID 942166.1]

Issues and solutions

cd /home/oracle/app/11.2.0/grid/log/linux2

cat alertlinux2.log

[ctssd(27935)]CRS-2409:The clock on host linux2 is not synchronous with the mean

cluster time. No action has been taken as the Cluster Time Synchronization Service is

running in observer mode.

2010-07-24 05:26:20.471

[ctssd(27935)]CRS-2412:The Cluster Time Synchronization Service detects that the local

time is significantly different from the mean cluster time. Details in

/home/oracle/app/11.2.0/grid/log/linux2/ctssd/octssd.log.

2010-07-24 05:26:20.471

[ctssd(27935)]CRS-2409:The clock on host linux2 is not synchronous with the mean

cluster time. No action has been taken as the Cluster Time Synchronization Service is

running in observer mode.

http://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi.html

http://arjudba.blogspot.com/2010/03/in-11gr2-grid-rootsh-fails-with-crs.html




How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation [ID 942166.1]

-- If root.sh has failed, run the below commands to clean up

-- first node

cd /home/oracle/app/11.2.0/grid/crs/install

./rootcrs.pl -verbose -deconfig -force

-- second node

cd /home/oracle/app/11.2.0/grid/crs/install

./rootcrs.pl -verbose -deconfig -force –lastnode

To add vip manually

[root@linux2 bin]# ./srvctl add vip -n linux2 -A 192.168.0.201/255.255.255.0/eth0 -k 1

[root@linux2 bin]# ./srvctlconfigvip -n linux2

TIP - to start the RAC services manually

cd /etc/oracle/scls_scr/racnode1/root/

Modify files crsstart and ohasdstr

change enable to disable


CONTENTSappsdba.info/docs/RAC/install/Install_RAC_using_OracleVM_on_OEL6… · Configure SCAN DNS...

Documents

Transcript of CONTENTSappsdba.info/docs/RAC/install/Install_RAC_using_OracleVM_on_OEL6… · Configure SCAN DNS...