Imagination at work.

Ajith Narayanan, Technical Lead-Oracle ERP GE Healthcare, Bangalore, IndiaBangalore, Feb 27th 2014

Oracle Real Application Clusters System Tests With Demo

2

Who am I?

Ajith NarayananTechnical Lead – Oracle ERP Configuration ManagementGE Healthcare, Bangalore, India

10 years of Oracle Apps & Oracle DBA experience.Blogger :- http://oracledbascriptsfromajith.blogspot.comWebsite Chair (2011-2013):- http://www.oracleracsig.org – Oracle RAC SIGMember – OAUG, AIOUG, RACSIG

3

AgendaReal Application Clusters Testing ObjectivesOracle Technologies Used For Tests• Test 1 :Planned Node Reboot• Test 2 :Clusterware and Fencing• Test 3 :Restart Failed Node• Test 4 :Reboot All Nodes Same Time• Test 5 :Unplanned Instance Failure• Test 6 :Planned Instance Termination• Test 7 :Clusterware and Fencing• Test 8: Service Failover• Test 9: Public Network Failure• Test 10: Interconnect Network Failure Sample Cluster Callout Script

Q & A

4

Real Application Clusters Testing objectives• To verify that the system has been installed and configured correctly… Check

that nothing is broken

• To verify that basic functionality still works in a specific environment and for a specific workload

• To make sure that the system will achieve its objectives… in particular, gigh availability and performance objectives

5

Oracle Technologies used for tests• Fast Application Notification (FAN) – Notification mechanism that alerts

application of service level changes of the database

• Fast Connection Failover (FCF) – Utilizes FAN events to enable database clients to proactively react to down events by quickly failing over connections to surviving database instances

• Transparent Application Failover (TAF) – Allows for connections to be automatically reestablished to a surviving database instance if the instance servicing the initial connection fails… but insert, update and delete transactions will be rolled back (12C overcomes this problem)

• Runtime Connection Load Balancing (RCLB) – Provides intelligence about the current service level of the database instances to application connection pools…his increases the performance of the application by utilizing least loaded (dynamic workload balancing)

6

Test 1 :Planned Node RebootProcedure• Start client workload and identify instance with most client connections• Reboot the node where the most loaded instance is running• For AIX, HPUX, Windows: “shutdown –r” , For Linux: “shutdown –r now” , For Solaris: “reboot”

Expected results1. The instances and other Clusterware resources go offline ( ‘SERVER’ field of crsctl stat res –t

output)2. The node VIP fails over the surviving nodes and will show a state of “INTERMEDIATE” with state

details of “FAILED_OVER”3. The SCAN VIP(s) that were running on the rebooted node will fail over to surviving nodes4. The SCAN Listener(s) running on that node will fail over to a surviving node5. Instance recovery is performed by another instance6. Services are moved to available instances7. Client connections are moved / reconnected to surviving… with TAF configured, select

statements should continue…active DML will be aborted8. After the database reconfiguration, surviving instances continue processing their workload

Measures9. Time to detect node or instance failure…Time to complete instance recovery. Alert Log helps!10. Time to restore client activity to same level11. Time before failed instance is restarted automatically by Clusterware and is accepting new

connections 12. Successful failover of the SCAN VIP(s) and SCAN Listener(s)

7

Test 2: Unplanned Node Failure of OCR MasterProcedure• Start client workload… Identify the node that is the OCR master using the following grep

command from any of the nodes:o grep -i "OCR MASTER" $GI_HOME/log/<node_name>/crsd/crsd.l*

NOTE: Windows users must manually review the $GI_HOME/log/<node_name>/crsd/crsd.l* logs to determine the OCR Master• Power off the node that is the OCR master

Expected results1. The instances and other Clusterware resources go offline ( ‘SERVER’ field of crsctl stat res

–t output)2. The node VIP fails over the surviving nodes and will show a state of “INTERMEDIATE” with

state_details of “FAILED_OVER”3. The SCAN VIP(s) that were running on the rebooted node will fail over to surviving nodes4. The SCAN Listener(s) running on that node will fail over to a surviving node5. Instance recovery is performed by another instance6. Services are moved to available instances7. Client connections are moved / reconnected to surviving instances. With TAF configured,

select statements should continue. Active DML will be aborted8. After the database reconfiguration, surviving instances continue processing their

workload

02/05/2023

8

Test 3: Restart Failed NodeProcedure• ajithpathiyil2:/home/oracle[RAC1]$ srvctl start instance –d RAC –I RAC1

Expected Results1. On clusters having 3 or fewer nodes, one of the SCAN VIPs and Listeners will be

relocated to the restarted node when the Oracle Clusterware starts2. The VIP will migrate back to the restarted node3. Services that had failed over as a result of the node failure will NOT

automatically be relocated4. Failed resources (ASM, listener, instance, etc) will be restarted by the

Clusterware

Measures5. Time for all resources to become available again, Check with “crsctl stat res –t”

9

Test 4: Restart All Nodes Same TimeProcedure• Issue a reboot on all nodes at the same time• For AIX, HPUX, Windows: ‘shutdown –r’• For Linux: ‘shutdown –r now’• For Solaris: ‘reboot’

Expected Results1. All nodes, instances and resources are restarted without problems

Measures2. Time for all resources to become available again, Check with “crsctl stat res –t”

10

Test 5: Unplanned Instance FailureProcedure• Start client workload, identify single database instance with the most client

connections and abnormally terminate that instance:

For AIX, HPUX, Linux, Solaris:Obtain the PID for the pmon process of the database instance:# ps –ef | grep pmonkill the pmon process:# kill –9 <pmon pid>

For Windows:Obtain the thread ID of the pmon thread of the database instance by running:SQL> select b.name, p.spid from v$bgprocess b, v$process p where b.paddr=p.addr and b.name=’PMON’;Run orakill to kill the thread:cmd> orakill <SID> <Thread ID>

11

Test 5: Unplanned Instance Failure (contd..)Expected Results1. One of the other instances performs instance recovery2. Services are moved to available instances, if a preferred instance failed3. Client connections are moved / reconnected to surviving instances (Procedure and

timings will depend on client types and configuration)4. After a short freeze, surviving instances continue processing the workload5. Failing instance will be restarted by Oracle Clusterware, unless this feature has been

disabled

Measures6. Time to detect instance failure7. Time to complete instance recovery. Check alert log for recovering instance8. Time to restore client activity to same level (assuming remaining nodes have sufficient

capacity to run workload)9. Duration of database freeze during failover.10. Time before failed instance is restarted automatically by Oracle Clusterware and is

accepting new connections

12

Test 6 :Planned Instance TerminationProcedure• Issue a ‘shutdown abort’

Expected Results1. One other instance performs instance recovery2. Services are moved to available instances, if a preferred instance failed3. Client connections are moved / reconnected to surviving instances (Procedure

and timings will depend on client types and configuration)4. The instance will NOT be automatically restarted by Oracle Clusterware due to

the user invoked shutdown.

Measures5. Time to detect instance failure.6. Time to complete instance recovery. Check alert log for recovering instance.7. Time to restore client activity to same level (assuming remaining nodes have

sufficient capacity to run workload).8. The instance will NOT be restarted by Oracle Clusterware due to the user

induced shutdown.

13

Test 7: Clusterware and FencingNode fencing is a general concept used by computer clusters to forcefully remove a malfunctioning node from it. This preventive technique is a necessary measure to make sure no I/O from malfunctioning node can be done, thus preventing data corruptions and guaranteeing cluster integrity.

Procedure• Start with a normal, running cluster with the database instances up and running• Monitor the logfiles for clusterware on each node. On each node, start a new window and

run the following command:

The network heartbeats are associated with a timeout called misscount, set from 11g Release 1to 30.

ajithpathiyil1:/home/oracle[+ASM1] $crsctl get css misscount30ajithpathiyil1:/home/oracle[+ASM1] $oifcfg getifbond0 192.168.78.51 global publicbond1 10.10.0.0 global cluster_interconnectAjithpathiyil1:]$ tail -f /u01/grid/oracle/product/11.2.0/grid_1/log/ajithpathiyil2/crsd/crsd.l*ajithpathiyil1:]$ tail -f /u01/grid/oracle/product/11.2.0/grid_1/log/‘hostname -s‘/cssd/ocssd.logajithpathiyil2:]$ ifconfig eth1 down

14

Test 7: Clusterware and Fencing (contd..)Expected ResultsFollowing this command, watch the logfiles you began monitoring in step 2 above. You should see errors in those logfiles and eventually (could take a minute or two, literally) you will observe one node reboot itself. If you used ifconfig to trigger a failure, then the node will rejoin the cluster and the instance should start automatically.

Alert Log[cssd(2864)]CRS-1612:Network communication with node rac1 (1) missing for 50% of timeout interval.Removal of this node from cluster in 14.920 seconds…[cssd(2864)]CRS-1610:Network communication with node rac1 (1) missing for 90% of timeout interval.Removal of this node from cluster in 2.900 seconds[cssd(2864)]CRS-1609:This node is unable to communicate with other nodes in the cluster and isgoing down to p reserve cluster integrity

More debugging information is written to the ocssd.bin process log file:[CSSD][1119164736](:CSSNM00008:)clssnmCheckDskInfo: Aborting local node to avoid splitbrain.Cohort of 1 nodes with leader 2, rac2, is smaller than cohort of 1 nodes led by node 1, rac1, based onmap type 2[CSSD][1119164736]###################################[CSSD][1119164736]clssscExit: CSSD aborting from thread clssnmRcfgMgrThread[CSSD][1119164736]###################################.

15

Test 8: Service FailoverProcedureCreate a Serviceajithpathiyil2:/home/oracle[RAC1]$ srvctl add service -d RAC -s svctest -r RAC1 -a RAC2 -P BASICajithpathiyil2:/home/oracle[RAC1]$ srvctl start service -d RAC -s svctestajithpathiyil2:/home/oracle[RAC1]$ srvctl status service -d RAC -s svctestService svctest is running on instance(s) RAC1ajithpathiyil2:/home/oracle[RAC1]$

Warning ! You should never directly change the SERVICE_NAMES init parameter on a RAC database!! This parameter is maintained automatically by the clusterware.

SQL> show userUSER is "SYS"SQL> select instance_name from v$instance;INSTANCE_NAME----------------RAC1SQL> shutdown abort;ORACLE instance shut down.SQL>

16

Test 9: Public Network failureProcedure• Unplug all network cables for the public network

NOTE: It is recommended NOT to use ifconfig to down the interface, this may lead to the address still being plumbed to the interface resulting in unexpected results.’

Expected ResultsCheck with “crsctl stat res –t”1. The ora.*.network and listener resources will go offline for the node.2. SCAN VIPs and SCAN LISTENERs running on the node will fail over to a

surviving node.

ajithpathiyil2:/home/oracle[grid]$ srvctl status scanSCAN VIP scan1 is enabledSCAN VIP scan1 is running on node ajithpathiyil2ajithpathiyil2:/home/oracle[grid]$ajithpathiyil2:/home/oracle[grid]$ srvctl status scan_listenerSCAN Listener LISTENER_SCAN1 is enabledSCAN listener LISTENER_SCAN1 is running on node ajithpathiyil2ajithpathiyil2:/home/oracle[grid]$

02/05/2023

17

Test 9: Public Network failure (contd..)3. The VIP for the node will fail over to a surviving node.4. The database instance will remain up but will be unregistered with the remote listeners.5. Database services will fail over to one of the other available nodes.6. If TAF is configured, clients should fail over to an available instance.

NODE VERSION=1.0 host=ajithpathiyil2 incarn=0 status=nodedown reason=public_nw_down timestamp=30-Aug-2009 01:56:12 reported=Sun Jan 30 01:56:13 CDT 2013NODE VERSION=1.0 host=ajithpahtiyil2 incarn=147028525 status=nodedown reason=member_leave timestamp=30-Aug-2009 01:57:19 reported=Sun Aug 30 01:57:20 CDT 2013

Measure1. Time to detect the network failure and relocate resources.

02/05/2023

18

Test 10: Interconnect Network FailureProcedure• Unplug all network cables for the interconnect networkNOTE: It is recommended NOT to use ifconfig to down the interface, this may lead to the address still being plumbed to the interface resulting in unexpected results

Expected resultsFor 11.2.0.2 and above:1. CSSD will detect split-brain situation and perform one of the following:2. In a two-node cluster the node with the lowest node number will survive.3. In a multiple node cluster the largest sub-cluster will survive.4. On the node(s) that is being evicted, a graceful shutdown of Oracle

Clusterware will be attempted. All I/O capable client processes will be terminated and all resources will be cleaned up. If process termination and/or resource cleanup does not complete successfully the node will be rebooted.

5. Assuming that the above has completed successfully, OHASD will attempt to restart the stack. In this case the stack will be restarted once the network connectivity of the private interconnect network has been restored.

Review the following logs:o $GI_HOME/log/<nodename>/alert<nodename>.logo $GI_HOME/log/<nodename>/cssd/ocssd.log

02/05/2023

19

Test 10: Interconnect Network Failure (Contd..)MeasureFor 11.2.0.2 and above:1. Oracle Clusterware will gracefully shutdown, should graceful shutdown fail

(due to I/O processes not being terminated or resource cleanup) the node will be rebooted

2. Assuming that the graceful shutdown of Oracle Clusterware succeeded, OHASD will restart the stack once network connectivity for the private interconnect has been restored

02/05/2023

20

Sample Cluster Callout Script#!/bin/ksh# # Author: Ajith Narayanan## http://oracledbascriptsfromajith.blogspot.com## Version 1.0## This callout script is extended to report/mail the affected weblogic services when any Oracle cluster event occurs.##umask 022FAN_LOGFILE=$ORACLE_HOME/racg/usrco/`hostname`_uptime.logEVENTLINE=$ORACLE_HOME/racg/usrco/`hostname`_eventline.logEVENTLINE_MID=$ORACLE_HOME/racg/usrco/`hostname`_eventline_mid.logMAIL_CONT=$ORACLE_HOME/racg/usrco/`hostname`_mail.logWEBLOGIC_DS=$ORACLE_HOME/racg/usrco/weblogic_dsecho $* "reported="`date` >> $FAN_LOGFILE &tail -1 $FAN_LOGFILE > $EVENTLINEawk '{for (f = 1; f <= NF; f++) { a[NR, f] = $f } } NF > nf { nf = NF } END {for (f = 1; f <= nf; f++) { for (r = 1; r <= NR; r++) { printf a[r, f] (r==NR ? RS : FS) } }}' $EVENTLINE > $EVENTLINE_MIDSER=`grep "service=" $EVENTLINE_MID|awk -F= '{print $2}'`DB=`grep "database=" $EVENTLINE_MID|awk -F= '{print $2}'`INST=`grep "instance=" $EVENTLINE_MID|awk -F= '{print $2}'`HOST=`grep "host=" $EVENTLINE_MID|awk -F= '{print $2}'`

02/05/2023

21

Sample Cluster Callout Script (contd..)STAT=`grep "status=" $EVENTLINE_MID|awk -F= '{print $2}'`if [ "$SER" != " " | "$DB" != " " | "$INST" != " " | "$HOST" != " " | "$STAT" != " " ]; thenif [ $STAT = nodedown ]; thencat $EVENTLINE_MID > $MAIL_CONTecho "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONTgrep -i "$DB_" $WEBLOGIC_DS >> $MAIL_CONTelif [ $STAT = up ]; thencat $EVENTLINE_MID > $MAIL_CONTecho "**============================SERVICES RESTORED===============================**" >> $MAIL_CONTgrep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONTelsecat $EVENTLINE_MID > $MAIL_CONTecho "**============================SERVICES AFFECTED===============================**" >> $MAIL_CONTgrep -i "$DB_" $WEBLOGIC_DS|grep "SERVICE_NAME=$SER" >> $MAIL_CONT#ficat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" ajithpathiyil@gmail.comfi#cat $MAIL_CONT| /bin/mail -s "Cluster $STAT event: $DB $INST $SER $HOST" ajithpathiyil@gmail.comfirm $EVENTLINE $EVENTLINE_MID $MAIL_CONT

22

Q&A

23

References• Oracle RAC Assurance Team:- Oracle

RACCheck

Thank youContact Me: ajithpathiyil@gmail.com