11gR2 RAC to RAC Data Guard The initial setup is not much different between RAC-RAC and RAC-Single instance data guard, in a way setting up the data guard is same with some additional setp s at the end of setup in the way of bringing the standby database under the cont rol of clusterware. The primary RAC environment consists of two nodes, hostname rac4 and rac5. The p rimary database name is rac11g2 and two instances are rac11g21 (running on rac4 node) and rac11g22 (on rac5). The standby RAC environment also consists of two nodes (symmetrical data guard) hostname rac4b and rac5b. The standby database will be named rac11g2s and the co rresponding instances will be rac11g2s1 (on rac4b) and rac11g2s2 (on rac5b). The software version is 11.2.0.2 on both primary and standby (both grid home and oracle home). Primary site has database created and standby site has grid infra structure and oracle home software installed. 1. On the standby ASM instances would have been created when grid infrastructure was installed but not the ASM diskgroups. Using asmca create diskgroup on the s tandby ASM instances. In this case both primary and standby ASM diskgroups are n amed +DATA and +FLASH 2. Create following directory structures in all standby nodes cd $ORACLE_BASE mkdir admin cd admin/ mkdir rac11g2s cd rac11g2s/ mkdir adump dpdump hdump pfile 3. It is assumed primary db is in archivelog mode if not put the db in archive l og mode. This is required since duplication is done using active database. 4. Enable force logging on the primary database alter database force logging; 5.Create standby log files for each thread on primary db. These should be same s ize as the online redo log files SQL> alter database add standby logfile thread 1 size 52428800; or SQL> alter database add standby logfile thread 1; SQL> alter database add standby logfile thread 2; There should be at least one more redo log group per thread than the online redo logs. 6. Both on primary and standby add static listener entries in $GRID_HOME($CRS_HO ME)/network/admin/listener.ora. If scan listener is used for connections create static listener for those as well. One key difference is both in 10gR2 and 11gR1 the static listener and default listeners have hostname appended to the end but on 11gR2 this is not the case. Find out the available listeners with srvctl config listener Name: LISTENER

Network: 1, Owner: oracle Home: End points: TCP:1521 srvctl config scan_listener SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521 and create the static listeners for default listener as well as the scan listene r. For standby on rac4b more listener.ora LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_S CAN1)))) # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = rac11g2s.domain.net) (SID_NAME = rac11g2s1) (ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1) ) ) SID_LIST_LISTENER_SCAN1 = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = rac11g2s.domain.net) (SID_NAME = rac11g2s1) (ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1) ) ) on rac5b more listener.ora LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_S CAN1)))) # line added by Agent LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = rac11g2s.domain.net) (SID_NAME = rac11g2s2) (ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1) ) )

SID_LIST_LISTENER_SCAN1 = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = rac11g2s.domain.net) (SID_NAME = rac11g2s2) (ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1) ) ) In 11gR2 there's another file in $ORACLE_HOME/network/admin folder called endpoi nts_listener.ora. This is there for backward compatibility with pre-11.2 databas es. More on metalink note 11.2 Scan and Node TNS Listener Setup Examples [ID 107 0607.1] Stop and start the listeners on standby srvctl stop listener -n `hostname -s` srvctl start listener -n `hostname -s` srvctl stop scan_listener srvctl start scan_listener After this stop and start of listeners doing a listener status listener would sh ow the statically registered instance and service as below Service "rac11g2s.domain.net" has 1 instance(s). Instance "rac11g2s1", status UNKNOWN, has 1 handler(s) for this service... The command completed successfully Do the same for primary environment on rac4 more listener.ora LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_S CAN1)))) # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = rac11g2.domain.net) (SID_NAME = rac11g21) (ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1) ) ) SID_LIST_LISTENER_SCAN1 = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = rac11g2.domain.net) (SID_NAME = rac11g21) (ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1) ) ) on rac5

more listener.ora LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_S CAN1)))) # line added by Agent LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent SID_LIST_LISTENER = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = rac11g2.domain.net) (SID_NAME = rac11g22) (ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1) ) ) SID_LIST_LISTENER_SCAN1 = (SID_LIST = (SID_DESC = (GLOBAL_DBNAME = rac11g2.domain.net) (SID_NAME = rac11g22) (ORACLE_HOME = /opt/app/oracle/product/11.2.0/db_1) ) ) 7. Create TNS entries on both primary and standby. On all primary node's $ORACLE _HOME/network/admin RAC11G2SCANB = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = racb-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2s.domain.net) ) ) RAC11G2S = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac4b-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = rac5b-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2s.domain.net) ) ) RAC11G2S1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac4b-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2s.domain.net) (INSTANCE_NAME = rac11g2s1) )

) RAC11G2S2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac5b-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2s.domain.net) (INSTANCE_NAME = rac11g2s2) ) ) RAC11G2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = rac5-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2.domain.net) ) ) RAC11G21 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2.domain.net) (INSTANCE_NAME = rac11g21) ) ) RAC11G22 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac5-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2.domain.net) (INSTANCE_NAME = rac11g22) ) ) On all standby node's $ORACLE_HOME/network/admin RAC11G2SCAN = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2.domain.net) ) ) RAC11G2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = rac5-vip)(PORT = 1521)) (CONNECT_DATA =

(SERVER = DEDICATED) (SERVICE_NAME = rac11g2.domain.net) ) ) RAC11G21 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2.domain.net) (INSTANCE_NAME = rac11g21) ) ) RAC11G22 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac5-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2.domain.net) (INSTANCE_NAME = rac11g22) ) ) RAC11G2S = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac4b-vip)(PORT = 1521)) (ADDRESS = (PROTOCOL = TCP)(HOST = rac5b-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2s.domain.net) ) ) RAC11G2S1 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac4b-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2s.domain.net) (INSTANCE_NAME = rac11g2s1) ) ) RAC11G2S2 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = rac5b-vip)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = rac11g2s.domain.net) (INSTANCE_NAME = rac11g2s2) ) ) 8. Before modifying initialization parameters create a pfile from the spfile on the primary db

create pfile='/home/oracle/primaryinitpfile.ora' from spfile; 9. Modify the initialization parameters on the primary db. Remote archive dest s tate is set to defer, enable it before running the standby duplication. LGWR wit h ASYNC and NOAFFIRM is used as the redo transport this could be changed to suit e the protection mode. Also FAL_CLIENT parameter has been set to each instance. This could be set to RAC11G2 TNS entry which has all entries. alter system set log_archive_config='dg_config=(rac11g2,rac11g2s)' scope=both ; alter system set log_archive_dest_10='location=use_db_recovery_file_dest valid_f or=(all_logfiles,all_roles) db_unique_name=rac11g2' scope=both; alter system set log_archive_dest_1='service=RAC11G2S LGWR ASYNC NOAFFIRM max_fa ilure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) d b_unique_name=rac11g2s' scope=both; alter system set log_archive_dest_state_1='defer' scope=both; alter system set log_archive_dest_state_10='enable' scope=both; alter system set fal_server='RAC11G2S1','RAC11G2S2' scope=both; alter system set fal_client='RAC11G21' scope=both sid='rac11g21'; --- since not using scan alter system set fal_client='RAC11G22' scope=both sid='rac11g22'; --- sicne not using scan alter system set log_archive_max_processes=10 scope=both; alter system set db_file_name_convert='+DATA/rac11g2s','+DATA/rac11g2' scope=spf ile; alter system set log_file_name_convert='+FLASH/rac11g2s','+FLASH/rac11g2','+DATA /rac11g2s','+DATA/rac11g2' scope=spfile; alter system set standby_file_management='AUTO' scope=both; To make these changes take effect it requires a database restart not an instance restart this is due to the fact that filename convert parameter must be same on all nodes. If an instance restart is done it will fail with the following error on alert log NOTE: Loaded library: System SUCCESS: diskgroup DATA was mounted SUCCESS: diskgroup FLASH was mounted NOTE: dependency between database rac11g2 and diskgroup resource ora.DATA.dg is established NOTE: dependency between database rac11g2 and diskgroup resource ora.FLASH.dg is established ORA-1105 signalled during: ALTER DATABASE MOUNT /* db agent *//* {1:7917:278} */ ... Fri Feb 18 14:49:17 2011 Shutting down instance (abort) License high water mark = 1 USER (ospid: 9322): terminating the instance Instance terminated by USER, pid = 9322 Fri Feb 18 14:49:18 2011 Instance shutdown complete on the command shell $ srvctl stop instance -d rac11g2 -i rac11g21 $ srvctl start instance -d rac11g2 -i rac11g21 PRCR-1013 : Failed to start resource ora.rac11g2.db PRCR-1064 : Failed to start resource ora.rac11g2.db on node rac4 CRS-5017: The resource action "ora.rac11g2.db start" encountered the following e rror: ORA-01105: mount is incompatible with mounts by other instances

ORA-01677: standby file name convert parameters differ from other instance CRS-2674: Start of 'ora.rac11g2.db' on 'rac4' failed If the dataguard traffic is only one way (don't plan to use current primary as a standby in the future, then filename convert parameters could be ommited and a instnace restart could be carried out), otherwise do a database restart which me ans brining the system down. 10. Another difference between 10gR2/11gR1 and 11gR2 is the local_listener and r emote_listener parameters. According to Real Application Clusters Installation G uide During Oracle Database creation, the LOCAL_LISTENER parameter is automatica lly configured to point to the local listener for the database. The Database Age nt sets the LOCAL_LISTENER parameter to a connect descriptor that does not requi re a TNS alias. You can set a value manually for LOCAL_LISTENER. However, Oracle recommends that you leave the parameter unset so that the Database Agent can maintain it automa tically. If you set LOCAL_LISTENER, then the Agent does not automatically update this value. If you do not set LOCAL_LISTENER, then the Database Agent automatic ally keeps the database associated with the Grid home's node listener updated, e ven as the ports or IP of that listener are changed. With Oracle Clusterware 11g release 2 and later, Database Configuration Agent (D BCA) no longer sets the LOCAL_LISTENER parameter. The Oracle Clusterware agent t hat starts the database sets the LOCAL_LISTENER parameter dynamically, and it se ts it to the actual value, not an alias. So listener_alias entries are no longer needed in the tnsnames.ora file. For the REMOTE_LISTENER parameter, Oracle Clus terware uses the EZ connect syntax scanname:scanport, so no entries are needed f or the REMOTE_LISTENER parameter in the tnsnames.ora file.More on 11gR2 Grid Inf rastructure Single Client Access Name (SCAN) Explained [ID 887522.1] 11. On 11gr2 remote listener is set to scan identify the standby scan name with srvctl config scan SCAN name: racb-scan, Network: 1/192.168.0.0/255.255.255.0/eth0 SCAN VIP name: scan1, IP: /racb-scan/192.168.0.92 This will be used later on. 12. Copy the primary password file to standby nodes and rename accordingly. On r ac4b where instance rac11g2s1 will be running password file will be named orapwr ac11g2s1 and on rac5b orapwrac11g2s2. 13. Select one node (rac4b in this case) on the standby cluster and create a pfi le with db_name entry which will have the standby database name (not the instanc e name). Rename the pfile with the instance name. cat initrac11g2s1.ora db_name=rac11g2s 14. Start the instance in nomount mode export ORACLE_SID=rac11g2s1 SQL> startup nomount pfile='?/dbs/initrac11g2s1.ora'; ORACLE instance started. Total System Global Area 217157632 bytes Fixed Size 2225064 bytes Variable Size 159386712 bytes Database Buffers 50331648 bytes Redo Buffers 5214208 bytes

15. On primary db enable the deferred log archive destination SQL> alter system set log_archive_dest_state_1='enable' SCOPE=both sid='*'; 16. On one node in the primary rac (rac4 in this case) connect to primary instan ce (target) as well standby instance (using rman auxiliary connection) rman target / auxiliary sys/******@rac11g2s1 Recovery Manager: Release 11.2.0.2.0 - Production on Fri Feb 18 15:12:39 2011 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. connected to target database: RAC11G2 (DBID=379668842) connected to auxiliary database: RAC11G2S (not mounted) 17. Run the duplication command with duplicate target database for standby from active database spfile parameter_value_convert 'rac11g2','rac11g2s','RAC11G2','RAC11G2S' set db_unique_name='rac11g2s' set db_file_name_convert='+DATA/rac11g2','+DATA/rac11g2s' set log_file_name_convert='+FLASH/rac11g2','+FLASH/rac11g2s','+DATA/rac11g2','+D ATA/rac11g2s' set control_files='+DATA','+FLASH' set instance_number='1' set log_archive_max_processes='5' set fal_client='RAC11G2S' set fal_server='RAC11G21','RAC11G22' set remote_listener='racb-scan:1521' reset local_listener set log_archive_dest_1='service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_c onnections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name= rac11g2' set log_archive_dest_10='location=use_db_recovery_file_dest valid_for=(all_logfi les,all_roles) db_unique_name=rac11g2s'; Key parameters are data file and log file name converters as well as the paramet er value converter which maps primary environment values to appropriate standby environment values. local_listener is reset as it is auto set by grid infrastruc ture. remote_listener value is set the scan ip and port identified earlier. rman target / auxiliary sys/******@rac11g2s1 Recovery Manager: Release 11.2.0.2.0 - Production on Fri Feb 18 16:10:16 2011 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. connected to target database: RAC11G2 (DBID=379668842) connected to auxiliary database: RAC11G2S (not mounted) RMAN> duplicate target database for standby from active database 2> spfile 3> parameter_value_convert 'rac11g2','rac11g2s','RAC11G2','RAC11G2S' 4> set db_unique_name='rac11g2s' 5> set db_file_name_convert='+DATA/rac11g2','+DATA/rac11g2s' 6> set log_file_name_convert='+FLASH/rac11g2','+FLASH/rac11g2s','+DATA/rac11g2', '+DATA/rac11g2s'

7> set control_files='+DATA','+FLASH' 8> set instance_number='1' 9> set log_archive_max_processes='5' 10> set fal_client='RAC11G2S' 11> set fal_server='RAC11G21','RAC11G22' 12> set remote_listener='racb-scan:1521' 13> reset local_listener 14> set log_archive_dest_1='service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 m ax_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_n ame=rac11g2' 15> set log_archive_dest_10='location=use_db_recovery_file_dest valid_for=(all_l ogfiles,all_roles) db_unique_name=rac11g2s'; Starting Duplicate Db at 18-FEB-11 using target database control file instead of recovery catalog allocated channel: ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: SID=13 device type=DISK contents of Memory Script: { backup as copy reuse targetfile '/opt/app/oracle/product/11.2.0/db_1/dbs/orapwrac11g21' auxiliary fo rmat '/opt/app/oracle/product/11.2.0/db_1/dbs/orapwrac11g2s1' targetfile '+DATA/rac11g2/spfilerac11g2.ora' auxiliary format '/opt/app/oracle/product/11.2.0/db_1/dbs/spfilerac11g2s1.ora' ; sql clone "alter system set spfile= ''/opt/app/oracle/product/11.2.0/db_1/dbs/sp filerac11g2s1.ora''"; } executing Memory Script Starting backup at 18-FEB-11 allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=155 instance=rac11g21 device type=DISK Finished backup at 18-FEB-11 sql statement: alter system set spfile= ''/opt/app/oracle/product/11.2.0/db_1/db s/spfilerac11g2s1.ora'' contents of Memory Script: { sql clone "alter system set audit_file_dest = ''/opt/app/oracle/admin/rac11g2s/adump'' comment= '''' scope=spfile"; sql clone "alter system set dispatchers = ''(PROTOCOL=TCP) (SERVICE=rac11g2sXDB)'' comment= '''' scope=spfile"; sql clone "alter system set db_unique_name = ''rac11g2s'' comment= '''' scope=spfile"; sql clone "alter system set db_file_name_convert = ''+DATA/rac11g2'', ''+DATA/rac11g2s'' comment= '''' scope=spfile"; sql clone "alter system set log_file_name_convert = ''+FLASH/rac11g2'', ''+FLASH/rac11g2s'', ''+DATA/rac11g2'', ''+DATA/rac11g2s'' c omment= '''' scope=spfile"; sql clone "alter system set control_files = ''+DATA'', ''+FLASH'' comment= '''' scope=spfile";

sql clone "alter system set instance_number = 1 comment= '''' scope=spfile"; sql clone "alter system set log_archive_max_processes = 5 comment= '''' scope=spfile"; sql clone "alter system set fal_client = ''RAC11G2S'' comment= '''' scope=spfile"; sql clone "alter system set fal_server = ''RAC11G21'', ''RAC11G22'' comment= '''' scope=spfile"; sql clone "alter system set remote_listener = ''racb-scan:1521'' comment= '''' scope=spfile"; sql clone "alter system set log_archive_dest_1 = ''service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_connections=5 reopen=18 0 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2'' comment= '''' scope=spfile"; sql clone "alter system set log_archive_dest_10 = ''location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_uniqu e_name=rac11g2s'' comment= '''' scope=spfile"; shutdown clone immediate; startup clone nomount; } executing Memory Script sql statement: alter system set audit_file_dest = ''/opt/app/oracle/admin/rac1 1g2s/adump'' comment= '''' scope=spfile sql statement: alter system set dispatchers = ''(PROTOCOL=TCP) (SERVICE=rac11g 2sXDB)'' comment= '''' scope=spfile sql statement: alter system set db_unique_name = ''rac11g2s'' comment= '''' sc ope=spfile sql statement: alter system set db_file_name_convert = ''+DATA/rac11g2'', ''+D ATA/rac11g2s'' comment= '''' scope=spfile sql statement: alter system set log_file_name_convert = ''+FLASH/rac11g2'', '' +FLASH/rac11g2s'', ''+DATA/rac11g2'', ''+DATA/rac11g2s'' comment= '''' scope=spf ile sql statement: alter system set control_files = ''+DATA'', ''+FLASH'' comment= '''' scope=spfile sql statement: alter system set instance_number = 1 comment= '''' scope=spfile sql statement: alter system set log_archive_max_processes = 5 comment= '''' sc ope=spfile sql statement: alter system set fal_client = ''RAC11G2S'' comment= '''' scope= spfile sql statement: alter system set fal_server = ''RAC11G21'', ''RAC11G22'' commen t= '''' scope=spfile sql statement: alter system set remote_listener = ''racb-scan:1521'' comment= '''' scope=spfile

sql statement: alter system set log_archive_dest_1 = ''service=RAC11G2 LGWR AS YNC NOAFFIRM max_failure=10 max_connections=5 reopen=180 valid_for=(online_logfi les,primary_role) db_unique_name=rac11g2'' comment= '''' scope=spfile sql statement: alter system set log_archive_dest_10 = ''location=use_db_recove ry_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=rac11g2s'' commen t= '''' scope=spfile Oracle instance shut down connected to auxiliary database (not started) Oracle instance started Total System Global Area Fixed Size Variable Size Database Buffers Redo Buffers 630501376 bytes 2229120 247467136 373293056 7512064 bytes bytes bytes bytes

contents of Memory Script: { sql clone "alter system set control_files = ''+DATA/rac11g2s/controlfile/current.256.743447505'', ''+FLASH/rac11g2s/controlf ile/current.256.743447507'' comment= ''Set by RMAN'' scope=spfile"; backup as copy current controlfile for standby auxiliary format '+DATA/rac11g2s /controlfile/current.257.743447507'; restore clone controlfile to '+FLASH/rac11g2s/controlfile/current.257.743447507 ' from '+DATA/rac11g2s/controlfile/current.257.743447507'; sql clone "alter system set control_files = ''+DATA/rac11g2s/controlfile/current.257.743447507'', ''+FLASH/rac11g2s/controlf ile/current.257.743447507'' comment= ''Set by RMAN'' scope=spfile"; shutdown clone immediate; startup clone nomount; } executing Memory Script sql statement: alter system set control_files = ''+DATA/rac11g2s/controlfile/ current.256.743447505'', ''+FLASH/rac11g2s/controlfile/current.256.743447507'' c omment= ''Set by RMAN'' scope=spfile Starting backup at 18-FEB-11 using channel ORA_DISK_1 channel ORA_DISK_1: starting datafile copy copying standby control file output file name=/opt/app/oracle/product/11.2.0/db_1/dbs/snapcf_rac11g21.f tag=T AG20110218T161053 RECID=1 STAMP=743443854 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07 Finished backup at 18-FEB-11 Starting restore at 18-FEB-11 allocated channel: ORA_AUX_DISK_1 channel ORA_AUX_DISK_1: SID=19 instance=rac11g2s1 device type=DISK channel ORA_AUX_DISK_1: copied control file copy Finished restore at 18-FEB-11

sql statement: alter system set control_files = ''+DATA/rac11g2s/controlfile/ current.257.743447507'', ''+FLASH/rac11g2s/controlfile/current.257.743447507'' c omment= ''Set by RMAN'' scope=spfile Oracle instance shut down connected to auxiliary database (not started) Oracle instance started Total System Global Area Fixed Size Variable Size Database Buffers Redo Buffers 630501376 bytes 2229120 247467136 373293056 7512064 bytes bytes bytes bytes

contents of Memory Script: { sql clone 'alter database mount standby database'; } executing Memory Script sql statement: alter database mount standby database RMAN-05529: WARNING: DB_FILE_NAME_CONVERT resulted in invalid ASM names; names c hanged to disk group only. contents of Memory Script: { set newname for tempfile 1 to "+data"; switch clone tempfile all; set newname for datafile 1 to "+data"; set newname for datafile 2 to "+data"; set newname for datafile 3 to "+data"; set newname for datafile 4 to "+data"; set newname for datafile 5 to "+data"; set newname for datafile 6 to "+data"; set newname for datafile 7 to "+data"; backup as copy reuse datafile 1 auxiliary format "+data" datafile 2 auxiliary format "+data" datafile 3 auxiliary format "+data" datafile 4 auxiliary format "+data" datafile 5 auxiliary format "+data" datafile 6 auxiliary format "+data" datafile 7 auxiliary format

"+data" ; sql 'alter system archive log current'; } executing Memory Script executing command: SET NEWNAME renamed tempfile 1 to +data in control file executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME executing command: SET NEWNAME Starting backup at 18-FEB-11 using channel ORA_DISK_1 channel ORA_DISK_1: starting datafile copy input datafile file number=00002 name=+DATA/rac11g2/datafile/sysaux.257.74077004 7 output file name=+DATA/rac11g2s/datafile/sysaux.258.743447539 tag=TAG20110218T16 1124 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:25 channel ORA_DISK_1: starting datafile copy input datafile file number=00001 name=+DATA/rac11g2/datafile/system.256.74077004 5 output file name=+DATA/rac11g2s/datafile/system.259.743447625 tag=TAG20110218T16 1124 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:15 channel ORA_DISK_1: starting datafile copy input datafile file number=00003 name=+DATA/rac11g2/datafile/undotbs1.258.740770 049 output file name=+DATA/rac11g2s/datafile/undotbs1.260.743447699 tag=TAG20110218T 161124 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25 channel ORA_DISK_1: starting datafile copy input datafile file number=00005 name=+DATA/rac11g2/datafile/undotbs2.264.740770 355 output file name=+DATA/rac11g2s/datafile/undotbs2.261.743447725 tag=TAG20110218T 161124 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07 channel ORA_DISK_1: starting datafile copy input datafile file number=00004 name=+DATA/rac11g2/datafile/users.259.740770049 output file name=+DATA/rac11g2s/datafile/users.262.743447733 tag=TAG20110218T161 124 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01 channel ORA_DISK_1: starting datafile copy input datafile file number=00006 name=+DATA/rac11g2/datafile/test.268.741633047 output file name=+DATA/rac11g2s/datafile/test.263.743447733 tag=TAG20110218T1611 24 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01 channel ORA_DISK_1: starting datafile copy

input datafile file number=00007 name=+DATA/rac11g2/datafile/test.269.741633183 output file name=+DATA/rac11g2s/datafile/test.264.743447735 tag=TAG20110218T1611 24 channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01 Finished backup at 18-FEB-11 sql statement: alter system archive log current contents of Memory Script: { switch clone datafile all; } executing Memory Script datafile 1 switched to datafile copy input datafile copy RECID=1 STAMP=743447740 stem.259.743447625 datafile 2 switched to datafile copy input datafile copy RECID=2 STAMP=743447740 saux.258.743447539 datafile 3 switched to datafile copy input datafile copy RECID=3 STAMP=743447740 dotbs1.260.743447699 datafile 4 switched to datafile copy input datafile copy RECID=4 STAMP=743447740 ers.262.743447733 datafile 5 switched to datafile copy input datafile copy RECID=5 STAMP=743447740 dotbs2.261.743447725 datafile 6 switched to datafile copy input datafile copy RECID=6 STAMP=743447740 st.263.743447733 datafile 7 switched to datafile copy input datafile copy RECID=7 STAMP=743447740 st.264.743447735 Finished Duplicate Db at 18-FEB-11 file name=+DATA/rac11g2s/datafile/sy file name=+DATA/rac11g2s/datafile/sy file name=+DATA/rac11g2s/datafile/un file name=+DATA/rac11g2s/datafile/us file name=+DATA/rac11g2s/datafile/un file name=+DATA/rac11g2s/datafile/te file name=+DATA/rac11g2s/datafile/te

18. Once the above command suceeds on primary on the standby(rac4b node) run the log apply command SQL> alter database recover managed standby database using current logfile disco nnect; Database altered. On the database alert log it could be observed archived logs being applied Fri Feb 18 17:19:56 2011 alter database recover managed standby database using current logfile disconnect Attempt to start background Managed Standby Recovery process (rac11g2s1) Fri Feb 18 17:19:56 2011 MRP0 started with pid=60, OS id=8639 MRP0: Background Managed Standby Recovery process started (rac11g2s1) started logmerger process Fri Feb 18 17:20:01 2011 Managed Standby Recovery starting Real Time Apply Parallel Media Recovery started with 2 slaves Waiting for all non-current ORLs to be archived... All non-current ORLs have been archived. Fri Feb 18 17:20:02 2011 Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_1_seq_53.269.743

447835 Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_2_seq_43.268.743 447835 Completed: alter database recover managed standby database using current logfile disconnect Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_1_seq_54.271.743 447847 Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_2_seq_44.270.743 447847 Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_2_seq_45.273.743 447887 Media Recovery Log +FLASH/rac11g2s/archivelog/2011_02_18/thread_1_seq_55.272.743 447883 Media Recovery Waiting for thread 1 sequence 56 (in transit) Recovery of Online Redo Log: Thread 1 Group 6 Seq 56 Reading mem 0 Mem# 0: +DATA/rac11g2s/onlinelog/group_6.270.743447749 Mem# 1: +FLASH/rac11g2s/onlinelog/group_6.263.743447751 Media Recovery Waiting for thread 2 sequence 46 (in transit) Recovery of Online Redo Log: Thread 2 Group 9 Seq 46 Reading mem 0 Mem# 0: +DATA/rac11g2s/onlinelog/group_9.273.743447755 Mem# 1: +FLASH/rac11g2s/onlinelog/group_9.266.743447755 Another way to find out is to connect into the standby instance (rac11g2s1 runni ng on rac4b node) and issue SQL> select sequence#,thread#,applied from v$archived_log; SEQUENCE# THREAD# APPLIED ---------- ---------- --------53 1 YES 43 2 YES 54 1 YES 44 2 YES 55 1 YES 45 2 IN-MEMORY 6 rows selected. which shows YES on archive log applied column. 19. With above configuration inplace the data guard is setup and working. The ne xt step is to bring the standby database under the control of the cluster. For t his on the standby instance create a pfile from the spfile that was created duri ng the duplication process. SQL> create pfile='/home/oracle/stdbypfile.ora' from spfile; 20. Edit the pfile and remove any references to primary db (rac11g2) except db_n ame this must be same as primary db. Set instance specific instance_number, undo _tablespace, fal_client and thread. pfile content after the edit is given below rac11g2s1.__db_cache_size=373293056 rac11g2s1.__java_pool_size=4194304 rac11g2s1.__large_pool_size=4194304 rac11g2s1.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment rac11g2s1.__pga_aggregate_target=209715200 rac11g2s1.__sga_target=633339904 rac11g2s1.__shared_io_pool_size=0 rac11g2s1.__shared_pool_size=239075328 rac11g2s1.__streams_pool_size=0 rac11g2s2.__db_cache_size=373293056

rac11g2s2.__java_pool_size=4194304 rac11g2s2.__large_pool_size=4194304 rac11g2s2.__oracle_base='/opt/app/oracle'#ORACLE_BASE set from environment rac11g2s2.__pga_aggregate_target=209715200 rac11g2s2.__sga_target=633339904 rac11g2s2.__shared_io_pool_size=0 rac11g2s2.__shared_pool_size=239075328 rac11g2s2.__streams_pool_size=0 *.audit_file_dest='/opt/app/oracle/admin/rac11g2s/adump' *.audit_trail='db' *.cluster_database=true *.compatible='11.2.0.0.0' *.control_files='+DATA/rac11g2s/controlfile/current.257.743447507','+FLASH/rac11 g2s/controlfile/current.257.743447507'#Set by RMAN *.db_block_size=8192 *.db_create_file_dest='+DATA' *.db_domain='domain.net' *.db_file_name_convert='+DATA/rac11g2','+DATA/rac11g2s' *.db_name='rac11g2' *.db_recovery_file_dest='+FLASH' *.db_recovery_file_dest_size=9470738432 *.db_unique_name='rac11g2s' *.diagnostic_dest='/opt/app/oracle' *.dispatchers='(PROTOCOL=TCP) (SERVICE=rac11g2sXDB)' rac11g2s1.fal_client='RAC11G2S1' rac1g2s22.fal_client='RAC11G2S2' *.fal_server='RAC11G21','RAC11G22' rac11g2s1.instance_number=1 rac11g2s2.instance_number=2 *.log_archive_config='dg_config=(rac11g2,rac11g2s)' *.log_archive_dest_1='service=RAC11G2 LGWR ASYNC NOAFFIRM max_failure=10 max_con nections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_uniq ue_name=rac11g2' *.log_archive_dest_10='location=use_db_recovery_file_dest valid_for=(all_logfile s,all_roles) db_unique_name=rac11g2s' *.log_archive_dest_state_1='enable' *.log_archive_dest_state_10='enable' *.log_archive_format='%t_%s_%r.dbf' *.log_archive_max_processes=5 *.log_file_name_convert='+FLASH/rac11g2','+FLASH/rac11g2s','+DATA/rac11g2','+DAT A/rac11g2s' *.open_cursors=300 *.pga_aggregate_target=209715200 *.processes=150 *.remote_listener='racb-scan:1521' *.remote_login_passwordfile='exclusive' *.sga_target=631242752 *.standby_file_management='AUTO' rac11g2s1.undo_tablespace='UNDOTBS1' rac11g2s2.undo_tablespace='UNDOTBS2' rac11g2s1.thread=1 rac11g2s2.thread=2 21. Shutdown the database and startup mount and create a spfile with +ASM diskgr oup as the locaiton SQL>alter database recover managed standby database cancel; SQL>startup mount pfile='/home/oracle/stdbypfile.ora';

ORACLE instance started. Total System Global Area 630501376 bytes Fixed Size 2229120 bytes Variable Size 247467136 bytes Database Buffers 373293056 bytes Redo Buffers 7512064 bytes Database mounted. SQL> create spfile='+DATA/rac11g2s/spfilerac11g2s.ora' from pfile='/home/oracle/ stdbypfile.ora'; File created. SQL> shutdown immediate; 22. Create pfile file on each node's oracle home (ORACLE_HOME/dbs) that points t o the spfile by adding one line that referes to the spfile vi initrac11g2s1.ora spfile='+DATA/rac11g2s/spfilerac11g2s.ora' Above is for rac4b, create initrac11g2s2.ora on rac5b. 23. Once above pfiles are created remove the spfile in $ORACLE_HOME/dbs that was created as part of the duplication. 24. Before bringing the standby instance under cluster control check it could be started on all other nodes with their respective instances. In this case rac5b since this is a two node rac, export ORACLE_SID to rac11g2s2 and start the insta nce in mount mode. Resolve any issues (init parameter related) before proceeding to the next step) 25. To add the database and instances to the cluster configuration run $GIRD_HOME/bin/srvctl add database -d rac11g2s -o $ORACLE_HOME -m local -p "+DAT A/rac11g2s/spfilerac11g2s.ora" -n rac11g2 -r physical_standby -s mount $GIRD_HOME/bin/srvctl add instance -d rac11g2s -i rac11g2s1 -n rac4b $GIRD_HOME/bin/srvctl add instance -d rac11g2s -i rac11g2s2 -n rac5b Without the "-s mount" the standby database would be started in open (read only) mode by default (both R1 and R2). With 11g (both R1 and R2) it is possible to a pply redo while the database is open in read only mode. But this requires active dataguard license and setting the default startup mode to mount prevents such a ctivity from happenning when active dataguard is not licensed. On 11gR1 db insta nces could be made dependent on that node's asm instance but this option is depr ecated in 11gr2 $CRS_HOME/bin/srvctl modify instance -d rac11g2s -i rac11g2s1 -s +ASM1 -s option has been deprecated and will be ignored. srvctl config database -d rac11g2s Database unique name: rac11g2s Database name: rac11g2 Oracle home: /opt/app/oracle/product/11.2.0/db_1 Oracle user: oracle Spfile: +DATA/rac11g2s/spfilerac11g2s.ora Domain: local Start options: mount Stop options: immediate

Database role: PHYSICAL_STANDBY Management policy: AUTOMATIC Server pools: rac11g2s Database instances: rac11g2s1,rac11g2s2 Disk Groups: Mount point paths: Services: Type: RAC Database is administrator managed 26. Test the configuration with starting the entire standby database srvctl status database -d rac11g2s Instance rac11g2s1 is not running on node rac4b Instance rac11g2s2 is not running on node rac5b srvctl start database -d rac11g2s srvctl status database -d rac11g2s Instance rac11g2s1 is running on node rac4b Instance rac11g2s2 is running on node rac5b Once the database is started as a cluster resource the asm dependencies are auto matically added as well as the local_listener values which were reset during the duplication. crsctl stat res ora.rac11g2s.db -p NAME=ora.rac11g2s.db TYPE=ora.database.type .. SERVER_POOLS=ora.rac11g2s SPFILE=+DATA/rac11g2s/spfilerac11g2s.ora START_DEPENDENCIES=hard(ora.DATA.dg,ora.FLASH.dg)weak(type:ora.listener.type,glo bal:type:ora.scan_listener.type,uniform:ora.ons,global:ora.gns)pullup(ora.DATA.d g,ora.FLASH.dg) START_TIMEOUT=600 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(intermediate:ora.asm,shutdown:ora.DATA.dg,shutdown:ora.FL ASH.dg) .. SQL> show parameter local NAME TYPE VALUE --------------- ---------- -----------------------------local_listener string (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP(HO ST=192.168.0.98)(PORT=1521)))) 26. Start the recovery process on one of the standby instances. SQL> alter database recover managed standby database using current logfile disco nnect; This concludes the setup of RAC-RAC data guard. 27. To make sure archive logs don't get deleted on primary without being applied on standby set the log deletion policy to applied on standby. For this at least one log archive locations must be mandatory.

SQL> alter system set log_archive_dest_1='service=RAC11G2S LGWR ASYNC NOAFFIRM m ax_failure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_ro le) db_unique_name=rac11g2s mandatory' scope=both sid='*'; Usign rman chagne the archive log deletion policy. RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY; old RMAN configuration parameters: CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; new RMAN configuration parameters: CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON STANDBY; new RMAN configuration parameters are successfully stored If it is configured APPLIED ON ALL STANDBY then archived redo log files are elig ible for deletion after being applied or consumed on all remote destinations, wh ether mandatory or not. With mandatory destination inplace if the standby is not up and accepting redo w hen the primary starts following error PING[ARC1]: Heartbeat failed to connect to standby 'RAC11G2S'. Error is 12543. Archived Log entry 202 added for thread 1 sequence 80 ID 0x16a13366 dest 10: ARCH: Archival stopped, error occurred. Will continue retrying ORACLE Instance rac11g21 - Archival Error ORA-16038: log 2 sequence# 80 cannot be archived ORA-12543: TNS:destination host unreachable ORA-00312: online log 2 thread 1: '+DATA/rac11g2/onlinelog/group_2.262.740770163 ' ORA-00312: online log 2 thread 1: '+FLASH/rac11g2/onlinelog/group_2.258.74077016 3' and any alter system switch logfile commands will hang. Distributing redo load It is possible to distribute the redo load among all standby instnaces. In this case instead of sending the redo from all primary instance to one standby instan ce each primary instance will send its redo to corresponding standby instance. T his configuration is ideal in a symmetric data guard environment and one signle standby instnace cannot keep up with all the redo generated. To do this set prim ary instance specific log_archive_dest values alter system set log_archive_dest_1='service=RAC11G2S1 LGWR ASYNC NOAFFIRM max_f ailure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2s mandatory' scope=spfile sid='rac11g21'; alter system set log_archive_dest_1='service=RAC11G2S2 LGWR ASYNC NOAFFIRM max_f ailure=10 max_connections=5 reopen=180 valid_for=(online_logfiles,primary_role) db_unique_name=rac11g2s mandatory' scope=spfile sid='rac11g22'; Here rac11g21 will send its redo to rac11g2s1 and rac11g22 will send to rac11g2s 22. But there can only be one apply instance. Otherway around, sending all primary redo once standby instance gives more flexi bility than above one-to-one redo transport configuration. In this case if the c urrent standby apply instance need to be shutdown all that is required is stop t he apply service on that instance and start in another instance. Since there's n

o instance specific redo transport configuration no additional steps are require d. (provided the TNS entry has addressed for all standby instances similar to ra c11g2s TNS entry above).