ORACLE 12C ACTIVE DATAGUARDFAST START FAILOVERTESTING AND FINDINGS

Boris Belous, Thomson Reuters7/27/2016

1

BACKGROUND TR – HA and DR technologies landscape FSFO – what is it and why do we need it?

2

TR DEPLOYMENT PATTERNS Definition:a deployment pattern is an implementation template that outlines which standard and vetted TR platform, storage, network and database technologies should be used and how they should be connected to and interplay with each other to deliver the specific functionality, high availability, disaster recovery, security and performance requirements of the pattern in question. In other words, a deployment pattern is concerned with how to deliver a service of certain quality. What are “Patterns” really? Where do they come from? Why do we need them?

3

A PATTERN COMES TO LIFE AS A SERVICEIMPLEMENTATION…

4

Give me:Oracle DB EE 12.1.0.2Physical SIP/POD v.X1 minute RTONear zero RPORolling outage capabilityPremium managed supportSecurity level 2Active/Passive10,000 IOPS200MBPS

Your database is served:

A UNIQUE NEED IS IDENTIFIED

5

Two unique use cases:• A currently DataGuarding application requires the databases to be replicated and have faster and/or automated failover capabilities between the sites.• A currently GoldenGated application requires the databases to be replicated and have near-zero-downtime failover capabilities between the sites.

ACTIVE DATAGUARDOBJECTIVE: BETTER DR RTO

6

Regular Failover

Fast Start Failover

Benefits:•Eliminate most time-consuming IM phases.•Speed up and error-proof the manual DBA tasks.

GOLDENGATE ACTIVE/PASSIVE / LOCAL SITEOBJECTIVE: SIMPLIFICATION AND COSTSAVINGS

7

Reconnect to Alternate Site

Fast Start Failover

Benefit:•Eliminate the cost and complexity of maintaining GoldenGate.

Key Question:Does DataGuard FSFO bring us sufficiently closer to data availability capabilities offered by GoldenGate?

TENTATIVE ROLLOUT SCHEDULE Multi-site GG (for concurrent data availability in multiple geographies) – NOT TARGETED Same-site GG (for HA) - TARGETED IN PHASE 1 Multi-site DG (for site disaster redundancy) –TARGETED IN PHASE 2 Same-site DG (for HA) – TARGETED IN PHASE 1

8

WHAT IS FAST START FAILOVER? A DataGuard failover mechanism It automates the failover process Monitors for and reacts to predefined conditions Optionally, re-instates the old standby into the new primary.

9

WHAT IS FAST START FAILOVER?

10

FSFO COMPONENTS DataGuard – the foundation Supported Protection Modes:

Maximum Availability – 10gR2 and later Maximum Performance – 11gR1 and later

Broker Keeps the configuration information Automates admin tasks

Flashback Database Need for re-instating the failed primary

Observer Lightweight OCI client built into the DGMGRL Can be on any other h/w and OS flavor / version The client Oracle version must be same or higher than the DB Primary job – monitor for failure condition and initiate a failover Secondary job – re-instate the new standby

11

FLASHBACK DATABASERECOMMENDATIONS Set the Flashback Recovery Area (FRA)

Set DB_RECOVERY_FILE_DEST_SIZE Set DB_RECOVERY_FILE_DEST

Set the flashback database as you would for its main intended purpose – CDP. Set DB_FLASHBACK_RETENTION_TARGET ALTER DATABASE FLASHBACK ON;

Make db_flashback_retention_target >= 60 Size the FRA appropriately.

Estimate Monitor disk space Monitor V$FLASHBACK_DATABASE_LOG and V$RECOVERY_FILE_DEST 12

FAILOVER RULES AND CONDITIONS Rules (if and only if all rules are true)

Observer is properly configured and running Observer and the standby both lose contact with the primary Observer is in contact with the standby Protection mode constraints are met Failover threshold timeout has elapsed

Configurable Conditions Datafile Offline (due to IO errors) Corrupted Controlfile Corrupted Dictionary Inaccessible Logfile (due to IO errors) Stuck Archiver 13

THE SETUP

14

OBSERVER PLACEMENT No “one solution fits all” answer, but depends on the number of factors:

How far apart are your sites? How isolated is the network between your sites? How is your application aligned to utilized the sites?

Equi-aligned One site designated as the “preferred” site

Official position from Oracle: the observer should be placed right alongside the application servers most likely to run in a primary capacity Lots of caveats here… 15

OBSERVER PLACEMENT – CONTINUED… Stand the Observer on a host dedicated to that purpose in one of the strategic data centers. Stand the Observer on a host different than either the primary or the standby hosts. Stand the Observer as close to the application servers(same datacenter, same domain, same trust zone) as possible. For cases of site application alignment, stand the Observer close to the application servers that would be used with the site that itself is more likely to play the primary role. Always enforce Broker thresholds longer than the round-trip-delay time (RTD) between the Observer site and the farthest of the DG sites. Having fulfilled the above three criteria and still having a choice of placement, stand the Observer as geographically far away as possible from all the DG sites. 16

ENABLING FAST START FAILOVER Set FSFO target:

Primary’s or Standby’s DGMGRL> EDIT DATABASE upup SET PROPERTY FastStartFailoverTarget = <site a or site b>; Set FSFO lag limit:

Primary’s or Standby’s DGMGRL> EDIT DATABASE upup SET PROPERTY FastStartFailoverLagLimit = <seconds>; Optionally, set some other database-wide parameters, such as ObserverReconnect, FastStartFailoverAutoReinstate, FastStartFailoverPmyShutdown and ObserverOverride. Set FSFO threshold:

Primary’s or Standby’s DGMGRL> EDIT CONFIGURATION SET PROPERTY FastStartFailoverThreshold = <seconds>; Enable FSFO:

Primary’s or Standby’s DGMGRL> enable fast_start failover; Observer’s > start observer;

17

BROKER CONFIGURATION show configuration verbose…

Protection Mode: MaxAvailabilityMembers:poc054a - Primary databasepoc054z - (*) Physical standby database

(*) Fast-Start Failover target

Properties:FastStartFailoverThreshold = '30'OperationTimeout = '30'TraceLevel = 'USER'FastStartFailoverLagLimit = '30'CommunicationTimeout = '180'ObserverReconnect = '0'FastStartFailoverAutoReinstate = 'TRUE'FastStartFailoverPmyShutdown = 'TRUE'BystandersFollowRoleChange = 'ALL'ObserverOverride = 'FALSE'ExternalDestination1 = ''ExternalDestination2 = ''PrimaryLostWriteAction = 'CONTINUE'

Fast-Start Failover: ENABLED

Threshold: 30 secondsTarget: poc054zObserver: boris-app-continunity-pocLag Limit: 30 seconds (not in use)Shutdown Primary: TRUEAuto-reinstate: TRUEObserver Reconnect: (none)Observer Override: FALSE 18

The protection mode

Number of seconds the observer attempts to reconnect to the primary database before initiating a fast-start failover

Second the standby is allowed to fall behind the primary in terms of redo applied, beyond which a fast-start failover will not be allowedHow often the observer establishes a new connection to the primary database

Causes the former primary database to be automatically reinstatedCauses the primary database to shut down if the primary has been STALLED for longer than FastStartFailoverThreshold

Allows failover to occur when the observer has lost connectivity to the primary, even if the standby has a healthy connection to the primary

BROKER CONFIGURATION – CONTINUED… show database verbose poc054a…Database - poc054a

Role: PRIMARYIntended State: TRANSPORT-ONInstance(s):poc054a1poc054a2

Properties:DGConnectIdentifier = 'poc054a'ObserverConnectIdentifier = ''LogXptMode = 'SYNC'RedoRoutes = ''DelayMins = '0'Binding = 'optional'MaxFailure = '0'MaxConnections = '1'ReopenSecs = '300'NetTimeout = '30'RedoCompression = 'DISABLE'LogShipping = 'ON'PreferredApplyInstance = ''ApplyInstanceTimeout = '0'ApplyLagThreshold = '0'

19

Maximum availability sync mode

BROKER CONFIGURATION – CONTINUED… show database verbose poc054acontinued…TransportLagThreshold = '0'

TransportDisconnectedThreshold = '30'ApplyParallel = 'AUTO'StandbyFileManagement = 'AUTO'ArchiveLagTarget = '1800'LogArchiveMaxProcesses = '4'LogArchiveMinSucceedDest = '1'DbFileNameConvert = ''LogFileNameConvert = '/n01/oradata1/poc054/POC054Z/onlinelog, /n01/oradata1/poc054/POC054A/onlinelog, /s01/oradata1/poc054/POC054Z/onlinelog, /s01/oradata1/poc054/POC054A/onlinelog'FastStartFailoverTarget = 'poc054z'InconsistentProperties = '(monitor)'InconsistentLogXptProps = '(monitor)'SendQEntries = '(monitor)'LogXptStatus = '(monitor)'RecvQEntries = '(monitor)'StaticConnectIdentifier(*)StandbyArchiveLocation(*)AlternateLocation(*)LogArchiveTrace(*)LogArchiveFormat(*)TopWaitEvents(*)(*) - Please check specific instance for the property value

20

Failover target – typically the opposite site’s database

BROKER CONFIGURATION – CONTINUED… show database verbose poc054z…Database - poc054z

Role: PHYSICAL STANDBYIntended State: APPLY-ONTransport Lag: 0 seconds (computed 0 seconds ago)Apply Lag: 0 seconds (computed 1 second ago)Average Apply Rate: 7.00 KByte/sActive Apply Rate: 753.00 KByte/sMaximum Apply Rate: 1.87 MByte/sReal Time Query: ONInstance(s):poc054z1 (apply instance)poc054z2

Properties:DGConnectIdentifier = 'poc054z'ObserverConnectIdentifier = ''LogXptMode = 'SYNC'RedoRoutes = ''DelayMins = '0'Binding = 'optional'MaxFailure = '0'MaxConnections = '1'ReopenSecs = '300'NetTimeout = '30'RedoCompression = 'DISABLE'LogShipping = 'ON'PreferredApplyInstance = ''ApplyInstanceTimeout = '0'ApplyLagThreshold = '0' 21

BROKER CONFIGURATION – CONTINUED… show database verbose poc054zcontinued…

TransportLagThreshold = '0'TransportDisconnectedThreshold = '30'ApplyParallel = 'AUTO'StandbyFileManagement = 'AUTO'ArchiveLagTarget = '1800'LogArchiveMaxProcesses = '4'LogArchiveMinSucceedDest = '1'DbFileNameConvert = ''LogFileNameConvert = '/n01/oradata1/poc054/POC054Z/onlinelog, /n01/oradata1/poc054/POC054A/onlinelog, /s01/oradata1/poc054/POC054Z/onlinelog, /s01/oradata1/poc054/POC054A/onlinelog'FastStartFailoverTarget = 'poc054a'InconsistentProperties = '(monitor)'InconsistentLogXptProps = '(monitor)'SendQEntries = '(monitor)'LogXptStatus = '(monitor)'RecvQEntries = '(monitor)'StaticConnectIdentifier(*)StandbyArchiveLocation(*)AlternateLocation(*)LogArchiveTrace(*)LogArchiveFormat(*)TopWaitEvents(*)(*) - Please check specific instance for the property value

22

BROKER CONFIGURATION – CONTINUED… show instance verbose poc054a1…Instance 'poc054a1' of database 'poc054a'

Host Name: makee-01PFILE:Properties:StaticConnectIdentifier = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.209.46.101)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=poc054a_DGMGRL.int.thomsonreuters.com)(INSTANCE_NAME=poc054a1)(SERVER=DEDICATED)))'StandbyArchiveLocation = '/n01/oraarch1/poc054a_standby/'AlternateLocation = '/n01/oraarch2/poc054a_standby/'LogArchiveTrace = '0'LogArchiveFormat = '%t_%s_%r.dbf'TopWaitEvents = '(monitor)'

23

show instance verbose poc054a2…Instance 'poc054a2' of database 'poc054a'

Host Name: makee-02PFILE:Properties:StaticConnectIdentifier =

'(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.209.46.102)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=poc054a_DGMGRL.int.thomsonreuters.com)(INSTANCE_NAME=poc054a2)(SERVER=DEDICATED)))'

StandbyArchiveLocation = '/n01/oraarch1/poc054a_standby/'AlternateLocation = '/n01/oraarch2/poc054a_standby/'LogArchiveTrace = '0'LogArchiveFormat = '%t_%s_%r.dbf'TopWaitEvents = '(monitor)'

Important to set the static connection identifier.HOST is set to the VIP address.SERVICE_NAME is set to the concatenated db_name+_DGMGRL string.

TEST 1 - LOSS OF THE PRIMARY INSTANCE(RAC) What was done:

A shutdown abort of a single instance was attempted.makee-01 {/home/oracle}: srvctl stop instance -d poc054a -i poc054a1 -o abort -f

The outcome: As expected, the primary database continued operating through the surviving instance.

24

TEST 2 - LOSS OF THE PRIMARY DATABASE(RAC) What was done:

A shutdown abort of all instances of the database was attempted.makee-01 {/home/oracle}: srvctl stop database -d poc054a -o abort

The outcome: The failover kicked in and completed in about 2 minutes, end to end. The old-primary-new-standby database went down and stayed down as expected. It required a manual startup:makee-01 {/home/oracle}: srvctl start database -d poc054aafter which the database was automatically reinstated in 2 minutes.

25

TEST 3 – LOSS OF THE CONTROLFILE What was done:

Writing garbage data info the live controlfile:echo "blah" > /n01/oradata1/poc054/POC054A/controlfile/o1_mf__102262659671_.ctl

The outcome: The failover kicked in and completed in about 2 minutes, end to end. The old-primary-new-standby database went down and stayed down as expected. It required a manual startup:cp -p /s01/oradata1/poc054/POC054A/controlfile/o1_mf__102262723422_.ctl /n01/oradata1/poc054/POC054A/controlfile/o1_mf__102262659671_.ctlsrvctl start database -d poc054aafter which the database was automatically reinstated in 3 minutes.

26

TEST 4 – STUCK ARCHIVER What was done:

Two simulations were attempted: 1: Revoke write privs on the archive dest:makee-01 {/n01/oraarch1}: chmod -w

/n01/oraarch1/poc054a2: Blocking IP access to the archive dest filer:iptables -I INPUT -s <filer_IP> -j DROP

The outcome:

27

Simulation Method

Primary Dest

Secondary Dest

Time taken to failover

FSFO failover

? CommentsBlock using iptable Block Active 25 Mins NO

Archive failover happens after 25 mins from primary to Secondary location and there was no impact on FSFO

Block using iptable Block Block N/A NO

Eternal loops between dest_1 and dest_4 and there was no impact on FSFO

Revoke write privilege Block Active 14 Mins NO

Archive failover happens after 14 mins from primary to secondary location and there was no impact on FSFO

Revoke write privilege Block Block N/A NO

Eternal loops between dest_1 and dest_4 and there was no impact on FSFO

Side note: patch 23148821 may be necessary to deal with MAX_FAILURE resets

TEST 5 – OFFLINE DATAFILE What was done:

alter system set "_datafile_write_errors_crash_instance"=false scope=both;rm/s01/oradata1/poc054/POC054A/datafile/o1_mf_test__12361943337127_.dbf

The outcome: The failover kicked in and completed in under 1 minute, end to end. The old-primary-new-standby database went down and stayed down as expected. The reinstatement of the database could not be done, due to the missing datafile, requiring a full reinstantiation. No automatic return to full fault tolerance! 28

TEST 6 – “MODIFIED” DATAFILE OFFLINE What was done:

Lazy unmount:umount –l /s01/oradata1

The Outcome: FSFO condition was not detected. Needed to manually execute:

DGMGRL> failover to poc054aFailover completed in 1 minute. Restoring the volume did allow for database mount and automatic reinstatement, which took about 2 minutes.mount –o <options> /s01/oradata1pavlik-01 {/home/oracle}: srvctl stop database -d poc054z -o abortcp -p /n01/oradata1/poc054/POC054A/controlfile/o1_mf__102262659671_.ctl /s01/oradata1/poc054/POC054A/controlfile/o1_mf__102262723422_.ctlpavlik-01 {/home/oracle}: srvctl start database -d poc054z 29

SIDELINE NOTES: HARDCOREEXPERIMENTATION What Oracle SMEs recommended to try:

1. MOUNT the failed Primary database.SQL> CONNECT / SQL> STARTUP MOUNTIf FSFO is enabled the Broker will try to reinstate but fail at this point.2. Connect with RMAN to the old Primary and setup encryption (because the manual says) RMAN> CONNECT TARGET "sysbackup@oldprimary_tns" AS SYSBACKUP";RMAN> SET ENCRYPTION ALGORITHM 'AES128';3. Restore the damaged/missing datafile from the New PrimaryRESTORE DATAFILE '/oradata/datafiles/whatever.dbf' FROM SERVICE newprimary_tns SECTION SIZE nnnM;4. Flash the newly fixed old primary back to the STANDBY_BECAME_PRIMARY_SCN retrieved from the New PrimarySQL> FLASHBACK DATABASE TO SCN standby_became_primary_scn;5. Convert it to a Physical StandbySQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;6. Fix the Database in the Broker configurationDGMGRL> CONNECT sysdg@newprimaryDGMGRL> DISABLE FAST_START FAILOVER FORCE;DGMGRL> REMOVE DATABASE <oldprimary>;DGMGRL> ADD DATABASE <oldprimary-newstandby> CONNECT IDENTIFIER IS <originalprimarytns>;Edit any database properties back to what they were originally and Reconfigure FSFO

The outcome:ORA-38791: flashback did not start because file 13 was not in a valid incarnation

30

TEST 7 – USER-DEFINED ORA- What was done:

DGMGRL> ENABLE FAST_START FAILOVER CONDITION 1578;DGMGRL> ENABLE FAST_START FAILOVER CONDITION 240;DGMGRL> SHOW FAST_START FAILOVER;…Oracle Error Conditions:

ORA-00240: control file enqueue held for more than %s secondsORA-01578: ORACLE data block corrupted (file # %s, block # %s)

Simulate ORA-1578: ORACLE data block corrupted (file # %s, block # %s) by using this command:alter session set events '16616 trace name context forever, level 283';

The Outcome: The failover kicked in and completed in about 2 minutes, end to end. The old-primary-new-standby database went down and stayed down as expected. It required a manual startup:

pavlik-01 {/home/oracle}: srvctl start database -d poc054zThe reinstatement did not occur automatically and required a manual kick-off:DGMGRL> reinstate database poc054z…completed in 2 minutes.

31

TEST 8 – APPLICATION INITIATED What was done:

cat appl_initiated_fsfo.sqlset serveroutput ondeclarestatus integer;beginstatus := dbms_dg.initiate_fs_failover('Failover Requested');

end;/ExitSQL> @appl_initiated_fsfo.sql

The Outcome:The failover kicked in and completed in about 1 minute, end to end. The old-primary-new-standby database went down and stayed down as expected. It required a manual startup:makee-01 {/home/oracle}: srvctl start database -d poc054aThe reinstatement did not occur automatically and required a manual kick-off:DGMGRL> reinstate database poc054a…completed in 2 minutes.

32

TEST 9 – INACCESSIBLE LOGFILE What was done:

Delete the online redo log files: makee-01 {/home/oracle/yas}: rm/n01/oradata1/poc054/POC054A/onlinelog/o1_mf_1__18637110189630_.log makee-01 {/home/oracle/yas}: rm/s01/oradata1/poc054/POC054A/onlinelog/o1_mf_1__18637117200858_.log makee-01 {/home/oracle/yas}: rm/n01/oradata1/poc054/POC054A/onlinelog/o1_mf_2__18637124324657_.log makee-01 {/home/oracle/yas}: rm/s01/oradata1/poc054/POC054A/onlinelog/o1_mf_2__18637131414786_.log

The Outcome:The failover kicked in and completed in about 2 minute, end to end. The old-primary-new-standby database went down and stayed down as expected. It required a manual startup:

makee-01 {/home/oracle}: srvctl start database -d poc054aManual reinstatement of the database failed, due to missing redo logs.… continued with the solution on next slide…

33

TEST 9 – INACCESSIBLE LOGFILE -CONTINUEDSQL> select first_change# from v$log where group#=2;

FIRST_CHANGE#-------------

102068695

SQL>SQL> select first_change# from v$log where group#=1;

FIRST_CHANGE#-------------

102068615Chose the larger of the two.Query the new primary for the logs of interest:SQL> select name from v$archived_log where FIRST_CHANGE#>=102068695;

NAME--------------------------------------------------------------------------------/n01/oraarch1/poc054z/2_1_910320360.dbf/n01/oraarch1/poc054z/1_71_910144956.dbf/n01/oraarch1/poc054z_standby/1_71_910144956.dbf/n01/oraarch1/poc054z/2_70_910144956.dbf/n01/oraarch1/poc054z_standby/2_70_910144956.dbf/n01/oraarch1/poc054z/1_1_910320360.dbf/n01/oraarch1/poc054z/2_2_910320360.dbf/n01/oraarch1/poc054z/1_2_910320360.dbf 34

TEST 9 – INACCESSIBLE LOGFILE -CONTINUEDTransferred the logs from the new primary to the new standby:pavlik-01 {/home/oracle}: scp /n01/oraarch1/poc054z/2_1_910320360.dbf oracle@makee-01:/n01/oraarch1/poc054a/… same for others…Cataloged the files:makee-01 {/home/oracle}: rman target /RMAN> catalog archivelog '/n01/oraarch1/poc054a/2_1_910320360.dbf';… same for others…Reinstated the standby manually:DGMGRL> reinstate database poc054aRefreshed the controlfile from the new primary to the new standby:SQL> alter database create standby controlfile as '/tmp/newstdbycntrl.ctl';scp /tmp/newstdbycntrl.ctl oracle@makee-01:/home/oraclepavlik-01 {/home/oracle}: scp /tmp/newstdbycntrl.ctl oracle@makee-01:/home/oraclemakee-01 {/home/oracle}: srvctl stop database -d poc054amakee-01 {/home/oracle}: cp /home/oracle/newstdbycntrl.ctl /n01/oradata1/poc054/POC054A/controlfile/o1_mf__102262659671_.ctlmakee-01 {/home/oracle}: cp /home/oracle/newstdbycntrl.ctl /s01/oradata1/poc054/POC054A/controlfile/o1_mf__102262723422_.ctlmakee-01 {/home/oracle}: srvctl start database -d poc054a

35

OVERALL CONCLUSIONS

36

OTHER READING Other companies’ HA, DR and MAA use cases and presentations:

http://www.oracle.com/technetwork/database/features/availability/ha-casestudies-098033.html Oracle MAA:

http://www.oracle.com/technetwork/database/features/availability/maa-096107.html Oracle MAA:

http://www.oracle.com/technetwork/database/availability/maa-roletransitionbp-2621582.pdf Other research:

http://www.oracle.com/technetwork/articles/smiley-fsfo-084973.html http://www.oracle.com/technetwork/database/availability/active-data-guard-wp-12c-1896127.pdf http://logicalread.solarwinds.com/increasing-oracle-availability-with-fast-start-failover-mc04/#.VwKfb_krIQ8 https://docs.oracle.com/database/121/DGBKR/sofo.htm#DGBKR390 https://docs.oracle.com/database/121/DGBKR/sofo.htm#DGBKR330 https://docs.oracle.com/database/121/HABPT/outage.htm#HABPT004 http://docs.oracle.com/database/121/DGBKR/dgmgrl.htm#DGBKR3434 SR 3-12345763911 : Fast Start Failover - Way to simulate "ORA-00240" and "Inaccessible Logfile" http://docs.oracle.com/database/121/DGBKR/dgmgrl.htm#BJEEIACG 37

Q&A38