Distinguished Product Manager - Oracle€¦ · Oracle Data Guard Page 13 of 76 Oracle Data Guard...
Transcript of Distinguished Product Manager - Oracle€¦ · Oracle Data Guard Page 13 of 76 Oracle Data Guard...
Data Guard Hands On Lab
Larry M. Carpenter Distinguished Product Manager
Oracle Data Guard
Page 2 of 76
Oracle Data Guard
Page 3 of 76
Data Guard Hands On Lab
Oracle Database 11g Release 2
MMMaaaxxxiiimmmuuummm AAAvvvaaaiiilllaaabbbiiillliiitttyyy AAArrrccchhhiiittteeeccctttuuurrreee (((MMMAAAAAA)))
OOOrrraaacccllleee BBBeeesssttt PPPrrraaaccctttiiiccceeesss FFFooorrr HHHiiiggghhh AAAvvvaaaiiilllaaabbbiiillliiitttyyy
Oracle Data Guard
Page 4 of 76
Oracle Data Guard
Page 5 of 76
INTRODUCTION.................................................................................................................................................... 7
OVERVIEW OF THE EXERCISES ..................................................................................................................... 9
SETUP AND CONFIGURATION ....................................................................................................................... 11
ORACLE DATA GUARD .................................................................................................................................... 13
CREATING A PHYSICAL STANDBY ......................................................................................................................... 14
Preparing the Primary Database..................................................................................................................... 14 Enabling Archiving ...................................................................................................................................... 14 Enable Force Logging .................................................................................................................................. 15 The Password File ........................................................................................................................................ 15 Adding Standby Redo Log Files .................................................................................................................. 16
Configuring OracleNet .................................................................................................................................... 17
Configure the Standby System ......................................................................................................................... 20
Creating the required directories .................................................................................................................. 20 Creating the Password File .......................................................................................................................... 21 Creating the Parameter File ......................................................................................................................... 21 Starting up the Standby Instance.................................................................................................................. 22
Creating the Physical Standby Database ........................................................................................................ 22
THE DATA GUARD BROKER ........................................................................................................................... 25
CONFIGURING THE BROKER .................................................................................................................................. 26 Broker Prerequisites ........................................................................................................................................ 26 Configuring the Listener .................................................................................................................................. 28
Create the Broker Configuration ..................................................................................................................... 31 Verifying the Broker Configuration ................................................................................................................. 33
Managing a Broker Configuration .................................................................................................................. 36
TRANSPORT SERVICES .................................................................................................................................... 43
MODIFYING THE TRANSPORT METHOD ................................................................................................................. 46 AUTOMATIC REDO GAP RESOLUTION ................................................................................................................... 48
PROTECTION MODES ....................................................................................................................................... 51
MAXIMUM PERFORMANCE .................................................................................................................................... 51
MAXIMUM AVAILABILITY..................................................................................................................................... 51 MAXIMUM PROTECTION ....................................................................................................................................... 51 CHANGING THE PROTECTION MODE ..................................................................................................................... 52
ROLE MANAGEMENT SERVICES .................................................................................................................. 53
SWITCHOVER ........................................................................................................................................................ 53 Performing a Switchover ................................................................................................................................. 53
FAILOVER ............................................................................................................................................................. 56
Enabling Flashback Database ......................................................................................................................... 56 Performing a Manual Failover ........................................................................................................................ 58 Performing a Manual Reinstate ....................................................................................................................... 60
FAST-START FAILOVER (FSFO) ............................................................................................................................ 63 Enabling Fast-Start Failover ........................................................................................................................... 64 Executing an Automatic Failover .................................................................................................................... 67
CONCLUSION ...................................................................................................................................................... 71
Oracle Data Guard
Page 6 of 76
RESOURCES ........................................................................................................................................................ 73
APPENDIX A – CREATE STANDBY SCRIPT ................................................................................................ 75
MAKENYC .......................................................................................................................................................... 75
Oracle Data Guard
Page 7 of 76
Introduction
Oracle Data Guard ensures high availability, data protection, and disaster recovery for enterprise data. Data
Guard provides a comprehensive set of services that create, maintain, manage, and monitor one or more standby
databases to enable production Oracle databases to survive disasters and data corruptions. These standby
databases are maintained as transaction consistent copies of the production database.
If the production database becomes unavailable because of a planned or an unplanned outage, Data Guard can
switch any standby database to the production role, minimizing the downtime associated with the outage. Data
Guard can be used with traditional backup, restoration, and cluster techniques to provide a high level of data
protection and data availability.
A Data Guard configuration consists of one production database and one or more standby databases. The
databases in a Data Guard configuration are connected by Oracle Net and may be dispersed geographically.
There are no restrictions on where the databases are located, provided they can communicate with each other.
For example, you can have a standby database on the same system as the production database, along with two
standby databases on other systems at remote locations.
A standby database can be:
Physical standby database
o Provides a physically identical copy of the primary database, with on disk database structures that
are identical to the primary database on a block-for-block basis. A physical standby database is
kept synchronized with the primary database, through Redo Apply, which recovers the redo data,
received from the primary database and applies the redo to the physical standby database.
Logical standby database
o Contains the same logical information as the production database, although the physical
organization and structure of the data can be different. The logical standby database is kept
synchronized with the primary database using SQL Apply, which transforms the data in the redo
received from the primary database into SQL statements and then executes the SQL statements on
the standby database. Some restrictions apply.
Active Data Guard standby database
o A Physical standby database that is open to read access with up-to-date data from the Primary
database. Ancillary writes are also possible but the writes must be redirected to a read write
database using database links.
Snapshot Standby database
o A fully read write standby database that is created by converting a physical standby database into a
Read write snapshot standby database.
o Receives and archives redo data from the primary database. A snapshot standby database does not
apply the redo data that it receives. The redo data is not applied until the snapshot standby is
converted back into a physical standby database, after first discarding any local updates made to
the snapshot standby database.
This handbook is your guide to the capabilities of Data Guard in Oracle Database 11g Release 2.
Oracle Data Guard
Page 8 of 76
Oracle Data Guard
Page 9 of 76
Overview of the Exercises
The exercises are set up to walk you through Oracle Database 11g Data Guard using SQL*Plus, RMAN and the
Broker working with a Physical standby (Redo Apply) database using Data Guard to create a standby for data
protection and to test the various features of Data Guard. The exercises are designed from the top down so that
each step builds upon the previous exercise. It is essential that you follow the exercises in order.
The following is an outline of the exercises that you will perform in Oracle Database 11g.
Create a Physical Standby Database
o Verify that data is getting into the Physical standby.
Configure the Data Guard Broker
Change the Transport mode using Broker Properties
Change the Protection mode to Maximum Availability
Perform a switchover from the Primary to the Standby
Enable Flashback database
Perform Manual Failover from the Primary to the standby.
Enable and use Fast-Start Failover.
Note: While these exercises are run on a single system for simplicity this is NOT the best practice when
implementing Data Guard.
As you go through these exercises you will find the commands that must be done for each step at each line
beginning with the word TASK.
The exercises assume that you have multiple terminal windows set up for SQL*Plus command, DGMGRL
commands or Linux commands. Each TASK line in the book list the commands (SQL*Plus, RMAN, DGMGRL
or Linux) that are to be executed in one of the windows, depending on the type of command. Any DGMGRL
commands are to be executed in your DGMGRL terminal window, SQL commands in your SQL*Plus terminal
window, etc.
For the first exercise, ‘Creating a Standby’, the RMAN command to actually create the standby database at the
end of the section is long so there is one script in the appendix of this book called CreateNYC that you will be
directed to execute at the appropriate time. All of the tasks leading up to this step have to be executed first.
Oracle Data Guard
Page 10 of 76
Oracle Data Guard
Page 11 of 76
Setup and Configuration
Install Oracle 11.2.0.3 and configure a database called SFO and enable archiving when it is created.
Oracle Data Guard
Page 12 of 76
Oracle Data Guard
Page 13 of 76
Oracle Data Guard
Oracle Data Guard provides the management, monitoring, and automation software infrastructure to create and
maintain one or more standby databases to protect Oracle data from failures, disasters, errors, and data
corruptions. There are two types of standby databases. A physical standby uses Redo Apply to maintain a block
for block, exact replica of the primary database. A logical standby uses SQL Apply and contains the same logical
information as the primary database, although the physical organization and structure of the data can be different.
Administrators can chose either manual or automatic failover of production to a standby system if the primary
fails in order to maintain high availability for mission critical applications. For the purposes of this lab, we will
concentrate on Physical Standby databases (Redo Apply).
A Data Guard configuration includes a production database, referred to as the primary database, and up to 30
standby databases. Primary and standby databases connect over TCP/IP using Oracle Net Services. There are no
restrictions on where the databases are located provided that they can communicate with each other. A standby
database is initially created from a backup copy of the primary database. Data Guard automatically synchronizes
the primary database and all of its standby databases by transmitting primary database redo - the information used
by Oracle to recover transactions - and applying it to the standby database.
Oracle Data Guard
Page 14 of 76
Creating a Physical Standby
To get started you will need to create your Physical standby database.
Preparing the Primary Database
There are some requirements that you need to meet before starting to create your Physical standby database.
Enabling Archiving
Enabling FORCE LOGGING
Creating the Password file
Adding Standby Redo Log files.
Enabling Archiving
Data Guard uses the redo (generated by the transactions and logged to the online redo log files by the LGWR
process) to maintain the standby database. As such you must enable archiving on your Primary database. In this
exercise, archiving is already enabled on SFO.
The archive logs are being archived to the Flash Recovery Area.
TASK:
In the SFO SQL*Plus window execute:
o sql
o archive log list
o show parameter recovery
SQL> archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 30
Next log sequence to archive 32
Current log sequence 32
SQL> show parameter log_archive_dest_1
NAME TYPE VALUE
------------------ ------- ------------------------------
log_archive_dest_1 string location=USE_DB_RECOVERY_FILE_
DEST
SQL> show parameter recovery
NAME TYPE VALUE
--------------------------- ----------- ----------------
db_recovery_file_dest string /u01/app/fast_recovery_area
db_recovery_file_dest_size big integer 8G
Oracle Data Guard
Page 15 of 76
Enable Force Logging
Force logging is a best practice since without the redo from the Primary database otherwise the datafiles on the
Physical standby will become unrecoverable and useless in a switchover or failover situation.
TASK:
In the SFO SQL*Plus window execute:
o select force_logging from v$database;
o alter database force logging;
o select force_logging from v$database;
The Password File
Data Guard uses Oracle Net sessions to transport redo data and control messages between the members of a Data
Guard configuration. These redo transport sessions are authenticated using either the Secure Sockets Layer (SSL)
protocol or a remote login password file. Your Data Guard configuration is configured to use a remote login
password file. Every physical standby database in the configuration must have an up-to-date copy of the
password file from the primary database which will be copied to the standby database during the creation process
later in this exercise.
Note: Whenever you grant or revoke the SYSDBA or SYSOPER privilege or change the login password of a
user who has these privileges, you must replace the password file at each physical or snapshot standby database
in the configuration with a fresh copy of the password file from the primary database.
TASK:
No task as the Primary already has a password file.
SQL> select force_logging from v$database;
FOR
---
NO
SQL> alter database force logging;
Database altered.
SQL> select force_logging from v$database;
FOR
---
YES
Oracle Data Guard
Page 16 of 76
Adding Standby Redo Log Files
Data Guard redo transport requires that the standby database have a set of standby redo log files also called the
SRL. An SRL is used to store redo received from the Primary (Chicago) database. If these standby redo log files
are created on the Primary before you create the standby database then they will automatically be added to the
standby database. In addition, having them created on the Primary database means that you are all set up to
function correctly when you perform a switchover. Standby redo log files are structurally identical to redo logs,
and are created and managed using the same SQL statements used to create and manage redo logs with the
‘STANDBY’ qualifier. Redo received from the Primary database via redo transport is written to the current
standby redo log group by a RFS background process. When a log switch occurs on the redo source database,
incoming redo is then written to the next standby redo log group, and the previously used standby redo log group
is archived by an ARCn background process on the standby. Oracle recommends that all redo log files in the
redo log at the redo source database and the standby redo log at a redo transport destination be of the same size.
The Primary has 50MB log files so you will create 50MB Standby redo log files. It is also a best practice for the
standby redo log to have at least one more redo log group per thread than the redo log on the Primary database.
TASK:
In the SFO SQL*Plus window execute:
o alter database add standby logfile size 50M;
o alter database add standby logfile size 50M;
o alter database add standby logfile size 50M;
o alter database add standby logfile size 50M;
This will create 4 standby redo log files (SRL) at the Primary SFO in the directory specified by the parameter
‘create_file_dest’, which in our case is the ‘oradata’ area. In addition a second copy will automatically be created
in the Fast Recovery Area (FRA).
You can examine the standby redo log files in the v$logfile view where the ‘TYPE’ is ‘STANDBY’.
SQL> alter database add standby logfile size 50M;
Database altered.
SQL> alter database add standby logfile size 50M;
Database altered.
SQL> alter database add standby logfile size 50M;
Database altered.
SQL> alter database add standby logfile size 50M;
Database altered.
Oracle Data Guard
Page 17 of 76
TASK:
In the SFO SQL*Plus window execute:
o select member from v$logfile where type=‘STANDBY’;
Configuring OracleNet
To enable your Primary database to talk to your standby database you must define a tnsnames entry on the
Primary system that connects to the standby database. In addition, to enable your standby database to talk to your
Primary database you must define a tnsnames entry on the standby system that connects to the Primary database.
These tnsnames connection descriptors will be used by Data Guard’s Redo Transport mechanism to send redo
from the Primary and to resolve any gaps in the redo when there is a connection failure.
To accomplish this you would first run ‘netmgr’ on your Primary system and add a tnsnames entry called ‘NYC’
pointing to your Standby system. You would then go to the Standby system and, again using ‘netmgr’, add a
tnsnames entry called ‘SFO’ pointing to your Primary system. Since both of your databases will reside on the
same system in this lab you would only need to add the entry for ‘NYC’ as ‘SFO’ already exists.
Since we are using the ‘LOCAL_LISTENER’ parameter in our databases it is also necessary to create tnsname
entries for our local listeners. These are called SFO_LISTENER and NYC_LISTENER.
SQL> select member from v$logfile where type = 'STANDBY';
MEMBER
---------------------------------------------------
/u01/app/oradata/SFO/onlinelog/o1_mf_4_6rppgb4b_.log
/u01/app/fast_recovery_area/SFO/onlinelog/o1_mf_4_6rppgb9z_.log
/u01/app/oradata/SFO/onlinelog/o1_mf_5_6rppghwg_.log
/u01/app/fast_recovery_area/SFO/onlinelog/o1_mf_5_6rppgj0r_.log
/u01/app/oradata/SFO/onlinelog/o1_mf_6_6rppgoqz_.log
/u01/app/fast_recovery_area/SFO/onlinelog/o1_mf_6_6rppgov1_.log
/u01/app/oradata/SFO/onlinelog/o1_mf_7_6rppgv6n_.log
/u01/app/fast_recovery_area/SFO/onlinelog/o1_mf_7_6rppgvb9_.log
8 rows selected.
Oracle Data Guard
Page 18 of 76
TASK:
At the Linux prompt execute the following command
o cat $ORACLE_HOME/network/admin/tnsnames.ora
NOTE: You will need to ensure that your TNSNAMES.ORA is setup correctly for your system.
bash-3.2$ cat $ORACLE_HOME/network/admin/tnsnames.ora
# tnsnames.ora Network Configuration File:
/u01/app/oracle/product/11.2.0/db_1/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.
SFO =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = SFO)
)
)
LISTENER_SFO =
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
NYC =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = NYC)
)
)
LISTENER_NYC =
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
Oracle Data Guard
Page 19 of 76
You will also need to add a static listener entry on the Standby system for the ‘NYC’ Physical standby database
so that RMAN will be able to connect to the instance (which will be started in a nomount state) and complete the
Physical standby database creation.
TASK:
At the Linux prompt execute the following command
o cat $ORACLE_HOME/network/admin/listener.ora
NOTE: You will need to ensure that your LISTENER.ORA is setup correctly for your system as above.
bash-3.2$ cat $ORACLE_HOME/network/admin/listener.ora
# listener.ora Network Configuration File:
/u01/app/oracle/product/11.2.0/db_1/network/admin/listener.ora
# Generated by Oracle configuration tools.
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = NYC)
(ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)
(SID_NAME = NYC)
)
)
LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
)
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = localhost)(PORT = 1521))
)
)
ADR_BASE_LISTENER = /u01/app
Oracle Data Guard
Page 20 of 76
Configure the Standby System
On the standby system you need to perform some setup before you can execute the RMAN script to create your
Physical Standby database using the new ‘over the network’ functionality.
Create directories for the Standby
o Audit dump directory and possibly the Data file directories.
Create a password file
Create an initialization parameter file
Startup the standby instance
Creating the required directories
RMAN will create the necessary dump directories for you with the exception of the Auditing directory as that
would normally be something that a DBA would be very careful about where they placed it. In your case you
will put it in the same path as the rest of the dump directories.
TASK:
At the Linux prompt execute the following command
o mkdir /u01/app/admin/NYC
o mkdir /u01/app/admin/NYC/adump
You also have to create the data file and log file directories because we are using the usual ‘oradata’ structure and
not using Oracle Managed Files (OMF). If the Primary and the Standby are both in ASM there would be no need
to create these directories; RMAN would create a NYC directory in ASM for you. In addition you need to create
the directories in the ‘Fast Recovery Area’ as well.
TASK:
At the Linux prompt execute the following commands
o mkdir /u01/app/oradata/NYC
o mkdir /u01/app/fast_recovery_area/NYC
o mkdir /u01/app/fast_recovery_area/NYC/archivelog
o mkdir /u01/app/fast_recovery_area/NYC/onlinelog
SQL> show parameter audit_file_dest
NAME TYPE VALUE
---------------- ----------- ------------------
audit_file_dest string /u01/app/admin/SFO/adump
bash-3.2$ mkdir /u01/app/admin/NYC
bash-3.2$ mkdir /u01/app/admin/NYC/adump
bash-3.2$ mkdir /u01/app/oradata/NYC
bash-3.2$ mkdir /u01/app/fast_recovery_area/NYC
bash-3.2$ mkdir /u01/app/fast_recovery_area/NYC/archivelog
bash-3.2$ mkdir /u01/app/fast_recovery_area/NYC/onlinelog
Oracle Data Guard
Page 21 of 76
Creating the Password File
Data Guard needs an exact copy of the Primary database’s password file unless you are using the SSL method of
authentication as mentioned earlier. Normally you would need to copy the password file from the Primary
system to the standby system in order to create your standby database.
With the new RMAN “FROM ACTIVE DATABASE” capability in Oracle Database 11g, you no longer need to
copy the password file to create the standby initially. You only need to create a password file on the standby
system in the parameter directory ‘$ORACLE_HOME/dbs’ that has the same password for the SYS user as the
Primary. RMAN will copy the password file from the Primary to the new Standby for you.
NOTE: Any subsequent changes to SYSDBA user passwords require that a new copy of the Primary
database’s password is copied to each Physical Standby in your configuration.
TASK:
Execute orapwd file=$ORACLE_HOME/dbs/orapwNYC password=oracle
Creating the Parameter File
Normally you would need to obtain a copy of the initialization parameter file from the Primary database and edit
it yourself before you could create the physical standby database. With the RMAN procedure you are going to
use this is no longer necessary. RMAN will obtain a copy of the Primary database’s initialization parameter file
copy it over and edit it for you. You can also perform any additional parameter editing in the RMAN script
without the need for a text initialization parameter file. The resulting spfile will be placed in the default
$ORACLE_HOME/dbs directory.
To get your standby database created you do need to create a text initialization parameter file for the standby but
it only needs to contain the DB_NAME parameter and it does not matter what value that parameter has. For
example, you could ‘cat’ a value for db_name into a temporary init.ora file.
As with the password file, the script you will execute does this task for you.
TASK:
Execute the following:
cat > /u01/app/oracle/product/11.1.0/db_1/dbs/tempini.ora <<EOF
db_name=anyname
EOF
As you will see when you execute the script, very few parameters actually need to be set as the bulk of the Data
Guard specific parameters will be set for you in the next exercise “Configuring the Data Guard Broker”.
However it is possible to set all the necessary parameters in the single RMAN script if you so choose. An
example of this is in Appendix A at the end of this book.
orapwd file=$ORACLE_HOME/dbs/orapwNYC password=oracle
cat > /u01/app/oracle/product/11.1.0/db_1/dbs/tempini.ora <<EOF
db_name=anyname
EOF
Oracle Data Guard
Page 22 of 76
Starting up the Standby Instance
For RMAN to attach to your future Physical standby database the instance must be started, but not mounted yet
as there is no control file nor any real set of parameters.
TASK:
Set your environment to NYC and execute the following:
sqlplus '/ as sysdba'
STARTUP PFILE='/u01/app/oracle/product/11.1.0/db_1/dbs/tempini.ora' NOMOUNT;
Creating the Physical Standby Database
Finally, the setup is complete. If you have performed all the tasks in this section then you are ready to execute
the script and create your standby database. As a recap, this script will.
Clean up in case this is a 2nd
run
Create temporary password file
o Must have same password as primary
Create the temporary parameter file
o With only the DB_NAME parameter set to an arbitrary name
Start up the auxiliary instance in nomount mode
Run RMAN to duplicate your Primary database as a Physical standby database
The setup phase of the script is as follows:
TASK:
Already done
sqlplus '/ as sysdba'
SQL> STARTUP PFILE='/u01/app/oracle/product/11.1.0/db_1/dbs/tempini.ora' NOMOUNT;
# Create temporary password file, must have same password as primary
orapwd file=$ORACLE_HOME/dbs/orapwNYC password=oracle
# Create temporary parameter file
cat > $ORACLE_HOME/dbs/tempini.ora <<EOF
db_name=anyname
EOF
# Start up the auxiliary instance nomount
sqlplus '/ as sysdba' <<EOF
SET ECHO ON
STARTUP PFILE='$ORACLE_HOME/dbs/tempini.ora' NOMOUNT;
EOF
Oracle Data Guard
Page 23 of 76
After those setup steps in your standby creation script, the RMAN script is executed. This script connects to the
Primary database ‘SFO’ (the TARGET) and the not yet created ‘NYC’ standby (the AUXILIARY), allocates
channels for parallelism, creates an in memory script of the spfile edits and as well as the backup procedure and
then executes its own script.
TASK:
Execute the above script from the Linux prompt
You will notice that the only parameters that are expressly set in the standby database are the bare minimum
required parameters and that there are no Data Guard specific parameters being set. This is done so that all of the
Data Guard parameters get created when you create the Data Guard Broker configuration in the next exercise.
One thing to note is that the RMAN ‘FROM ACTIVE DATABASE’ duplication method will place the SPFILE
of the standby in the normal ‘$ORACLE_HOME/dbs’ directory. If you wish you can move it into ASM with the
CREATE SPFILE command in SQL*Plus if you are using ASM.
# Run RMAN to duplicate your database.
rman <<EOF
set echo on
connect target sys/oracle@SFO; # Primary
connect auxiliary sys/oracle@NYC; # Standby: used by primary
run {
# Create channels for parallism
allocate channel prmy1 type disk;
allocate channel prmy2 type disk;
allocate channel prmy3 type disk;
allocate channel prmy4 type disk;
allocate channel prmy5 type disk;
allocate auxiliary channel stby1 type disk;
duplicate target database for standby from active database
spfile
parameter_value_convert 'SFO','NYC'
set 'db_unique_name'='NYC'
nofilenamecheck
;
}
EOF
exit 0
Oracle Data Guard
Page 24 of 76
When the script has completed you will see the following at the end of the output on your screen.
At this point you have a standby database but it is not receiving any redo from the Primary nor is it ready to apply
any redo. This is due to the fact that we did not set any of the Data Guard parameters in the Standby or in the
Primary yet. You can verify that the Physical Standby database is in fact running as a standby with SQL*Plus.
TASK:
Execute the following commands in your NYC SQL*Plus window
o . oraenv (Enter NYC if it is not already the default)
o sql
To complete the process manually you could now log into the Standby as above and set all the Data Guard
parameters and start Redo Apply. After that you would return to the Primary and setup the same parameters
including the redo transport parameters. Once complete you would switch logs on the Primary and redo would
begin being sent to the Physical standby database. However, in our case we will be using the Broker so we’ll
leave things as they are and let the Broker define everything for us in the next exercise.
You have successfully completed the “Creating a Physical Standby Database” exercise. The next step is to add in
a Data Guard Broker configuration and watch how the Broker takes care of all those parameters for you. No
need for more and more scripts!
RMAN>
Recovery Manager complete.
bash-3.2$ . oraenv
ORACLE_SID = [NYC] ? NYC
The Oracle base remains unchanged with value /u01/app
bash-3.2$ sql
SQL*Plus: Release 11.2.0.2.0 Production on Thu Mar 24 16:48:40 2011
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit…
With the Partitioning, OLAP, Data Mining and Real Application…
You are logged in as sys with sysdba privileges!
Unique Name Current Role Open Mode Protection Mode
---------------- ---------------- ------------ --------------------
NYC PHYSICAL STANDBY MOUNTED MAXIMUM PERFORMANCE
Oracle Data Guard
Page 25 of 76
The Data Guard Broker
The Oracle Data Guard Broker is a distributed management framework that automates and centralizes the
creation, maintenance, and monitoring of Data Guard configurations. The following list describes some of the
operations the broker automates and simplifies:
Creating Data Guard configurations that incorporate a primary database, one or more (physical, logical, or
snapshot) standby databases, redo transport services, and log apply services. Any of the databases could
be Oracle Real Application Clusters (RAC) databases.
Managing an entire Data Guard configuration, including all databases, redo transport services, and log
apply services, through a client connection to any database in the configuration.
Managing the protection mode for the broker configuration.
Invoking switchover or failover with a single command to initiate and control complex role changes
across all databases in the configuration.
Configuring failover to occur automatically upon loss of the primary database, increasing availability
without manual intervention.
Monitoring the status of the entire configuration, capturing diagnostic information, reporting statistics
such as the redo apply rate and the redo generation rate, and detecting problems quickly with centralized
monitoring, testing, and performance tools.
The Data Guard Broker logically groups primary and standby databases into a broker configuration that allows
the broker to manage and monitor them together as an integrated unit. Administrators may interact with the
Broker using either Enterprise Manager Grid Control or the Broker’s command-line interface (DGMGRL).
Enterprise Manager Grid Control includes wizards that further simplify the creation of a Data Guard
configuration. Key Data Guard metrics such as apply lag, transport lag, redo rate and configuration status, are
included in a new consolidated HA Console, shown below.
Oracle Data Guard
Page 26 of 76
Enterprise Manager enables historical trend analysis on the Data Guard metrics that it monitors - for example,
how the metric’s performance has been in the last 24 hrs, or last 5 days, etc. Also, through Enterprise Manager, it
is possible to set up notification-alarms such that administrators may be notified in case the metric crosses the
configured threshold value. Use of Enterprise Manager for Data Guard requires that you enable the Broker.
For the rest of the exercises we will be using the Data Guard Broker.
Configuring the Broker
For the purposes of these exercises you will be using the Command Line Interface called ‘dgmgrl’. To get your
Broker configuration created and enabled you will need to execute very few Broker commands as shown below.
Connect
Create configuration
Add database
Enable configuration;
However there are a few things that must be done before you can create and use a Broker configuration.
Broker Prerequisites
The following conditions must be met before you can use the Broker:
1. The primary and standby databases must be using the same version of Oracle Database
2. You must use a server parameter file (SPFILE)
3. Oracle Net Services network files must be set up on the primary database and on the standby database.
4. The value of the DG_BROKER_START initialization parameter must be set to TRUE
5. A service with a specific name must be statically registered with the local listener of each instance.
More information on these prerequisites can be found in Chapter 2 of the Data Guard Broker manual.
Oracle Data Guard
Page 27 of 76
The first 3 of the above prerequisites have already been met. The other two 2 prerequisites need to be met before
you can execute these commands and create your Data Guard Broker configuration. These two need to be met
for all Broker configurations.
To enable the Broker to create and manage the Data Guard configuration you must first enable the Data Guard
Broker background processes on each database in the configuration. This is done by setting the parameter
“dg_broker_start” to ‘TRUE” on both databases (SFO and NYC).
Before you enable the Broker you first need to check the placement of the Broker configuration files. These are
controlled by the parameters “dg_config_file1” and “dg_config_file2”. These files default to the
$ORACLE_HOME/dbs directory and we will leave them there. On our standby they are as follows.
For the Primary database SFO they would be in the same location but with SFO in the filename.
Normally you would modify these parameters so that both copies of the files on each system in the configuration
reside in different directories for redundancy. And if the database is a Real Application Cluster (RAC) they then
must be modified so that the files are visible across the entire cluster as there can only be one copy of each file
for the entire RAC.
TASK: (Optional)
Execute the following commands in each SQL*Plus window
o show parameter broker
SQL> show parameter broker
NAME TYPE VALUE
---------------------- ----------- ------------------------------
dg_broker_config_file1 string /u01/app/oracle/product/11.2.0
/db_1/dbs/dr1NYC.dat
dg_broker_config_file2 string /u01/app/oracle/product/11.2.0
/db_1/dbs/dr2NYC.dat
dg_broker_start boolean TRUE
SQL> show parameter broker
NAME TYPE VALUE
---------------------- ----------- ------------------------------
dg_broker_config_file1 string /u01/app/oracle/product/11.2.0
/db_1/dbs/dr1SFO.dat
dg_broker_config_file2 string /u01/app/oracle/product/11.2.0
/db_1/dbs/dr2SFO.dat
dg_broker_start boolean TRUE
Oracle Data Guard
Page 28 of 76
If you had not set these parameters correctly before enabling the Broker (as shown below) your Broker
configuration creation would fail and you would have to reset the ‘dg_broker_start” parameters back to FALSE
before you could change the file location parameters.
TASK:
Execute the following command in the SQL*Plus window of SFO and NYC
o alter system set dg_broker_start=TRUE scope=both;
Configuring the Listener
The second prerequisite is to configure a static listener entry on both systems so that the Broker can attach to any
of the databases in the configuration even when they are shutdown, as in a switchover operation or a restart and
automatic reinstatement of a failed Primary database in a Fast-Start Failover configuration. The Broker tries to
connect to a specific listener entry which is the database unique name with the string ‘_DGMGRL’ and the
domain, as in “SFO_DGMGRL.domain”. You can override this by modifying a Broker property called
‘StaticConnectIdentifier’ but for the purpose we will establish the default static entries in our listener.
Following Oracle best practices you would use the Network Manager utility (netmgr) to add these static entries to
the listener files at the Primary and Standby systems. In our case we’re on one system so we can just run it once
and add both entries.
TASK:
From the Linux prompt start ‘netmgr’ to make the changes to your listener file.
TASK:
Expand the configuration and select the LISTENER.
SQL> alter system set dg_broker_start=TRUE scope=both;
System altered.
SQL>
bash-3.2$ netmgr
Oracle Data Guard
Page 29 of 76
TASK:
Select the ‘Database Service’ drop down menu at the right.
TASK:
Click the ‘Add Database’ button.
TASK:
Add Database 2 with SFO_DGMGRL as the name and SFO as the SID.
Oracle Data Guard
Page 30 of 76
TASK:
Click ‘Add Database’ again
Add Database 3 with NYC_DGMGRL as the name and NYC as the SID
o Remember, if you were using two systems you would do this step on the Standby system
TASK:
Select ‘File’ from the menu and click ‘Exit’.
TASK:
And save the configuration changes by clicking the ‘Save’ button.
Oracle Data Guard
Page 31 of 76
The last thing to do is to reload the listener to enable the changes you just made.
TASK:
Execute the following command at the Linux prompt
o lsnrctl reload listener
Verify that the changes have been made by executing ‘lsnrctl status’. You should see an entry for both SFO and
NYC with the ‘_DGMGRL’ addition.
TASK:
Execute the following command at the Linux prompt
o lsnrctl status
Create the Broker Configuration
Now that the prerequisites have been met it is time to create the Data Guard Broker configuration.
TASK:
Execute the following commands in the DGMGRL window
o dgmgrl sys/oracle@SFO
You must connect to the Primary database to create a configuration.
bash-3.2$ lsnrctl reload listener
.
.
.
The command completed successfully
bash-3.2$ lsnrctl status
… <edited for clarity>
Service "NYC_DGMGRL" has 1 instance(s).
Instance "NYC", status UNKNOWN, …
Service "SFO_DGMGRL" has 1 instance(s).
Instance "SFO", status UNKNOWN, …
bash-3.2$ dgmgrl
DGMGRL for Linux: Version 11.2.0.2.0 - 64bit Production
Copyright (c) 2000, 2009, Oracle. All rights reserved.
Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/oracle@SFO
Connected.
Oracle Data Guard
Page 32 of 76
Next you create the configuration, identifying how the Broker should connect to the Primary database by
specifying a tnsnames descriptor that includes all of the Primary instances. In our case the database is a single
instance database so our tnsname points to one host.
TASK:
Execute the following commands in the DGMGRL window
o create configuration DG2011 as primary database is SFO connect identifier is SFO;
Next you add the standby database that we just created by specifying the tnsname for the standby. As with our
Primary you specify a tnsnames descriptor that includes all of the Standby instances. In our case the database is a
single instance database so our tnsname points to one host.
TASK:
Execute the following commands in the DGMGRL window
o add database NYC as connect identifier is NYC maintained as physical;
At this point the configuration is complete, a Primary database with our one standby database. But up to now all
we have done is update the Broker configuration files at the Primary. Until we enable the configuration, nothing
will be written to the Standby Broker configuration files nor will any Data Guard parameters be created. So the
next step is to enable the configuration. Once this step is complete Data Guard will be in full operation.
TASK:
Execute the following commands in the DGMGRL window
o enable configuration;
If you switch to the alert log windows where you should be tailing the alert log files, you can see the various
ALTER SYSTEM and ALTER DATABASE commands that the Broker executes to define all of the Data Guard
parameters, to start redo transport and start redo apply at the standby.
DGMGRL> create configuration DG2011
> as primary database is SFO
> connect identifier is SFO;
Configuration "dg2011" created with primary database "sfo"
DGMGRL>
DGMGRL> add database NYC as connect identifier is NYC
> maintained as physical;
Database "nyc" added
DGMGRL>
DGMGRL> enable configuration;
Enabled.
DGMGRL>
Oracle Data Guard
Page 33 of 76
Verifying the Broker Configuration
It will take a few minutes for the Broker configuration to be enabled and ready to go. You can check the Broker
configuration with the ‘SHOW CONFIGURATION’ command.
TASK:
Execute the following command in the DGMGRL window
o show configuration;
At this point the Broker has defined all the parameters that you would normally have to do manually. In addition
it has started redo transport from the Primary to the Standby in asynchronous mode and it has started the Redo
Apply process at the standby.
You can verify that the redo is being shipped and applied by using SQL*Plus on the standby to select data from
the ‘V$MANAGED_STANDBY’ view and display it, Primary first, standby second. In the output you will see
that there is an LNS process on the Primary database that is sending the current redo Sequence to our standby.
There would be multiple of ‘LNS’ lines if there were multiple standbys.
TASK:
Execute the following command in the SFO SQL*Plus window
o select client_process, process, sequence#, status from v$managed_standby;
DGMGRL> show configuration;
Configuration - dg2011
Protection Mode: MaxPerformance
Databases:
sfo - Primary database
nyc - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
SQL> select client_process,process,sequence#,status
2 from v$managed_standby;
CLIENT_P PROCESS SEQUENCE# STATUS
-------- --------- ---------- ------------
ARCH ARCH 43 CLOSING
ARCH ARCH 0 CONNECTED
ARCH ARCH 46 CLOSING
ARCH ARCH 47 CLOSING
LNS LNS 48 WRITING
Oracle Data Guard
Page 34 of 76
Performing the same command on the standby you will see the receiving RFS process of the redo and a process
called MRP that is coordinating the redo apply services.
TASK:
Execute the following command in the NYC SQL*Plus window
o select client_process, process, sequence#, status from v$managed_standby;
From this output you can see that the Primary is currently shipping sequence #48 and the standby is applying that
same sequence in real time apply mode. Real time apply is enabled because we created standby redo log files on
the primary database before we created the standby. Since they existed on the Primary, RMAN created them for
us on the Standby and the Broker automatically uses Real Time Apply when they exist.
You can examine the standby redo log files on NYC in the v$logfile view where the ‘TYPE’ is ‘STANDBY’.
TASK:
In the NYC SQL*Plus window execute:
o select member from v$logfile where type=‘STANDBY’;
SQL> select client_process,process,sequence#,status
2 from v$managed_standby;
CLIENT_P PROCESS SEQUENCE# STATUS
-------- --------- ---------- ------------
ARCH ARCH 47 CLOSING
ARCH ARCH 46 CLOSING
ARCH ARCH 0 CONNECTED
ARCH ARCH 44 CLOSING
N/A MRP0 48 APPLYING_LOG
LGWR RFS 48 IDLE
UNKNOWN RFS 0 IDLE
UNKNOWN RFS 0 IDLE
ARCH RFS 0 IDLE
SQL> select member from v$logfile where type = 'STANDBY';
MEMBER
---------------------------------------------------------------
/u01/app/oradata/NYC/onlinelog/o1_mf_4_6rqcggh4_.log
/u01/app/fast_recovery_area/NYC/onlinelog/o1_mf_4_6rqcgglm_.log
/u01/app/oradata/NYC/onlinelog/o1_mf_5_6rqcgnd7_.log
/u01/app/fast_recovery_area/NYC/onlinelog/o1_mf_5_6rqcgnhs_.log
/u01/app/oradata/NYC/onlinelog/o1_mf_6_6rqcgvmn_.log
/u01/app/fast_recovery_area/NYC/onlinelog/o1_mf_6_6rqcgvr3_.log
/u01/app/oradata/NYC/onlinelog/o1_mf_7_6rqch4d9_.log
/u01/app/fast_recovery_area/NYC/onlinelog/o1_mf_7_6rqch4hx_.log
8 rows selected.
Oracle Data Guard
Page 35 of 76
To see which standby redo log is currently being used you can query the view ‘v$standby_log’ and look for
‘ACTIVE’ in the ‘STATUS’ column.
TASK:
In the NYC SQL*Plus window execute:
o select group#, sequence#, status from v$standby_log;
So from the previous mixture of SQL*Plus and DGMGRL exercises you can see how the Broker simplifies
things as you only had to log in once and use the same window to examine both databases in your Data Guard
configuration. We will discuss Redo Transport and the various processes associated with it later.
SQL> select group#,sequence#,status from v$standby_log;
GROUP# SEQUENCE# STATUS
---------- ---------- ----------
4 0 UNASSIGNED
5 48 ACTIVE
6 0 UNASSIGNED
7 0 UNASSIGNED
Oracle Data Guard
Page 36 of 76
Managing a Broker Configuration
We can examine the details for each database in our configuration by typing ‘show database verbose <database
name>;’ which will show you the current status of the database and its current property settings. So for SFO we
would see the following (some properties edited out for space).
TASK:
In the DGMGRL window execute:
o show database verbose sfo;
DGMGRL> show database verbose sfo;
Database - sfo
Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s):
SFO
Properties:
DGConnectIdentifier = 'sfo'
ObserverConnectIdentifier = ''
LogXptMode = 'ASYNC'
DelayMins = '0'
…
ReopenSecs = '300'
NetTimeout = '30'
RedoCompression = 'DISABLE'
LogShipping = 'ON'
PreferredApplyInstance = ''
ApplyInstanceTimeout = '0'
ApplyParallel = 'AUTO'
StandbyFileManagement = 'MANUAL'
ArchiveLagTarget = '0'
LogArchiveMaxProcesses = '4'
LogArchiveMinSucceedDest = '1'
DbFileNameConvert = ''
LogFileNameConvert = ''
FastStartFailoverTarget = ''
InconsistentProperties = '(monitor)'
InconsistentLogXptProps = '(monitor)'
SendQEntries = '(monitor)'
LogXptStatus = '(monitor)'
RecvQEntries = '(monitor)'
…
TopWaitEvents = '(monitor)'
Database Status:
SUCCESS
Oracle Data Guard
Page 37 of 76
And for NYC we would see the following (some properties edited out for space).
TASK:
In the same DGMGRL window execute:
o show database verbose nyc;
DGMGRL> show database verbose nyc;
Database - nyc
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds
Apply Lag: 0 seconds
Real Time Query: OFF
Instance(s):
NYC
Properties:
DGConnectIdentifier = 'nyc'
ObserverConnectIdentifier = ''
LogXptMode = 'ASYNC'
DelayMins = '0'
…
ReopenSecs = '300'
NetTimeout = '30'
RedoCompression = 'DISABLE'
LogShipping = 'ON'
PreferredApplyInstance = ''
ApplyInstanceTimeout = '0'
ApplyParallel = 'AUTO'
StandbyFileManagement = 'MANUAL'
ArchiveLagTarget = '0'
LogArchiveMaxProcesses = '4'
LogArchiveMinSucceedDest = '1'
DbFileNameConvert = ''
LogFileNameConvert = ''
FastStartFailoverTarget = ''
InconsistentProperties = '(monitor)'
InconsistentLogXptProps = '(monitor)'
SendQEntries = '(monitor)'
LogXptStatus = '(monitor)'
RecvQEntries = '(monitor)'
…
TopWaitEvents = '(monitor)'
Database Status:
SUCCESS
Oracle Data Guard
Page 38 of 76
From your DGMGRL session you can and MUST effect all changes to your configuration. Once you start using
the Broker any changes you make to Data Guard parameters will be reset by the Broker to the last known value in
the Broker configuration. The list of ‘Properties’ you saw in the preceding exercise are mostly related to init.ora
parameters of the Oracle Database. For example, the Broker considers ‘log_archive_max_processes’ to be a
Data Guard parameter at this point so if we change it in SQL*Plus the Broker will report an error.
TASK:
In the SFO SQL*Plus window execute:
o show parameter log_archive_max_processes;
o alter system set log_archive_max_processes=6 scope=both;
At this point, when the Broker performs the next health check it will report a parameter inconsistency between
what it think the value should be and what it actually is in the database.
TASK:
In the DGMGRL window execute:
o show configuration
SQL> show parameter log_archive_max_processes
NAME TYPE VALUE
------------------------- ----------- -----------------------------
log_archive_max_processes integer 4
SQL> alter system set log_archive_max_processes=6 scope=both;
System altered.
DGMGRL> show configuration
Configuration - dg2011
Protection Mode: MaxPerformance
Databases:
sfo - Primary database
Warning: ORA-16792: configurable property value is
inconsistent with database setting
nyc - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
WARNING
Oracle Data Guard
Page 39 of 76
You can just tell the Broker to fix the problem blindly (which it will do if the database is restarted) or you can
examine the parameter that is ‘inconsistent’. There are a few ‘properties’ in the Broker that are flagged as
‘monitor’ properties. One of these is ‘InconsistentProperties’. The following output is edited.
TASK:
In the DGMGRL window execute:
o show database sfo InconsistentProperties;
To clear the error you can either edit the Broker configuration to set the Broker property value to 6 or you can tell
the Broker to set it back by using an ‘enable database’ command.
TASK:
In the DGMGRL window execute:
o enable database sfo;
o show configuration;
DGMGRL> show database sfo InconsistentProperties;
INCONSISTENT PROPERTIES
INSTANCE_NAME PROPERTY_NAME
SFO LogArchiveMaxProcesses
MEMORY_VALUE SPFILE_VALUE BROKER_VALUE
6 6 4
DGMGRL>
DGMGRL> enable database sfo;
Enabled.
DGMGRL> show configuration;
Configuration - dg2011
Protection Mode: MaxPerformance
Databases:
sfo - Primary database
nyc - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
Oracle Data Guard
Page 40 of 76
If you go to your alert log window you will see that the Broker reset the parameter to what it considers the correct
value based on the configuration setting.
TASK:
Examine the SFO alert log output.
Likewise, any changes to the state of the transport or apply services must be performed with DGMGRL otherwise
you will see errors when you monitor your configuration. For example, stopping the redo apply services using
SQL*Plus instead of the Broker will result in an error state.
TASK:
In the NYC SQL*Plus window execute:
o alter database recover managed standby database cancel;
SQL> alter database recover managed standby database cancel;
Database Altered
Thu Mar 31 06:05:20 2011
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='SFO';
ALTER SYSTEM SET log_archive_format='%t_%s_%r.dbf' SCOPE=SPFILE
SID='SFO';
ALTER SYSTEM SET standby_file_management='MANUAL' SCOPE=BOTH
SID='*';
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';
Thu Mar 31 06:05:20 2011
Shutting down archive processes
Thu Mar 31 06:05:20 2011
ARCH shutting down
ARC4: Archival stopped
Thu Mar 31 06:05:20 2011
ARCH shutting down
ARC5: Archival stopped
ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';
Oracle Data Guard
Page 41 of 76
This will stop the redo apply services but the Broker expects them to be running so it will display an error.
TASK:
In the DGMGRL window execute:
o show configuration;
A simple ‘ENABLE DATABASE’ command will put things back the way they were supposed to be just like it
did with the parameter we set earlier. Or you can use the ‘EDIT’ command to restart the apply services.
TASK:
In the DGMGRL window execute:
o edit database nyc set state=apply-on;
DGMGRL> show configuration;
Configuration - dg2011
Protection Mode: MaxPerformance
Databases:
sfo - Primary database
nyc - Physical standby database
Error: ORA-16766: Redo Apply is stopped
Fast-Start Failover: DISABLED
Configuration Status:
ERROR
DGMGRL>
DGMGRL> edit database nyc set state=apply-on;
Succeeded.
DGMGRL>
Oracle Data Guard
Page 42 of 76
As in the parameter exercise, you can see the Broker restarting the Redo Apply services on the standby NYC by
examining the alert log.
TASK:
Examine the NYC alert log output.
To recap, you should never make any changes to a Broker controlled Data Guard configuration using SQL*Plus.
These are called ‘Out of Band changes’ and will always be flagged by the Broker as errors. To change a
parameter or the ‘state’ of anything in the configuration you must always use the ‘EDIT’ command of DGMGRL
or Enterprise manager. In the above case, to stop the redo apply services on NYC you would execute ‘edit
database nyc set state=apply-off;’ from DGMGRL.
Now that you have created, managed and monitored your Data Guard configuration it is time to move on to
modifying the ‘Redo Transport’ method and ‘Protection Modes’.
Thu Mar 31 06:19:00 2011
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL
SWITCHOVER DISCONNECT USING CURRENT LOGFILE
Attempt to start background Managed Standby Recovery process (NYC)
Thu Mar 31 06:19:00 2011
MRP0 started with pid=29, OS id=11750
MRP0: Background Managed Standby Recovery process started (NYC)
started logmerger process
Thu Mar 31 06:19:05 2011
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 2 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Waiting for thread 1 sequence 80 (in transit)
Recovery of Online Redo Log: Thread 1 Group 5 Seq 80 Reading mem 0
Mem# 0: /u01/app/oradata/NYC/onlinelog/o1_mf_5_6rqcgnd7_.log
Mem# 1:
/u01/app/fast_recovery_area/NYC/onlinelog/o1_mf_5_6rqcgnhs_.log
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH
ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE
Oracle Data Guard
Page 43 of 76
Transport Services
Data Guard Redo Transport Services coordinate the transmission of redo from a primary database to the standby
database. At the same time that the primary database LGWR process is writing redo to its ORL, a separate Data
Guard process called the Network Server (NSS or NSA) is reading from the redo buffer in SGA and passing redo
to Oracle Net Services for transmission to the standby database. Data Guard’s flexible architecture allows a
primary database to transmit redo directly to a maximum of 30 standby databases. Data Guard is also well
integrated with Oracle RAC. Each primary instance that is active generates its own thread of redo and has its
own NSS/NSA process per standby database to transmit redo to that standby database.
Data Guard transport services transmit the redo to a standby database either synchronously or asynchronously,
where it is written to a standby redo log file (step one in the figure below).
The processes involved in redo transport are outlined in the next figure. Redo records transmitted by the
NSS/NSA are received at the standby database by another Data Guard process called the Remote File Server
(RFS). The RFS receives the redo at the standby database and writes it to a sequential file called a standby redo
log file (SRL). In a multi-standby configuration, the primary database has a separate NSS/NSA process that
manages redo transmissions for each standby database. In a configuration with three standby databases, for
example, three NSS/NSA processes are active on each primary database instance. As users commit transactions
at a primary database, Oracle generates redo records and writes them to a local online log file.
Oracle Data Guard
Page 44 of 76
Synchronous redo transport (SYNC) causes the primary database to wait for confirmation from the standby
database that redo has been hardened to disk before it will acknowledge commit success to the application -
providing zero data loss protection. Primary database performance is impacted by the sum of the time required
for the standby redo log file I/O to complete and network round-trip time.
Data Guard 11g Release 2 is designed to reduce the impact to primary performance of synchronous transport.
Redo is now transmitted to the remote standby in parallel with the local online log file I/O on the primary
database, effectively eliminating standby I/O from impacting total round trip time. This is shown in the figure
below as Step 1 where the NSS (Network Server Synchronous) process sends the redo at the same time the
LGWR is queuing the I/O to the Online Redo Log (ORL) file. Once the LGWR has the acknowledgement from
the disk I/O and the Standby round trip, Step 2, it will release the commit to the user, Step 3.
Oracle Data Guard
Page 45 of 76
This enables greater geographic separation between primary and standby databases in a synchronous zero data
loss configuration. On low latency networks it can reduce the impact of SYNC replication on primary database
performance to near zero, making it attractive to complement a remote ASYNC standby with a local SYNC
standby for zero data loss HA protection against component and database failures (SAN failure for example).
Asynchronous redo transport (ASYNC) avoids impact to primary database performance by having the primary
database acknowledge commit success to the application without waiting for acknowledgment that redo has been
received by the standby database. Generally there is no interaction between the NSA process and the LGWR
process other than startup. Data Guard 11g enhancements have virtually eliminated any impact on primary
database performance by shipping directly from the primary log buffer (instead of from an online redo log file),
as well as improving network throughput on high latency wide area networks (WAN). The performance benefit
of ASYNC, however, is accompanied by the potential for a small amount of data loss since there is no guarantee
that all redo has been received by the standby database.
Note: The asynchronous transport method is able to stream redo to a standby database so efficiently with no
impact to the Primary database that the old ‘ARCH’ method of shipping redo was deprecated in Oracle Database
11g and the default transport mode for Data Guard is ASYNC.
Oracle Data Guard
Page 46 of 76
Modifying the Transport Method
The transport method is shown in the Broker configuration by the “LogXptMode = 'ASYNC'” property
which was displayed when you did the ‘show database verbose’ commands on the two databases. Like the rest of
the properties associated with each database, the LogXptMode is how you specify how redo should be sent to that
standby. The odd thing about this property is that it is never defined on a Standby in reality; it is part of the
‘LOG_ARCHIVE_DEST_n’ that is setup by the Broker on the Primary database to send redo to the standby
database. This is true for all the properties that are actually attributes of the ‘LOG_ARCHIVE_DEST_n’
parameter definition.
DelayMins
LogXptMode
MaxConnections
MaxFailure
NetTimeout
RedoCompression
ReopenSecs
For the purposes of this exercise we are just going to concentrate on the LogXptMode property. To begin with
let’s look at the property as it is configured at present.
TASK:
In the DGMGRL window execute:
o show database sfo logxptmode;
o show database nyc logxptmode;
Since SFO is the Primary this property just shows how redo will be sent to SFO when it becomes a Standby. And
the property for NYC, our standby database, shows how the redo is currently being sent to NYC from SFO. You
can verify this by looking at the ‘LOG_ARCHIVE_DEST_n’ parameters as they are currently defined on SFO.
TASK:
In the SFO SQL*Plus window execute:
o show parameter log_archive_dest_2
DGMGRL> show database sfo logxptmode;
LogXptMode = 'ASYNC'
DGMGRL> show database nyc logxptmode;
LogXptMode = 'ASYNC'
DGMGRL>
SQL> show parameter 'log_archive_dest_2'
log_archive_dest_2 string service="nyc", LGWR ASYNC NOAF
FIRM delay=0 optional compress
ion=disable max_failure=0 max_
connections=1 reopen=300 db_un
ique_name="nyc" net_timeout=30
, valid_for=(all_logfiles,prim
ary_role)
Oracle Data Guard
Page 47 of 76
You can see the ‘LGWR ASYNC NOAFFIRM’ which corresponds to the current setting of ‘LogXptMode’ for
the standby ‘NYC, which is visible by the ‘service=”nyc” part of the parameter (the “nyc” is our tnsname we
created and entered as the connect identifier in exercise one). To change the transport mode you would edit the
property on the standby NYC, which will become effective immediately, and also for the Primary SFO, which
will become effective when SFO becomes the standby later on,. You want to change them both at the same time
otherwise you will end up with a configuration that is not the same after you switchover.
TASK:
In the DGMGRL window execute:
o edit database sfo set property logxptmode=‘SYNC’;
o edit database nyc set property logxptmode=‘SYNC’;
This change to NYC will become immediately visible on the Primary SFO.
You can see the results of the action by examining the alert log file of the Primary SFO.
TASK:
Examine the SFO alert log output.
Changing the transport mode to ‘SYNC’ is the first step to configuring a ‘Zero Data Loss (ZDL)’ Data Guard
configuration but by itself will not provide ZDL. You must also modify the Protection Mode, something we will
do right after we discuss the last part of Transport Services, Automatic Redo Gap Resolution.
DGMGRL> edit database sfo set property logxptmode='SYNC';
Property "logxptmode" updated
DGMGRL> edit database nyc set property logxptmode='SYNC';
Property "logxptmode" updated
DGMGRL>
SQL> show parameter 'log_archive_dest_2'
log_archive_dest_2 string service="nyc", LGWR SYNC AFFIR
M delay=0 optional compression
=disable max_failure=0 max_con
nections=1 reopen=300 db_uniqu
e_name="nyc" net_timeout=30, v
alid_for=(all_logfiles,primary
_role)
Thu Mar 31 09:48:16 2011
ALTER SYSTEM SET log_archive_dest_2='service="nyc"','LGWR SYNC
AFFIRM delay=0 optional compression=disable max_failure=0
max_connections=1 reopen=300 db_unique_name="nyc"
net_timeout=30','valid_for=(all_logfiles,primary_role)' SCOPE=BOTH;
ALTER SYSTEM SWITCH ALL LOGFILE start (SFO)
Thu Mar 31 09:48:16 2011
Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED
Oracle Data Guard
Page 48 of 76
Automatic Redo Gap Resolution
A log file gap occurs whenever a primary database continues to commit transactions while the NSS or NSA
process has ceased transmitting redo to the standby database. This can occur whenever the network or the standby
database is down, depending on how you have chosen to implement your Data Guard configuration (discussed in
the section “Protection Modes” next). While in this state, the primary database LGWR process continues writing
to the current ORL, fills it, and then switches to a new ORL while an archive (ARCH) process archives the
completed ORL locally. This cycle can repeat itself many times over on a busy system before the connection
between the primary and standby is restored, creating a large log file gap.
Once the standby becomes available again, the ARCH process that has been designated as the ‘Standby Ping
Process’ will discover it and, as long as the specified reopen time (specified by the ‘ReopenSecs’ property) has
passed, immediately begin resolving any missing archive log files that were not sent to that standby. In addition
it will force a log switch which will get the NSS/NSA process for that standby restarted and connected to the
standby so that the current redo stream can also begin to be shipped to the standby.
Oracle Data Guard
Page 49 of 76
The simplest way to create a redo gap is to shutdown the standby and then perform a couple of log switches at the
Primary database. These archive logs will not be able to be shipped to the standby and will be placed on the
‘Fetch Archive Log (FAL)’ queue to be shipped at a later time when the standby is reachable again. Before we
begin we will reduce the reopen time from the default of 5 minutes to 15 seconds to enable the gap to be resolved
faster.
TASK:
In the DGMGRL window execute:
o edit database sfo set property reopensecs=‘15’;
o edit database nyc set property reopensecs=‘15’;
Now, to create the gap we will abort our standby NYC and switch logs twice at the Primary SFO.
TASK:
In the NYC SQL*Plus window execute:
o shutdown abort
Then switch logs twice at the Primary database.
DGMGRL> edit database sfo set property reopensecs='15';
Property "reopensecs" updated
DGMGRL> edit database nyc set property reopensecs='15';
Property "reopensecs" updated
DGMGRL>
SQL> shutdown abort
ORACLE instance shut down.
SQL> select sequence# from v$log where status like 'CURRENT%';
SEQUENCE#
----------
86
SQL> alter system switch logfile;
System altered.
SQL> alter system switch logfile;
System altered.
SQL> select sequence# from v$log where status like 'CURRENT%';
SEQUENCE#
----------
88
SQL>
Oracle Data Guard
Page 50 of 76
TASK:
In the SFO SQL*Plus window execute:
o select sequence# from v$log where status like ‘CURRENT%’;
o alter system switch logfile;
o alter system switch logfile;
o select sequence# from v$log where status like ‘CURRENT%’;
We now have a gap where Sequences 86 and 87 have not been shipped nor has the current redo stream in
sequence 88. Within 60 seconds of starting the standby, the ARCH ping process will discover it is back and
immediately begin gap resolution as well as forcing a log switch so that sequence 89 becomes the current stream
and sequence 88 gets sent as a gap after 86 and 87.
TASK:
In the NYC SQL*Plus window execute:
o startup mount
When you look at the alert log for the standby you will see where those missing log files have been shipped and
applied and the current redo stream (89) is ‘(in transit)’. Your actual sequence numbers will vary.
TASK:
Examine the NYC alert log output.
Now that you understand how Redo transport works we will finish our modifications and change the Protection
Mode of our configuration to a Zero Data Loss mode.
SQL> startup mount
ORACLE instance started.
Total System Global Area 2438529024 bytes
Fixed Size 2228920 bytes
Variable Size 1358957896 bytes
Database Buffers 1056964608 bytes
Redo Buffers 20377600 bytes
Database mounted.
SQL>
/u01/app/fast_recovery_area/NYC/archivelog/2011_03_31/o1_mf_1_86_6s
951xwd_.arc
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH
ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE
Media Recovery Log
/u01/app/fast_recovery_area/NYC/archivelog/2011_03_31/o1_mf_1_87_6s
951xxn_.arc
Media Recovery Log
/u01/app/fast_recovery_area/NYC/archivelog/2011_03_31/o1_mf_1_88_6s
951y54_.arc
Media Recovery Waiting for thread 1 sequence 89 (in transit)
Oracle Data Guard
Page 51 of 76
Protection Modes
Data Guard provides three modes of data protection to balance cost, availability, performance, and data
protection. Each mode uses a specific redo transport method, and establishes rules that govern the behavior of the
Data Guard configuration should the primary database ever lose contact with its standby. The following table
outlines the characteristics of each mode.
DATA GUARD PROTECTION MODES
MODE RISK OF DATA LOSS TRANSPORT IF NO ACKNOWLEDGEMENT FROM THE STANDBY DATABASE, THEN:
Maximum Protection Zero data loss Double failure protection
SYNC Stall primary database until acknowledgement is received from the standby database
Maximum Availability Zero data loss Single failure protection
SYNC Stall primary database until acknowledgement is received or NET_TIMEOUT
threshold period expires – then resume processing
Maximum Performance Potential for minimal data loss
ASYNC Primary never waits for standby acknowledgment
Data Guard protection modes implement rules that govern how the configuration will respond to failures,
enabling you to achieve your specific objectives for data protection, availability, and performance. Data Guard
can support multiple standby databases in a single Data Guard configuration, and they may all have the same, or
different, protection mode setting, depending on your requirements. The different Data Guard protection modes
are Maximum Performance, Maximum Availability, and Maximum Protection.
Maximum Performance
This mode emphasizes primary database performance over data protection. It requires ASYNC redo transport so
that the LGWR process never waits for acknowledgement from the standby database. Primary database
performance and availability are not impacted by redo transport, by the status of the network connection between
primary and standby, or by the availability of the standby database.
Maximum Availability
This mode emphasizes availability as its first priority and zero data loss protection as a very close second priority.
It requires SYNC redo transport, thus primary database performance may be impacted by the amount of time
required to receive an acknowledgement from the standby that redo has been written to disk. SYNC transport,
however, guarantees 100-percent data protection during normal operation in the event that the primary database
fails. However, events that have no impact on the availability of the primary database can impact its ability to
transmit redo to the standby.
Maximum Protection
As its name implies, this mode places utmost priority on data protection. It has the same requirements as
Maximum Availability but the primary will not acknowledge a commit to the application unless it receives
acknowledgement from at least one standby database in the configuration that the data needed to recover that
transaction is safely on disk. If the primary does not receive acknowledgement from a SYNC standby database, it
will stall and eventually abort, preventing any unprotected commits from occurring.
Oracle Data Guard
Page 52 of 76
Changing the Protection Mode
As you can see from this chart, we started with our configuration in ‘Maximum Performance’ mode using the
ASYNC transport. We then changed our transport mode to SYNC but until we change the protection mode to
either ‘Maximum Availability’ or ‘Maximum Protection’ we are not running in a Zero Data Loss mode. If we
had to failover now we would lose data.
Reducing the protection mode has never required a restart of the Primary database but increasing the protection
mode in a Data Guard configuration in older versions required that you bounce the Primary database as the
Protection mode is always controlled from the Primary where the redo is being generated. Now the protection
mode can be elevated or reduced without downtime. If you were not using the Broker you could execute the
SQL*Plus command to raise the Protection mode to Maximum Availability even if you had not yet met the
requirements, you would just be running at the resynchronization level until the requirements had been met.
Moving to Maximum Protection does require that all the requirements are met before the protection mode can be
raised to the highest level otherwise the Primary database would be aborted due to violating the rules.
The Broker requires that the prerequisites for a Protection Mode have been met before it will allow you to change
it. In our case, to get to a zero data loss mode, we need to change the transport mode to SYNC, which we have
done, and we need to have Standby Redo Log files (SRL) at the standby, which we also have since we created
them on the Primary before we created our standby. All that is left is to edit our configuration.
TASK:
In the DGMGRL window execute:
o edit configuration set protection mode as MaxAvailability;
o show configuration;
DGMGRL> edit configuration set protection mode as MaxAvailability;
Succeeded.
DGMGRL> show configuration;
Configuration - dg2011
Protection Mode: MaxAvailability
Databases:
sfo - Primary database
nyc - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
Oracle Data Guard
Page 53 of 76
Role Management Services
The next area of Data Guard architecture we will discuss are role management services that enable the rapid
transition of a standby database to the primary database role. Data Guard documentation uses the term switchover
to describe a planned role transition, usually for the purpose of minimizing downtime during planned
maintenance activities. The term failover is used to describe a role transition in response to unplanned events.
In both cases, Data Guard Role Management Services quickly transition a designated standby database to the
primary role.
Switchover
A switchover is a planned operation used to reduce downtime during planned maintenance, such as operating
system or hardware upgrades, rolling upgrades of the Oracle database, and other database maintenance.
Regardless of the transport service (SYNC or ASYNC) or protection mode utilized, a switchover is always a zero
data loss operation.
Performing a Switchover
If you were not using the Broker then a switchover would be a combination of several commands on both
databases and a manual database restart of the original primary. The order of the process would be as follows:
1. On the Primary database
a. Shutdown all but one instance of the Primary
b. alter database commit to switchover to standby with session shutdown;
c. shutdown immediate
2. On the Standby Database
a. alter database commit to switchover to primary;
b. alter database open;
3. On the original Primary (now a standby)
a. startup mount
b. alter database recover managed standby database using current logfile disconnect;
c. Startup the other instances
The Broker simplifies all of that process for you down into on command “switchover to <database name>;”
DGMGRL> show configuration;
Configuration - dg2011
Protection Mode: MaxAvailability
Databases:
sfo - Primary database
nyc - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
Oracle Data Guard
Page 54 of 76
As far as the users are concerned, the switchover was finished when the Broker displayed “New primary database
"nyc" is opening...” as the new Primary is up and ready for business at that point. The rest of the time was spent
on getting the old primary up as a standby and Data Guard resynchronized. So, your primary database is now
NYC and SFO is your standby database as shown by another ‘show configuration’ command.
TASK:
In the DGMGRL window execute:
o show configuration;
o switchover to nyc;
o show configuration;
You can verify this by examining the role of the NYC database in your SQL*Plus window.
TASK:
In the NYC SQL*Plus window execute:
o select db_unique_name, database_role from v$database;
DGMGRL> switchover to nyc;
Performing switchover NOW, please wait...
New primary database "nyc" is opening...
Operation requires shutdown of instance "SFO" on database "sfo"
Shutting down instance "SFO"...
ORACLE instance shut down.
Operation requires startup of instance "SFO" on database "sfo"
Starting instance "SFO"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "nyc"
DGMGRL> show configuration;
Configuration - dg2011
Protection Mode: MaxAvailability
Databases:
nyc - Primary database
sfo - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
SQL> select db_unique_name, database_role from v$database;
DB_UNIQUE_NAME DATABASE_ROLE
--------------- ------------------
NYC PRIMARY
Oracle Data Guard
Page 55 of 76
Your original SQL*Plus connection to SFO has been lost because the Broker had to restart SFO as part of the
switchover process. So to examine the role of SFO you will have to connect again.
TASK:
In the SFO SQL*Plus window execute:
o select db_unique_name, database_role from v$database;
o connect sys/oracle as sysdba
o select db_unique_name, database_role from v$database;
To return to our original configuration, perform a 2nd
switchover and make SFO the Primary database again.
TASK:
In the DGMGRL window execute:
o switchover to sfo
Note: Don’t forget to reconnect your NYC SQL*Plus window!
SQL> select db_unique_name, database_role from v$database;
select db_unique_name, database_role from v$database
*
ERROR at line 1:
ORA-03135: connection lost contact
Process ID: 11282
Session ID: 9 Serial number: 5
SQL> connect sys/oracle as sysdba
Connected.
SQL> select db_unique_name, database_role from v$database;
DB_UNIQUE_NAME DATABASE_ROLE
--------------- ------------------
SFO PHYSICAL STANDBY
SQL>
DGMGRL> switchover to sfo;
Performing switchover NOW, please wait...
New primary database "sfo" is opening...
Operation requires shutdown of instance "NYC" on database "nyc"
Shutting down instance "NYC"...
ORACLE instance shut down.
Operation requires startup of instance "NYC" on database "nyc"
Starting instance "NYC"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "sfo"
Oracle Data Guard
Page 56 of 76
Failover
A failover brings a standby database online as the new primary database during an unplanned outage of the
primary database. As with a switchover, a failover operation does not require the standby database to be restarted
in order to assume the primary role. Also, as long as the database files on the original primary database are intact
and the database can be mounted, the original primary can be reinstated and resynchronized as a standby database
for the new primary using Flashback Database – it does not have to be restored from a backup.
Manual failover is initiated by the administrator using the Oracle Enterprise Manager GUI interface, the Data
Guard Broker’s command line interface, or directly through SQL*Plus. Optionally, Data Guard can perform
automatic failover in a very controlled manner using Fast-Start Failover (FSFO).
Enabling Flashback Database
Flashback Database is part of the Oracle Flashback technologies and is basically a ‘rewind’ button for an Oracle
database. For more information on the various flashback technologies including Flashback Database, please refer
to the Oracle documentation.
Flashback Database plays a very important part in any Data Guard configuration as it allows you to ‘reinstate’ a
failed Primary database as a standby database after a failover has occurred. In many cases it is not a complete
failure of the Primary databases that necessitates a failover. It might have been caused by power, system or site
failures but the actual database files are fine. Or it might have been caused by corruption to some of the data files
of the Primary but the rest of the database is not corrupted. In any case though, without Flashback Database
enabled on the Primary database prior to a failover operation, you would be required to completely rebuild a new
standby database from a backup of the new Primary (aka the old Standby). To reinstate a failed primary database
Data Guard requires 60 minutes of flashback retention time. If you plan on using Flashback Database for more
than just reinstating failed primary databases you can set the retention time to whatever you need.
Flashback database is not mandatory for Data Guard configurations (although strongly recommended) unless you
plan on using the automatic failover capability of the Broker (FSFO). If so, then it is mandatory on the Primary
and the failover target and you will not be able to enable FSFO without it. As FSFO is the last exercise in this lab
we will enable it now on both SFO and NYC.
First set the retention time to 60 minutes on both databases as the default retention period is 24 hours..
TASK:
In the both the SFO and NYC SQL*Plus windows execute:
o show parameter db_flashback_retention_target
o alter system set db_flashback_retention_target=60 scope=both;
SQL> show parameter db_flashback_retention_target
NAME TYPE VALUE
------------------------------ ----------- ------------------------
db_flashback_retention_target integer 1440
SQL> alter system set db_flashback_retention_target=60 scope=both;
System altered.
Oracle Data Guard
Page 57 of 76
Now we’re ready to enable Flashback Database on both of our databases. In previous versions of Oracle it was
required that the database be in the mount state to enable flashback. With Oracle Database 11g Release 2 (11.2)
this is no longer necessary.
TASK:
In the SFO SQL*Plus window execute:
o select flashback_on from v$database;
o alter database flashback on;
o select flashback_on from v$database;
However, the standby redo apply services must be stopped first on the standby before you can enable flashback
database. DGMGRL has a SQL command capability but it requires that your session is connected to the database
where you wish to execute the SQL command. Since we want to change the standby we will reconnect to NYC.
TASK:
In the DGMGRL window execute:
o connect sys/oracle@NYC
o edit database nyc set state=apply-off;
o sql “alter database flashback on”;
o edit database nyc set state=apply-on;
SQL> select flashback_on from v$database;
FLASHBACK_ON
------------------
NO
SQL> alter database flashback on;
Database altered.
SQL> select flashback_on from v$database;
FLASHBACK_ON
------------------
YES
DGMGRL> connect sys/oracle@NYC
Connected.
DGMGRL> edit database nyc set state=apply-off;
Succeeded.
DGMGRL> sql "alter database flashback on";
Succeeded.
DGMGRL> edit database nyc set state=apply-on;
Succeeded.
DGMGRL>
Oracle Data Guard
Page 58 of 76
To ensure that there is sufficient information in the Flashback logs both databases, switch log files at the Primary
database SFO.
TASK:
In the SFO SQL*Plus window execute:
o alter system switch logfile;
Now we are ready to perform a failover and force NYC to become the Primary database.
Performing a Manual Failover
As with Switchover, if you were not using the Broker then a failover would be several commands that you would
execute on the Standby. The order of the process would be as follows:
1. On the Primary database
a. Nothing, it’s gone!
2. On the Standby Database
a. alter database recover managed standby database finish;
b. alter database commit to switchover to primary;
c. alter database open;
The Broker simplifies all of that process for you down into on command “failover to <database name>”.
To simulate our failure we will abort the Primary and then failover to the standby.
TASK:
In the SFO SQL*Plus window execute:
o shutdown abort
Panic ensues! Production is down! Never fear, Data Guard is here! You can examine the state of your
configuration with the usual ‘show configuration’ command. The error will be a TNS error because the Broker
has lost its connection with the Primary database. Remember, in the last exercise we reconnected our DGMGRL
session to the standby NYC. If you forgot to do that then you will not be able to examine the state of the
configuration since SFO is down!
SQL> alter system switch logfile;
System altered.
SQL>
SQL> shutdown abort
ORACLE instance shut down.
SQL>
Oracle Data Guard
Page 59 of 76
TASK:
In the DGMGRL window execute:
o show configuration
This error could be due to the primary being down or just a network disconnect between the Standby and the
Primary. You have to make the decision whether or not to failover. And if you decide to manually failover you
have to be sure that the Primary is actually down and that you prevent it from starting up normally later on
otherwise you will have two Primary databases.
TASK:
In the DGMGRL window execute:
o failover to nyc;
You can verify this by examining the database role of NYC.
TASK:
In the NYC SQL*Plus window execute:
o select db_unique_name,database_role from v$database;
DGMGRL> show configuration
Configuration - dg2011
Protection Mode: MaxAvailability
Databases:
sfo - Primary database
nyc - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
ORA-12514: TNS:listener does not currently know of service
requested in connect descriptor
ORA-16625: cannot reach database "sfo"
DGM-17017: unable to determine configuration status
DGMGRL> failover to nyc;
Performing failover NOW, please wait...
Failover succeeded, new primary is "nyc"
DGMGRL>
SQL> select db_unique_name,database_role from v$database;
DB_UNIQUE_NAME DATABASE_ROLE
--------------- ------------------
NYC PRIMARY
Oracle Data Guard
Page 60 of 76
As with switchover, the failover is complete for the users as soon as the new Primary database is open for
business. In a failover there is no more work to be done since the original Primary database is gone. To get the
old Primary back now as a standby we will have to perform a manual ‘reinstate’.
Performing a Manual Reinstate
A manual reinstate is a sequence of commands that you execute on both the new Primary and the old Primary
using Flashback Database to rewind the failed primary database to a point just before the failover occurred. Then
it is converted into a Physical standby, restarted and the Redo Apply services restarted. At that point Data Guard
will reconnect and resolve the gaps in the redo until the configuration is synchronized again.
The important thing to remember is not to allow the old Primary database to startup normally. There is nothing
in place to stop it from opening read write. Fast-Start Failover is the only thing that can stop that from
happening. So our first step to reinstating the old Primary is to start it up in the MOUNT state.
TASK:
In the SFO SQL*Plus window execute:
o startup mount
Because we are using the Broker we really just have one command to put the old Primary back as a standby.
Looking at the configuration again you can see that the Broker is telling you to reinstate SFO.
SQL> startup mount
ORACLE instance started.
Total System Global Area 2438529024 bytes
Fixed Size 2228920 bytes
Variable Size 1526730056 bytes
Database Buffers 889192448 bytes
Redo Buffers 20377600 bytes
Database mounted.
DGMGRL> show configuration;
Configuration - dg2011
Protection Mode: MaxAvailability
Databases:
nyc - Primary database
Warning: ORA-16629: database reports a different protection
level from the protection mode
sfo - Physical standby database (disabled)
ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: DISABLED
Configuration Status:
WARNING
Oracle Data Guard
Page 61 of 76
TASK:
In the DGMGRL window execute:
o show configuration;
The ORA-16629 above is due to the fact that we are in Maximum Availability mode but have no SYNC standby
to support a Zero Data Loss mode so we are in error. The ORA-16661 tells us why. All that is left is to reinstate
SFO and then, when the configuration is synchronized again, switchover from NYC to SFO to return to our
original Production configuration.
TASK:
In the DGMGRL window execute:
o reinstate database sfo;
Once more show configuration to ensure that the configuration is resynchronized and then you can switchover
again returning SFO to the Primary status.
TASK:
In the DGMGRL window execute:
o show configuration;
DGMGRL> reinstate database sfo;
Reinstating database "sfo", please wait...
Operation requires shutdown of instance "SFO" on database "sfo"
Shutting down instance "SFO"...
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "SFO" on database "sfo"
Starting instance "SFO"...
ORACLE instance started.
Database mounted.
Continuing to reinstate database "sfo" ...
Reinstatement of database "sfo" succeeded
DGMGRL>
DGMGRL> show configuration
Configuration - dg2011
Protection Mode: MaxAvailability
Databases:
nyc - Primary database
sfo - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
Oracle Data Guard
Page 62 of 76
TASK:
In the DGMGRL window execute:
o switchover to sfo;
TASK:
In the DGMGRL window execute:
o show configuration;
Now that you have gotten this far into this lab book, it is time to take a walk on the wild side. If you thought that
we had a reason talking about the Broker in these exercise you are right. We wanted you to see how the Broker
can simplify the management of Data Guard. But we had an ulterior motive as well. And that is the subject of
our last exercise for this lab – Fast-Start Failover.
DGMGRL> switchover to sfo;
Performing switchover NOW, please wait...
New primary database "sfo" is opening...
Operation requires shutdown of instance "NYC" on database "nyc"
Shutting down instance "NYC"...
ORACLE instance shut down.
Operation requires startup of instance "NYC" on database "nyc"
Starting instance "NYC"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "sfo"
DGMGRL>
DGMGRL> show configuration
Configuration - dg2011
Protection Mode: MaxAvailability
Databases:
sfo - Primary database
nyc - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
Oracle Data Guard
Page 63 of 76
Fast-Start Failover (FSFO)
Over the years, database administrators have been confronted with the requirement of automating as much as
possible of their disaster recovery plans so that anyone on duty can engineer a move to the disaster recovery site
at the drop of a hat. Everyone is afraid that if only one person knows how to recover the database, that person
won’t be around when the failure happens. However, despite many people’s best efforts and no matter how
scripted you make the process, you still need people to get involved before the transition can take place. Once
people get involved, decisions have to be made, doubt is thrown into the mix, investigations occur, phone calls
happen, and, usually, errors are made. These are not things you want to happen when your databases are down
and your business is on hold. That is why Data Guard implemented Fast-Start Failover. And Fast-Start Failover
requires the Broker. Fast-Start Failover (FSFO) removes the people from the equation. That is not as harsh as it
sounds. It means that when there is a problem, there is no room for error, and people make mistakes, especially
when they are stressed. It also means that you do not end up relying on a script or document that was written by
someone who may no longer work for the company.
FSFO allows Data Guard to automatically fail over to a previously chosen, standby database without requiring
manual intervention to invoke the failover. A Data Guard Observer process continuously monitors the status of a
Fast-Start Failover configuration. If both the Observer and the standby database lose connectivity to the primary
database, the Observer attempts to reconnect to the primary database for a configurable amount of time before
initiating a fast-start failover. Fast-start failover is designed to ensure that out of the three fast-start failover
members - the primary, the standby and the Observer - at least two members agree to major state transitions to
prevent split-brain scenarios from occurring. Once the failed primary is repaired and mounted, it must establish
connection with the Observer process before it can open. When it does, it will be informed that a failover has
already occurred and the original primary is automatically reinstated as a standby of the new primary database.
The simple, yet elegant architecture of fast-start failover makes it excellent for use when both high availability
and data protection is required.
The architecture employed by FSFO is a third-member quorum that ensures that the failover occurs only when
everything meets the rules you have defined, and the failed primary is never allowed to open read write after a
failover to avoid any chance of the split brain scenario. The main two members are the primary database and the
standby database, of course, running in a normal configuration setup as we have setup in this lab.
.
The third member is called the Observer, and its job is to maintain a connection with the primary and the target
standby, monitoring the health of the configuration and performing the failover when required. The Observer is
also responsible for performing the automatic reinstatement of the failed primary database when it comes back
online, if possible. It has rules on where it should be placed but it does not require an Oracle license nor does it
have to be on the same platform or operating system as the databases.
The first thing you need to do is decide where you are going to put the Observer. In our case since we only have
one system we’re going to put it in its own window in the ‘Observer’ workspace of your desktop. In reality you
would want to place it on a 3rd
system with redundant network paths to the Primary and the Standby.
Oracle Data Guard
Page 64 of 76
Enabling Fast-Start Failover
Now that you have decided where you are going to place the Observer and prepared the system for it, you can
start setting up FSFO. You must meet several prerequisites before setting up FSFO, most of which we have
implemented already:
Use the broker with all of its prerequisites. Done!
Enable Flashback Database on both the primary and the standby. Done!
Set up the configuration correctly for the protection mode. Done!
Standby redo log files on both sides. Done!
Redo transport setup the same in both directions. Done!
Configure the Broker for FSFO. Not Done!
Install the Observer system and configure TNSNAMES. Not Done!
The first thing we can set is the ‘FastStartFailoverTarget’ property for both databases so that no matter which
database is the Primary the other is the FSFO Target standby. Since we only have one standby this is automatic
but we will execute the commands anyway.
TASK:
In the DGMGRL window execute:
o edit database sfo set property FastStartFailoverTarget = nyc;
o edit database nyc set property FastStartFailoverTarget = sfo;
We will also set the ‘FastStartFailoverThreshold’ to enable our failover to happen faster. The actual setting for
this property depends on your configuration (if the Primary is a RAC you have to add cluster miscount and
reconfiguration time to the threshold) and how long you want the Primary to stall before aborting and a failover
occurring. There are other properties that relate to FSFO but we’ll just accept the defaults for them.
TASK:
In the DGMGRL window execute:
o edit configuration set property FastStartFailoverThreshold=30;
DGMGRL> edit database sfo set property FastStartFailoverTarget = nyc;
Property "faststartfailovertarget" updated
DGMGRL> edit database nyc set property FastStartFailoverTarget = sfo;
Property "faststartfailovertarget" updated
DGMGRL>
DGMGRL> edit configuration set property FastStartFailoverThreshold=30;
Property "faststartfailoverthreshold" updated
DGMGRL>
Oracle Data Guard
Page 65 of 76
Now we are ready to enable Fast-Start Failover although it will not be ready to go until we start the observer.
TASK:
In the DGMGRL window execute:
o enable fast_start failover;
o show fast_start failover;
Finally we start the Observer in its own window and things will be ready to go! In the Observer workspace of
your desktop, or in new terminal window somewhere, connect DGMGRL to the Primary or Standby.
TASK:
In the OBSERVER window execute:
o dgmgrl sys/oracle
o start observer;
DGMGRL> enable fast_start failover;
Enabled.
DGMGRL> show fast_start failover;
Fast-Start Failover: ENABLED
Threshold: 30 seconds
Target: nyc
Observer: (none)
Lag Limit: 30 seconds (not in use)
Shutdown Primary: TRUE
Auto-reinstate: TRUE
Configurable Failover Conditions
Health Conditions:
Corrupted Controlfile YES
Corrupted Dictionary YES
Inaccessible Logfile NO
Stuck Archiver NO
Datafile Offline YES
Oracle Error Conditions:
(none)
bash-3.2$ dgmgrl sys/oracle
DGMGRL for Linux: Version 11.2.0.2.0 - 64bit Production
Copyright (c) 2000, 2009, Oracle. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Connected.
DGMGRL> start observer
Observer started
Oracle Data Guard
Page 66 of 76
The observer runs in a foreground window and will not stop until you execute a ‘stop observer’ from another
DGMGRL session. Enterprise Manager offers high availability for the Observer. In our manual mode we would
have to script it to restart if it fails.
That’s it; FSFO is cocked and ready to go. To verify that all is well we’ll examine the Fast-Start Failover setup
in our configuration.
TASK:
In the DGMGRL window execute:
o show fast_start failover;
As you can see there are other properties you can set for a FSFO configuration that we have not explicitly set.
Refer to the Data Guard Broker manual for more information on these properties.
We are now ready for the last exercise. An Automatic Failover!
DGMGRL> show fast_start failover;
Fast-Start Failover: ENABLED
Threshold: 30 seconds
Target: nyc
Observer: ip-10-124-233-12
Lag Limit: 30 seconds (not in use)
Shutdown Primary: TRUE
Auto-reinstate: TRUE
Configurable Failover Conditions
Health Conditions:
Corrupted Controlfile YES
Corrupted Dictionary YES
Inaccessible Logfile NO
Stuck Archiver NO
Datafile Offline YES
Oracle Error Conditions:
(none)
Oracle Data Guard
Page 67 of 76
Executing an Automatic Failover
Remember how we said that FSFO takes the ‘person’ out of a failover? Well, it does but we need you to crash
the Primary. As you will see in the manual, there are several events that will trigger a Fast-Start Failover.
The usual way that a failover will be triggered is when the Observer and the target standby lose their connections
with the primary database, but the Observer and the standby database are still communicating. This can be a
database crash, a system crash, the loss of the network, or an entire primary site outage—as in a complete power
failure of the locality where the primary is running. The threshold will be honored and the failover will not
commence until the timer has expired. Since this is an unexpected event and there is no information as to the
health of the primary database, the Observer will attempt to reinstate the failed database automatically as a
standby when it is restarted.
You can also force a FSFO by performing a SHUTDOWN ABORT on the primary database. Of course, if the
primary is a RAC, a SHUTDOWN ABORT will only cause a failover if the instance being aborted is the last
surviving instance and your FastStartFailoverThreshold is exceeded. As with the normal lost connection event,
the Observer will reinstate the failed primary when it comes back up.
In addition, you can configure FSFO conditions that the Broker will watch for, and, if it sees one of them
happening, FSFO will initiate an immediate abort of the primary and a failover as long as the requirements for the
failover have been met. These are set in the Broker using the ENABLE command and there are five specific
conditions you can configure which are documented in the Broker manual.
So, we’re going to simulate a failure with a SHUTDOWN ABORT.
TASK:
In the SFO SQL*Plus window execute:
o select db_unique_name, database_role from v$database;
To verify that we really are on the Primary
o shutdown abort
That will trigger a failover to NYC since the Broker (the Observer) recognizes immediately that our last
surviving Primary instance has crashed. It will wait until our threshold has passed and then it will make NYC the
Primary database ‘automatically’!
SQL> select db_unique_name, database_role from v$database;
DB_UNIQUE_NAME DATABASE_ROLE
--------------- ------------------
SFO PRIMARY
SQL> shutdown abort
ORACLE instance shut down.
SQL>
Oracle Data Guard
Page 68 of 76
If you return to the Observer window you can observe the Observer actually performing the failover.
TASK:
In the OBSERVER window
o Watch the Observer performing the failover.
Returning to the DGMGRL window you can see the current state of the configuration, NYC as the Primary and
SFO needing reinstatement.
TASK:
In the DGMGRL window execute:
o show configuration;
This is exactly the same state we were in when we performed our manual failover, NYC is the Primary and SFO
needs to be reinstated. The difference is (and it is a big one) that the Primary cannot open read write no matter
how it is started and the Observer will automatically reinstate it as a standby when it does restart.
DGMGRL> start observer
Observer started
11:41:17.61 Friday, April 01, 2011
Initiating Fast-Start Failover to database "nyc"...
Performing failover NOW, please wait...
Failover succeeded, new primary is "nyc"
11:41:21.07 Friday, April 01, 2011
DGMGRL> show configuration
Configuration - dg2011
Protection Mode: MaxAvailability
Databases:
nyc - Primary database
Warning: ORA-16817: unsynchronized fast-start failover
configuration
sfo - (*) Physical standby database (disabled)
ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: ENABLED
Configuration Status:
WARNING
Oracle Data Guard
Page 69 of 76
Return to your SFO SQL*Plus window and execute a ‘startup’, which normally would open the database read
write (since it is still a Primary database at this point).
TASK:
In the SFO SQL*Plus window execute:
o startup
As you can see, the Broker knows this database was in a FSFO enabled configuration and will prevent it from
opening. Going back to the Observer window you will see that the Observer has detected that SFO is back and
automatically reinstates it as a standby.
TASK:
In the OBSERVER window
o Watch the Observer performing the reinstate.
SQL> startup
ORACLE instance started.
Total System Global Area 2438529024 bytes
Fixed Size 2228920 bytes
Variable Size 1526730056 bytes
Database Buffers 889192448 bytes
Redo Buffers 20377600 bytes
Database mounted.
ORA-16649: possible failover to another database prevents this
database from being opened
11:49:26.29 Friday, April 01, 2011
Initiating reinstatement for database "sfo"...
Reinstating database "sfo", please wait...
Operation requires shutdown of instance "SFO" on database "sfo"
Shutting down instance "SFO"...
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "SFO" on database "sfo"
Starting instance "SFO"...
ORACLE instance started.
Database mounted.
Continuing to reinstate database "sfo" ...
Reinstatement of database "sfo" succeeded
11:50:33.27 Friday, April 01, 2011
Oracle Data Guard
Page 70 of 76
At this point we are back in business and FSFO is ready to go if NYC should fail. You can check the
configuration with another show configuration command in the DGMGRL window.
TASK:
In the DGMGRL window execute:
o show configuration;
Success! You can leave things like they are and move production back to SFO when NYC fails or you can
perform a switchover to move production from NYC back to SFO. Of course you could also do a SHUTDOWN
ABORT on NYC and let the Observer move production to SFO.
This ends our exercises with Data Guard but feel free to redo or play around with your configuration.
DGMGRL> show configuration
Configuration - dg2011
Protection Mode: MaxAvailability
Databases:
nyc - Primary database
sfo - (*) Physical standby database
Fast-Start Failover: ENABLED
Configuration Status:
SUCCESS
Oracle Data Guard
Page 71 of 76
Conclusion
This concludes the exercises for the Oracle Database 11g Data Guard Hands On Lab. During this session you
have explored and exercised Data Guard’s unique capabilities for protecting your data. You were able to create,
manage and use Data Guard standby databases in a variety of situations.
You have completed the following exercises.
Created a Physical Standby Database
Configured the Data Guard Broker
Edited Broker Properties
Changed the Protection mode
Performed a Switchover
Performed a Manual Failover
Performed a Manual Reinstate.
Enabled and used Fast-Start Failover.
It is our sincere hope that these exercises have helped to guide you on your path to using Data Guard and
understanding how to use the features that it brings to the Oracle Database world to provide you with Disaster
Protection for your Oracle data.
Thank you for your participation.
Larry M. Carpenter
Oracle Data Guard
Page 72 of 76
Oracle Data Guard
Page 73 of 76
Resources
The following documentation is available for Data Guard
Oracle® Data Guard Concepts and Administration 11g Release 1 (11.1)
o http://download.oracle.com/docs/cd/B28359_01/server.111/b28294/toc.htm
Oracle® Data Guard Broker 11g Release 1 (11.1)
o http://download.oracle.com/docs/cd/B28359_01/server.111/b28295/toc.htm
Oracle® Data Guard Concepts and Administration 11g Release 2 (11.2)
o http://download.oracle.com/docs/cd/E11882_01/server.112/e10700/toc.htm
Oracle® Data Guard Broker 11g Release 2 (11.2)
o http://download.oracle.com/docs/cd/E11882_01/server.112/e10702/toc.htm
The Maximum Availability Architecture Best Practices papers are on Oracle OTN.
Best Practices for High Availability -- Maximum Availability Architecture (MAA)
o http://www.oracle.com/goto/maa
Best Practices for Data Guard
o http://www.oracle.com/goto/dataguard
Oracle Data Guard
Page 74 of 76
Oracle Data Guard
Page 75 of 76
Appendix A – Create Standby Script
MakeNYC #!/bin/bash
. oraenv <<EOF
NYC
EOF
#Abort the NYC instance in case it was running from a prior attempt
# Start up the auxiliary instance nomount
sqlplus -s '/ as sysdba' <<EOF
SHUTDOWN ABORT
EOF
# Perform some clean up in case this is a 2nd run.
rm -f /u01/app/oracle/product/11.2.0/db_1/dbs/*NYC*
rm -f /u01/app/oracle/product/11.2.0/db_1/dbs/tempini.ora
# Create temporary password file, must have same password as primary
orapwd file=/u01/app/oracle/product/11.2.0/db_1/dbs/orapwNYC password=oracle
# Create temporary parameter file
cat > /u01/app/oracle/product/11.2.0/db_1/dbs/tempini.ora <<EOF
db_name=anyname
EOF
# Start up the auxiliary instance nomount
sqlplus '/ as sysdba' <<EOF
SET ECHO ON
STARTUP PFILE='/u01/app/oracle/product/11.2.0/db_1/dbs/tempini.ora' NOMOUNT;
EOF
# Run RMAN to duplicate your database.
rman <<EOF
set echo on
connect target sys/oracle@SFO; # Primary
connect auxiliary sys/oracle@NYC; # Standby: tnsname used by primary
run {
# Create channels for parallism
allocate channel prmy1 type disk;
allocate channel prmy2 type disk;
allocate channel prmy3 type disk;
allocate channel prmy4 type disk;
allocate channel prmy5 type disk;
allocate auxiliary channel stby1 type disk;
duplicate target database for standby from active database
spfile
parameter_value_convert 'SFO','NYC'
set 'db_unique_name'='NYC'
nofilenamecheck
;
}
EOF
exit 0
Oracle Data Guard
Page 76 of 76
Data Guard Hands on Lab
January 2013
Authors: Larry M. Carpenter
Oracle USA
World Headquarters
500 Oracle Parkway
Redwood Shores, CA 94065
U.S.A.
Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200
oracle.com
Copyright © 2010, Oracle. All rights reserved.
This document is provided for information purposes only and the contents hereof are subject to change without
notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether
expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a
particular purpose. We specifically disclaim any liability with respect to this document and no contractual
obligations are formed either directly or indirectly by this document. This document may not be reproduced or
transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written
permission. Oracle, JD Edwards, and PeopleSoft are registered trademarks of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.