MIGRATING FROM SINGLE INSTANCE TO RAC AND UPGRADING: A DATA GUARD

SUCCESS STORY

Fuad Arshad @fuadar

Walgreen Company

About me

• More than 16+ years of Oracle experience

• Worked in Textiles, Telecom & Retail/Healthcare

• Currently work for Walgreens in the Enterprise Shared

Services area

• Areas of Interest: backup & recovery, high availability,

MAA, performance tuning

Socially Networkable

• Twitter @fuadar

• Linkedin

http://www.linkedin.com/in/fuadar

• G+ +Fuad Arshad

• Blog http://www.fuadarshad.com

IOUG - Collaborate

Community Access

• RAC Attack

• IOUG RAC SIG

• IOUG

Blogs / Blogs /Blogs .

• http://www.orana.info

• http://blogs.oracle.com

• http://www.oaktable.net

Ask Questions

• Freelists Oracle L

• Stack Overflow

• Quora

• MOS Communities

• Oracle Forums

• WhitePaper for this Session

Agenda

• Problem

• Analysis

• Decision

• Strategy

• Migration

• Things to look out for!

• Credits

Our Problem

• The application was growing and adding features

• Customer access slowdown was reported

Our Problem

• System built in the early 2000’s to handle the new

wave of technology

• More work as newer features were rolled out

• Architecture & design not intended to handle the

rate of change

• Database is always the bottleneck

Analysis

• Active/Passive servers

• Oracle database version close to desupport

• Old OS version

• Old SAN infrastructure

• CPU and disk I/O issues

Result -----> IT WAS OLD

Analysis

Decision

• Let’s migrate to RAC

• Can it be done yesterday?

• Can we do it with zero downtime?

• The application team doesn’t want to change

anything

Strategy

• Export/Import

• Transportable Table spaces

• Disk based replication

• GoldenGate

• Data Guard - The Choice

Pre-Migration

• Build of servers

• Build of network

• RAC starter kits

Pre-Migration

• The strategy was to build a RAC database from a single

instance environment

• Oracle provided instructions on how to prep the single instance

environment so that it is RAC-aware. This includes running

catclust scripts and adding undo tablespaces for the number of

nodes that the new cluster will have

Pre-Migration

• The server builds need to have all the patches based on the OS

that is being used

• Oracle provides a variety of starter kits that can be used to

validate the configuration

• It is also important that the network be configured and available

based on best practices

• 11.2.0.2 and above have added requirements like SCAN that

need to be accounted for

Pre-Migration

• RAC Check is important tool to ensure that the environment is

built based on best practices

• RAC Check can be downloaded at MOS # 1268927.1 and is

updated frequently

– OS settings including kernel parameters and patches

– CRS/Grid Infrastructure

– RDBMS memory and database parameters

– ASM parameters and statistics

Pre-Migration (Primary)

Data Guard setup steps need to be followed which include :

• Enable supplemental logging & force logging on primary

• Add additional REDO threads (Based on # of RAC nodes)

• Add additional UNDO tablespaces (Based on # of RAC nodes)

• Backup the primary database

• Create a PFILE from the primary database

• Copy TNSNAMES and LISTENER files into a staging directory

• Configure the primary database for Data Guard

Pre-Migration (Standby RAC)

• Data Guard setup steps need to be followed which include:

• NFS mount or copy backup and auxiliary files

• Configure Oracle Net which includes setting up TNSNAMES and LISTENER

for 11.2.0.2 (This will be done in the Grid infrastructure home)

• Modify the TNSNAMES.ORA to include the primary as well as the standby

tns entries

• Create a tns entry for the listener and make sure it is referenced in the

REMOTE_LISTENER (REMOTE_LISTENER specifies the global list of

listeners in a RAC configuration)

• Make sure the listener alias referenced by the LOCAL_LISTENER

initialization parameter is in each standby tnsnames.ora file

• Ensure that the standby server has the same software version and patches

installed as the primary

Pre-Migration (Standby RAC,cont.)

Data Guard setup steps need to be followed which include:

• Ensure that a password file is configured on both the primary and the

standby

• Copy the PFILE to the $ORACLE_HOME/dbs in the standby node

• Add DB_UNIQUE_NAME and CLUSTER_DATABASE parameters in the

PFILE

• Modify all the other parameters outlined in the white paper

• Connect to the standby and create an SPFILE in the ASM diskgroup

• Startup the proposed standby environment in NOMOUNT mode

• Duplicate the database using “DUPLICATE TARGET DATABASE FOR

STANDBY”

• Setup Server Control (SRVCTL) to register the instances

Pre-Migration (Standby RAC, cont.)

Data Guard setup steps need to be followed which include:

• Add additional standby REDO logs (The number of standby redo logs should

be one more than the number of online REDO logs per thread and should be

the same size)

• Enable Managed Recovery Mode on the standby database

• Enable archive destination on the primary to start shipping logs to the

standby

• Validate the transfer

• Setup Data Guard Broker

• Validate the configuration and ensure that logs are being applied

• Bring up the other nodes in the RAC environment

Migration

Migrating is simple after the Data Guard database is setup

• Ensure all ARCHIVELOGS have been applied

• Put the primary database in restricted mode and switch ARCHIVELOGS

• After all the outstanding logs have been applied, shutdown REDO apply on

the standby

• Validate SCN across the primary and the standby

• Two options from here:

- Switchover & upgrade

- Shutdown & upgrade

**We Chose Shutdown & Upgrade**

Migration

• All standby RAC nodes can be up if using DBUA

• Ensure pre-checks are done

• DBUA is brutal if you don’t remove the obsolete parameters

• DST changes are important

• If manually upgrading DBMS_DST needs to be run to upgrade DST

Things to Look Out For

Ensure Environments Are Validated

• RDA should be run and all issues fixed

• RAC check will give you a detailed overview of the system

• Ensure all starter kits and the RAC manuals are consulted: the result can be

a good best practices evolving document

• Use services to better manage how the application connects and load

balance your system

• Private network is key with RAC: follow Oracle’s best practices.

• XA changes in 11.2.0.2 & can cause issues: important to understand them

• “Compatible” parameter testing is important.

• Monitoring the environment after go-live is very important

Shameless Plug

A couple of Sessions that myself and my colleagues will be presenting

CON3776 - Implementing the Oracle Database Appliance as an Enterprise Solution

Tuesday, Oct 2, 1:15 PM - 2:15 PM

CON12593 - Success Story: Indirect Procurement Transformation at Walgreens

Tuesday, Oct 2, 5:00 PM - 6:00 PM –

CON11894 - Achieving Business Transformation with Oracle E-Business Suite R12 Applications

Tuesday, Oct 2, 10:15 AM - 11:15 AM

.

Credits

I’d like to thank the following people for helping me with the

presentation and accompanying white paper:

• Matthew Gallagher, Architect -- Walgreens

• Brice Lahl, Manager DB Architecture -- Walgreens

• Daniel Morgan, ACE Director -- Drugstore.com

• The Team At IOUG for encouraging new speakers.

A special shout out to Arup Nanda & Alex Gorbachev for encouraging me to present and

Leighton Smith for reviewing my session

Also special credit to the RAC Assurance Team at Oracle for helping is get through this migration and

subsequent issues. Some content in the whitepaper and presentation has been republished with permission from

the Oracle RAC Assurance Team whose goal had been to proactively provide input to customers with new and

existing CRS, RAC, ASM and MAA implementations.

Questions

/Comments

SAVE THE DATE!

COLLABORATE 13

April 7-11, 2013

Colorado Convention Center

Denver, Colorado

http://collaborate13.ioug.org