MIGRATING FROM SINGLE INSTANCE TO RAC AND UPGRADING: A DATA
GUARD SUCCESS STORYFuad Arshad @fuadar
Walgreen Company
Me, Myself & I
•More than 15 years of Oracle experience
•Worked in textiles, telecom & retail/healthcare
•Currently work for Walgreens in the Enterprise Shared Services area
•Areas of Interest: backup & recovery, high availability, MAA, performance tuning
Socially Networkable • Twitter @fuadar
• Linkedin http://www.linkedin.com/in/fuadar
• G+ +Fuad Arshad
• Blog http://www.fuadarshad.com
Agenda• Problem
• Analysis
• Decision
• Strategy
• Migration
• Things to look out for!
• Credits
Community Access
• RAC Attack
• IOUG RAC SIG
• IOUG
Blogs / Blogs /Blogs .
• http://www.orana.info
• http://blogs.oracle.com
• http://www.oaktable.net
Ask Questions
• Freelists Oracle L
• Stack Overflow
• Quora
• MOS Communities
• Oracle Forums
The Problem
“It was very early, and we were still like beta or alpha stage, and so we started receiving a ton of download. The server became overloaded, and that's when I realized that this had a huge market.”
Shawn Fanning (Napster)
Paper #371 7
Our Problem
•The application was growing and adding features
•Customer access slowdown was reported
Paper #371 8
Our Problem
•System built in the early 2000’s to handle the new wave of technology
•More work as newer features were rolled out
•Architecture & design not intended to handle the rate of change
•Database is always the bottleneck
Paper #371 9
Analysis
•Active/Passive servers
•Oracle database version close to desupport
•Old OS version
•Old SAN infrastructure
•CPU and disk I/O issues
Result -----> IT WAS OLD Paper #371 10
Analysis
•Active/Passive Servers
•Oracle Database version close to desupport
•Old OS version
•Old SAN infrastructure
•Result ---------------- IT WAS OLD Paper #371 11
Paper #371 12
Veritas VolumeManager Upgrade
(Stage 2)
Veritas VolumeManager Upgrade
(Stage 1)
Veritas File SystemManager Upgrade
OS Upgrade toSolaris 10 (Stage 2)
OS Uprade toSolaris 10 (Stage 1)
Rebuild the stagingcluster (to test the
productionmigration)
Current serverNew ServerDependency (disk)
Veritas ClusterUpgrade
Migrate productionstorage to EMCVemax storageframe (here)
Expand Disk performance (IOPS)
Migrate HighPhysical IO objects
to SSD
Create a dedicatedPerformance Test
Environment
Migrate Storage toASM
ETA: Feb. / March 2011
Disk rehost case:DMX800 End of LifeAvailability8G network & additional IOPSSupport Oracle 11g upgrade
Time to Server CapacityEnd of Life (2011 Holidays) 6 - 7 Months
Active - Passive Server Failover Rolling Upgrades (6 hours of TTL Outages, 10 hours single node)Predecessors1) Setup ODS2) Migrate Data Guard
?
DBUpgrades
3 outages - 2 hours each
VDI & CodeRepository Backups
Decision
•Let’s migrate to RAC
•Can it be done yesterday?
•Can we do it with zero downtime?
•The application team doesn’t want to change anything
Paper #371 13
Strategy
•Export/Import
•Transportable Table spaces
•Disk based replication
•GoldenGate
•Data Guard - The Choice
Paper #371 14
Pre-Migration
•Build of servers
•Build of network
•RAC starter kits
Paper #371 15
Pre-Migration
• The strategy was to build a RAC database from a single instance environment
• Oracle provided instructions on how to prep the single instance environment so that it is RAC-aware. This includes running catclust scripts and adding undo tablespaces for the number of nodes that the new cluster will have
Paper #371 16
Pre-Migration
• The server builds need to have all the patches based on the OS that is being used
• Oracle provides a variety of starter kits that can be used to validate the configuration
• It is also important that the network be configured and available based on best practices
• 11.2.0.2 and above have added requirements like SCAN that need to be accounted for
Paper #371 17
Pre-Migration
• RAC Check is important tool to ensure that the environment is built based on best practices
• RAC Check can be downloaded at MOS # 1268927.1 and is updated frequently
– OS settings including kernel parameters and patches– CRS/Grid Infrastructure– RDBMS memory and database parameters– ASM parameters and statistics
Paper #371 18
Pre-Migration (Primary)Data Guard setup steps need to be followed which include :
• Enable supplemental logging & force logging on primary
• Add additional REDO threads (Based on # of RAC nodes)
• Add additional UNDO tablespaces (Based on # of RAC nodes)
• Backup the primary database
• Create a PFILE from the primary database
• Copy TNSNAMES and LISTENER files into a staging directory
• Configure the primary database for Data GuardPaper #371 19
Pre-Migration
• RAC Check is important tool to ensure that the environment is built based on best practices
• RAC Check can be downloaded at MOS # 1268927.1 and is updated frequently
– OS settings including kernel parameters and patches– CRS/Grid Infrastructure– RDBMS memory and database parameters– ASM parameters and statistics
Paper #371 20
Pre-Migration (Standby RAC)• Data Guard setup steps need to be followed which
include:
• NFS mount or copy backup and auxiliary files
• Configure Oracle Net which includes setting up TNSNAMES and LISTENER for 11.2.0.2 (This will be done in the Grid infrastructure home)
• Modify the TNSNAMES.ORA to include the primary as well as the standby tns entries
• Create a tns entry for the listener and make sure it is referenced in the REMOTE_LISTENER (REMOTE_LISTENER specifies the global list of listeners in a RAC configuration)
• Make sure the listener alias referenced by the LOCAL_LISTENER initialization parameter is in each standby tnsnames.ora file
• Ensure that the standby server has the same software version and patches installed as the primary
Paper #371 21
Pre-Migration (Standby RAC, cont.) Data Guard setup steps need to be followed which include:
• Ensure that a password file is configured on both the primary and the standby
• Copy the PFILE to the $ORACLE_HOME/dbs in the standby node
• Add DB_UNIQUE_NAME and CLUSTER_DATABASE parameters in the PFILE
• Modify all the other parameters outlined in the white paper
• Connect to the standby and create an SPFILE in the ASM diskgroup
• Startup the proposed standby environment in NOMOUNT mode
• Duplicate the database using “DUPLICATE TARGET DATABASE FOR STANDBY”
• Setup Server Control (SRVCTL) to register the instancesPaper #371 22
Pre-Migration (Standby RAC, cont.)Data Guard setup steps need to be followed which include:
• Add additional standby REDO logs (The number of standby redo logs should be one more than the number of online REDO logs per thread and should be the same size)
• Enable Managed Recovery Mode on the standby database
• Enable archive destination on the primary to start shipping logs to the standby
• Validate the transfer
• Setup Data Guard Broker
• Validate the configuration and ensure that logs are being applied
• Bring up the other nodes in the RAC environmentPaper #371 23
MigrationMigrating is simple after the Data Guard database is setup
• Ensure all ARCHIVELOGS have been applied
• Put the primary database in restricted mode and switch ARCHIVELOGS
• After all the outstanding logs have been applied, shutdown REDO apply on the standby
• Validate SCN across the primary and the standby
• Two options from here:
- Switchover & upgrade
- Shutdown & upgrade
**We Chose Shutdown & Upgrade**24Paper #371
Migration
• All standby RAC nodes can be up if using DBUA
• Ensure pre-checks are done
• DBUA is brutal if you don’t remove the obsolete parameters
• DST changes are important
• If manually upgrading DBMS_DST needs to be run to upgrade DST
Paper #371 25
Things to Look Out For Ensure Environments Are Validated
• RDA should be run and all issues fixed
• RAC check will give you a detailed overview of the system
• Ensure all starter kits and the RAC manuals are consulted: the result can be a good best practices evolving document
• Use services to better manage how the application connects and load balance your system
• Private network is key with RAC: follow Oracle’s best practices.
• XA changes in 11.2.0.2 & can cause issues: important to understand them
• “Compatible” parameter testing is important.
• Monitoring the environment after go-live is very importantPaper #371 26
CreditsI’d like to thank the following people for helping me with the presentation and accompanying white paper:
• Matthew Gallagher, Architect -- Walgreens
• Brice Lahl, Manager DB Architecture -- Walgreens
• Daniel Morgan, ACE Director -- Drugstore.com
A special shout out to Arup Nanda & Alex Gorbachev for encouraging me to present and Leighton Smith for reviewing my session
Also special credit to the RAC Assurance Team at Oracle for helping is get through this migration and subsequent issues. Some content in the whitepaper and presentation has been republished with permission from the Oracle RAC Assurance Team whose goal had been to proactively provide input to customers with new and existing CRS, RAC, ASM and MAA implementations.
Paper #371 27
Questions• Please fill
out evaluations
Paper #371 28
Top Related